While I personally don't mind whether Visual Novels are categorized in one way or the other, I do believe that the fact that they've been marketed as games all this time has been detrimental to how they've been perceived, and the expectations people tend to have when they go into them. Leaving aside all the people who get bored with a lot of text in their games, most gamers tend to get quite hung up on the interactivity aspect of story-based games. In fact, a lot of the negative reviews I've seen on VNs tend to be related to how "your choices don't really matter" or how "they don't change much of the story!" I think it's quite telling that the most popular VN in the west are either the detective trial-based murder mysteries like Ace Attonery or Dangan Ronpa, or stat-based dating sims. Why of course, those are the ones that tend to have the most interactivity! On the other hand I've seen people who are mainly readers be bothered by having to "constantly click to move forward" and finding all the interactive elements like choices, bad ends and mini games to be very distracting.
I think it's ultimately all about expectations and what the VN audience goes to them for. For me personally Visual Novels are really just their own category. A very important part of what makes a VN experience is the combination of written prose+audio visual elements. It's all about immersion. All the elements are put together in a way that puts the audience in the world right besides all the characters.
There's no better example to illustrate the point than the way Little Busters mini games are incorporated into the story. The game's main themes are friendship and youth, and that absolutely comes through with them. I especially find the baseball minigame absolutely brilliant. To elaborate, the first couple of times I played it I actually got a little mad by how unbalanced and, well, random it seemed. Rin's throws never seemed to make sense, and she sometimes failed completely making me unfairly miss one ball. But as the game went along and I got to know the characters better and we recruited more members, I realized that I wasn't playing to beat some score or defeat some CPU opponent or get the most hits, I was just playing around with these fun characters. Rather than hit the highest combo plays, I wanted to see all the characters react to all the different places the ball could go to. Seeing Rin succeed or fail in different/new ways. Seeing Kyosuke give a lame nickname to her when she sucked at throwing or seeing the girls just chilling besides the tree under the sunset and how they reacted when the stray balls went their way (usually towards resident buttmonkey Komari). Rin getting mad at the cats getting hit and Riki's sometimes witty responses or cheering as Kurugaya or Haruka were in the right spot to catch the ball and saving our nice long combo. And at some point it hit me. It was never really about the gameplay, or the baseball mechanics themselves (really, it could barely be called baseball). When Rin failed her throws, it wasn't for any gameplay/mechanic reason that you were supposed to "solve", it simply was because she sucked at throwing! You see, the whole conceit of the minigame was simply about the mood it puts the readers in and getting to hang out with these people, Riki's friends. Your friends. That, to me, is the quintissential VN thing that I can't get in any other medium.
Even choice-less games like Umineko put together the whole audio, visuals and narrative together in a way that makes me feel like I'm actually taking part in solving the mystery even though I'm literally not interacting with the game beyond "clicking through the story", and even seemingly trivial things like the text's color are made to be a significant part of the narrative. All for the sake of immersion in a way that a normal book never, ever could.
I honestly believe that there's too many things that separate Visual Novels from both games and books in terms of structure and audience expectations that it feels kind of unfair to lump them in the same category. Then again, the game market is rapidly evolving into something quite different, especially with the rapid increase of growingly niche indie games, so who knows!
(Also, dear almighty, please forgive the massive wall of text )