I refuse to believe I'm the only person who think's that's crazy.
Personally, I've always felt that stories in video games are more of a bonus than a driving force, and too often they're not even that. You see, I don't think many game creators know how to tell a great story. Sure, there are exceptions. Metal Gear Solid is more cutscene than game but that's very easy to forgive, since the story is just so damn crazy and interesting. Still, even then, it's pretty overwhelming. Playing through the games a second time can be a real pain in the butt, seeing as you're constantly stopping to skip cutscenes or codec conversations.
This brings me to my next point. I think there are too many times when the story has gotten in the way of an otherwise great game. Take Final Fantasy X, for example. (Yes, I'm calling out an RPG for having too much story.) Final Fantasy X had a lot of things going for it. The battle system flowed amazingly well, the character customization system was completely unique, the world was immersive, the music was memorable, and the graphics were beautiful. But the story? The story was completely invasive. Every few minutes you'd have to stop and listen to one of the characters explain to the main character, Tidus, some little detail you likely don't care about. The pacing was completely destroyed by this constantly nagging flaw. The fact that cutscenes are unskippable makes a second playthrough nearly impossible. Final Fantasy XII took the opposite approach, and quickly became one of my favourite games because of that. There's a story, but the game focuses more on gameplay and, interestingly enough, immersion. Story and immersiveness are not necessarily deeply connected. There are lots of things going on in Ivalice. Plot points may be few and far between, but things like the bestiary (in which each monster entry also comes with a bit of lore regarding something else in Ivalice, such as an event or place) manage to draw you in.
There are times when a story is important, though. In visual novel games such as the Ace Attorney series or Hotel Dusk, the story and the gameplay are closely related. The main objective of these games is to follow the events of the story closely and react in the correct way, usually be solving puzzles or catching people in their lies.
In the end, is it really worth it? Most game stories aren't even that good. They're hurt by the fact that they have to be told around the gameplay (or they'd just stop being video games all together). It's hard to come up with an interesting story when you have restrictions like the need to explain why the game takes place in a jungle, then on a snowy mountain, then at the bottom of the sea. For every one tolerable video game story that feels like may almost be worthwhile, it's pretty easy to find five or ten movies or books with much better stories that were released around the same time. This is because movies and books don't have so many restrictions, and writers are free to focus entirely on the story without worrying about anything else.
So maybe those crazy gamers who play games exclusively for the stories would be better off taking up reading.