Well, the thing is that the industry doesn't really make the difference really defined. Most games, you can still play seriously, like a hardcore gamer. My personal definition of a casual game is one that they didn't put enough depth into to allow you to play competitively; games that they just made too easy to master, or didn't do enough to add the needed depth to.
I find it hard to see fighters like Streetfighter being in the same category as some of the fighters out there. I'd almost say Soul Calibur, but there is still a fair bit of depth to that game. They just make it easy to be really cheap.
In any case, you compare something like Smash Brothers to something like, hell half the stuff released for the Wii, and you will see the huge difference in depth.
So it's less about the category, really, and more about the real quality of the game, and the fact that most low quality games lack the depth of production that separates good games from bad ones.