Anyone ever sit beside a 12" alpine sub when it's blasting? I have. Now that is an experience. The sub took up the driver side and middle seat in the back of the car (it was a temporary thing). But man, sitting beside one of those at full blast is one hell of an experience.
In other news, having hearing loss can be a bit of a problem every now and again, but not as much as some people might think... (well, mine's not all that bad anyway, it's more frequency loss than a uniform drop in perceived sound level.)
Also, ear buds pump out the exact same volume that headphones or a car stereo would, generally. They adjust the output to deal with the difference in sound amplification that your causes, as well as for the fact that they are much closer to the ear drum.
All in all, unless you consistently put your music above about 80 db, you're gonna be fine, but if you listen to 80+ db music for any length of time, you're likely to lose your hearing over time.