With visual medium it is much easier to notice technical problems. You don’t need a “golden ear” to notice audio issues, but it helps.
A short video on psycho-acoustics, which stereo technology is very much a part of.
http://en.wikipedia.org/wiki/Psychoacoustics is another article on the subject.
I don’t know if they go into the backwards headphone thing, but it may give you some idea of the complexities of decoding sound by the brain and ears. Just as an example, binaural recording use a device called a Neumann head, which is a vaguely human head shaped device with microphones exactly where the ears are. Due to the way the human head, and the shape of the ears affects the sound (cover the tops of your ears to notice the change), when the audio hits the microphones, the stereo signal is an almost exact real time replica of what happens when we hear sounds. So when a binaural recording is played back using earbuds or some other headphones that sits in the same place the microphones did on the Neumann head, the sound “decodes” into a 3D soundscape, where you can hear noises in the recording outside of your head, like listening to a surround system, but with 2 channels. Reverse the channels, and the illusion collapses. To some extent almost all modern stereo recordings use tricks similar to a simple binaural recording. Pink Floyd for example used HRTFs to make it appear sounds are coming from over your shoulder from a stereo recording.
I’ve listened to some modern pop music and it is basically mono, with no dynamics, so if you are listening to stuff like that, I doubt it matters much if your headphones are reversed. Not criticizing that type of music, just pointing out it is basically not stereo, not in the way live jazz or classical is, or wonderfully produced albums like Thriller or Aja.
Finally, if it sounds good to you with the headphones reversed, it is good!
Sorry for the off topic, but I really am an audio/headphone geek.
The headphones Frasier is wearing are Sony MDR V-6 or Sony MDR V-7506 (the pro model, the only diff is a gold plug and a higher price tag). I highly recommend them if you want good indestructible headphones for under $100.