[Francis Vaughan]

As one might gather, there is good reason for thinking either way. At reasonable levels there is no mechanism in the ear that can discern absolute polarity. It simply isn't how the ear works. So from that point of view a purist approach says that absolute polarity cannot matter. But between the sound source and your ear there can be non-linear processes that are polarity dependant. So these can change the harmonic content of the sound, and that you can hear.

As Ethan says, - speakers are asymmetric devices - just look at a cone - it isn't symmetric about the plane normal to its movement. A high energy pulse will distort the cone in a different manner moving one way to the other. But you do need a significant force to do this.

Curiously, air itself is asymmetric - trivially - if the sound is loud enough that the high pressure half of the cycle reaches 2 atmospheres the low pressure side must be a vacuum. You can't get less air than this so the air clips. (The gas laws result in asymmetric distortion long before this point - but either way we are talking sound pressure levels that would have your brains running out of your ears as a thin grey goop.)

And of course sound processing devices are asymmetric - a single ended class A device that is driven hard is intentionally so - and so will respond differently. So within a recording chain there should be no surprise that there are elements for which absolute phase matters.

The difference is however clear - you can't directly hear absolute phase - but you can hear the effects of absolute phase.