Before attempting to explain why some people might or might not be sensitive, it's necessary to establish, with fair rigor, whether that is actually true.
And here we have the classic "audio debate" territory.
Your posit puts you in the "engineer/objective" camp, which loves meaurements and DBTs before they will believe anything. I myself am in the "other" (non-measurement/non DBT/non-engineer/subjective? camp!
If someone claims they can hear when absolute polarity is reversed, as I keep an open mind, I'm happy to believe them ... and will also be interested in conducting experiments to see whether I can, too. Morever, in case I am one who is relatively insensitive to absolute polarity, I
will take care to ensure my system is in the best possible situation to allow me to hear this effect, when I play the Cardas Test Record - so electrical phase is set correctly in all my components (by having the power tranfsormer connected optimally to 'active' & 'neutral') and I have speakers which don't have 1 driver reversed. To this end, I will shortly be changing the active XOs in my Maggies to all 6dB (to match the latest Maggies), to see whether that helps. If it doesn't then I will conclude that I am one of the "lucky" majority that aren't sensitive to this phenonmenon ... but I won't diss people who say they are!
Whereas the objective camp won't lift a finger until someone "proves" to them that it's worth their while to do so.