To be clear, changing the polarity of a single speaker does not create any weird or "off" sounds. In fact, the sound coming from the speaker alone is indistinguishable between normal and reversed polarity.
To achieve seamless integration between speakers in adjacent passbands, they MUST be phase aligned at the crossover frequency. The easiest, and most accurate way of achieving this is through time alignment. You are delaying the closer speaker to you, such that the wave from the farther speaker arrives at your head at the same angle (phase). If you do not have the luxury of T/A, then phase is your next best option. If the sound waves from the midbass and sub speakers are 180 degrees (or close) apart, then you swapping the phase on the sub is basically the same thing as delaying the speaker closer to you by the time that is equivalent to half the wavelength.
People may hear different effects (outside the crossover frequency range) when the polarity of one speaker is reversed, because phase is frequency dependent. So at 80Hz the waves may be perfectly aligned, but at 60 and 100 Hz they will be off again, due to changing wavelengths. That's why many people prefer steeper crossover slopes, because it reduces such interference.
Mobridge DA2 > JBL MS-8 > JL Audio HD600/4 > (3) Hybrid Audio L3SE & (2) Rainbow Cal25; JL Audio HD900/5 > (2) Morel HCW-10 & (1) AE SBP15