I was in the same boat.
Drove an e36 m3 for 9 years sold it with the intentions of getting an e46 m. My dad has an e93 which I was very impressed with. Although after owning an m for almost a decade I couldn't bring myself to even consider the 335's.
Just one thing I couldnt get over with the e46, it was designed in 1999 so while its an awesome car it's out dated. The n54 is an awesome engine and once I saw the capabilities of the motor I quickly changed my mind.
With that said I do miss the e36 only b/c it would of made a great track car but mine only had 88k on it.
I do plan on an e46 m down the road to fill that void. For now I'm enjoying the hell out of my e92.