I don't think you need to get rid of the M3. Long story short, I purchased 2007 rdx cuv when I was single (don't ask why, but fell in love with the handling on a beefy car). 2 years later, got into relationship, had a child, and still rarely drove the cuv. I had small cars.
10 months ago, we had our second child; 5 months ago, I purchased a 335iS. The rdx (my daily) and Prius (not me) does all the daily hauling. The iS is the toy. I still don't find the need for a minivan, although I've given it some thought on long road trips.
Strollers take up a lot of trunk space; carseats will stick with the car until the kids are heavy enough to be without one. Afterwards, a minivan or suv for a family of 4 is still not a requirement.
If I did it all again, I'd purchase a larger luxury sedan for the family instead of the small suv (cuv).