I also had a M3 recently, an E90 Sedan up until I sold last fall. I have gone to the X3 because of a lot of reasons: size (just right for my needs), power (35i), all-weather capability (something the M3 was not so good for), still fun to drive, good for long hiway runs, carrying capacity, etc, etc, etc.
Who cares what others think? Get what works for you, and ignore ppl who are shallow enough to label you by your car....meh....