I'll too will be coming from current M3 to X3 30D Msport at some stage and very interesting to read your feedback. For me changing to a normal car (so wife can drive it) would always be a big step down but going to a SUV type vehicle with the different advantages it offers, would not.
I'me very surprised you think of the E46 CSL as an all round car. Most people on forums comment that it's gear change is awful (unless one is going for a very spirted drive) and the ride to stiff for UK roads.