I'm an audiophile and have been tossing up whether to do anything to my new X3 and I'm still undecided.
Biggest issue you face if you want a top end system is the lack of standard speaker hole space. Your really limited to fitting high end speakers in the stock holes which you basically have to weigh up the cost to sound improvement.
My biggest gripe is the limited tuning options with the standard system - you'd have to fit an aftermarket processor to do some serious tuning and that comes with big $$ once again.
You can always basic tune with amps however, you mentioned you didn't want to change amps so that once again limits your options....