I've owned a 550, so I can give you some perspective.
If you like a lighter, more nimble feeling car, then you'll love the X4 vs. the 5 series. As much as I enjoyed the 550, they are at the end of the day large, heavy vehicles. Smaller, lighter cars are more fun to drive. The X4 35i is fast and fun to drive. You'll love it.
That said, the X4 isn't exactly small, or light,
but it does feel more lively and entertaining to drive than a 5 series. If you have some budget flexibility, test drive the M40i. Both the 35i and m40i will be terrific - you may find an appreciable difference between the two, or you may not, but if I were you, I'd drive both to be sure you are happy with the 35i. FYI - the 35i no longer exists as of 2017. Certainly you'll be able to get a terrific deal on a '16 35i (or a '16 m40i) on the lot vs. a '17 m40i.