Thanks jonsc and scott@bmwhn !
Took your guys' advice and gave it one last college try, this time armed with a few more tools:
1. Mechanics gloves - Just needed one, to help with with pulling the grille and preserving my hand

2. A longer flat-head screw driver.
3. Head-lamp (so I didn't have to fumble around with a flash light) while pulling the grille and using a screwdriver.
I started off by unlatching the top 3 tabs by using the screwdriver while pulling the grille towards me. If I wasn't careful, the tabs would "re-lock" themselves as I was trying to get all 3 unlocked. I ended up putting a small screw driver in between the grille and the bumper to keep the tabs from going back in as I started to work on the side tabs.
Using a little bit more force than last time, I was able to pull the bumper far out enough to use my long screwdriver to push on the tab closest to the headlight while simultaneously pulling the grille towards me. It took a little bit of time, but the headlamp sure helped in this case (was working in a dimly lit garage and night).
Once that was out, I did the same on the tab closest to the wheel. Popped that out with a screwdriver, and was then able to pull the grille far enough away from the bumper to reach my hand in and used my fingers to release the remaining bottom 3. It took a bit of trial and error, but I eventually got it off
Once one of the grilles were off, I repeated the steps for the other side. It was much easier since I could put my hand/arm in the removed grille opening and work from behind.
All-in-all, it was much harder than I thought it would be but I'm happy with the end result (and happy that I didn't have to pay the dealer)
Thanks again guys! Looking forward to seeing more mods on your X3