The only time our fuel door came off was a bizarre incident that occurred when our X3 was about a week old. We were leaving a parking garage and all of the sudden two guys came running out between the lanes of cars, one chasing the other. The first one tried to jump between our car and the one next to us, pushing himself up off ours with his hand on the filler door. As he did it the door came off. Fortunately traffic was at a stand still so I was able to retrieve the door and put it back on when we got home.
It has not ever come off again. [And I don't know what happened to the guys]