I've tried other methods on scrap enclosures. Dremel tool to carefully cut off either the head or the bulge in back was too 'hit or miss'. Trying to pry up or otherwise deform the front flange also resulted in damage more often than not.
I generally pick a drill bit just a little larger than the hole and start drilling from the front where the rivet head is located. Most of the time, if I pick the appropriate bit size, (and use careful pressure - no need to rush it), the bit will take away just enough metal and the rivet head will twist loose and stay stuck on the drill bit.
At that point, you can shut down, vacuum up the bits, remove the metal head from the drill bit and the rest of the rivet should just slide out.
I wouldn't disagree with any of the above comments re: drilling out pop-rivets.
Just one further thought .If you are just removing the XLR to get the solder connections or to replace with a new xlr insert, you might be able to do this just by removing the plastic insert in the middle. Normally held in with one screw somewhere on the body of the XLR. The plastic insert will then pull out. You can then change/resolder or whatever without removing the XLR cases.
Its probable that this isn't what you want to do, but I thought it worth mentioning anyway.