Had a few near-misses, almost deleting a selected user, rather than "Deselecting" them. The two buttons are right next to each other, and both start with "De".
Suggest changing the color of "Delete" to red, and possibly adding a "Are you sure?" popup (if this is not already present).