Do you fear being called a racist? A sexist? A ablist? Homophobic? Bigoted?
You know the biggest issue I run into with most privileged people is that when they're told that they said something shitty and bigoted, they immediately go into a defensive flap because they're so afraid of being thought as inappropriately behaved rather than afraid of being inappropriate.
Or they remain silent in the face of things like honor killings because they're so scared of being called racist that they won't stand up to protect the vulnerable against murder even when protecting the vulnerable is their job.
Why are these labels so scary that they control people's ability to act and close their minds?
The important thing is not to be a bigot, not to avoid being accused of being one. If you're too busy arguing about how you're not a bigot, or getting upset because someone said you did something bad? You're probably not actively not being a bigot.
Fact is, we all will be accused of being horrible at some point, rightly or wrongly.
"But it's a false accusation".
If it is a false accusation, I know that stings, I find it hard when people falsely accuse me as well. But I've yet to ever convince anyone who slings false accusations to retract such statements by arguing with them. People will believe what they want to believe at the end of the day, if they want to believe you're a bigoted asshole, nothing will change that belief, not reality, not a good argument, nothing. What is of more benefit to you and society? Arguing with someone who wants to believe you're a bigot or doing something to actively fight bigotry instead?
Seriously, if you fear being called a term that means you're bigoted more than you fear being bigoted and hurting your peers. Then you fear the wrong thing.
If the accusation is true? Examine why you were called it and what you can do to tackle it.
If it's false? You'll never convince them otherwise so why argue over it?
Regardless, do not fear what others can and may say about you. You cannot always be right, or always popular and some people are just assholes anyway and you shouldn't take them seriously.