I went through a similar situation at a crossfit affiliate and was accused of starting the whole thing even though the guy outweighed me by 100 pounds. Thats slightly exaggerated. It was probably more like 50 or 60 pounds.
I should have just called the police on the guy.
I went back a couple times and I decided the place just wasnt worth going to if I had to deal with that kind of bullshit.
I dont like the attitude of a lot of crossfitters also. A lot Ive met seem to think they are better than anyone else just because they do a fitness program.