The reason there is a “craziness narrative” with Carson is that he regularly makes offensively hyperbolic statements to make his points. This is the same man who said Obamacare was the worst thing since slavery. He said Americans are currently living in a “Gestapo age” that is much “like Nazi Germany.” The good doctor compared homosexuality to bestiality and pedophilia. Carson has also made ridiculous comments about Ferguson, claiming the women’s lib movement was to blame for protests and comparing the demonstrators to Hamas.
So, yeah, when you say crazy things, people are going to label you as crazy. Of course, for some far-right voters out there, the crazier a candidate it, the better.