You speak as though people pick the politicians they support out of a hat. Stupid, I can deal with. If a lot of immoral people supported a politician I liked, however, I'd have to at least wonder what they did to earn that support.
If I liked a politician and then I found out a bunch of Nazis supported him (...for instance) I'd at least look at why. Maybe nothing and they've just convinced themselves this person is on their side, like with Taylor Swift? These are stupid people after all. But you'd at least take a look.
You mentioned Libertarians, and if I supported them I'd wonder why they attract so much support from people who are extremely interested in the age of consent. Like, maybe that tells you something about how Libertarian policies might play out if they were ever introduced into the real world?