"I seem to draw damaged young men who desperately need a father figure. I could pull together a core of 'acolyte/enforcers' fairly easily."

And what's the problem in working with that? What other kinds of followers do you expect to get until people start seeing the bubbles rising around the frog in the pot?