Some people would call certain Anabaptist groups "extreme" because they live and work peacefully among themselves and do not interact a whole lot with modern society. Personally, I call some Christian groups rather extreme, such as the prosperity groups or the ones who are so charismatic that they run up and down the church aisles or throw themselves on the floor or against the wall. I would call ISIS pretty extreme.
Then you have the cults. I think something is wrong with their core beliefs, such as narcissism or need to control. I don't necessarily think mental illness comes into it, though some might. How do cults start? Isn't it usually with a leader who has fantastic people skills, so he can lure in vulnerable people, eventually turning into an abuse of power? I don't know. The only one I can think that was likely psychotic - or probably sociopathic - was the one from Helter Skelter.