I would say the "psychology" of the characters. My favorite scary movie is The Shining. Yeah, Jack Nicholson is freak-y, but the whole movie is that way. All of the characters have their own psychological mindset that comes out in the film. One thing I appreciate about the film is that there was such an antagonistic relationship between Shelly Duvall ("Wendy") and Stanley Kubrick (director/producer) that it added to her performance. She was actually physically sick for much of the filming. And I love how the film was shot - Kubrick was a master of 'perspective'.
I remember going to see a newer 'scary' movie - one of the Insidious movies, I think. It was okay until a girl started levitating in a chair, and the cables lifting the thing in the air were visible. Everyone was freaking out. I thought "really...?" and started laughing.