I think it was when they started the point of telling stories of the witches and what they did with the people it just started to spook them out and they feared that maybe they were honestly true. And I think at that time it really wasn't the best time to stand out and/or say/do something out of their league because they would think you were a witch.
(And we all saw Sleepy Hollow sooo) :)
