The first reason he gave was that Elizabeth was sick during the winter season. However, he finally told Hale that he did not like how Reverend Parris operated the church. He believed that he was corrupt, and used the church to gain money.
Another likely reason that he hasn't been attending is that he is ashamed of his affair with Abigail. He doesn't want to see her or be seen with her, so he is avoiding her.
