There needs to be at least five justices who agree. This is due to the fact that the Supreme Court is made up of nine justices total. An odd number of justices helps ensure that there are no "ties" in their rulings. The US Supreme Court has had nine justices since 1869.
