During it's upward motion, gravity acts on the ball which points towards the center of the earth. Since displacement of ball and force are in opposite direction, work done on ball while going up is negative.
During it's downward motion, gravity acts on the ball which points towards the center of the earth. Since displacement of ball and force are in same direction, work done on ball while coming down is positive.