Mandela means that someone who decides to take someone else's freedom is prjudiced and narrowminded. He can not be free if he takes away someone else's freedom just as a person who does not have freedom.
A person who takes away someone else's freedom is not free.