There were legions of Americans who could not vote, including Native Americans, some immigrants, women and black Americans.