Fisher was born in Saugerties, New York. His father was a teacher and a Congregational minister, who raised his son to believe he must be a useful member of society. Despite being raised in religious family, he later on became an atheist. As a child, he had remarkable mathematical ability and a flair for invention. A week after he was admitted to Yale College his father died, at age 53. Irving then supported his mother, brother, and himself, mainly by tutoring. He graduated first in his class with a B.A degree in 1888, having also been elected as a member of the Skull and Bones society.