Klein was born in the Bronx, New York City. He was an only child and both his parents were schoolteachers. After graduating from James Monroe High School at the age of 14, he attended Columbia University, where he received a bachelor's degree in mathematics in 1942 and a master's degree in physics in 1944. In 1948, he received a Ph.D. in physics under László Tisza at the Massachusetts Institute of Technology.
From 1949 to 1966, Klein was a member of the staff of the physics department of Cleveland's Case Institute of Technology, starting as an instructor and becoming a full professor in 1960. Throughout the 1950s, he became more interested in the history of physics. During the academic year 1958–1959 he was a Guggenheim Fellow at the Lorentz Institute of the University of Leiden. He joined Yale University's Department of the History of Science and Medicine in 1967 and in 1971 became the chair of the department. In 1977, due to fiscal concerns, Yale University eliminated the department and Klein became a professor in the physics department, where he remained until his retirement.
From 1963 to 1979, Klein wrote 20 articles devoted exclusively to Einstein's work. From 1988 to 1998, he was the editor-in-chief of The Collected Papers of Albert Einstein under the aegis of Princeton University Press. The Einstein Papers Project started in the mid-1970s and published 2 volumes before Klein took over. He led the team that produced volumes 3 through 6, covering Einstein's papers from 1909 through 1917.
At Yale University, he was the Eugene Higgins emeritus professor of the history of physics and an emeritus professor of physics. He was elected to the Académie Internationale d'Histoire des Sciences (1971), the National Academy of Sciences (1977) and the American Academy of Arts and Sciences (1979).
In 2005 Klein was the first recipient of the Abraham Pais Prize for History of Physics, a joint award of the American Physical Society and the American Institute of Physics. His doctoral students include Russell McCormmach.
He died in Chapel Hill, North Carolina.
- 1970: Paul Ehrenfest: The Making of a Theoretical Physicist. Biography of Paul Ehrenfest. Amsterdam: Elsevier. ISBN 0-7204-0163-1. 1985 edition: ISBN 0-444-86948-4
- 1993: Physicists' Inaugural Lectures in History. Amsterdam University Press. ISBN 90-5356-057-2.
- 1993-1996: Einstein Papers Project. Lead editor, Volumes 3, 4, 5; editor, Volume 6, The Collected Papers of Albert Einstein. Princeton University Press: