Scherer was born in Schönborn, Austrian Empire (present-day Göllersdorf, Lower Austria, Austria). He was educated at the academic gymnasium in Vienna and afterwards at the University of Vienna, where he was a favorite pupil of the distinguished Germanist, Karl Müllenhoff (1818–1884). Having taken the degree of doctor philosophiae, he became privatdozent for German language and literature in 1864.
In 1868 he was named a full professor at Vienna, and in 1872 received a call in a like capacity to the University of Strasbourg, and in 1877 to the University of Berlin, where in 1884 he was made a member of the Academy of Sciences.
Scherer's literary activity falls into three categories: in Vienna was the philologist, at Strasbourg the professor of literature and Berlin the author.
His earliest work was a biography of the philologist Jakob Grimm (1865, 2nd ed. 1885); the next, in conjunction with his former teacher Müllenhoff, published Denkmäler deutscher Poesie und Prose aus dem 8. bis 12. Jahrhundert (1864, ed. 1892). His first major work was Zur Geschichte der deutschen Sprache (Berlin, 1868; 3rd ed., 1890), a history of the German language with special reference to phonetic laws.
He contributed the section on Alsatian literature to Ottokar Lorenz's Geschichte des Elsasses (1871, 3rd ed. 1886). Other important works are Geschichte Poeten der deutschen Kaiserzeit (Strassburg, 1875); Geschichte der deutschen Dichtung im 11. und 12. Jarhundert (1875); and Vorträge und Aufsätze zur Geschichte des geistlichen Lebens in Deutschland und Österreich (1874).
Scherer's best known work is his history of German literature, Geschichte der deutschen Literatur (Berlin, 1883; 10th ed., 1905; English translation Mrs F. C. Conybeare, 1883; new ed., 1906). This work is distinguished by the clearness with which details are co-ordinated with general and comprehensive survey of German literature from the beginning to the death of Goethe.
Besides many other philological treatises, Scherer wrote largely on Goethe (Aus Goethes Frühzeit, 1879; Aufsätze über Goethe, 1886), and took an active part in the foundation of the Goethe archives at Weimar. A small treatise on Poetik, a biography of Karl Müllenhoff, and two volumes of Kleine Schriften were published after his death.