Using Google Scholar to Assess the Impact of Philosophical Work (J. Stanley)

Academics spend much time trying to assess the relative merits of work in an area. There is no sure-fire way to do this of course. But citation indices are one method to assess the impact work has had on an area. Though philosophers are loathe to use them, they are widely used in other disciplines. Citation indices of course do not tell us everything we need to know to make such a judgment. Much work is of very high quality, but sufficiently specialized to be of interest to only a very few. Conversely, someone can write a paper that sparks a great deal of interest for its obvious flaws. Nevertheless, one can hope that citation indices could give us at least some sense of the major themes in a subject area. My sense is that as philosophy has become more specialized, more and more philosophers have simply lost contact with what is being currently discussed in journals and books. One might hope that citation indices could provide a rough objective map of the terrain of an area that can be used in place of word-of-mouth. 

Since I discovered Google Scholar about six months ago, I’ve been comparing its citation results to my general sense of what is going on in fields in which I work. Generally, it seems quite accurate – papers that have had a significant impact in an area have had correspondingly greater hits on Google Scholar than papers that have had smaller impacts. For example, some test cases: two much-admired recent papers that have created significant literatures in epistemology are Jim Pryor’s paper, “The Skeptic and the Dogmatist” and Adam Elga’s “Self-Locating Belief and the Sleeping Beauty Problem”. Google Scholar correctly reveals this; these are two of the most cited papers in epistemology since 2000 (68 for Pryor’s paper, and 40 for Elga’s paper). Keith DeRose’s “Solving the Skeptical Problem” is one of the most influential papers in epistemology written in the past thirty years, and Google Scholar again reveals this; it has 187 hits, despite being published as recently as 1995. Ted Sider’s book Four-Dimensionalism and Timothy Williamson’s Knowledge and Its Limits have been hugely important works, and Google Scholar clearly reveals this (266 hits and 303 hits respectively, despite publication dates of 2001 and 2000, respectively). One paper that has had no impact whatsoever in its field is Jason Stanley’s 1998 contribution to the literature on personal identity, “Persons and their Properties”. Again, Google Scholar correctly reveals this, since this paper has no hits.

There are of course pitfalls to using Google Scholar. First, one should refrain from comparing hit numbers across areas of philosophy. Some areas of philosophy (e.g. philosophy of mind that borders on philosophy of psychology and philosophy of language that borders on linguistics) are cross-disciplinary, and so have created literatures in multiple fields. This naturally increases the number of researchers reacting to these papers, and correspondingly the number of hits. If one wants to compare the impact just in philosophy of a certain work, this makes things difficult. Furthermore, some areas of philosophy seem to involve more citation than others, or simply more researchers. So one must take care to compare (e.g.) only work in history of modern to other work in history of modern, or work in meta-ethics to other work in meta-ethics. Finally, it takes a number of years for the impact of a work to register on Google Scholar. The publication date of an article is a very large factor, as the older an article or book is the more hits it will receive. It is not yet possible to use Google Scholar to assess the impact of publications from 2004 or after. So in judging the relative impact of work, it’s best to compare work that was published at roughly the same time. Nevertheless, after several months of procrastinating with it, in areas such as metaphysics, epistemology, philosophy of language, and philosophy of mind, when used with appropriate caution, it does deliver results that accord with my sense of what papers and books have created some of the major debates in these areas.