In Moving beyond the photo album, Kevin Smith describes the journal article as “a “snapshot” of research… increasingly far-removed from the actual research process and have less and less relevance to it.”

Smith summarizes a talk by G. Sayeed Choudhury, Associate Dean for Library Digital Programs at Johns Hopkins University, in which:

Choudhury called on libraries to move past a vision of themselves as merely a collection of these snapshots and become more active participants in the research process. He recounted a conversation he had with one researcher who, in focusing on the real need he felt in his own work, told Sayeed that he did not care if the library ever licensed another e-journal again, but he did need their expertise to help preserve and curate his research data. The challenge for libraries is to radically rethink how we spend our money and allocate the expertise of our staffs in ways that actually address felt needs on our campuses and do not leave us merely pasting more snapshots into a giant photo album that fewer people every day will look at.

Of course, as Smith notes, promotion and tenure systems still rely on the “outmoded system of scholarly communications that is represented by the scientific journal”, even as actual scholarly communication is happening outside of, and in spite of, published journal articles.

What are libraries to do? Dwindling budgets are forcing many of us to cancel journal subscriptions anyway; but as long as tenure and promotion rely on traditional citation and publication counts, we can’t very well stop collecting the snapshots that are traditional scholarship. But, we can play a role in promoting new modes of scholarly communication, and we ought to be participating in conversations about establishing better, more efficient, more relevant ways of evaluating scholarship.

Edited 8/27/09: Thanks to @ericrumsey for the better, more accurate title suggestion.

Several recent papers and/or blog posts have highlighted the social nature of citation practices, challenging the assumption that citation counts stand as impartial measures of the quality and impact of a piece of scholarly work:

1. In Position matters Philip Davis summarizes Positional Effects on Citation and Readership in arXiv , in which the authors find that articles listed at the top of arXiv’s daily publication announcements were cited more frequently than articles listed lower. The authors go on to note:
   

   we’ve documented here that accidental forms of visibility can drive early readership, with consequent early citation potentially initiating a feedback loop to more readership and citation, ultimately leaving measurable and significant traces in the citation record.

2. Although Henry at Crooked Timber rather convincingly argues that Charles Rowley’s article (behind paywall), ‘The curious citation practices of Avner Greif: Janet Landa comes to grief’; is “one of the sorriest hack-jobs that I’ve ever had the misfortune to read in an academic journal”; the Rowley article does highlight the general issue of the very real influence that individual citation choices can have. Scholarly careers can be made or broken by citation (or lack thereof) in a subsequently influential piece of scholarship. Once cited in one influential paper, one’s work is more likely to become part of the regularly cited literature on a topic, as subsequent scholars cite the same works.
3. Finally, Steven Greenberg’s How citation distortions create unfounded authority: analysis of a citation network highlights the social side of citations by demonstrating how “the persuasive use of citation–bias, amplification, and invention–can be used to establish unfounded scientific claims as fact.”

Clearly lots of factors affect when, where, why, by whom and how often a given scholarly work is cited. It seems like it would be good for us all to remember not to place undue weight on either individual citations or citation patterns; and not to blindly rely on citation count as an unbiased, objective measure of quality.

The question of “how do you do a lit review?” is being discussed by folks at my favorite Sociology blog, scatterplot. As a librarian, it is very interesting to “eavesdrop” on faculty having this discussion, and see what advice they have:

1. ISI Citation Databases/Social Science Citation Index
-If you aren’t already familiar with this then you need to be! Extremely useful for finding key articles because of the ability to sort by number of citations. This isn’t a sure-fire way to find the most important articles so don’t just use number of citations but it pretty much always leads you to some key articles and from there you can quickly get to others.

2. Google Scholar
-Also can be good for articles but I use it primarily for books. Particularly nice for books because of the google books project that let’s you flip through some of the book to get a sense of whether or not it might be useful before you run to the library or purchase it.

3. Comp reading lists from our department and other top departments
-A good place to find an introduction to a general field. It will give you a sense of some of the most important articles and the general topics for the field and you can dig deeper from there. Some times reading lists will be fairly comprehensive, though they rarely have the latest “cutting-edge” research.

4. Syllabi from key people in the field or really respected institutions
-Works similarly to the comp reading list though it may be better in some instances. Of course, if you’re looking for syllabi based on the key people in the field this requires you to know who these key people are in the first place. If it’s a fresh syllabus then it will often have newer research. Depending on the time put into the syllabus and the level of detail it may also give you a much better sense of how the field is loosely organized.

5. Annual Reviews
-Not surprisingly, these tend to have diminishing returns as they age. If there’s one that’s recent and for your specific interest then these are often money. If the article is less recent but not especially “old” it should still give you a nice framework upon which to build. If it is older then it can still be useful though. Some times the key features of a debate last a long time, often debates are cyclical, and, if nothing else, they can give you a bit of a history lesson to help you understand where the current literature is coming from.

Some of the comments contain great tidbits too … like “Reviewing the literature and writing the literature review are not the same thing.”

This makes me think that we (librarians) might consider pitching our drop-in workshops not as “How to use library databases”, but as “How to do a literature review”. And maybe we should get some interested faculty to co-teach a session with us.

Once again, the folks at hangingtogether are my source for new research — in Efficiency and scholarly information practices, they describe a new study published in Journal of the American Society for Information Science and Technology : JASIST that purports to refute an earlier study published in Science magazine that shows that as journal content becomes available online, citations get narrower.

(Full citation: See: Larivière, Vincent, Yves Gingras, and Eric Archambault. “The Decline in the Concentration of Citations, 1900-2007.” Journal of the American Society for Information Science and Technology : JASIST. 60. 4 (2009): 858-862. Stanford-only link).

The original article “Electronic Publishing and the Narrowing of Science and Scholarship”, by friend and colleague James Evans, attracted quite a bit of attention (I wrote a bit about it myself), so I am not surprised that someone has already tried to replicate (or refute) his findings.

In “The Decline in the Concentration of Citations, 1900-2007.” the authors claim that contrary to Evans’ findings, citations are becoming increasingly dispersed over time. The major problem here is that Evans has done a fairly sophisticated analysis (including negative binomial models) that controls for time. This allows him to isolate the estimated effect of journal online availability on trends and patterns of citations. In contrast, Lariviere, Gingras, and Arhambault readily admit that their models “do not take into account ‘the online availability’ variable.”

In other words, the study that claims to refute Evans’ work actually fails to account for the key independent variable, and instead shows simple trends over time.