How ROI killed the academic library

Edited to add: @jacobsberg makes the excellent point that this talk might better be titled “How ROI fails the academic library”. Despite the fact that it means giving up the allusion to this classic, I think he’s right.

I gave this talk at the ABLD/EBSLG/APBSLG Joint Meeting at Stanford University back in April and wrote about it in I think I’ve become a Feral Humanist. The theme of that conference was “Business Library ROI: Measuring Usage and Identifying Value”, so I gave an opening talk called “How ROI killed the academic library: A cautionary tale.” Barbara Fister’s recent column Let’s (Not) Do the Numbers inspired me to publish the text of the talk here. I think it is important that we take a serious, critical look at the movement towards reducing the value of academic libraries (and higher education, more generally) to a numbers game.
The talk was about 30 minutes long, so I’ve cut some stuff here; including the part where I explain why a Stanford librarian goes by “mchris4duke” on twitter.

How ROI killed the academic library: A cautionary tale

My current job with the Stanford libraries is Assistant University Librarian for Public Services – I am responsible for all the Social Sciences & Humanities libraries and librarians and for our Special Collections and University Archives. Let me tell you – that’s a lot of humanities responsibilities for a Sociologist. Especially a Sociologist from from one of the most quantitatively rigorous sociology programs in the nation at that. What I have learned from my amazing colleagues about the humanities, about humanities research, and about library support for the humanities has very much informed my evolving perspective on the future of academic libraries.

So, as you might gather from the title of my presentation, I want to talk today about my concerns about the ROI (Return on Investment) framework – especially as it applies to large academic libraries like Stanford.
For me, an ROI framework is dangerous for academic libraries for 2 big reasons:

1. ROI tends to focus on the short-term & quantitative; and real impact of academic libraries tends to be long-term & qualitative.
2. An ROI framework doesn’t account very well for “rare events”. And I think Academic Libraries are about, at least in part, facilitating rare events.

Let’s start with the short-term versus long-term tension.  When we talk about ROI for higher education, especially for research universities, we really aren’t talking about economic returns – at least not in any straightforward money-in, money-out kind of way– at least I hope we aren’t. Academic Libraries in the US are non-profits, so strict financial returns are not really our thing.  To understand and assess the value academic libraries bring to universities, I think you have to look at the mission of the university– which is not about making money.

Let’s look at Stanford’s mission – with the caveat that Stanford doesn’t actually have a current, officially labeled Mission Statement document. So we have to do a little archival research and look at our Founding Grant:

Founding Grant, Stanford University

From our Founding Grant 1891, we get this nugget about the original mission of Stanford University:

Its object, to qualify its students for personal success, and direct usefulness in life;

And its purposes, to promote the public welfare by exercising an influence in behalf of humanity and civilization, teaching the blessings of liberty regulated by law, and inculcating love and reverence for the great principles of government as derived from the inalienable rights of man to life, liberty, and the pursuit of happiness.

Well that should be fairly straightforward to measure, right? We just need to see if the investments we make in library services are contributing to students’ “personal success and usefulness” and whether our collection development work and our digital library infrastructure development “promote public welfare”.  That sounds easy enough, right?

I’m clearly being more than a bit facetious, but the truth is that deep down I do think that the investments we are making – in research services to students and scholars, in building deep collections, and in the development of next generation digital library tools and infrastructure do advance those original aims of the university, just not necessarily in tidy yearly increments that can be measured and reported as metrics used to gauge Return on our Investments.

But let’s see if we can find something more recent, to see if the current university administration here at Stanford give us some more measurable goals to work toward.  In kicking off the recently concluded Stanford Challenge, Pres. Hennessey articulated some goals:

“Our goal for The Stanford Challenge is nothing short of building a university for the 21^st century and beyond: A university that will better serve the world through the quality, impact, and vision of its research, and through the new generation of leaders it will produce.”

We’re still talking about some really long-term and lofty goals here. Basically, we want to produce great leaders, and solve world problems. Stanford’s true purpose (and I would argue the purpose of most major research universities) is not higher graduation rates, good retention rates, or even higher employment rates for our graduates. So the returns on Stanford’s investments, or even the return on any given students’ investment in Stanford, can’t be measured that way.  Those measures won’t tell you much about how well we accomplish our actual goals – which are lofty and inspiring and long-term.

So you see, I have concerns about the ROI focus on higher education generally as well.  And as I said, a big part of my concern about the increased emphasis on ROI and Assessment in higher education is that an ROI framework tends to encourage a focus on short-term outcomes, when higher education in general; and academic libraries specifically, are in the business of pursuing and producing long-term outcomes.

That doesn’t mean that we don’t expect a return on our investment – only that it may take a very long time for those returns to be realized; and those that want us to prove our value by showing adequate returns on their/our investments must be very patient.

Academic libraries, of course, exist to further the goals of their parent institutions.  At Stanford Libraries, we support Stanford’s goals and missions by doing what libraries do – we collect, describe, interpret, share, and preserve information.

We do that in all the old traditional ways — for example, we still purchase about 120,000 physical books each year; and we logged over 628 thousand circulation transactions last year. We answer nearly 154 thousand reference questions and conduct 1000 workshops every year.

We also protect, collect and provide access to information in many new ways; as information production, discovery, use, re-use and consumption is happening in all kinds of new and innovative ways. Ways that our traditional measures of usage may not capture very well.

One example of Stanford Libraries’ innovative spirit is our involvement in the Google Books project. In 2005, we took a leap of faith as one of the original 5 libraries, agreeing to let Google digitize our collections. We did this in the hopes that getting the words inside our books indexed and therefore searchable would enhance discovery and would open up the treasures of our collection to a broader audience.  While we are disappointed in the lack of a settlement agreement, we remain pleased at the positive impact the Google Books project has had on discoverability and use of collections.

It is important to remember that the text-mining research made possible because of the enormous corpus of digital texts depended on us collecting and retaining a whole lot of books over time — and lots of those books never circulated.  Many of the kinds of research questions that can now be asked because the words in the books have now become data, can only be answered because of the sheer size and comprehensiveness of the corpus.

For example, there is a graduate student here at Stanford using both Google Books and HathiTrust files of Portugese language publications as a means of tracing the evolution of Brazilian Portugese.  I suspect many of the books that make up the data for this project  have rather dismal circulation histories.  If libraries like Stanford had only collected and preserved books with immediate and measurable use, the ever growing corpus of digitized texts would be even more skewed and biased than it already is.

In addition, I’m certain that the librarians who selected the titles that are now part of this scholar’s data never anticipated this sort of use for their selections. We need to be very cognizant of the fact that the objects we collect today (physical or digital) will almost certainly be used in ways we cannot yet anticipate. Or, as Rufus Pollock of the Open Knowledge Foundation says “The coolest thing to do with your data will be thought of by someone else” Or, as Dan Cohen, Director of the Center for History & New Media at George Mason University likes to say: “scholars have uses for archives that archivists cannot anticipate.”

OK – so any assessment of the ROI on the collections of a large academic library has to account for long-term impacts. Perhaps that is not particularly controversial to this audience, but I do hope you are all preaching this story to your university administrators.

What about library services? Surely we can apply ROI to assess the effectiveness of our reference and instruction efforts. Of course we can, but again I caution against relying on simple use statistics and short-term returns.

That said, I have already bragged about the fact that we continue to answer over 150 thousand reference questions a year, and that number is actually up from last year. But before you let me celebrate that figure – and believe me I want to and I have – let’s stop and think about what it really means?

How do the number of reference questions asked and answered help us understand our impact? Is the number of reference questions a good proxy for the effectiveness of our reference program at contributing to teaching, learning and research at Stanford? Probably a better measure than some, but still not a very direct measure. After all – one could argue that lots of reference questions just means our online tools are too confusing (they are), that our website is not particularly user-centered (it is now) or that it is hard to negotiate our stacks (it is).

Instruction is an area where I think we are doing a pretty good job at assessing our effectiveness. Here at Stanford, we have partnered closely with the Freshman writing program for many, many years – providing a designated librarian and a library tour & workshop for every first year  writing class.

When I first took responsibility for the instruction program, our only assessment of this rather significant investment of librarian time was a survey we asked the students to complete at the end of their workshop. Those surveys were nice – we usually got high marks, and most of the  librarians could use the results to learn how to improve their presentation styles (lots of us talk too fast, apparently); but those surveys told us nothing about how the workshops, and the availability of an assigned librarian, contributed to the goals of the first year writing program, or to our goals of developing students’ enthusiasm for and skill in finding and using scholarly resources.

So, we added a survey at the end of each quarter, after students have submitted their research papers. And the results are quite encouraging:

• 99% of students use the library catalog; and although we don’t have any comparative data – we don’t withhold the workshop from a random sample of students in order to have a control group – I feel pretty confident that 99% is a much higher percentage than we would get w/o our workshops
• The Library catalog and the Library databases were rated most useful (ahead of Google and Wikipedia)
• Nearly 40% of students consulted a librarian about their research paper
• Students who consulted a librarian were more likely to use Library databases and the online Research Guides, and rated the Library databases more useful than those who did not consult a librarian

To my mind, this is good data to show that our investment in instruction is paying off in terms of use of library resources and an appreciation of the value of library resources – including the librarians themselves.

The next logical step for us would be to conduct even longer term assessments – it would be great to know if the work we are doing with the freshman pays off throughout four years and beyond.

So, by now you can see that I’m not categorically opposed to library assessment or to the practice of calculating the returns on our investment. I am merely cautious about it, especially when an ROI approach leads us to focus on short-term outcomes that might be very far removed from the long-term goals we have of supporting research and learning in the service of developing educated citizens who will solve world problems. I think we need to really think carefully about the statistics that we collect and the metrics that we use, lest we start to mistake circulation or reference traffic as the goals.

This is the philosophy, in part, that drives our investments in areas like digital preservation, Digital Forensics and web archiving.

This is long-term stuff … where we are preserving and collecting “just in case”.  Digital forensics and web archiving are exercises in both digital collecting and digital preservation – both of which are long-term investments.  An example is our web archiving of Middle Eastern political sites and Iranian blogs – we happen to have researchers at Stanford using these now; but even if we didn’t, we still think there is long-term value in archiving these bits of world history.

As we continue to collect archival materials – based on our judgment about what and whose archives will have long-term value to scholars and to society – we are increasingly collecting items that are born-digital.  Email archives and drafts of articles and papers often come to the archives on hard drives or computer discs only.  And here is the key difference in investment in digital archiving and paper archiving – with fairly minimal intervention, we can take a box of letters or paper manuscripts and put them in appropriate storage conditions and trust that when we get around to processing them, they will still be usable.  This is sometimes referred to as preservation by benign neglect.

Digital preservation is hard

Not so with digital archives – those floppy discs we got from a Nobel laureate in physics that contain his email archives and some other “stuff” he assures us might be interesting, have to be dealt with quickly, before the data deteriorates to an unusable state. A coffee stain on a paper manuscript is unfortunate, but with the right treatment, the manuscript is still “available”, and still readable. In the case of bit rot on a digital manuscript –we are often looking at complete file loss.  So the investment in extracting, preserving, and reformatting born digital materials is often considerable.

And for much of what we collect and preserve, that investment represents a leap of faith. We are making our best guesses (as librarians and archivists always have) at what is valuable to scholars today and what will be valuable to future generations. And in the case of digital preservation, we are making our best guesses at what formats will work in the future, with the full realization that continual integrity checking and reformatting are part of the new responsibilities of digital archivists.

In addition to a willingness to patiently focus on long-term returns when assessing value of academic libraries investments, I would argue that we also need to recognize the qualitative rather than merely the quantitative nature of our contributions.

Circulation is one of the quantitative measures that I fear is way over-emphasized in many libraries and by many university administrators.

Lord of the Rings trilogy

When we look at Stanford Libraries’ circulation numbers, the Lord of the Rings DVDs would seem to yield the highest return on our collection investments – since it is our most heavily circulated item in the last 5 years.  Now, as perhaps the only American librarian who has actually never seen nor read Lord of the Rings; I feel that I must pause here and note that I am not saying we shouldn’t have the  Lord of the Rings DVD, or that we shouldn’t be quite happy that it circulates.  I’m sure it is a fine movie,with considerable academic value. But I do have an issue with using circulation as a key measure of value, if for no other reason than it would lead us to over-value  Lord of the Rings and undervalue collections like our historical newspapers. After all,  Lord of the Rings is our most heavily circulated item, and the microfilm reels that contain the text and images from 100s of years of historic US newspapers are much less frequently used.

Railroaded by Richard White

But if we care about actual impact on research, we might want to look at historian Richard White’s recent book Railroaded, which provides a new and controversial vision of the so-called Gilded Age in the US, and the impact of the Transcontinental railroads on the making of modern America.  White relied heavily on archival materials rarely used by others, and on dusty reels on microfilm that he may well have been the first to pull out of the file cabinets.   The centrality of the archives to White’s research is acknowledged, quite literally, in the acknowledgements section:

“the legendary Margaret Kimball helped me go through the holdings and find what I needed. Jim Kent, who runs the media and microfilm room at Green Library, and his staff helped me in ways probably best kept between us…”

I like this example for at least 2 other reasons:

1. It highlights again the long-term nature of library work – as we all know, books take a long time to write. The time between research and eventual publication is usually many years. In this case, the librarians Richard White acknowledges had both long since retired before the publication of Railroaded.
2. I think acknowledgments of libraries, archives, librarians, and archivists in published materials constitute a direct and real measure of our impact on scholarship. I don’t know of any academic library that measures and tracks acknowledgements systematically; and I’m not even sure how it might be done. But I think it is an idea with considerable potential. Of course, it works best for book based disciplines, as journal authors rarely acknowledge the library work involved in providing them access to all those prior works they read and cited.

Books still matter

I also want to emphasize the value of simply having a large collection of books—even if some, maybe many of them, don’t circulate.  Students and scholars benefit from being surrounded by lots of books.  Writer, editor, book reviewer Kristy Logan wrote recently about the impact of the 800 unread books on her shelves:

Sometimes I hold these books in my hands and imagine what I will learn from them. These books have affected my writing, and I haven’t even read them. Maybe we can learn as much from our expectations of a story as we can from the actual words on the page.

In the chapter titled “Library Life” in the book “Stop what you’re doing and Read this”, Zadie Smith writes of the the influential role studying in her local library had on her development as a scholar:

It was a community of individuals, working to individual goals, in a public space. It’s short-sighted to think all our goals were bookish ones. I happened to be in the library in the hope it would lead to me to other libraries, but my fellow students were seeking all kinds of futures: in dentistry, in social work, in education, in catering, in engineering, in management. We all learned a lot of things in Willesden Green Library, and we learned how to learn things, which is more important…
But I know I never would have seen a single university library if I had not grown up living a hundred yards from that library in Willesden Green.

Let me stop here and say that if you have not yet read “Stop what your’re doing and read this”, well … you should stop what you’re doing and read it.

And if these qualitative testaments don’t convince you, there is data!

The results of a 2010 cross-national study of family scholarly culture and children’s educational attainment showed that:
“Children growing up in homes with many books get 3 years more schooling than children from bookless homes, independent of their parents’ education, occupation, and class.”

My point is this – there is value to libraries and to collections that are no less real and no less impactful for the fact that they can’t really be measured. That is a hard truth for a quantitatively trained sociologist to admit, but I have come to believe it. So, yes, we should practice continual assessment and we should gather as much evidence as we can that shows the impact of our collections and services on the goals of our institutions – but we should do so with an eye towards the long-term and the big picture; lest we fall prey to measuring (and therefore doing and funding) only what is quick and easy.

So now let me turn to my real, real concern with ROI –which is that quantitative assessments will always miss one of the most important functions of an academic library – which is to facilitate the rare event. Yes, I know, how novel—a librarian talking about serendipity. But remember, I’m not a real librarian, and my belief in serendipity has developed slowly and skeptically – but I am a convert. I have come to believe that it is absolutely the responsibility of libraries to encourage, support and in all ways make possible the unanticipated discoveries that lead to new knowledge, new ways of thinking and new contributions to the cultural and scholarly record. In fact, I think providing the context in which new, unanticipated, unique discoveries, thoughts, connections, and inspirations are sparked may be the most important value-added contribution that libraries make.

Allow me to share a couple of fairly recent examples of the kind of serendipity made possible by the careful work of libraries.

In 1989, in honor of Condoleeza Rice, Walter Hewlett gifted to Stanford an autographed fragment of a musical score from German composer Robert Schumman. Some 20 years later, Frederick Moyer, a concert pianist, and his uncle, Paul Green, an engineer; tracked the score down in the Stanford Libraries, requested and received via InterLibrary Borrowing a digital scan of the score, and created the first ever playable version of Robert Schumman’s hitherto unfinished 4th sonata.

In 2010, two Harvard professors “discovered”, by scouring newspaper microfilm in the basement of the Widener Library at Harvard, two new short stories by famed Harlem Renaissance author Zora Neale Hurston. These two stories were never listed in any of the published collected works of Hurston and had not yet been studied or analyzed by other scholars.

US release date for Super Mario Bros. remains a mystery

And here’s an example of the failure of the archives (archives writ large, not any particular archive). One of the great unknowns in video game history is the U.S. release date for Super Mario Brothers. We take video game history rather seriously here at Stanford, in fact we hold one of the largest historical collections of video games in the world. The lack of careful documentation and archiving of that documentation actually represents a fairly substantial gap in the history of video gaming, as Super Mario Brothers is one of the most successful, iconic and influential video games in the history of the industry. But it is hard to confidently trace its influence on the development of the industry when historians can’t yet agree even on the year of its US release.

I’m sure many of you could provide other powerful examples of the same sorts of serendipitous discoveries of and rare uses of materials hiding in libraries. Or of missing archives that hinder scholarly progress.

And yet — in the back of my head (and perhaps in the mutterings of this very audience), I hear a little voice reminding me that “the plural of anecdote is not data”.
But let’s all remember that the fact that these accounts are merely anecdotal does not render them any less true. These stories, and countless others, represent real contributions to scholarship and to our understanding and appreciation of the world.

Serendipity by definition is a rare, unexpected, and unanticipated occurrence. But it is still real. One of the most well-known quotes about serendipity comes from the French scientist Louis Pasteur, who claimed that “in the field of observation, chance favors the prepared mind”. Surely that is true — but allow me to offer the librarian addendum that “chance also favors those with access to great libraries.”

Perhaps I have presented an overly romantic, even mystical portrait of academic libraries – and at a time when libraries and higher education are under the gun to get practical. But I guess what I am suggesting is that if we don’t defend the hard to define and even harder to measure qualitative importance of libraries, who will?

And, I suspect that many of you probably agree with me, at least in principle, that universities ought to have great libraries, with expert staff and large collections and a range of services in support of teaching and research. But of course, we all face constraints in the forms of budgets, space, and competing priorities.

So, yes, by all means find good ways to measure our contributions to the aims of higher education. But also, please, take opportunities to evangelize on behalf of the non-measurable impact of libraries – make sure your administration knows that there is value in books that aren’t read, in data that hasn’t been used yet, in archives yet to be discovered, and in the mere fact of great libraries.

My final slide. Pretty hokey, huh?