Our Library Concierge project is in full swing, and because we included an assessment plan in the project from the start, we can already report some decent results.
To recap, our Library Concierge project is an ambitious initiative designed to promote exceptional public services across the Stanford Libraries, and to empower and equip all staff to provide that service. It refers to both a service perspective and a set of training experiences for all library staff. The training experiences are designed to increase staff familiarity with the full range of resources and services our organization offers, so that they can provide better public service, give more effective referrals, and can take advantage of opportunities to serve as fully informed ambassadors for the Stanford Libraries. Ultimately, our goal is to maximize Stanford scholars’ knowledge of and access to our resources and services; but we know we have to start by providing ways for our own staff to learn about the services offered by their colleagues throughout the organization. For a more thorough introduction to the project, you can watch this 8 minute video from our Introductory Session.
The core part of the project is our monthly Concierge sessions for library staff. Each session is designed to expose staff to some part of the Stanford Libraries and give them enough information about that service or resource that they can provide basic information to a patron and/or make an effective referral. We hold 3 sessions for each topic, which allows us to reach 150 staff each month with the live session. We also video each session and provide the videos on a secure CourseWork site dedicated to the project. The CourseWork site also serves as the home for discussion forums and supplemental materials on each topic.
Since our primary goal is to increase staff’s knowledge of various library resources and services, so that they can more confidently provide Concierge-level service to our scholars, our assessment plan consists of a series of pre-and post-test surveys assessing staff knowledge and confidence about the topics convered before and after the sessions on those topics. We also ask several questions designed to gather feedback on elements of the project itself, including suggestions for future topics.
To measure the impact of the training sessions on staff knowledge, we asked folks to answer the following question on a 1-100 slider scale:
“How confident do you feel in providing basic information about the following topics:”; and then we list both upcoming topics (pre-test), and past topics (post-test). We plan to deploy the surveys quarterly, so each survey will provide pre-test data for 2-3 topics, and post-test data for 2-3 topics.
We have now completed the initial baseline pre-test survey, and the first post-test survey, measuring change in staff knowledge for our first two topics: Copyright, Licensing, & E-Resources; and Numeric & Spatial Data Support. Preliminary results are very encouraging. On Copyright, Licensing, & E-Resources, average staff confidence in basic knowledge went from 40 before training to 46 after training. For Numeric & Spatial Data Support, average staff confidence ratings shot up from 21 to 44.
Write in comments on the Copyright, Licensing, & E-Resources session indicate that the smaller increase may be due to many staff coming away from the training with a realization that Copyright issues are more complicated than they had previously thought.
We employed a panel design, which will allow us to match individual answers over time (by means of an anonymous, randomly assigned ID). This means we will be able to look not just at aggregate change (which could be due to selection bias — e.g. people who felt they learned something may have been more likely to fill out the survey), but also at individual knowledge change over the course of the project. We also collected data on what parts of the training staff engaged in (live session, video, discussion forums, chat room, supplemental materials, etc.), what part of the organization they work in, and whether their job involves direct public service to scholars; so we will have the opportunity to do more sophisticated analysis (and better charts!) as we have time and more data. As always, stay tuned.
(Wrote this post on CalTrain, finished it up at the ballpark. Go Giants!)