How (some) Readers Navigate, or why sampling is important

How Readers Navigate to Scholarly Content describes results of a survey asking researchers their preferred starting point for navigating scholarly content. Given that their sample came from a list of scholars who were already signed up for alerting services, it is not at all surprising that they found a preference for alerting services over personalization or search.

This is a classic example of the common methodological error known as “sampling on the dependent variable.” The study does NOT show that scholars prefer alerting services. Their results show that scholars who subscribe to alerting services like alerting services.

Basically, this is like asking students who are studying in the library where they like to study, and then announcing that your study shows that students like to study in the library.

The study also claims to compare changing user behavior from 2005 to 2008, but there isn’t a single significance test in the entire report. Without significance testing, there is no way to know if the differences they found are due to chance, or to a real change in behavior or preferences.

I actually think alerting services are very cool, and I think understanding how scholars get to scholarly content is important. But because of the serious sampling bias, this study really doesn’t tell us much about scholars’ behavior (except for those scholars who already use alerts, which is likely to be a very small percentage of scholars.)

0 Responses to “How (some) Readers Navigate, or why sampling is important”

  1. Leave a Comment

Leave a Comment

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s

%d bloggers like this: