I am really, really excited about intercept recruiting for web site usability testing. I recently experienced this method of recruiting for some usability testing we did on a major auto manufacturer’s web site. Basically, it works like this:
- Site visitors are randomly “intercepted” at a designated location within the site by a pop-up asking them if they’d like to participate in a usability test RIGHT NOW in exchange for a reward;
- Person who says “yes” answers a few questions (screener) to ensure that they match the target profile;
- Remote facilitator scans the answers and, if they qualify, contacts them immediately by phone to engage them in a remote usability test (using online conference tool).
What I loved about this method of recruitment is the quality of the participants we encountered. These were not just folks who were interested in earning a few market research bucks; these were people truly interested in shopping for a vehicle at that very moment. In my opinion, the quality of the participants’ comments as they shopped the site were unusually insightful and helpful.
What made the testing even more interesting is that we allowed participants to proceed with shopping the site as they were planning to do anyway. Rather than a rigid discussion guide, we focused more on observing the participants look for answers to THEIR questions, and made note of instances where they had difficulty. By studying this natural shopping behavior, we gained some significant insights that we may not have encountered with a more scripted test. (If time permitted, however, we did throw in a few specific, task-oriented questions.)
This was our first usability test using this method of recruitment. While I still feel there are some projects better suited to traditional, face-to-face testing, our experience was so positive, I expect we’ll be doing much more of this, with continued analysis of natural shopping behavior.
I’ve been looking forward to remote usability testing a major manufacturer’s website for some weeks now. While some people may have doubts about the value of remote usability testing compared to face-to-face testing, I’m here to attest that the value is comparable, and in some respects, perhaps even higher.
In our arrangement, we observed the testing in a conference room on a large projection screen, while a facilitator (in another state) moderated the test sessions, and the participants sat at home at their own computers.
While you don’t typically see a user’s facial expressions in this remote testing set-up, here’s what you do get:
- You hear what the user is saying (and perhaps listen better since you’re not distracted by what he/she looks like).
- You see the user’s on-screen activity including what browser they use and how it’s set up (e.g., number of toolbars).
- You get a better understanding of the user’s natural environment (whether their speakers are hooked up, whether there’s a dog in the house, etc.).
- You may get more honest feedback. I’m not sure what the research says about this, but my impression was that people not meeting face-to-face with a facilitator may be more comfortable saying what they really think.
- No need to sit in a dark room behind one-way glass, nor keep our voices down when we wanted to discuss users’ comments during a test. (Our conference line was on mute, of course.)
- No travel costs!
A few things that I thought made our remote testing experience even better included:
- A two-way video conference set up with the remote facilitator so that we could see and talk to her after each session — as if she were in the room with us.
- A research director who sat with our team and could chat online with the facilitator when we wanted to interject a question in the middle of a test.
- Because the usability vendor was local, it was feasible for more project team members and the client to attend the test sessions. This should help ensure that the findings are seriously considered, since nothing beats seeing users interact with your site first-hand.
A few things I would keep in mind for next time include:
- Add a question in the screener to ensure the participant has a LAN line which can be called for the test session. Cell phone connnections can be unreliable.
- As in face-to-face testing, consider hiring floaters who can be available to replace a participant who doesn’t answer the telephone for the test, has technical problems, or turns out not to be a great match to the target audience.
I look forward to more remote usability testing experiences!
Most often, when you think of usability tests, you think of a facilitator guiding and observing a test partipant as they complete a set of predefined tasks. Recently, I’ve been considering a different approach to testing that involves letting participants loose on a web site (still encouraging them to think-out-loud) and observing what is perhaps a truer user experience, since the user is making most of the decisions about where to go and what’s worth doing (e.g., Download, Share). One reason that a facilitator may still be of use in these situations is that sometimes a user will leave the site to do research elsewhere and spend less time commenting on the site being tested. While this may be an important learning, it may not be the best use of time, and having a facilitator available to guide the user back to the focus of the study can be important. I’m curious to know if anyone has experience with this “let ’em loose” approach, and would love to hear suggestions.