Two Days of Remote Usability Testing

I’ve been looking forward to remote usability testing a major manufacturer’s website for some weeks now.  While some people may have doubts about the value of remote usability testing compared to face-to-face testing, I’m here to attest that the value is comparable, and in some respects, perhaps even higher.

In our arrangement, we observed the testing in a conference room on a large projection screen, while a facilitator (in another state) moderated the test sessions, and the participants sat at home at their own computers.

While you don’t typically see a user’s facial expressions in this remote testing set-up, here’s what you do get:

  • You hear what the user is saying (and perhaps listen better since you’re not distracted by what he/she looks like).
  • You see the user’s on-screen activity including what browser they use and how it’s set up (e.g., number of toolbars).
  • You get a better understanding of the user’s natural environment (whether their speakers are hooked up, whether there’s a dog in the house, etc.).
  • You may get more honest feedback.  I’m not sure what the research says about this, but my impression was that people not meeting face-to-face with a facilitator may be more comfortable saying what they really think. 
  • No need to sit in a dark room behind one-way glass, nor keep our voices down when we wanted to discuss users’ comments during a test.  (Our conference line was on mute, of course.)
  • No travel costs!

A few things that I thought made our remote testing experience even better included:

  • A two-way video conference set up with the remote facilitator so that we could see and talk to her after each session — as if she were in the room with us.
  • A research director who sat with our team and could chat online with the facilitator when we wanted to interject a question in the middle of a test.
  • Because the usability vendor was local, it was feasible for more project team members and the client to attend the test sessions.  This should help ensure that the findings are seriously considered, since nothing beats seeing users interact with your site first-hand. 

A few things I would keep in mind for next time include:

  • Add a question in the screener to ensure the participant has a LAN line which can be called for the test session.  Cell phone connnections can be unreliable.
  • As in face-to-face testing, consider hiring floaters who can be available to replace a participant who doesn’t answer the telephone for the test, has technical problems, or turns out not to be a great match to the target audience.

I look forward to more remote usability testing experiences!

2 thoughts on “Two Days of Remote Usability Testing

  1. Great article! A few other benefits I’d include:

    —Remote research enables “live remote testing”: recruit visitors to your website to participate in your study, and then begin the study right away; that means you can watch participants doing self-motivated tasks they were planning on doing anyway, rather than assigning artificial tasks they may not care about. (More on live recruiting <a href=""here.)

    —You can easily record sessions using software like Camtasia Studio, Screenflow, or iShowUHD

    —Along with the user’s natural environment, you can see their computing environment as well: bookmarks, other browser tabs that they’re using in conjunction with their current task, desktop shortcuts, and so on (with the user’s consent, of course)

    —The facial expression issue may be solved soon as webcams become more standard; services like iChat and Adobe Connect already have integrated webcam / screensharing. (Anyway, I don’t really believe that facial expressions matter all that much in user research: people have long since become comfortable with conveying emotion in tone of voice, and in any case, it’s the behavior you want to pay attention to, not just the facial expressions, which can be easy to misinterpret)

    And as for the other considerations:
    —Also remember that users may be on wifi connections, which can make the screensharing choppy. Make sure that the user is using a landline phone and a wired internet connection if possible.

    —Live recruiting can make hiring “floaters” unnecessary, because you can simply recruit visitors as they come to your site.

    Nate Bolt and I wrote a <a href="; book on Remote Research, which will be coming out soon, and which covers all of these issues. Check it out if you’re interested!

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s