| As more usability practitioners start
conducting remote usability testing, there seems to be a
demand for some tips and guidelines around this technique. New
screen-sharing tools like Breeze, Co-Pilot, and GoToMeeting, and remote
usability tools like Ethnio and The Astoria
Project Beta, make it easier to conduct moderated remote
usability testing. Dealing with video and audio recordings
keeps getting simpler as well. But observing people remotely
presents a unique set of obstacles, so this is a guide to what
we’ve learned from conducting 149 remote studies with 1,213
participants over the last seven years. We can’t get that time
back, but hopefully some of what we’ve picked up will be
helpful.
Moderated Remote What?
Basically, there are two kinds of remote usability –
moderated and automated.
-
Moderated is one person watching
another person use a computer, viewing their screen
movements with a screen-sharing tool, and talking with them
over the phone. The moderator watches and listens to where
the participant runs into difficulty while interacting with
the application or site, and records both the conversation
and the participant’s screen. It’s pretty much the same as
in-person testing, but minus the user’s facial expressions.
Also known as facilitated remote research.
-
Automated means hundreds or thousands
of participants report their own behavior through a browser
window or frame that has an open text field and survey
questions. As the user navigates through a web site, they
enter their feedback and answer page-specific questions in
the browser frame. It’s more than a survey, because it’s
still behavioral, but there is nobody watching and talking
with the users. Some people call this unattended remote
usability. The data is usually automatically distilled into
a report with aggregate verbatim answers, click data, or
exercise results. Some of the best new automated remote
tools are MindCanvas, UserZoom, ClickTale, RelevantView, and KDA
Revelations.
You can and should mix these methods, but this article is
going to stick to moderated. If you’re just getting started
with remote usability, there are articles here, here, here,
and here,
that can give you more background.
Recruiting Doesn’t Have to Suck
In order to get started, you’ll need some users, and there
are basically three options for how you can recruit for remote
testing:
-
Traditional includes all the ways you
would recruit for an in-person study - email campaigns,
customer lists, recruiting agencies, craigslist ads, bus
stop flyers, whatever.
-
Online scheduled is where you use an
online survey or recruiting screener of some kind to
schedule participants for sessions in advance.
-
Live Recruiting means you use an online
screener to intercept people in the middle of their
real-life tasks, and watch them live as they complete those
tasks, in their native task environment. No scheduling in
advance. This method is so effective that I’m going to
dedicate the entire next section to it. Here we go.
Native Task Environment: A Really Big Deal
The ability to intercept someone live as they begin a task,
have them quickly share their screen and comments over the
phone, and then watch them continue to use their computer the
way they were about to anyway, is by far the greatest benefit
remote testing offers. The criteria by which participants make
click decisions are totally different when they have a
temporal, emotional, and logistical attachment to completing a
task. Throw in their native physical and computing
environments, and you are looking at the most accurate form of
behavioral usability research possible. Well, that might be a
stretch, but you see what I’m getting at. In order to catch
people in their native task environment, you’ll have to do
some form of live online recruiting.
Live Online Recruiting
This method assumes two things – you are usability testing
something that has a web site associated with it, and you can
convince the I.T. or Web director to let you temporarily place
some code on that page. Not always possible, of course. This
code can be a static link that goes to your Zoomerang survey,
or in the case of Ethnio, one line of JavaScript that triggers
a DHTML layer recruiting invite. Here are the details of this
method:
-
Got webernet? You’ll need that
editorial access to a web site or web application related to
the tool you are testing. Why do you need this? Because
that’s where you’re going to place the invite to your live
recruiting form. You can recruit live for software by asking
people from your organizations web site if they are going to
be using, say, QuickBooks, for anything that day. Then ask
them to describe the task they will be performing. You’ll
get real insight just from that alone, but then you can call
them before they’ve opened up QuickBooks for the day.
-
We all hate pop-ups but this is the
best way to do live recruiting. We use a DHTML layer that
cannot be blocked by a pop-up blocker inside Ethnio, but you
can also insert a link or a feature box onto a single page.
-
Call Them. Use your phone to make first
contact with participants instead of email. You may lose the
ability to get people to pre-setup for screen-sharing, but
they have to be able to start right away if you want to get
the true live behavior.
-
Six live recruits per hour is the
minimum response rate to do live recruiting (that we
suggest). This means you need to have at least six people to
choose from, in our experience, in order to not have to
schedule anybody in advance for your study, ever again. If
the site or web app you’re testing doesn’t have that kind of
traffic, you could sit around and wait forever, but that
generally isn’t much fun. With six per hour, you can conduct
one session per hour and get 5-6 sessions per day in easily.
Samurai moderators can do 8-10 per day. Grab a second
moderator and you can do 20 users in a day.
-
Use any old survey tool to put your
recruiting screener questions into. Just be prepared to do a
whole lot of exporting to excel to get the verbatim
responses. You could also build your own live recruiting
tool in Ruby on Rails. That would be mad Web 2.0. There is a
certain live
recruiting product out there as well.
-
It’s Tricky to Call Fast. Calling Live
Recruiting Respondents within a few seconds can be tricky.
It involves a lot of monitoring of your live responses
coming in, but when you get someone within a few seconds, it
yields awesome data.
-
Extremely Narrow Recruiting Quotas don’t really work with live recruiting. This means that as
you narrow down your potential target audience you won’t be
able to easily get live users. If you need to recruit a
married database administrator from the east coast who
enjoys marmoset photography and live folk music, you’re
going to need to schedule them in advance.
-
But I want new users. A common
misconception is that you cannot recruit brand new users
with an online invite of some kind from an existing web
site. We’ve found that using the live recruiting method to
call within 10 seconds of them arriving at the site and
filling out the screener will give you nearly the same
results as getting users from an agency or craigslist ad
that swear they have never used your web site before. Just
add a question in your screener that asks: “How many times
have you visited the site before?” You can also do a mix of
“new user” email recruits and live recruits and see for
yourself.
-
Lady Luck. In reality, live recruiting
takes a lot of luck to get people before they have started
actually using your site. But even calling within a few
minutes can yield good results, and you can absolutely mix
and match recruiting methods A common setup for us is 8
users live, and 8 users recruited by an agency.
Observing
This is where you watch someone else’s desktop behavior
using a screen-sharing tool. You just have to pick the
screen-sharing tool that’s best for you. No matter what tool
you use, some people won’t be able successfully share their
screen. We see about an 85% success rate just getting someone
to successfully install a screen-sharing tool. The participant
often has firewalls, no admin installation privileges, etc.
Here is a list of my favorite tools, and you can find updates
and a more detailed list at the remote usability wiki:
-
Breeze (It’s in Flash. Its slick. Very reliable. Kinda pricey)
-
Astoria
Project Beta (Coming soon from TechSmith,
pricing not yet determined)
-
GoToMeeting (Least expensive. Speed is much better than WebEx. Also, if
you are using the “Reverse Morae + GoToMeeting” method to
conduct remote sessions, you can use the features and flags
inside Morae to log behavior, which is cool. )
-
Co-Pilot (Based on
open source VNC, which rocks.)
-
WebEx (Slow, and
they once billed us $3,000 for two sessions. I hope they
fixed that glitch.)
-
Ethnio (Early release, hard to get an account, and only tool with
integrated audio recording and recruiting as of July,
‘06.)
Tips for Remote Moderating
Once you’ve chosen a tool for the screen sharing, you’ll
have some questions about how to contact participants and
interact with them over the phone. Do you prefer taking notes
by hand? You can still do that with remote testing but you
might consider giving the old evil machine a try for
note-taking. Coming up with a system for tagging quotes,
behavior, and video time code as it happens will make your
life so much easier. Here is a list of things to keep in mind
when you call the participant:
-
Practice Muting. Come up with a phrase
you can use frequently to put the participant on hold.
Practice saying, “Please hang on a sec while I adjust
something on my end.” You can say this to your participant
for ANY reason, so that you can put them on mute. If a fire
breaks out at your desk (it happens), “Can you hang on while
I adjust something on my end?” works great. If an observer
starts laughing hysterically and calling your participant a
moron, just go “Can you hold on again one more time while I
adjust something again?”, press mute, then slap the observer
with a reverse open palm.
-
Success Rate. Only 80-85% of the people
you call will be able to participate (and sometimes less).
Some will have insane firewalls that don’t even work with
the workhorse WebEx. Some will have kids that trip over
Ethernet cables. Many will be whispering to you from their
cubicle so quietly you can’t hear a d*mn thing they’re
saying. Just plan for this in the same way you recruit extra
lab users to accommodate for flakes.
-
Scary Willingness. Almost everyone you
call will be willing to install any crazy screen-sharing
tool you ask them too. This is scary, and I don’t know why
people are so willing. If someone doesn’t want to, thank
them and move on. Users recruited live are plentiful.
-
Outside Observers. The absolute most
fun way to do remote testing is in a big conference room
on-site with whoever is paying for your study (client, your
company, etc). Use a projector and speakers hooked up to
your phone tap, so that all the engineers, designers, and
stakeholders can come in to watch live. If the moderator
uses a headset, people can have low-level conversations in
the same room and the directional mic won’t pick it up. They
can also tap the moderator on the shoulder or send them an
IM if they want follow up questions or something changed
about the moderating. You can also setup remote observers
with most of the tools out there by having them join the
meeting. Make sure observers are invisible by hiding the
participant list, and make sure observers are muted if you
have them call in to a teleconference to listen to the
session. Your participant may guess there is more than one
person listening to them, but there is no need to make it
obvious by having outside observers giggle or sneeze when
the user does something silly. I mean interesting. Something
interesting. Set these three ground rules posted on the
door:
- No loud laughing or giggling.
-
Speak at whisper level.
- No yelling out what they think the participant should
click on.
-
Study Design is Up to You. You can use
the live recruiting method but have participants do all
pre-determined tasks, or even measure time-on-task. You can
mix online scheduled with live recruiting, or divert some
users to a prototype, just as you could in person.
Recording
Ahh, recording. You’d probably like to have the video and
audio from your sessions afterwards so you can view, edit, and
share them with your team. The easiest way would be to use a
screen-sharing tool that has built-in audio and video
recording, but until the Astoria Project Beta is complete or
Ethnio is fully launched, you’ll have to use a combination of
methods:
-
Camtasia + Phone Patch. You can use a
screen recording tool, like Camtasia, combined with this phone tap from JK
audio to record to AVI. You just plug this tap in
between the handset and your PBX or Analogue phone, and then
the microphone input on the computer you’re using to test.
The JK tap is inexpensive and the best quality phone audio
tap on the market. We’ve tried the ones from RadioShack,
HelloDirect, and WebEx, and they all pick up interference or
send low quality signals.
-
Teleconferencing. Another method is to
use a teleconferencing service that records the phone call
for you – Breeze alledgedly does this although we’ve never
got it to work. You still have to use a screen recording
tool such as Camtasia.
-
Speakerphone Rigged. Yet another method
is to position a microphone next to the speaker on your
phone, or speakerphone. This is officially jury-rigged. It
can work, but it’s susceptible to all sorts of problems. Mic
getting knocked over. Noises in the room. Plus, nobody wants
to listen to you over a speakerphone because you just don’t
sound like yourself.
-
Reverse GoToMeeting + Morae is a tricky
but popular method. You have to give the participant control
of a computer (through GoToMeeting or WebEx, etc.) on your
end that has Morae installed. Then you use Morae to record
all the participants behavior and somehow get the audio in.
The main problem is that the participant is surfing super
slow, because they are remote controlling the Morae
computer. This will be unnecessary with the Astoria Project
release, I’m sure.
International Love
Many people get into remote usability strictly for the
ability to test international participants. Here are the most
important things you need to know:
- Plan for about double the time that a
domestic remote usability session would take.
- Use a native language moderator if you can, or AT&T language line
service to get a native language operator on the phone
doing real-time translation between you and your user. Be
prepared to put subtitles in your highlight video using a
video editing program for the coolest sounding highlight
clips ever.
- You can also recruit English speaking participants in
any country, as long as they verify they have been a
resident for more than, say, 10 years..
- Realize that your cultural biases may have a huge, and
difficult to predict, impact on the results of your
international remote research. Just like in-person, but
easier to forget about since remote testing is just so much
easier than flying to Beirut all the time.
Incentives
We use Amazon gift certificates because they only require
an email address to fulfill. You can use any incentive you
like, but if it’s a check or an American Express gift card,
you might have to spend a considerable amount of time with
every person verifying their address and name spelling. I’m sorry, could you please spell Moheekuwaka Drive for
me?
Make it happen.
If you’d like to get started with remote testing, you have
three easy steps. First, choose a recruiting method, then
decide on a screen-sharing tool, and third, figure out if
audio and video recording is important to you and decide on a
method for that. Then you just have to give it a whirl,
ideally with a trial study, or you can just use it for an
actual project. If you’re already doing some remote usability,
don’t forget to experiment with different study designs and
new tools. In 1999, we did 5% of our studies remotely at Bolt
| Peters using Timbuktu or PcAnywhere. By 2005, almost 95% of
our studies were conducted remotely. Now we hardly ever leave
the computer screen, which is everyone’s goal, right? Nate Bolt is co-founder and CEO of Bolt | Peters where he
works on remote usability and ethnography. Having overseen
hundreds of moderated remote usability studies for clients
like Oracle, Time Warner, Princess Cruises, and Hallmark, he
led the creation of the first moderated remote usability
application, Ethnio. Nate speaks regularly about remote
research and created a degree titled “Digital Technology and
Society,” at the University of California, San Diego, which
focused on the intersection of technology and mass population
usage. He also studied at the Sorbonne in Paris, where he was
jailed briefly for playing drums in public without a
license. |
