We all want to create products that make people’s lives easier and more enjoyable, products that delight people around us and ourselves. Creating great experiences starts with understanding your users, their behavior, motivations and goals.
Remote user studies are ideal for getting insights from end-users who may be located anywhere in the world. These studies provide access to a larger pool of potential customers, cut out travel time, and can significantly lower the cost of usability testing.
In this article I would like to discuss methods and tools for effective remote UX research, as well as how to make sure your research findings are put to good use.
When remote study works
Remote user research has two key advantages over traditional testing:
- Flexibility. The remote method allows you to connect with your customers from anywhere in the world. There’s also greater user diversity and a quicker set-up time thanks to online implementation.
- Environment. User interviews are conducted in the participant’s usual working environment with the hardware and software they use on a day-to-day basis, or while they are on the go.
When remote study doesn’t work
There are a lots of reasons you might want to conduct an in-person study instead.
- Information security.You may want to keep control over confidential test materials.
- Inability to use screen sharing.Some studies may require you to talk to users who don’t have reliable high-speed internet connections.
- Face and body movements.In some studies, it’s importance to be able to see the user’s body or face, or track eye movements.
When you need to do a lab study, use reliable equipment, keep the interface 100% secure, and monitor the user’s physical movements when you can’t use screen sharing tools.
Moderated vs automated research
Remote research can be divided into two categories: moderated and unmoderated research.
Moderated
Creating great experiences starts with understanding your users, their behavior and goals
In moderated research, a moderator speaks directly to the participants. Moderated research allows you to gather in-depth qualitative feedback, as the researcher can make personal observations about the participants.
Since moderated studies provide lots of context and insight into exactly what users are doing and why, these methods are best for research projects that are looking for new ideas to come from observation.
Automated
Unlike moderated research, automated research does not involve any conversation between the researcher and the participants. Instead, you use online tools and services to collect information automatically.
You can conduct automated research to get feedback from a large number of participants on a specific set of tasks. Automated research is usually quantitative and good at addressing specific questions like:
- What percent of users can successfully log in?
- How long does it take for users to find a product?
If all you need is performance data, and not the reasons users behave the way they do, then automated testing is for you.
How to design a remote user study
There are several similarities between planning a standard or a remote user study, as you’ll need to establish:
- What. You need to decide what you want to learn in the study. Before starting, everyone on the team should agree on the questions you plan to ask and the assumptions you plan to test.
- Who. You’ll need to carefully recruit people based on your goals—do you want to study existing customers, prospective customers, or representative customers? I personally like a collaborative approach here where you work with your team to identify goals for the study, as well list the characteristics of the people you want to interview or exclude from your study. After you’ve done this, figure out precise criteria you can use to identify those people and write questions to screen them.
- How. Determine the best research methodology for a particular study that will allow you to reach your goals, and write an interview guide or that keeps you on track during the interviews.
Next, I would like to focus on “Who” and “How” steps and discuss what is different when you plan and manage a remote study.
Recruiting for web or mobile study
Live recruiting
It is very important to understand the usage context of an interface—the physical environment where people are using a product. Remote research allows you to instantly recruit people who are right in the middle of performing the task you’re interested in, in a process called Time-Aware Research.
Time-awareness means that users are interested in what they’re doing because they’re doing it for their own reasons, not because you’re asking them to.
(For more details on Time-aware research, I highly recommend Nate Bolt’s amazing book “Remote Research.” It also has a lot of practical tips and templates that you can apply in your own remote study).
An effective strategy for live recruiting is to use a popup form that will open the recruiting screener, which users can fill out to opt-in to your study. Alternatively, you can place a link to your survey in some noticeable spot on your site. The easiest way to create a survey is to link to a form in created in Google Forms, or use a tool like Ethnio.
Ethnio is a customizable screener that can be placed as a pop-up on your site or accessed through a direct link to an Ethnio URL. Ethnio provides a 2-line JavaScript code, which you should place on the page where you want to intercept your users.
It will take some trial and error to determine when exactly to show the popup and the right balance between session length and incentive.
Ethnio also works to find participants in real-time from mobile apps.
For mobile survey requests, you’ll need to place code from Android or iOS libraries into your app, then trigger an Ethnio screener based on actions you define in your app.
An alternative approach is to just send a push notification to the selected audience with a request to take part in the study.
Other channels for recruiting
In addition to the channels listed above, you can also use other channels to recruit someone for a study. Use agencies when you have higher budget—Usertesting is one such agency that screens people very thoroughly. Facebook or Craiglist ads work in some cases too.
Live recruiting is typically your best option if you have an existing site or an app. But in order to get people to participate, you’ll most likely need to provide an incentive.
Incentive
The incentive amount you should offer to the participants depends heavily on whom you are looking to recruit.
From my experience the average participation incentive for a typical remote study is somewhere between $35 and $50, which is lower than most in-person studies. For example, if you are testing doctors or lawyers, you will most likely need to hike up the amount
On the other hand, there are plenty of people who are willing to donate their time for no incentive at all. This is especially true for services with loyal users, who are quite often eager to participate in the study. They may bring up problems they have encountered with your product.
There is a great article by Charles Liu, design researcher from KISSmetrics, sharing his experiments with recruiting people via emails with a zero budget.
One of the tips he gives is to change the conversation from “Help me” to “Could this help you?”
In the article, he shares several email templates that you can use for recruiting people for your own studies. It’s definitely worth a look.
Tools for remote UX studies
There are several different tools that can be used for remote research.
Moderated—Screen sharing & recording
In the screener form, I’ll typically ask the participants what screen sharing service is convenient for them. More often than not, Skype is the first choice, but sometimes they are not familiar with screen sharing. In this case, I suggest what better works for us.
For the remote studies at Stanfy, we usually use the following tools:
- Skype—you can observe the face of the participant and their screen.
- Google Hangout —you can observe either their screen or face.
- GoToMeeting($49/month) —you can observe their face and the screen, and you can also give keyboard and mouse control to the participant.
GoToMeeting lets you view your participants’ screen, and allows your participants to see and even control your screen. This feature is very useful when you want to test a mobile app launched in an emulator.
For screen recording, you can use tools such as the following:
- Camtasia Studio(Mac, PC), $150.
- IShowU HD(Mac), $60
- QuickTime (Mac), Free
The first two solutions may be a bit pricey, but they will cover all your screen recording needs. All three have nice video and audio recording options to record portions of the screen or windows.
Automated—Usability study and task elicitation
There are a number of automated tools that prompt users to perform a task and then record their behavior. If you want to learn how users interact with some part of your website or app, task elicitation is the way to go.
Task elicitation is a flexible way of getting a specific answer to common usability issues, such as the following:
- Do users have difficulty performing the task?
- How many people abandon the task, and where?
- What is the most common path users take through the interface to complete a task?
- Where do users go to perform the task and where do they expect to go?
There are several different tools you can use for task elicitation.
Loop11. A browser bar prompts users to perform a task. Automated analysis includes task completion rate, time on task, first click, click stream, most frequent success and fail points.
UserTesting. Users’ spoken are captured and synced with videos of their site behavior as they perform your tasks. Videos are downloadable and can be annotated with comments. You can also see users’ demographics and system info. Users are recruited by UserTesting.com from a preselected panel.
Moderated or automated mobile studies
When running a mobile study, there are two different tools I recommend.
Lookback. Lookback helps you learn how people are actually using your mobile app. It records the screen of your app, along with the face and voice of the user, as well as all their taps and touches. With recordings like these, it’s super simple for anybody to send you feedback or bug reports, straight from within your app.
Lookback for Android records everything on screen, including any native app, mobile website or prototype. (iOS blocks third-party apps from recording this data).
Once a record is completed and uploaded, it appears on Lookback’s website where you and your collaborators can view, discuss, rename and share them.
Unicorns.io. Unicorns is an app that streams whatever you’re doing on your phone—playing a game, texting, swiping through Tinder, etc. Unicorns allows you to conduct usability tests, showcase pre-release apps or upcoming features, and chat with the users live.
PlaytestCloud. PlaytestCloud is an online service for crowdsourced testing for mobile games. The video recordings show you the player’s screen during their playthrough and highlight their touches on the screen. You can also discuss and share the videos inside your team.
Invisionapp. Invisionapp allows you to transform your static screens into clickable, interactive prototypes complete with gestures, transitions, and animations. You can share a link with your users to open on their desktop or mobile device, then conduct usability tests with the users live. Invisionapp is great for quickly demonstrating concepts, collecting feedback in one place and making rapid iterations.
Recently, Invisionapp partnered with Lookback so that you can record a user as they navigate through your Invisionapp prototype on their iOS device. You can capture their screen as well as audio and video reactions.
UserTesting. I already mentioned this tool before in relation to un-moderated web studies, but it works great for mobile studies as well. You can watch users complete the tasks you specify (including gestures), and listen to them speak their minds as they interact with your app. Their mobile recorder works with whatever is displayed on the mobile device, including wireframes and prototypes.
Observers
Remote research allows a large number of observers to record their thoughts and observations at their computers.
Having developers, project managers, marketers and more observe and engage with the research can make them more invested in the outcome. Often, that makes it more likely that the findings will be applied.
Including everyone might sound like a waste, but it’s very important. Each team member has a unique perspective and will interpret your findings in different ways, leading to a more complete understanding of how people interact with your app/product/service.
Making notes
While watching the videos, the goal for the team is to capture insights and ideas about what the user is saying and doing. The tools we use to do this are simple:
- Post-it notes in three colors
- Markers
Instruct everyone to capture two kinds of things on paper while watching. Write important insights (good and bad) learned from the user on the different Post-it notes.
Let’s say yellow is for good and pink is for bad. Write any specific implementation ideas on green post-it notes. Then, ask everyone to write each down on their post-it notes using a thick, black marker.
Combine everyone’s insights
After several remote sessions, gather around a white board and ask everyone to put their Post-its on the wall. The goal of this step is to distill everyone’s notes into a common list of insights.
Once data is grouped together, work with the team to try and find trends, patterns, and themes that are relevant to the research question. Then, discuss them.
Make sure you do it all together—get the goals and questions from the team, then do the up-front work conducting a survey or getting participants to the interview. The team should be there during the interviews, usability tests, and debriefing sessions, so they can observe users with their own eyes.
Remote collaboration
One really important aspect of a collaborative process during your research study is that if any member of your team is remote, then everyone should be remote.
Why? We at Stanfy did a lot of experiments with this, trying to involve a remote person via Google Hangout or GoToMeeting, but it never worked. Sooner or later, all discussions start to happen between people who are in the same space, ignoring those who are outside.
There are many tools that you could use for remote team collaboration. With the design team at Stanfy we tried Murally, Realtimeboard and Boardthing. They have different pricing models but basically do the similar job – providing you with a shared canvas for online brainstorming and collaboration, which allows you to visually organize your ideas. Murally currently is our favorite space for such collaborative work.
Summary
Here’s the quick 6-step summary of a test setup:
- Choose a research method based on what you want to find out.
- Select the tool or service that best supports your research goals.
- Create and post a recruiting screener and confirm valid participants.
- Design a set of tasks or write an interview guide.
- Conduct research session and pay out the incentives (don’t forget to include your team).
- Summarize findings together with your team.
If you’re working with an audience scattered across the globe, all under budget and time constraints, remote user testing can provide the rich data to help make informed design decisions, and to understand how people interact and behave with the interface you’ve made.
Images by Anna Iurchenko.
