In the early 90’s, Jakob Nielsen declared in-person user research as state of the art. “User testing with real users is the most fundamental usability method and is in some sense irreplaceable, since it provides direct information about how people use computers […]”.1 Sometimes in-person user research can be logistically impractical or cost prohibitive, so remote user research is often employed as an alternative. As well, companies that specialize in user research often combine both in-person and remote user research.
In-person user research has been around the longest, and is still widely used as a great way to gather feedback on websites, advertisements, or software. In-person research usually involves letting users perform tasks on a computer while asking them questions, observing their behaviors and body language, or having them think out loud. Additional hardware can be used, such as eye tracking devices.
Most research firms that specialize in this kind of testing have a physical laboratory for in-person test sessions. If it’s difficult to set up a full-blown testing lab, a temporary lab can be created in an office or café. A pleasant and informal test environment makes participants feel comfortable and helps them to act more naturally.
In remote user research, on the other hand, the physical location is no longer important because the research subjects can work independently of the researchers. There are two forms of remote user research: moderated and automated. Moderated tests require the researcher to interact with the participant during the session. During automated tests, the researcher does not interfere, which allows people to participate whenever it fits their personal schedule.
Moderated remote test sessions are similar to in-lab user testing. Participants are asked to perform certain tasks and the researcher observes and asks questions. In automated remote test sessions, the researcher needs to set up the whole test beforehand. Since he is not personally involved in the test session, it is not possible to make any changes to the test once participants have been invited.
Some things are much harder to do remotely than in person:
- Observe and record what the user says or sees
- Track eye movements
- Offer a secure testing environment and keep control over confidential test materials
But there are advantages to remote user research:
- It can be much cheaper
- It is much faster and requires fewer resources
- The test is taken in the context of real-world use
- Timing data is more reliable
- It is much easier to create a diverse test panel with participants from all over the world
However, the most interesting insights and results derive from combining both remote and in-person research. In my role at Usabilla, which is a web-based product for conducting remote research, I work with clients to understand the context in which our tool is being used. Those who have the luxury of combining in-person research with remote research all agree on the benefits. In this article, I will be talking about combining in-person research with unmoderated remote research.
Ways of Combining In-Person and Remote User Research
Choosing the optimal combination of user research methods is highly case-dependent. But in general, each of the different phases in the development process has associated best practices.
In the inspiration phase of a project, broad ideas can be gathered with in-person user research. These ideas can then be verified in with remote user research.
In the design phase, flaws in the design can be detected by doing in-person user research. These flaws should then be verified and prioritized with remote research.
In the final evaluation phase, in-person user testing can be used to verify the design and collect concrete feedback. A final remote research session can be conducted to gather more general and representative feedback.
Inspiration phase
At the very beginning of a project as you’re coming up with the basic concept, it is important to involve your future users. Prospective users can tell you best what they really want, such as which features they think are handy, or how they prioritize the features. By asking real users, you will discover aspects you had not considered before and learn to relate to the product from the user’s perspective.
Focus groups are a great in-person user research technique for conducting this kind of enquiry. You and a group of prospective users get together to brainstorm about the product. A fun and informal atmosphere triggers people’s creativity and you will get a bunch of ideas, inspirations, and suggestions of things you should avoid. You can also present a first concept to the group and collect feedback on that. However, beware that untrained participants might have difficulty judging abstract concepts, so don’t expect too much. Read more about how to set up a focus group in the book Focus Groups – A practical guide to applied research.
Remote research, on the other hand, will not bring a group of communicative people around one table, but it offers a great way to verify and extend the input you got from focus groups. You might want to translate the ideas from the focus group into visual concepts for use in the remote research; this makes it easier for people to share their opinions without a thorough introduction to the topic. For example, with a survey you can have a large number of people verify your concept. Of course, you can also ask for additional feedback and ideas that go beyond your findings from the focus group.
Design phase
After an initial inspiration phase, you should continue to include your users in the design process. Ideally, you’d have the capability to test early and often with users. Ongoing user research can actually save more time and money than it costs in the long run, and also spares your nerves by helping you stay on the right track throughout the whole design process.
In-person research at different points in the design process offers in-depth insights into how users interact with the product, or at least a prototype of it. You can detect design flaws, observe where users encounter difficulties, and ask them why. Basically, you can detect usability issues that need to be addressed in the next design iteration. According to Jakob Nielsen, you don’t even need to test with a bunch of people; he suggests that testing repeatedly and with as few as five participants can detect 80% of the existing usability issues.
Remote user research has two primary benefits during the design phase. First, you can easily set up your first test very early in the design phase with as little as a paper sketch. This way, you can focus on and verify early ideas and basic features. Once your design gets more detailed, you might want to consider shifting to in-person tests. Qualitative findings from these test sessions can easily be verified, generalized, and prioritized with a remote test. Just build a remote user test around a specific usability issue and invite a large number of people. Verifying usability issues that were detected during in-lab user session with remote tests will help you to determine priorities for your next development iteration.
Final evaluation phase
In the past, user testing was considered more of a post hoc design evaluation technique than something part of the whole design process. For the sake of great products this general perception has changed and companies have learned to listen to their users before the product has reached its final stage. Still, the final evaluation of a design is a popular practice.
When evaluating your final product with in-person user research, you get high quality and detailed feedback about your product. When this feedback is positive, not only do you get confirmation that you did a good job, but you also get great testimonials for your marketing. However, you can only test a limited number of participants and their opinion will hardly be representative for all your users. Basically, you can get an idea of what few people think, which you probably want to back up with some big numbers.
With remote user research you can ask feedback from a big sample of users, big enough even to be fully representative of all of your users. This way you know whether you did a good job, whether people like it, and even why they like it. Of course you can also test remaining negative aspects, but I wouldn’t recommend that at this point. The automated tests that are typically part of QA are very focused and you only get answers to what you actually ask, so focus your user research on general feedback instead.
Conclusion
Both in-person and remote user research have advantages and disadvantages. There is no ultimate guide to effective user research, and the way you combine the two approaches strongly depends on the specific project and available resources. However, if resources are available, a combination of different methods has proven to be very effective.
Most of the time, it is reasonable to not conduct in-person and remote research simultaneously, but rather to do it sequentially, giving them the chance to supplement each other. In general, in-person research reveals more qualitative insights such as usability issues or new ideas. Remote research, on the other hand, is great to underpin these insights with numbers to verify and possibly generalize them.
Best practices of combining in-person and remote research depending on the developmental phase:
Inspiration phase: Use in-person tests to gather and define features. Use remote tests to verify and generalize the concept based on your initial findings.
Design phase: Use remote tests to collect early feedback and later to generalize and prioritize in-person tests findings. Use in-person tests to evaluate more detailed designs.
Final evaluation: Use in-person tests to collect specific feedback and high quality quotes. Use remote tests to collect broad and representative feedback.
- Usability Engineering, 1993, p.165 (back)
