The mobile domain is growing rapidly. Mobile devices currently make up 20% of our online traffic.
That’s one-fifth of our user base accessing content through a mobile device: we cannot ignore the mobile channel.
Even though mobile usage of websites continues to grow, mobile users are still not as prevalent as desktop users. I’ve been conducting in-person mobile usability tests since late 2009, but I’ve had difficulty getting the samples needed to conduct in-person usability studies.
To expand the sample, I’ve been evaluating and testing remote usability testing tools since the summer of 2012. When I began, such tools seemed impossible to find, but now, many exist on the market. People frequently ask about the tools I have tried and what features and limitations are associated with each, so I put together this two-part article to help others understand what tools are available, what their key capabilities are, and what tool will be best for their organization.
The first part of the article will focus on mobile testing tools for moderated usability studies. In a forthcoming article, part two will focus on mobile testing tools for unmoderated studies.
Please note: this is not a complete list of all mobile remote usability testing tools. It is just a list of tools that I have reviewed. These tools are constantly being updated with additional features and functionalities, so please visit the tool’s website for the most up-to-date information.
An Overview Of Features By Tool
What Tools Can I Use For Moderated Remote Usability Testing?
Moderated remote usability testing tools provide many of the benefits of in-lab tests with the convenience of a remote study. To conduct a mobile moderated remote usability test, the researcher needs to be able to view the user’s mobile device screen in real-time while speaking with them on a conference call or using VOIP. This process poses some challenges for smart phone studies because the user must have a service provider that allows for voice and data use at the same time (AT&T, 4GLTE, or Wi-Fi,) or they must have access to another phone. Apple has security restrictions in place that do not allow for screen sharing. Therefore screen-sharing products that work with iOS have been difficult to find. Some products are using desktop mirroring or webcams to overcome Apple’s security restrictions.
Moderated remote usability testing tools provide many of the benefits of in-lab tests.
In spite of the challenges associated with remote mobile usability testing, several viable tools exist that can enable researchers to collect rich user feedback about mobile websites and applications, at any stage during their development. Accordingly, the next section provides an overview of the tools I have evaluated.
GoToAssist
GoToAssist displays the user’s mobile device screen onto the researcher’s computer in real-time, making it possible to view any app or website that can be displayed on the user’s mobile device. GoToAssist allows you to recruit your own users and have them share their screens by downloading an app. Typically, GoToAssist is used for desktop support specialists to interact with a device for troubleshooting, but it has the unintended benefit of providing the capabity of moderated remote usability testing. Currently, screen sharing via GoToAssist is only available on Samsung and Blackberry devices due to the constraints of the mobile manufactures.
Works on Samsung and BlackBerry; mobile app and website.
Features/Functionality
- Moderated usability testing
- Recruit your own users
- Ability to see the user’s screen in real time—Including layers, scrolling, and orientation changes between landscape and portrait
Limitations
- Samsung and Blackberry only
- Researcher must be very familiar with app or website because the tool has no indication of what the user has tapped
- User must install GoToAssist app
- GoToAssist does not have built-in screen recording capabilities
UserTesting.com
UserTesting.com has repurposed their desktop solution to enable mobile usability testing, providing video files to analyze and annotate. The tool also gives you the ability to add time-stamped comments to the video, jump between tasks, and create video clips. Using a dual webcam set-up, UserTesting.com records what is displayed on the user’s computer, the mobile image, and the picture-in-picture of the person’s face. This setup can be complicated to arrange, so UserTesting offers the use of a panel of users who have their systems configured to access any remote mobile studies. If using a panel for a study is a viable option for your business, then UserTesting.com could be a good fit. If not, recruiting your own users is possible, but would require you to send your users a webcam or two and have them configure their systems in a specific way.
Works on any device; mobile app and website.
Features/Functionality
- Moderated usability testing
- Utilizes a panel of mobile device users that have two high-resolution webcams designed to record the entire mobile experience
- Ability to add time-stamped comments to video, jump between tasks, and create video clips
- iPhone, iPad, Android, and Windows
Limitations
- If you use your own users, you need to send HD camera and help the user with setup
- Cannot randomize tasks
- The videos are stored on UserTesting.com servers
Reflector
In addition to being able to see a user’s mobile screen directly (via a webcam or screen sharing), another way to provide remote moderated usability testing is through the combination of mirroring and video conferencing. Reflector can be used to mirror the user’s mobile device onto the user’s computer. The user can then share their computer screen using any video conferencing tool. This process also involves some setup and configuration on the user’s end.
Works on iPhone or iPad; mobile app and website.
Features/Functionality
- Mirroring what is on the device onto a computer
- User can share their mirrored screen via a video conferencing tool
- Wirelessly connects to iPhone 4S, iPad 2, the new, or any Mac or PC
Limitations
- iPhone or iPad only
- All devices that are mirrored must be on same network
- Cannot mirror to another mobile device
- Researcher must be very familiar with app or website because the tool has no indication of what the user has tapped
- Some apps may look different when mirrored
- Reflector does not have built-in screen recording capabilities
GoToMeeting
The first place I looked for a remote usability testing solution was in e-meeting tools, because we use e-meeting tools today to conduct remote usability testing for the desktop. Most of the tools allow people to attend an e-meeting from a mobile device, but don’t allow the mobile participant to share their screen. GoToMeeting has added the capability of sharing mobile websites from an iPad. This can help with usability testing responsive websites on an iPad, but it does not help with usability testing mobile apps.
Works on iPad only; mobile website.
Features/Functionality
- Start meetings from your iPhone or iPad (with paid GoToMeeting account)
- Can present from an iPad
- Connect to audio through your device’s built-in mic and speakers or dial in over the phone
- Pinch- zoom meeting content
- Easily switch between portrait and landscape
- Ability to record meeting
Limitations
- iPad only
- Mobile websites only, no app capabilities
- User must install GoToMeeting app
- Researcher must be very familiar with app or website because the tool has no indication of what the user has tapped
- The GoToMeeting browser doesn’t support opening multiple tabs and bookmarks, or provide a way to pass a URL from the in-meeting chat
- Observers do not see the keyboard when it is open and there is a bit of a scrolling delay
What Tools Can I Use For Moderated In-Person Mobile Usability Testing?
So far, we’ve discussed moderated remote usability testing tools, but I’ve also been asked about ways to record in-person studies using the mobile device itself rather than a webcam. Using an app to record the session straight from the device allows for the user to move freely without the restrictions of holding the device under a camera.
Magitest and UX Recorder both capture the user’s screen, a picture-in-picture view of their facial expressions, as well as audio. These tools also provide an indicator in the video that shows when and where the user has tapped. These tools are resource intensive, in that they can use a lot of battery power, and take some time to process the video files after the sessions. It’s also worth noting that some performance issues can arise by running the recording app while conducting the session.
Magitest
Magitest allows recording from either a mobile website or an iOS app. To record an app, code needs to be added to your native app’s SDK. The Magitest app must be downloaded to the device and the recording will run from the Magitest app. Originally, the tool was designed for in-lab studies, however the FAQs section of their site explains a remote workaround that I have not tried.
Works on iPhone or iPad; mobile app or websites.
Features/Functionality
- Shows where the user taps, swipes and pinches
Limitations
- iPhone or iPad only, Magitest is developing an Android solution they hope to have ready soon
- For in-lab studies, remote capabilities are available using a workaround with Dropbox
- Magitest code must be built into the app SDK
- Does not support landscape view
- Some tasks, like typing, may be slowed down while running the testing tool
- Recordings can take a while to encode and save
- If a user hits the home button or gets a phone call it stops the recording
UX Recorder
UxRecorder allows you to record from a mobile website only. No app recording is possible. The UX Recorder app must be downloaded to the user’s mobile device, and the recording runs from the UX Recorder app. The tool is designed for in lab-studies, but files can be saved to dropbox.
Works on iPhone or iPad; mobile websites.
Features/Functionality
- Shows where the user taps, swipes, and pinches
Limitations
- iPhone or iPad only
- Mobile websites only, no app capabilities
- For in-lab studies; remote capabilities are available using a workaround with Dropbox
- Resource intensive; must keep device plugged in
- Issues arise when switching between landscape and portrait in same session
- Some tasks, like typing, may be slowed down
- Recordings can take a while to encode and save
- If a user hits the home button or gets a phone call it stops the recording
I hope this article was educational and helpful in distributing the knowledge of what tools are available to assist with moderated mobile usability testing. In part two of this article I will share my knowledge of un-moderated mobile tools.
Image of person using smartphone courtesy Shutterstock
