Flag

We stand with Ukraine and our team members from Ukraine. Here are ways you can help

Home ›› Mobile Testing Toolbox: Part 1

Mobile Testing Toolbox: Part 1

by Amber DeRosa
8 min read
Share this post on
Tweet
Share
Post
Share
Email
Print

Save

A survey of mobile testing tools for moderated usability studies finds products and features that suit many different design needs.

The mobile domain is growing rapidly. Mobile devices currently make up 20% of our online traffic.

That’s one-fifth of our user base accessing content through a mobile device: we cannot ignore the mobile channel.

Even though mobile usage of websites continues to grow, mobile users are still not as prevalent as desktop users. I’ve been conducting in-person mobile usability tests since late 2009, but I’ve had difficulty getting the samples needed to conduct in-person usability studies.

To expand the sample, I’ve been evaluating and testing remote usability testing tools since the summer of 2012. When I began, such tools seemed impossible to find, but now, many exist on the market. People frequently ask about the tools I have tried and what features and limitations are associated with each, so I put together this two-part article to help others understand what tools are available, what their key capabilities are, and what tool will be best for their organization.

The first part of the article will focus on mobile testing tools for moderated usability studies. In a forthcoming article, part two will focus on mobile testing tools for unmoderated studies.

Please note: this is not a complete list of all mobile remote usability testing tools. It is just a list of tools that I have reviewed. These tools are constantly being updated with additional features and functionalities, so please visit the tool’s website for the most up-to-date information.

An Overview Of Features By Tool

What Tools Can I Use For Moderated Remote Usability Testing?

Moderated remote usability testing tools provide many of the benefits of in-lab tests with the convenience of a remote study. To conduct a mobile moderated remote usability test, the researcher needs to be able to view the user’s mobile device screen in real-time while speaking with them on a conference call or using VOIP. This process poses some challenges for smart phone studies because the user must have a service provider that allows for voice and data use at the same time (AT&T, 4GLTE, or Wi-Fi,) or they must have access to another phone. Apple has security restrictions in place that do not allow for screen sharing. Therefore screen-sharing products that work with iOS have been difficult to find. Some products are using desktop mirroring or webcams to overcome Apple’s security restrictions.

Moderated remote usability testing tools provide many of the benefits of in-lab tests.

In spite of the challenges associated with remote mobile usability testing, several viable tools exist that can enable researchers to collect rich user feedback about mobile websites and applications, at any stage during their development. Accordingly, the next section provides an overview of the tools I have evaluated.

 

GoToAssist

GoToAssist displays the user’s mobile device screen onto the researcher’s computer in real-time, making it possible to view any app or website that can be displayed on the user’s mobile device. GoToAssist allows you to recruit your own users and have them share their screens by downloading an app. Typically, GoToAssist is used for desktop support specialists to interact with a device for troubleshooting, but it has the unintended benefit of providing the capabity of moderated remote usability testing. Currently, screen sharing via GoToAssist is only available on Samsung and Blackberry devices due to the constraints of the mobile manufactures.

Works on Samsung and BlackBerry; mobile app and website.

Features/Functionality

  • Moderated usability testing
  • Recruit your own users
  • Ability to see the user’s screen in real time—Including layers, scrolling, and orientation changes between landscape and portrait

Limitations

  • Samsung and Blackberry only
  • Researcher must be very familiar with app or website because the tool has no indication of what the user has tapped
  • User must install GoToAssist app
  • GoToAssist does not have built-in screen recording capabilities
UserTesting.com

UserTesting.com has repurposed their desktop solution to enable mobile usability testing, providing video files to analyze and annotate. The tool also gives you the ability to add time-stamped comments to the video, jump between tasks, and create video clips. Using a dual webcam set-up, UserTesting.com records what is displayed on the user’s computer, the mobile image, and the picture-in-picture of the person’s face. This setup can be complicated to arrange, so UserTesting offers the use of a panel of users who have their systems configured to access any remote mobile studies. If using a panel for a study is a viable option for your business, then UserTesting.com could be a good fit. If not, recruiting your own users is possible, but would require you to send your users a webcam or two and have them configure their systems in a specific way.

Works on any device; mobile app and website.

Features/Functionality

  • Moderated usability testing
  • Utilizes a panel of mobile device users that have two high-resolution webcams designed to record the entire mobile experience
  • Ability to add time-stamped comments to video, jump between tasks, and create video clips
  • iPhone, iPad, Android, and Windows

Limitations

  • If you use your own users, you need to send HD camera and help the user with setup
  • Cannot randomize tasks
  • The videos are stored on UserTesting.com servers
Reflector

In addition to being able to see a user’s mobile screen directly (via a webcam or screen sharing), another way to provide remote moderated usability testing is through the combination of mirroring and video conferencing. Reflector can be used to mirror the user’s mobile device onto the user’s computer. The user can then share their computer screen using any video conferencing tool. This process also involves some setup and configuration on the user’s end.

Works on iPhone or iPad; mobile app and website.

Features/Functionality

  • Mirroring what is on the device onto a computer
  • User can share their mirrored screen via a video conferencing tool
  • Wirelessly connects to iPhone 4S, iPad 2, the new, or any Mac or PC

Limitations

  • iPhone or iPad only
  • All devices that are mirrored must be on same network
  • Cannot mirror to another mobile device
  • Researcher must be very familiar with app or website because the tool has no indication of what the user has tapped
  • Some apps may look different when mirrored
  • Reflector does not have built-in screen recording capabilities
GoToMeeting

The first place I looked for a remote usability testing solution was in e-meeting tools, because we use e-meeting tools today to conduct remote usability testing for the desktop. Most of the tools allow people to attend an e-meeting from a mobile device, but don’t allow the mobile participant to share their screen. GoToMeeting has added the capability of sharing mobile websites from an iPad. This can help with usability testing responsive websites on an iPad, but it does not help with usability testing mobile apps.

Works on iPad only; mobile website.

Features/Functionality

  • Start meetings from your iPhone or iPad (with paid GoToMeeting account)
  • Can present from an iPad
  • Connect to audio through your device’s built-in mic and speakers or dial in over the phone
  • Pinch- zoom meeting content
  • Easily switch between portrait and landscape
  • Ability to record meeting

Limitations

  • iPad only
  • Mobile websites only, no app capabilities
  • User must install GoToMeeting app
  • Researcher must be very familiar with app or website because the tool has no indication of what the user has tapped
  • The GoToMeeting browser doesn’t support opening multiple tabs and bookmarks, or provide a way to pass a URL from the in-meeting chat
  • Observers do not see the keyboard when it is open and there is a bit of a scrolling delay

What Tools Can I Use For Moderated In-Person Mobile Usability Testing?

So far, we’ve discussed moderated remote usability testing tools, but I’ve also been asked about ways to record in-person studies using the mobile device itself rather than a webcam. Using an app to record the session straight from the device allows for the user to move freely without the restrictions of holding the device under a camera.

Magitest and UX Recorder both capture the user’s screen, a picture-in-picture view of their facial expressions, as well as audio. These tools also provide an indicator in the video that shows when and where the user has tapped. These tools are resource intensive, in that they can use a lot of battery power, and take some time to process the video files after the sessions. It’s also worth noting that some performance issues can arise by running the recording app while conducting the session.

Magitest

Magitest allows recording from either a mobile website or an iOS app. To record an app, code needs to be added to your native app’s SDK. The Magitest app must be downloaded to the device and the recording will run from the Magitest app. Originally, the tool was designed for in-lab studies, however the FAQs section of their site explains a remote workaround that I have not tried.

Works on iPhone or iPad; mobile app or websites.

Features/Functionality

  • Shows where the user taps, swipes and pinches

Limitations

  • iPhone or iPad only, Magitest is developing an Android solution they hope to have ready soon
  • For in-lab studies, remote capabilities are available using a workaround with Dropbox
  • Magitest code must be built into the app SDK
  • Does not support landscape view
  • Some tasks, like typing, may be slowed down while running the testing tool
  • Recordings can take a while to encode and save
  • If a user hits the home button or gets a phone call it stops the recording
UX Recorder

UxRecorder allows you to record from a mobile website only. No app recording is possible. The UX Recorder app must be downloaded to the user’s mobile device, and the recording runs from the UX Recorder app. The tool is designed for in lab-studies, but files can be saved to dropbox.

Works on iPhone or iPad; mobile websites.

Features/Functionality

  • Shows where the user taps, swipes, and pinches

Limitations

  • iPhone or iPad only
  • Mobile websites only, no app capabilities
  • For in-lab studies; remote capabilities are available using a workaround with Dropbox
  • Resource intensive; must keep device plugged in
  • Issues arise when switching between landscape and portrait in same session
  • Some tasks, like typing, may be slowed down
  • Recordings can take a while to encode and save
  • If a user hits the home button or gets a phone call it stops the recording

I hope this article was educational and helpful in distributing the knowledge of what tools are available to assist with moderated mobile usability testing. In part two of this article I will share my knowledge of un-moderated mobile tools.

 

Image of person using smartphone courtesy Shutterstock

post authorAmber DeRosa

Amber DeRosa,

Amber DeRosa is a senior usability engineer working for a large financial company.  She has tested over 150 mobile users across nearly 20 mobile studies, examining native iOS and Android applications as well as responsive websites. Since 2009, she has adapted her mobile testing techniques to maximize learnings gained from every test session. To ensure usability tests positively impact the business and our users’ experience, she regularly presents test findings with executive audiences throughout the company. Beyond testing, executive stakeholders and product owners look to her for consultative guidance on mobile best practices and heuristic feedback. She was the founding Vice-Chair of PhillyCHI a local chapter of the ACM SIG CHI and has been an active member of the local Philadelphia UX community since 2004. Feel free to follow her on twitter at @ajderosa

Tweet
Share
Post
Share
Email
Print

Related Articles

How UX teams can provide key insights using technology adoption models

Article by Lawton Pybus
What Makes Users Adopt a New Feature or Tool?
  • The article explores the evolution of technology adoption theories, focusing on models like Diffusion of Innovations, TAM, TAM 2 and 3, UTAUT, and HMSAM, providing UX practitioners with tools to understand and enhance user behavior in product development.
Share:What Makes Users Adopt a New Feature or Tool?
6 min read

What do Architecture, Computer Science, Agile, and Design Systems have in common?

Article by Kevin Muldoon
A Pattern Language
  • The article explores Christopher Alexander’s impact on diverse fields, from architecture to software development, introducing the concept of design patterns and their influence on methodologies like Agile and the evolution of Design Systems.
Share:A Pattern Language
7 min read

In this article, I’ll share seven incredibly useful design resources that can elevate your skills as a UX designer in 2024 and make you a well-rounded designer overall. These resources not only help you work more efficiently but also automate repetitive tasks and enable you to create outstanding user experiences. So, let’s dive into this without further delay.

Article by Abhi Chatterjee
7 Must-Have Resources for UX Designers in 2024
  • The article explores seven indispensable resources for UX designers in 2024, offering insights into design methodologies, case studies, UX laws, and practical challenges to elevate designers’ skills and efficiency.
Share:7 Must-Have Resources for UX Designers in 2024
3 min read

Did you know UX Magazine hosts the most popular podcast about conversational AI?

Listen to Invisible Machines

This website uses cookies to ensure you get the best experience on our website. Check our privacy policy and