Flag

We stand with Ukraine and our team members from Ukraine. Here are ways you can help

Home ›› Business Value and ROI ›› 6 Key Questions to Guide International UX Research ›› Mobile Testing Toolbox, Part 2

Mobile Testing Toolbox, Part 2

by Amber DeRosa
7 min read
Share this post on
Tweet
Share
Post
Share
Email
Print

Save

A survey of mobile testing tools for unmoderated usability studies finds products and features that suit many different design needs.

This is a continuation of a two part article on mobile testing tools. In Mobile Testing Toolbox: Part 1, I talked about tools that can be used for moderated remote usability testing and for recording in-person mobile usability studies.

In part two of this article I will share my knowledge of unmoderated mobile tools. Unmoderated tools are good for collecting behavioral data about what people are doing with a mobile website or an app.

If a researcher wants to know why people are doing certain things on a mobile site or app they should use the moderated tools discussed in my previous article.

Please note: this is not a complete list of all mobile remote usability testing tools. It is just a list of tools that I have reviewed. These tools are constantly being updated with additional features and functionalities, so please visit the tool’s website for the most up-to-date information.

Unmoderated tools are good for collecting data about what people are doing with a mobile site or app

An Overview Of Features By Tool

What Tools Can I Use For Unmoderated Remote Usability Testing?

Unmoderated remote usability testing tools provide users with a link to a website evaluation. The website evaluation includes set of tasks that the user can complete at any time. The task instructions serve as the usability test moderator. A survey can be shown after the tasks to gain additional information. Userzoom and userlytics are two products on the market that can provide unmoderated remote usability testing.

UserZoom

UserZoom is an enterprise level software tool that enables you to conduct unmoderated remote usability testing and integrate with a suite of other mobile tools, such as mobile Voice of Customer (VoC) and real-time recruiting from your mobile sites. Mobile VoC is a feedback feature that allows users to comment on various portions of the mobile website. The comments are connected to a page or feature and can be sorted and analyzed. Real-time recruiting allows the researcher to intercept a user who is browsing on the site and invite them to participate in a survey or an online study. UserZoom is a quantitative tool that records analytical data such as clickstreams, time-on-task, and completion rates, offering a video session replay. (The video session is a screen capture with no picture-in-picture or audio.) The user accesses the study by downloading an app and using the app to access a mobile website.

Works on iOS and Android; mobile website.

Features/Functionality

  • Unmoderated usability testing of mobile websites
  • Email invitation to download app, allowing for screening of certain types of users
  • Clickstreams (no tagging), time-on-task, and net promoter scores
  • Video session replay and the ability to create video clips
  • Mobile surveys and voice of client

Limitations

  • Mobile websites only; no app capabilities
  • User must install UserZoom app
  • No picture-in-picture or audio in video
  • Videos stored on UserZoom servers

Userlytics

Userlytics is a more qualitative tool that allows for unmoderated remote usability testing. The tool provides video files to analyze and annotate. This tool does not provide any analytical data such as clickstreams and time-on-task, it only provides the raw video file of the session. Userlytics requires the user to use their mobile device as well as a laptop and webcams—one pointed at the mobile device and a second to record the picture-in-picture of the user’s face. Again, similar to UserTesting, this setup can get complicated, therefore Userlytics offers the use of a panel of users who have their systems configured to be able to access any remote mobile studies. If using a panel for a study is a viable option for your business, the tool can be a good fit. If not, recruiting your own users is possible, but it requires you to send your users a webcam they will have to configure to their systems in a specific way.

Works on iPhone or iPad; mobile app or websites.

Features/Functionality

  • Unmoderated usability test of an iPhone or iPad app or website
  • Utilizes a panel
  • Record the iPad/iPhone screen, picture-in-picture, and user’s audio
  • Download the videos for advanced editing and analysis
  • Ability to annotate videos and request transcripts of the recording

Limitations

  • iPhone and iPad only
  • If you test with your own users, they must have laptop, iOS device, and webcam
  • Apps only, no mobile website capabilities
  • In order to ask follow up questions after the tasks, you must either create a zip file with a presentation or ask the questions after the recording part of the videos
  • The videos are stored on the Userlytics servers

What Tools can I use for Analytical Data?

Unlike research tools where you have to recruit participants, analytical data tools run in the background for everyone who accesses your site or app. All of these tools are for apps only and additional code must be built into your app’s SDK.

Appsee

Appsee (formerly UserVOD) is an analytic tool that provides both quantitative data as well as qualitative video files of the user’s interaction with your app. Reports include touch analysis heat maps, unresponsive gestures, and crash records. The unresponsive gestures report can help you understand what parts of the screen users are tapping, swiping, or pinching that are not currently active. This can help identify ways to improve usability by potentially enabling gestural interactions that users expect. In addition to its usability benefits, the app has some quality assurance features because it includes crash records. Video files are stored on Appsee’s servers, but they have the ability to set private screens that will not be recorded to ensure user privacy.

Works on iPhone or iPad; mobile app.

Features/Functionality

  • Unmoderated click stream recording; records application screens; and captures all gestures
  • Touch analysis heat maps, crash records, and unresponsive gestures are recorded
  • Ability to set private screens that will not be recorded
  • In order to preserve data useage the app has an option to select if the sessions should only be recorded and uploaded through Wi-Fi

Limitations

  • iPhone or iPad only; Appsee is developing an Android solution they hope to have ready soon
  • Apps only, no mobile website capabilities
  • Appsee code must be built into the app SDK
  • The videos are stored on Appsee servers

 

Delight

Delight is an analytics tool that provides qualitative video files of the user’s interaction with your app. Delight also has the capability to capture picture-in-picture and audio by having the user consent to a recording of their session. Delight does not provide quantitative reports of the data, but it provides a dashboard to watch recordings of specific user flows that can enable you to gain insights from actual user sessions. Video files are stored on Delights servers, but Delight has the ability to set private screens that will not be recorded to ensure user privacy.

Works on iPhone or iPad; mobile app

Features/Functionality

  • Unmoderated click stream recording, records application screens, and captures all gestures
  • Ability to capture picture-in-picture and audio
  • Ability to set private screens that will not be recorded
  • An option to select if the sessions should only be recorded and uploaded through Wi-Fi or not
  • Ability to track events

Limitations

  • iPhone or iPad only, Delight is developing an Android solution they hope to have ready soon.
  • Apps only, no mobile website capabilities
  • Delight code must be built into the app SDK
  • The videos are stored on Delight.io servers

Swrve

Swrve is an A/B testing tool that measures the differences between two variations of an app. The tool is tailored to marketing or return on investment (ROI) tasks such as revenue, conversion, user retention, and engagement, but can also be used to determine the best designs for improving traffic or usability as well. Swrve has the ability to target specific users for A/B tests by using their interface to send the versions to specific users.

Works on iPhone or iPad; mobile app

Features/Functionality

  • Mobile app A/B testing tool
  • Full key performance indicator (KPI) reporting (revenue, conversion, user retention, engagement, etc)
  • Full direct marketing platform delivers in-app messages to target customer groups with relevant offers and content

Limitations

  • iPhone or iPad only
  • Apps only; no mobile website capabilities
  • Swrve code must be built into the app SDK

Arise

Similar to Swerve, Arise is an A/B testing tool that measures the differences between two variations of an app. For example, it provides data that makes it possible to decide between button color, font size, and text labels. It can show you which version of the app has the best results based on pre-determined goals such as return on investment, new account creation, or traffic to certain pages. Arise is still in the beta version of its product, although they claim to be production ready.

Works on iOS or Andriod; mobile app

Features/Functionality

  • Mobile app A/B testing tool
  • Simple, clear graphs allow you to instantly see which version of your app is driving results
  • Can export test data to other software programs for viewing
  • iOS and Android

Limitations

  • Apps only; no mobile website capabilities
  • Arise code must be built into the app SDK.

I hope this article was educational and helpful in distributing the knowledge of what tools are available to assist with mobile usability testing. I expect it at least serves as a good overview and starting point for researchers looking for tools in the mobile space.

 

Image of person using smartphone courtesy Shutterstock

post authorAmber DeRosa

Amber DeRosa,

Amber DeRosa is a senior usability engineer working for a large financial company.  She has tested over 150 mobile users across nearly 20 mobile studies, examining native iOS and Android applications as well as responsive websites. Since 2009, she has adapted her mobile testing techniques to maximize learnings gained from every test session. To ensure usability tests positively impact the business and our users’ experience, she regularly presents test findings with executive audiences throughout the company. Beyond testing, executive stakeholders and product owners look to her for consultative guidance on mobile best practices and heuristic feedback. She was the founding Vice-Chair of PhillyCHI a local chapter of the ACM SIG CHI and has been an active member of the local Philadelphia UX community since 2004. Feel free to follow her on twitter at @ajderosa

Tweet
Share
Post
Share
Email
Print

Related Articles

What do Architecture, Computer Science, Agile, and Design Systems have in common?

Article by Kevin Muldoon
A Pattern Language
  • The article explores Christopher Alexander’s impact on diverse fields, from architecture to software development, introducing the concept of design patterns and their influence on methodologies like Agile and the evolution of Design Systems.
Share:A Pattern Language
7 min read

In this article, I’ll share seven incredibly useful design resources that can elevate your skills as a UX designer in 2024 and make you a well-rounded designer overall. These resources not only help you work more efficiently but also automate repetitive tasks and enable you to create outstanding user experiences. So, let’s dive into this without further delay.

Article by Abhi Chatterjee
7 Must-Have Resources for UX Designers in 2024
  • The article explores seven indispensable resources for UX designers in 2024, offering insights into design methodologies, case studies, UX laws, and practical challenges to elevate designers’ skills and efficiency.
Share:7 Must-Have Resources for UX Designers in 2024
3 min read
Article by Eleanor Hecks
8 Key Metrics to Measure and Analyze in UX Research
  • The article outlines eight essential metrics for effective UX research, ranging from time on page to social media saturation
  • The author emphasizes the significance of these metrics in enhancing user experience and boosting brand growth.

Share:8 Key Metrics to Measure and Analyze in UX Research
6 min read

Did you know UX Magazine hosts the most popular podcast about conversational AI?

Listen to Invisible Machines

This website uses cookies to ensure you get the best experience on our website. Check our privacy policy and