UX Magazine

Defining and Informing the Complex Field of User Experience (UX)
Article No. 1098 September 26, 2013

Mobile Testing Toolbox: Part 1

The mobile domain is growing rapidly. Mobile devices currently make up 20% of our online traffic.

That's one-fifth of our user base accessing content through a mobile device: we cannot ignore the mobile channel.

Even though mobile usage of websites continues to grow, mobile users are still not as prevalent as desktop users. I’ve been conducting in-person mobile usability tests since late 2009, but I've had difficulty getting the samples needed to conduct in-person usability studies.

To expand the sample, I’ve been evaluating and testing remote usability testing tools since the summer of 2012. When I began, such tools seemed impossible to find, but now, many exist on the market. People frequently ask about the tools I have tried and what features and limitations are associated with each, so I put together this two-part article to help others understand what tools are available, what their key capabilities are, and what tool will be best for their organization.

The first part of the article will focus on mobile testing tools for moderated usability studies. In a forthcoming article, part two will focus on mobile testing tools for unmoderated studies.

Please note: this is not a complete list of all mobile remote usability testing tools. It is just a list of tools that I have reviewed. These tools are constantly being updated with additional features and functionalities, so please visit the tool’s website for the most up-to-date information.

An Overview Of Features By Tool

What Tools Can I Use For Moderated Remote Usability Testing?

Moderated remote usability testing tools provide many of the benefits of in-lab tests with the convenience of a remote study. To conduct a mobile moderated remote usability test, the researcher needs to be able to view the user’s mobile device screen in real-time while speaking with them on a conference call or using VOIP. This process poses some challenges for smart phone studies because the user must have a service provider that allows for voice and data use at the same time (AT&T, 4GLTE, or Wi-Fi,) or they must have access to another phone. Apple has security restrictions in place that do not allow for screen sharing. Therefore screen-sharing products that work with iOS have been difficult to find. Some products are using desktop mirroring or webcams to overcome Apple’s security restrictions.

Moderated remote usability testing tools provide many of the benefits of in-lab tests.

In spite of the challenges associated with remote mobile usability testing, several viable tools exist that can enable researchers to collect rich user feedback about mobile websites and applications, at any stage during their development. Accordingly, the next section provides an overview of the tools I have evaluated.

 

GoToAssist

GoToAssist displays the user’s mobile device screen onto the researcher’s computer in real-time, making it possible to view any app or website that can be displayed on the user’s mobile device. GoToAssist allows you to recruit your own users and have them share their screens by downloading an app. Typically, GoToAssist is used for desktop support specialists to interact with a device for troubleshooting, but it has the unintended benefit of providing the capabity of moderated remote usability testing. Currently, screen sharing via GoToAssist is only available on Samsung and Blackberry devices due to the constraints of the mobile manufactures.

Works on Samsung and BlackBerry; mobile app and website.

Features/Functionality

  • Moderated usability testing
  • Recruit your own users
  • Ability to see the user’s screen in real time—Including layers, scrolling, and orientation changes between landscape and portrait

Limitations

  • Samsung and Blackberry only
  • Researcher must be very familiar with app or website because the tool has no indication of what the user has tapped
  • User must install GoToAssist app
  • GoToAssist does not have built-in screen recording capabilities
UserTesting.com

UserTesting.com has repurposed their desktop solution to enable mobile usability testing, providing video files to analyze and annotate. The tool also gives you the ability to add time-stamped comments to the video, jump between tasks, and create video clips. Using a dual webcam set-up, UserTesting.com records what is displayed on the user’s computer, the mobile image, and the picture-in-picture of the person’s face. This setup can be complicated to arrange, so UserTesting offers the use of a panel of users who have their systems configured to access any remote mobile studies. If using a panel for a study is a viable option for your business, then UserTesting.com could be a good fit. If not, recruiting your own users is possible, but would require you to send your users a webcam or two and have them configure their systems in a specific way.

Works on any device; mobile app and website.

Features/Functionality

  • Moderated usability testing
  • Utilizes a panel of mobile device users that have two high-resolution webcams designed to record the entire mobile experience
  • Ability to add time-stamped comments to video, jump between tasks, and create video clips
  • iPhone, iPad, Android, and Windows

Limitations

  • If you use your own users, you need to send HD camera and help the user with setup
  • Cannot randomize tasks
  • The videos are stored on UserTesting.com servers
Reflector

In addition to being able to see a user’s mobile screen directly (via a webcam or screen sharing), another way to provide remote moderated usability testing is through the combination of mirroring and video conferencing. Reflector can be used to mirror the user’s mobile device onto the user’s computer. The user can then share their computer screen using any video conferencing tool. This process also involves some setup and configuration on the user’s end.

Works on iPhone or iPad; mobile app and website.

Features/Functionality

  • Mirroring what is on the device onto a computer
  • User can share their mirrored screen via a video conferencing tool
  • Wirelessly connects to iPhone 4S, iPad 2, the new, or any Mac or PC

Limitations

  • iPhone or iPad only
  • All devices that are mirrored must be on same network
  • Cannot mirror to another mobile device
  • Researcher must be very familiar with app or website because the tool has no indication of what the user has tapped
  • Some apps may look different when mirrored
  • Reflector does not have built-in screen recording capabilities
GoToMeeting

The first place I looked for a remote usability testing solution was in e-meeting tools, because we use e-meeting tools today to conduct remote usability testing for the desktop. Most of the tools allow people to attend an e-meeting from a mobile device, but don’t allow the mobile participant to share their screen. GoToMeeting has added the capability of sharing mobile websites from an iPad. This can help with usability testing responsive websites on an iPad, but it does not help with usability testing mobile apps.

Works on iPad only; mobile website.

Features/Functionality

  • Start meetings from your iPhone or iPad (with paid GoToMeeting account)
  • Can present from an iPad
  • Connect to audio through your device’s built-in mic and speakers or dial in over the phone
  • Pinch- zoom meeting content
  • Easily switch between portrait and landscape
  • Ability to record meeting

Limitations

  • iPad only
  • Mobile websites only, no app capabilities
  • User must install GoToMeeting app
  • Researcher must be very familiar with app or website because the tool has no indication of what the user has tapped
  • The GoToMeeting browser doesn’t support opening multiple tabs and bookmarks, or provide a way to pass a URL from the in-meeting chat
  • Observers do not see the keyboard when it is open and there is a bit of a scrolling delay

What Tools Can I Use For Moderated In-Person Mobile Usability Testing?

So far, we’ve discussed moderated remote usability testing tools, but I’ve also been asked about ways to record in-person studies using the mobile device itself rather than a webcam. Using an app to record the session straight from the device allows for the user to move freely without the restrictions of holding the device under a camera.

Magitest and UX Recorder both capture the user’s screen, a picture-in-picture view of their facial expressions, as well as audio. These tools also provide an indicator in the video that shows when and where the user has tapped. These tools are resource intensive, in that they can use a lot of battery power, and take some time to process the video files after the sessions. It’s also worth noting that some performance issues can arise by running the recording app while conducting the session.

Magitest

Magitest allows recording from either a mobile website or an iOS app. To record an app, code needs to be added to your native app’s SDK. The Magitest app must be downloaded to the device and the recording will run from the Magitest app. Originally, the tool was designed for in-lab studies, however the FAQs section of their site explains a remote workaround that I have not tried.

Works on iPhone or iPad; mobile app or websites.

Features/Functionality

  • Shows where the user taps, swipes and pinches

Limitations

  • iPhone or iPad only, Magitest is developing an Android solution they hope to have ready soon
  • For in-lab studies, remote capabilities are available using a workaround with Dropbox
  • Magitest code must be built into the app SDK
  • Does not support landscape view
  • Some tasks, like typing, may be slowed down while running the testing tool
  • Recordings can take a while to encode and save
  • If a user hits the home button or gets a phone call it stops the recording
UX Recorder

UxRecorder allows you to record from a mobile website only. No app recording is possible. The UX Recorder app must be downloaded to the user’s mobile device, and the recording runs from the UX Recorder app. The tool is designed for in lab-studies, but files can be saved to dropbox.

Works on iPhone or iPad; mobile websites.

Features/Functionality

  • Shows where the user taps, swipes, and pinches

Limitations

  • iPhone or iPad only
  • Mobile websites only, no app capabilities
  • For in-lab studies; remote capabilities are available using a workaround with Dropbox
  • Resource intensive; must keep device plugged in
  • Issues arise when switching between landscape and portrait in same session
  • Some tasks, like typing, may be slowed down
  • Recordings can take a while to encode and save
  • If a user hits the home button or gets a phone call it stops the recording

I hope this article was educational and helpful in distributing the knowledge of what tools are available to assist with moderated mobile usability testing. In part two of this article I will share my knowledge of un-moderated mobile tools.

 

Image of person using smartphone courtesy Shutterstock

ABOUT THE AUTHOR(S)

User Profile

Amber DeRosa is a senior usability engineer working for a large financial company.  She has tested over 150 mobile users across nearly 20 mobile studies, examining native iOS and Android applications as well as responsive websites. Since 2009, she has adapted her mobile testing techniques to maximize learnings gained from every test session. To ensure usability tests positively impact the business and our users’ experience, she regularly presents test findings with executive audiences throughout the company. Beyond testing, executive stakeholders and product owners look to her for consultative guidance on mobile best practices and heuristic feedback. She was the founding Vice-Chair of PhillyCHI a local chapter of the ACM SIG CHI and has been an active member of the local Philadelphia UX community since 2004. Feel free to follow her on twitter at @ajderosa

Add new comment

Comments

64
63

Good read! But my personal experience of using is DeviceAnywhere FREE http://www.keynotedeviceanywhere.com/da-free-product-overview.php

Key features like Cloud-Based Architecture which allow us to access popular mobile handsets over the Internet. A time saving tool

66
69

Yes, Keynotedeviceanywhere is a device emulator that allows you to access and interact with a mobile device via a desktop. Note: Some apps may look slightly different when emulated. I prefer a usability participant to be able to interact with the actual device so I did not include emulators in this review. It could be used for early concept feedback.

66
70

Thanks for putting these posts together, Amber--if nothing else they re-affirmed that all the limitations I've run up against aren't just me being silly ;)

Given the limitations of "easy to use" apps like Magitest and UXrecorder, we've tended to do a lot of moderated, in-person mobile testing. Recently we did a study where we used Reflector app to share the device screen to a Macbook, which we broadcast to observers and recorded using a private UStream.tv feed. While we had device screen and audio capture, we lacked a participant cam and gestures--but the set up still provided a rich set of data to work from.

While Reflector was great for iOS devices, we used AirDroid's 'screenshot' function to mirror device screen on Android devices to the Macbook for broadcast/capture. While a little bit laggy (you actually access and view the 'screenshot'/mirroring via Safari browser) this set up worked pretty well, and wasn't difficult to install and use. AirDroid does require rooted access on the Android device, so this might be the only drawback. We used our own devices, but users' devices may not be rooted.

65
76

Thanks for sharing your experience Lynne! I still do mostly in person testing and we built a few tools for ourselves because the perfect tool (for our situation) does not yet exist. I am glad you shared your experience maybe someone who reads the article can benefit from your setup.

85
76

Thanks for the article, although it's a little depressing to see so many solutions with so many limitations! The perfect tool still doesn't exist.

I was stoked about UXRecorder and Magitest, but they reduced performance on my phone so much, the app I was testing wasn't really usable anymore.

60
78

Yes Mark unfortunately all of the tools I reviewed had some limitation or another, the limitations tend to vary depending on the tool. I thought this article and part 2 (coming soon) would help people choose which tool, if any, was right for them given their unique situation. If there was one solution this article would not be necessary, I would have just told my friends to use (insert the one solution here) when they asked me about a mobile tool.

63
82

Hi Amber, thanks for your article. Just a small correction: Reflector has screen recording capabilities built in the PC/Mac software but, as you noted, taps and other user gestures are not recorded (unless the feature is implemented in the application you are testing)

70
82

Thank you for the correction. I will update my notes. Are you suggesting that the PC/Mac has recording software built in so you can use that software to record or is it a feature of the Reflector capabilities specifically?

61
74

Reflector has built-in recording capabilities. It records audio and video, and encodes the video as MP4 if I remember correctly.

62
77

Thank you after further review I found the information about recording on the Reflector site.

Recording on OS X requires at least OS X 10.7 Lion. Recording on OS X 10.6 is not available.
Recording for Windows requires at least Windows 7. Recording is not available on Windows 8, Vista or XP. A bug currently prevents Windows 8 users from recording. We're aware of this and will issue a fix in an update.

Recording on Mac can be started/stopped from the device menu or by pressing ⌘+R.
Windows recording can be started/stopped by pressing ALT+R.

68
75

Hi Amber. There's another solution out there in Mr. Tappy.
He's a camera rig designed specifically for filming interaction with mobile devices.
check him out at www.mrtappy.com

66
75

Thanks Amber, great roundup!

Mr Tappy is awesome - need iOS AND Android? Just swap the phone out. I use mine with a GoPro Hero camera, which can create it's own wifi network to pipe the feed to another device in your observer room running the GoPro app (in this case, an iphone or ipad 5+ with a projector adapter). Does sound also, so your observer set up is sorted, and (very importantly in my case) totally outside of and not affected by any ridiculous enterprise firewall network nonsense. This solution is hardware (and battery) intensive, but I've tried a lot of the tools above, and this has been by far the most reliable and flexible (performance and reliability for UX Recorder and Magitest made them basically unusable, and reflector just didn't play nice with our corporate network AT ALL). I find users are pretty ok with Mr Tappy too - he doesn't weigh much so they can still walk around and be mobile if they want to. No capture of the user's face though (downside.)

77
87

Thank you Mr. Tappy for your comment. The article is focused on software tools not hardware or camera rigs. I use a HD HUE camera for my studies because it has a bendable neck. I could see where your tool could be used if someone wanted to attach something to the device.