UX Magazine

Defining and Informing the Complex Field of User Experience (UX)
Article No. 518 April 15, 2010

Usabilla and Loop11: Taking remote research tools on a test drive

User experience research can mean the difference between a product's success and failure. You might, for example, watch users who are playing a video game for the first time to see where they get lost, confused, or bored. You might review clickstream data to see where customers abandon their shopping carts. You could do formative research to help define features for a new product by interviewing people about their needs and preferences.

Not all of this research is done face-to-face. Remote research—recruiting and interviewing people without face-to-face contact—offers lots of advantages over more traditional research. Remote methods allow you to talk to people all over the world without travel expenses. A broader range of users provide a broader range of perspectives. Easier, cheaper, and better research? Rock n' roll.

Now several services promise to make remote testing even easier, cheaper, and faster, but how well do they work? To find out, I used two of them, Usabilla and Loop11, to help with some real research I am doing for a company called Livefyre.

Livefyre is a San Francisco-based startup that has developed a conversation platform that can be used as an online community at livefyre.com, or as a commenting widget incorporated into blogs. The team at Livefyre asked me to help them answer some questions about how people are using and understanding their new site.

Livefyre home page

Some of their main concerns were:

  • new visitors to livefyre.com might not understand the purpose of site
  • new users might have difficulty getting started
  • creating an account might pose barriers
  • some users might be turned off that accounts needed to be tied to either a Facebook or Twitter accounts

We crystalized these issues into the following user research study questions:

Homepage

  1. What do you think that people do on this website?
  2. What's the first thing you'd like to do on this page?
  3. Where would you go to create an account?

Login Overlay

  1. Is this login page what you expected to see?
  2. Do you feel comfortable entering your Facebook or Twitter username/password when you create accounts on other sites?

And so I used these questions to test whether Usabilla and Loop11 could help answer these questions through remote user research.

Livefyre join screen

Usabilia Testing

I started with Usabilla. Usabilla promises "hassle free usability testing" through "one-click" tests: "You ask your users a question and they answer it by clicking anywhere on the screen." Access starts at $49 to test 10 pages and goes up to $950 to test 250 pages.

I wasn't exactly sure how it worked, but I jumped in to figure it out.

Usabilia homepage

First you upload a screenshot of your site. There is an option to paste in a site's URL, but it's a bit misleading; Usabilla just takes a screenshot of that page, and the screenshot it takes is only above the fold. I wanted to ask questions about the entire homepage, not just the part above the fold, so I had to download and install a Firefox extension to do a full-screen grab (I used Screengrab). I took screenshots of both the homepage and the homepage with the "Join Livefyre" overlay.

After uploading them to Usabilla, I then moved on to the question component. Usabilia recommends default questions, but they didn't seem very useful. I added my own questions: the three questions about the homepage and the two about the account creation dialog listed above.

Usabilia add question screen

Once I had uploaded the pages I wanted testers to evaluate and added (or selected) the questions to ask on those pages, I was done. Usabilia then generated a URL to share the test.

Overall, I found the process to be fairly painless once I figured out about the full-page screenshot. The downside, though, is that since you're only looking at static images you can't show site interactions. The Livefyre site has some lovely interface behaviors that a picture of the site can't convey.

I sent the URL to four test users. Unfortunately, it didn't work well. The users didn't know that they had to click on the page to add a note, and then add an annotation to this note in order to answer a question. They would just click to progress to the next question and there was no way to return to respond to a previous question. The process—clicking to add a note—was described in an introductory movie that none of them watched.

Because of this significant usability issue, I would not trust the data I would get from a larger-scale study.

Would I try Usabilla again? Probably not. The same answers could have been gained for free, and more easily, by posting screenshots along with a form created in Google Docs. If you're primarily interested in how a large group of users navigates your site, you would be better served by traditional traffic analysis using Web analytics tools.

Pros

  • Easy to set up
  • Somewhat inexpensive

Cons

  • Usability problems
  • Impossible to test interactions

Summary

You're probably better off doing a survey along with basic traffic analysis.

Loop11 Testing

Loop11 required more effort to get started and the pathway to beginning a new project was not as clear as with Usabilla. I signed up for a free trial and forged ahead.

Loop11 homepage

The biggest difference I saw at first was that Loop11 uses live HTML pages rather than screenshots. An obvious benefit is that you don't have to go through the hassle of getting screenshots. But a less obvious problem is that you can't easily test a dynamic page that changes without changing the URL.

Here's what I mean: my little study was looking at two pages, the homepage (livefyre.com) and the overlay that appears when you click on "Join." The URL is the same for both. I was stymied as to how I could ask a question about the "Join" overlay. I ended up just telling the users to click on the "Join" button to complete the test.

Loop11 allows you to enter either "questions" or "tasks." I entered the first question, "What do you think that people do on this website?" and was pleased to see that I could choose the question format.

Loop11 Create Question Screen

What I didn't realize, though, was that I could not ask a question and display the Livefyre site at the same time. The site can only be displayed for a "task" such as "Click on a button." Any questions appear separately on otherwise blank pages. In order to ask a question about the site, I created a task to "Spend a minute looking over this page. Then click 'Task Complete' to continue."

Loop11 Task Screen

Then on the next screen I asked a question: "What do you think that people do on this website?"

Loop11 Question Screen

This felt clunky, but I couldn't figure out a better alternative.

As I added more questions and tasks I got a better sense of the real value of Loop11. The ability to alternate survey-type questions with task analysis is something that I haven't seen before, and it's potentially very powerful. You could ask, for example, "How many times have you gone to the grocery store this month?" and then ask the participant to navigate part of a recipe site. Would people who shop for groceries more often find it easier to navigate a menu of ingredients? There are lots of possibilities for how this mix of questions and tasks could add a useful dimension to user research.

I launched the test and shared the URL with four participants. They all found the process of using Loop11 straightforward and did not have trouble using it. They did note, though, that it was difficult to answer a question about a page that they had previously seen without being able to look at it at the same time.

The dashboard displays the number of responses, the paths that the participants took when responding to tasks, and the answers to questions. For each participant you also get a breakdown of total time spent on the test, average time taken on tasks, and average page views per tasks. For the Livefyre test, most of these metrics weren't very useful.

Going through the process of creating, launching, and analyzing a test helped me understand that Loop11 is optimized for research into specific task completion such as, "How quickly can users find an answer in the help center?" rather than more open-ended tasks like the ones I was working with.

I would say that Loop11 will be a great tool further down the line with Livefyre. At this point, as we do work that requires a nuanced understanding of messaging comprehension, Loop11 provides some insight. As we do more research into user task flows it will be a great tool. Also, it integrates with Ethnio, which is a great remote testing recruiting tool. It will be interesting to see how the partnership between these two services continues.

Pros

  • Flexible question format
  • Ability to track and measure tasks
  • Integration with Ethnio

Cons

  • Can't ask questions on a page with visuals
  • Relatively expensive

Summary

If you have a budget to work with and tasks to measure, this would be a great tool to add to your arsenal.


As we wrap up, remember that these are just tools; they support, but don't replace, the work that a researcher does. Just as with all research tools, you need to understand what you want to discover, what questions to ask, who you should ask, and what insights you can glean from the resulting data.

Have you used remote testing tools? What's your experience? Please share in the comments below.

ABOUT THE AUTHOR(S)

User Profile

Juliette Melton is a user experience researcher and design strategist based in San Francisco. You can find her on Twitter at @j.

Add new comment

Comments

17
27

I'd like to recommend Justinmind Prototyper (http://www.justinmind.com) as a tool to make HTML prototypes that can be tested using both Usabilla and Loop11

25
24

Is there any ficility to capture the flow of application using Usabilla ? or does it capture only screenshots?

24
28

Thanks again for the feedback. We've just released an update for our test interface, which makes it easier for participants to share feedback with notes:
http://revw.us/bP2SkJ

24
29

This is a useful article and discussion. I'd like to note that usertesting.com is really unmoderated usability sessions that you don't gain metrics/qual data from like you do with Usabilla and Loop11. They're just different beasts.

Bolt | Peters also checked out both Usabilla and Loop11 this week (May 14) for a test and we have a few notes to add to Julie's write-up:

Usabilla:
*Usabilla does take screenshots below the fold now (that was one thing we liked)
*I'm sure no one watches the introductory movie: which is why we timed it (30-seconds) and left an explicit comment to watch it to learn how to interact
*We also explicitly added a comment in the "help text" for the first question reminding users of how to click and add anote
*Usabilla has now added the feature to deactivate, edit, and re-run your test — you can't duplicate one yet, but you can deactivate to tweak which we weren't able to do before

Loop11:
*it does not work with Flash websites (of which there are a lot)
*it does not work with Facebook.com which blocks users from logging in if the site is hosted on another domain — smart, but it means you can't use Loop11 to test interactions on or with Facebook

20
27

Brynn - thanks for the note. 

 

Usabilla:

*Usabilla does take screenshots below the fold now (that was one thing we liked)

*** I'm glad to know that Usabilla does take full-screen screenshots now -- that is a great improvement to the product.

 

*I'm sure no one watches the introductory movie: which is why we timed it (30-seconds) and left an explicit comment to watch it to learn how to interact

*We also explicitly added a comment in the "help text" for the first question reminding users of how to click and add a note

*** These are good workarounds. Ideally, though, users would be able to use the product without needing to watch a video or lots of additional prompting.

 

*Usabilla has now added the feature to deactivate, edit, and re-run your test — you can't duplicate one yet, but you can deactivate to tweak which we weren't able to do before

 

*** It's good they added this to the product and that it's still evolving.

 

Loop11:

*it does not work with Flash websites (of which there are a lot)

*it does not work with Facebook.com which blocks users from logging in if the site is hosted on another domain — smart, but it means you can't use Loop11 to test interactions on or with Facebook

*** Great to know. Thanks!

23
29

The space is heating up.. thanks for the write up! We did one of our own back in the Loop11 closed beta days. Including a "crowd-source" usability analysis trend headed up by Feedback Army

http://limina-ao.com/blog/2009/07/01/crowdsourcing-usability-or-not/

26
31

Hi there,

Interesting article, but you forgot the most useful tool (well, in my totally biased opinion anyway) http://intuitionhq.com - we've tried to be a bit different from sites like Usabilla and Loop11, easier to use, and requiring much less of an investment of time.

I think it could be quite helpful for you at this stage in your design process, and you could get some pretty revealing information pretty quickly.

If you'd like to have a play, hit us up on twitter @intuitionhq or through the website - we'd happily set you up with a free test if you'd like to try it out.

Thanks, and good luck with your new site!

22
27

Remote methods allow you to talk to people all over the world without travel expenses. A broader range of users provide a broader range of perspectives.

23
34

Thanks for this review. The Austin Usability Professionals Association recently held a panel in which UXers discussed Morae, UserVue, ClickTale, UserFly, Silverback, VMWare and WebEx. My notes from that meeting are posted here: http://www.erinlynnyoung.com/335/user-research-notes/

I hadn't heard of conceptfeedback.com (comment above) so I look forward to checking that out.

19
24

You may also wish to look at fivesecondtest.com and conceptfeedback.com

21
27

Thanks for the review Juliette.

Usabilla is designed for studies with a large number of participants (100 - 1000 participants. The answers of a larger test group are presented with scatterplots and heatmaps. A study with 4 participants is like taking a Porsche for a test drive and only use the first gear.

You can use Usabilla to set up behavioral tests and to collect feedback (attitude). We offer two types of tasks: standard tasks and "One-Click" tasks. A standard task allows participants to answer a question by adding multiple points to the screen. They can add a notes to these points (optional) to share some additional feedback. The second type of tasks (One-Click) allows participants to only add one point. This way you can analyze task performance (time, position of a click).
Especially your second question 'What do you think that people do on this website' is difficult to answer with Usabilla and is a question you could better ask in a post test survey. We can definitely improve the way we guide our users in setting up a test and help them to select the right tasks. We published some demo cases on our blog and we're definitely going to add more examples in the near future.

I really appreciate that you point out some interesting issues we need to solve. We're constantly improving Usabilla and it's great to hear how people are using our product. We're working on some exciting new features. In about 2 months you'll be able to test user flows (participants can click through multiple screens) and we're working on the first prototypes for live feedback on a webpage (test without a screenshot).

19
29

Hi Paul, thanks for your note. I'm looking forward to trying out the user flow testing.

Also, to clarify -- to your point "A study with 4 participants is like taking a Porsche for a test drive and only use the first gear" -- I wasn't testing the validity of the research results of the tool, but rather the usability of it. Because the participants had so much trouble responding I was dissuaded from running a large-scale test with it. 

Thanks,

Julie

24
21

Having practised user experience research for years, I appreciate and welcome gladly both of these cost-efficient online tools.

Unfortunately http://www.usertesting.com cannot be used in Scandinavia. We do not have the possibility to use the TechSmith Morae's extension UseVue here either even though I regularly use my Morae.

Thanks for the useful review!

22
34

Interestingly, I've found http://www.usertesting.com to be the best of both worlds. For $40/user, you can give them a task (there's no page limit) to complete and get a screen video of their actions as well as a narration as they work through the scenario. They also write responses to four follow up questions you provide.

The best part is you don't have to provide the users and you can get results in short order (we has complete results in an hour). You can also select the demographic information on who does the testing.

It's not perfect, but it's cheap enough that you can get real world feedback in short order, and flexible that you aren't limited in testing scope.