UX Magazine

Defining and Informing the Complex Field of User Experience (UX)
Article No. 1146 November 20, 2013

Create Great UX in an Agile World by Conducting Lean UX Research

The agile software development process has gained tremendous popularity recently, adopted by many companies to deliver high-quality products through iterative launches and testing.

In contrast to the traditional water fall model, in an agile environment design and development teams collaborate very closely and there is little step-by-step procedure or upfront planning—decisions are made and solutions are implemented on the fly, in a highly iterative and flexible manner.

However, the lack of planning and lead-time in the process poses a major challenge to user experience research. Remember, UX research is supposed to bring a strategic perspective into software development, helping the product team figure out priorities and focus on the right things to work on based on user insight. But the making-decisions-on-the-fly mindset underlying the agile process seems to make conducting UX research improbable and unnecessary.

This leads to the questions, is UX research even needed any more in an agile environment? And, if so, how do we conduct effective UX research in this context?

The answer: lean UX research—conducting research in a quick, but not dirty, way.

Garbage in, Garbage out

A common objection to conducting UX research in the agile model is that we can simply launch, conduct A/B testing, and learn from the testing results. Using real market data to validate products, there’s no more need for UX research.

A fatal flaw exists in this premise. Test-and-learn—choosing winners among variations through A/B testing—can tell you which version is better but fails to answer the question of why it’s better. More importantly, it cannot tell you if there’s another, untested version, that’s better than all the variations you A/B tested, due to the lack of insight into “why.”

Furthermore, test-and-learn doesn’t come free. It takes time and effort, not to mention great skills, to do it right. To begin with, you need to form good hypotheses, create variations that can effectively test the hypotheses, and then spend a lot of time analyzing the results. Most A/B tests don’t even give you clear data because the variations are not that different in terms impact on user behavior.

It’s what I call “garbage in, garbage out.” Good consumer insight, the output of testing, depends on high-quality input, things like valid hypotheses and well-constructed design alternatives. If you don't know what matters to users, your A/B tests will have low-quality input that will only get you misinformed output.

To know what really matter to users, you have to rely on UX research that generates an in-depth, qualitative understanding of users and products.

More Effective UX Research

Looking closer, you'll find that the agile process not only makes sense in supporting a faster, more effective software development process, it’s also a great opportunity for UX research to truly influence the final output!

Under the old water fall model, the product team moves through strategy, research, design, and development in clearly defined steps. In this environment, we can plan research far in advance and we have the luxury of conducting very comprehensive and thorough studies. However, because research and development are separated in the process, researchers have limited influence over the team in terms of actually implementing the research findings.

In an agile world, there’s no wrong time for conducting UX research

In an agile environment, however, due to the fact that teams interact more frequently and more closely, UX researchers are able to conduct research to support the ever-changing team needs and turn insights into actions quickly. That’s what lean UX research is all about.

A Quick Guide to Conducting Lean UX Research

So how do we conduct effective research under tight timeline to improve business results? It requires researchers to apply a few different approaches, and I’ll go through them one by one.

Conduct UX research throughout the product development cycle

Traditionally UX research is conducted before the development stage. Given the rapid iterations of product development in any setting (agile or traditional), that leaves little room for conducting research extra. Due to the iterative nature of the agile process, you can conduct research during the development phase or after a product is launched, when insight turns into action for the next release, which could be just weeks away.

Another benefit is, testing QA versions or live products saves us lots of time when creating prototypes for testing and can yield better insight because the product is fully functional and users can explore it in natural ways and give context-relevant feedback.

In an agile world, there’s no wrong time for conducting UX research.

Lean UX Research
Conduct UX research to complement A/B tests

Given that a big part of the Agile process is test-and-learn (typically meaning A/B testing) we need to conduct UX research to complement A/B testing. A typical way to do this is to conduct a usability study on the different variations currently being A/B tested. Given that the product is already live with the different variations, it’s very easy for us to test the product, as there’s no need to do prototyping or wireframing in preparing for the usability study.

Conducted in conjunction with A/B testing, the usability study can tell us “why” one variation is better than the other, and if a better solution outside of the variations tested exists. During one project I was asked to conduct a usability study to evaluate the variations of a live-site A/B test in order to encourage users’ shopping behavior. Combining insight from the UX research with data from the A/B test, I helped the client create an experience in which users were much more likely to go through the shopping flow, and we saw a truly dramatic lift in revenue as a result.

Develop ongoing user research program

You can put in place an ongoing research program with a pre-determined testing schedule and an ongoing recruiting effort. For example, your company can conduct user interviews on a monthly basis with different topics for each month. Because the schedule is fixed in advance, the team is able to plan and recruit participants with a lot of lead-time. In an agile environment, due to the speediness of development and design changes, you will always find something worth testing every month (see the graph below).

Lean UX Research

I’ve conducted this kind of ongoing UX research for a client, and each month, we received requests from various product teams in terms of topics to test and were able to plan accordingly. This approach informed the company of up-to-date usability issues and user concerns on an ongoing basis, leading to quick solutions and many new product ideas to fill the gaps.

Be creative and flexible about process

In an agile environment, we need to drastically shorten the cycle by doing more things at once, taking out unnecessary steps, and applying alternative research approaches. For instance, you can use friends and colleagues as participants, assuming they fit your target user’s profile. You can also create a high-level discussion guide that addresses key research questions but that doesn’t contain detailed verbiage used for the interview, and you can write a quick findings report rather than a fully-developed, polished report.

I conducted an informal usability study in the form of diary-style short surveys to influence the design requirements and improve usability of the first launch of an iPhone app for a major financial company. The app, which greatly benefitted from the informal user research, later won a Silver award from w3c and received a five-star average rating in the app store.

Agile ≠ Poor Quality

UX and market researchers often fear that conducting research the lean way means sacrificing quality for speediness. Speaking from personal experience, I find the fear unfounded. If anything, this fear reveals a deeply rooted misconception of quality.

Quality of UX research is not about research deliverables like polished reports and pretty PowerPoint presentation graphics, it’s all about the quality of the product and the kind of user experienced it offers. Defined as such, the lean UX research approach leads to higher quality because we can more effectively influence product teams in producing great products this way.

This is, of course, requires that we’re thoughtful about research methodology: interviewing users with the right kind of profile, asking relevant questions, taking great care in extracting user insight, and making actionable recommendations.

Lean UX Research

Conclusion

At first glance, the agile process poses challenges to conducting UX research. Some might even think UX research is no longer relevant under this new development model. But looking closer, you’ll see that the agile process can actually present more opportunities for leveraging UX research to actually achieve visible business results. As long as product teams commit themselves to listening to user feedback, having open communication within the team, staying responsive in leveraging user insight, and adopting some of the techniques mentioned above, they will be able to conduct and benefit UX research in this new environment.

 

Image of agile orangutan courtesy Shutterstock

ABOUT THE AUTHOR(S)

User Profile

Frank Guo provides UX strategy and design consulting through his firm, UX Strategized LLC, helping leading companies like eBay, PayPal, Yahoo!, StubHub, and IMVU improve user experience, usability, product strategy, and digital marketing. He has published 15+ professional papers and co-authored a book chapter related to user experience.

Add new comment

Comments

36
43

Team members moving in and out of projects can be a challenge as we know there is little to no documentation in agile. How do you seamlessly transfer knowledge between UX research/insights/designs?

I'd add that for UX Research to truly be incorporated into the agile process buy in from stakeholders is important. Stakeholders can become the champions of UX work once they see the influence, and can help involve/partner up with UX research/teams to drive success as project moves along.

34
42

Hi Len, these are great questions! For the first question, I've been using the so called lean UX research report, which is typically in the format of Word document, with all the findings, recommendations, and screen shots, to document the findings. That won't take too long to put together and that can help knowledge transfer. That said, when there're new team members joining, we should walk the new members through the lean research reports to get them up to speed about past findings.

Yes I completely agree that stakeholder buy-in is crucial, otherwise the findings will not be leveraged to improve the UX. Lean UX research is quite effective in getting stakeholders' buy-in, because of its quick turnaround time and immediate results. Also, lean means the UXers work very closely with the stakeholders, which should also help with buy-in.

49
40

Interesting article Frank. We've been trying some more lean research techniques in my team and they do work. Taking designers along to sessions with users while research is going on cuts out the need for lengthy reporting.

However, clients and designers still want the detail. So for us it's a juggling act between having to constantly provide the context and providing enough information that an ever-evolving product can leverage the same insight as it moves from one channel to another.