In contrast to the traditional water fall model, in an agile environment design and development teams collaborate very closely and there is little step-by-step procedure or upfront planning—decisions are made and solutions are implemented on the fly, in a highly iterative and flexible manner.
However, the lack of planning and lead-time in the process poses a major challenge to user experience research. Remember, UX research is supposed to bring a strategic perspective into software development, helping the product team figure out priorities and focus on the right things to work on based on user insight. But the making-decisions-on-the-fly mindset underlying the agile process seems to make conducting UX research improbable and unnecessary.
This leads to the questions, is UX research even needed any more in an agile environment? And, if so, how do we conduct effective UX research in this context?
The answer: lean UX research—conducting research in a quick, but not dirty, way.
Garbage in, Garbage out
A common objection to conducting UX research in the agile model is that we can simply launch, conduct A/B testing, and learn from the testing results. Using real market data to validate products, there’s no more need for UX research.
A fatal flaw exists in this premise. Test-and-learn—choosing winners among variations through A/B testing—can tell you which version is better but fails to answer the question of why it’s better. More importantly, it cannot tell you if there’s another, untested version, that’s better than all the variations you A/B tested, due to the lack of insight into “why.”
Furthermore, test-and-learn doesn’t come free. It takes time and effort, not to mention great skills, to do it right. To begin with, you need to form good hypotheses, create variations that can effectively test the hypotheses, and then spend a lot of time analyzing the results. Most A/B tests don’t even give you clear data because the variations are not that different in terms impact on user behavior.
It’s what I call “garbage in, garbage out.” Good consumer insight, the output of testing, depends on high-quality input, things like valid hypotheses and well-constructed design alternatives. If you don't know what matters to users, your A/B tests will have low-quality input that will only get you misinformed output.
To know what really matter to users, you have to rely on UX research that generates an in-depth, qualitative understanding of users and products.
More Effective UX Research
Looking closer, you'll find that the agile process not only makes sense in supporting a faster, more effective software development process, it’s also a great opportunity for UX research to truly influence the final output!
Under the old water fall model, the product team moves through strategy, research, design, and development in clearly defined steps. In this environment, we can plan research far in advance and we have the luxury of conducting very comprehensive and thorough studies. However, because research and development are separated in the process, researchers have limited influence over the team in terms of actually implementing the research findings.
In an agile environment, however, due to the fact that teams interact more frequently and more closely, UX researchers are able to conduct research to support the ever-changing team needs and turn insights into actions quickly. That’s what lean UX research is all about.
A Quick Guide to Conducting Lean UX Research
So how do we conduct effective research under tight timeline to improve business results? It requires researchers to apply a few different approaches, and I’ll go through them one by one.
Conduct UX research throughout the product development cycle
Traditionally UX research is conducted before the development stage. Given the rapid iterations of product development in any setting (agile or traditional), that leaves little room for conducting research extra. Due to the iterative nature of the agile process, you can conduct research during the development phase or after a product is launched, when insight turns into action for the next release, which could be just weeks away.
Another benefit is, testing QA versions or live products saves us lots of time when creating prototypes for testing and can yield better insight because the product is fully functional and users can explore it in natural ways and give context-relevant feedback.
In an agile world, there’s no wrong time for conducting UX research.
Conduct UX research to complement A/B tests
Given that a big part of the Agile process is test-and-learn (typically meaning A/B testing) we need to conduct UX research to complement A/B testing. A typical way to do this is to conduct a usability study on the different variations currently being A/B tested. Given that the product is already live with the different variations, it’s very easy for us to test the product, as there’s no need to do prototyping or wireframing in preparing for the usability study.
Conducted in conjunction with A/B testing, the usability study can tell us “why” one variation is better than the other, and if a better solution outside of the variations tested exists. During one project I was asked to conduct a usability study to evaluate the variations of a live-site A/B test in order to encourage users’ shopping behavior. Combining insight from the UX research with data from the A/B test, I helped the client create an experience in which users were much more likely to go through the shopping flow, and we saw a truly dramatic lift in revenue as a result.
Develop ongoing user research program
You can put in place an ongoing research program with a pre-determined testing schedule and an ongoing recruiting effort. For example, your company can conduct user interviews on a monthly basis with different topics for each month. Because the schedule is fixed in advance, the team is able to plan and recruit participants with a lot of lead-time. In an agile environment, due to the speediness of development and design changes, you will always find something worth testing every month (see the graph below).
I’ve conducted this kind of ongoing UX research for a client, and each month, we received requests from various product teams in terms of topics to test and were able to plan accordingly. This approach informed the company of up-to-date usability issues and user concerns on an ongoing basis, leading to quick solutions and many new product ideas to fill the gaps.
Be creative and flexible about process
In an agile environment, we need to drastically shorten the cycle by doing more things at once, taking out unnecessary steps, and applying alternative research approaches. For instance, you can use friends and colleagues as participants, assuming they fit your target user’s profile. You can also create a high-level discussion guide that addresses key research questions but that doesn’t contain detailed verbiage used for the interview, and you can write a quick findings report rather than a fully-developed, polished report.
I conducted an informal usability study in the form of diary-style short surveys to influence the design requirements and improve usability of the first launch of an iPhone app for a major financial company. The app, which greatly benefitted from the informal user research, later won a Silver award from w3c and received a five-star average rating in the app store.
Agile ≠ Poor Quality
UX and market researchers often fear that conducting research the lean way means sacrificing quality for speediness. Speaking from personal experience, I find the fear unfounded. If anything, this fear reveals a deeply rooted misconception of quality.
Quality of UX research is not about research deliverables like polished reports and pretty PowerPoint presentation graphics, it’s all about the quality of the product and the kind of user experienced it offers. Defined as such, the lean UX research approach leads to higher quality because we can more effectively influence product teams in producing great products this way.
This is, of course, requires that we’re thoughtful about research methodology: interviewing users with the right kind of profile, asking relevant questions, taking great care in extracting user insight, and making actionable recommendations.
At first glance, the agile process poses challenges to conducting UX research. Some might even think UX research is no longer relevant under this new development model. But looking closer, you’ll see that the agile process can actually present more opportunities for leveraging UX research to actually achieve visible business results. As long as product teams commit themselves to listening to user feedback, having open communication within the team, staying responsive in leveraging user insight, and adopting some of the techniques mentioned above, they will be able to conduct and benefit UX research in this new environment.
Image of agile orangutan courtesy Shutterstock