Product Validation In A Real-Life Scenario: Intercept Field Study

This article was originallly published by UX Studio.

Is it enough to validate a product with in-lab studies? We went to the field with our football app to get real-life insights. The results have changed our beliefs.

Our client creates digital products to help football fans follow matches, favorite teams, news or even play web or mobile games. Some years ago their team decided to put a new app in football-lovers’ pockets. This app would guide fans as they travel to a match anywhere in Europe. Packed with clever tips, news, sightseeing and public transport info and more – all focusing on football.

Before design, the team already had

• Insights from a massive discovery research on the 2018 World Cup
• Over 3000 detailed observation points from our research system

With enough research ammo, they began to build the app. The plan was to introduce it during the EURO 2020 where millions of fans crazy about football would be active. Of course, these plans were made long before the COVID-19 outbreak.

From the Lab to the Jungle

Design and development were going smoothly and genuinely fast. By May 2019 the team already had a lab-validated test app with an early version of all the major features. We as researchers constantly supported the team with insights.

But we had zero knowledge of real-life usage. What will happen when the first user downloads and touches this app? How will they react?

The team decided that the product has to leave the convenient and familiar lab environment, so the researchers were asked to go onto the field to collect insights:

• from real ticket holders,
• in real-life situations,
• using our live app.

Luckily, the finals week of UEFA Nations League appeared to be a convenient time and occasion for testing our application live. The team adapted the product to the competition and we flew to Porto, Portugal.

Research streams

Our goal was set: evaluate our app in a real-life context. But how? We listed the questions we wanted to get answers for and divided them into two categories:

1. How do people think and behave while adapting the app on a long run?
2. How easy is it to use the current app and what contexts people might need to be able to use them?

In order to find answers, we’ve defined two separate research streams for these groups of questions. These research streams operated independently on the field to ensure unbiased findings. Stream 1 focused on longer adoption of the app while Stream 2 collected intercept insights about first-time users.

Stream 1: Concept evaluation

Method: Diary study

With this stream, we wanted to find answers to these questions: What do people think about this app? How do they adapt to it? How does it align with their habits? Is it really something that fulfills the needs of tournament goers in the long run?

Stream 2: User experience evaluation

Method: Intercept field study

With this stream, we wanted to find answers to these questions: How do tournament goers understand our current app’s offering? How do they complete the given tasks? How easy or difficult is it to use the app? What journey steps do they go through when they start using it?

I organized and led Stream 2: UX evaluation.

Challenges

Probably the most time-consuming part of our initial planning was to brace ourselves for every possible scenario – both in terms of research and logistics. We basically sat down on a daily basis and gave a lot of thought to how to best cover everything.

Challenge 1: Screening

In order to have a large variety of different users, we had to set up our criteria.

We identified two main user groups for the testing: Travelling fans (unfamiliar with the city, living 50+ km away or abroad) and Local fans (living in or nearby the city). In order to adapt our most suitable scenario, we also wanted to ask about their favorite teams, days of staying in the city, basic demographics, and some of their planned programs.

But how can one verify these on the field? How can we screen them if we only have a few moments before they lose their interest?

We have printed a screener spreadsheet with all the necessary information we wanted to learn. However, instead of approaching them with our boring questions upfront, we immediately started with the fun part: showing them the app. Since people don’t usually go alone to watch football games, the second researcher could target another group member (Secondary user) and ask the screening questions from them.

After a few minutes, once our primary user got enthusiastic about the app, we could slowly ask our questions from them as well, while completing the tasks.

This way we were able to have an intercept and one or two short interviews from a group.

Tips based on my learnings:

• Screen your participants, but collect only necessary information. Time is extremely tight when intercepting.
• Take notes as fast as you can: even though you do screen recording, essential bits of information can be lost in the noise.
• Use the power of your team: don’t do field study with less than 2 researchers working together. Distribute tasks wisely.

Challenge 2: Location

We aimed to intercept football fans in environments most suitable for tests and short interviews. In order to do so, we had to run through our initial assumptions about locations and cross-check it with our football experience. So I ran in-head mind-palace Q&A sessions with myself on a daily basis. It sounded something like this in my head:

1. What are the most frequent places and times we can stop fans for a few minutes?

→ I guess airports, train stations, fan meeting points, football events in the city center, and so on.

2. Can we use those locations?

→ Probably yes. Let’s get written permission from officials to do research on the location sites.

3. What scenarios should we use at each location?

→ Well, fans are likely to get directions to the city center when arriving at the airport and train stations. They’ll possibly wait at baggage claim. I assume they would be interested in football-related programs close to their hotels and the city center. They probably want to read news and facts about the events to kill time while they are waiting.

4. How do we build up a schedule to touch all locations during our limited time?

→ Check arriving flights from supporters’ countries. Check train schedules between cities where games take place. Build a plan to follow the fans’ journey throughout the week.

…and so on.

Tips based on my learnings:

• Start your plan with a lot of thinking and map out your on-site journey thoroughly.
• Share your plans early on with the team as they have fresh eyes.

Challenge 3: Improvisation

Planning: check. Now let’s turn your plan upside down. If you’ve ever done any field research, you know planning isn’t everything.

You get there, suddenly realize there’s a construction on your desired location, potential users are nowhere close by and your app keeps on loading due to poor mobile network. What do you do?

These are exactly the reasons that make us go and observe the real context of our product in the first place. Alright, none of the above really happened to us this time, but it could have. Here’s something we experienced though:

Example: participants not stopping

By the time we got to the first location (airport), we already had a spreadsheet printed with arrivals of countries with upcoming matches. But as we tried catching people coming out of the arrival gates almost everyone refused to help us.

Makes sense, right? You get off a plane, you collect your luggage, you’re probably tired and just want to get to your accommodation.

So we revised our tactics and used our Beaver scout observational senses. Instead of catching people on the move, we started looking for people who waited for something. Where do people wait at the airport? Taxi pickups, public transport stops, cafés. We followed the crowd and realized that the train from the airport was leaving every 10-15 minutes.

Findings

Fortunately, we managed to overcome every one of our challenges and carried out a successful study. The results were invaluable: we have collected insights we wouldn’t have done in any other way. Here’s a handful of them:

Content structure

Initially, our app presented information about the finals week in a tree-like system: you have main sections, then sub-sections within, then even more granular information on a third level. This system worked on the lab tests, but when we went on the field, our system kind of fell on us.

During the intercepts, a number of users started spending too much time jumping back and forth between sections and submenus to collect the small bits of information in the necessary articles. This was overwhelming, and they couldn’t even find the desired content in some cases. Their mental model was built-up in a horizontal structure: “Show me everything I need to know today!”

So the product team put a new “mix” to the home page with relevant info and interesting details of multiple sections. It served kind of as a feed of ‘most relevant information at the moment’. The team kept the original navigation as well but restructured and renamed sections for better findability.

Best feature?

Before the study, we thought ticket holders would spend most of their time using our custom map that had built-in football interests, shortcuts to football activities, route planning and more.

On the field we learned that although people took their time to use it, they did not find that much value in it. Our football fans, such as other audiences, were likely to use their favorite apps they’re already familiar with – Google Maps, Apple Maps, Maps.me.

On the other hand, they found value in our Tips: a section where we would show them interesting facts, notifications and events going on in the city at the moment that are connected to the games.

Copywriting, Local names

We used a variety of names in our article, search, and map to identify locations. Some examples:

• Estádio do Dragão
• São Bento
• Guimaraes

Can you quickly tell what these names are? Neither did I. The first one Estádio do Dragão is actually Porto’s home stadium, where most of the games have been played at. São Bento is a train station in Porto city center, while Guimaraes is a city in Porto’s agglomeration.

We’ve learned that users had problems remembering local names, and identifying what is what. They used terms like “Porto stadium” or similar in their mother tongue. So we had to improve our app by adding English alternatives to locations. After a bit of Googling, I found out that this was an issue that Google Maps also identified and solved with alternative street/interest names.

So the app team simply added subcategories and alternative names:

• Estádio do Dragão – Porto stadium
• São Bento – Porto train station
• Guimaraes city

We also came up with new name ideas to almost all sections within the app as users had problems identifying the content behind them.

Takeaways

Every field research is an opportunity to check real-life situations. We had the chance to talk and evaluate our app with more than 50 participants in 10+ scenarios. Then we compared our intercept study with the diary stream to have the full picture.

Our main takeaways from this project:

1. Invest in field studies. You can never know how your users will use the app in real-life situations. Intercept offers quick and relatively easy evaluation – combine with longer studies to the best results.
2. Have a plan but dare to change it. Review and enhance your plan on site and improvise if necessary.
3. Attitude is key. Be friendly, open, to the point, and smile a lot to gain trust early on. However, prepare for plenty of rejections – it’s part of the job.
4. Distribute tasks well amongst team members. Time is extremely limited, use it the best you can.

Now is not the best time to go on the field while COVID-19 is out there. Check out our other articles on how to build up a remote team and overcome remote work challenges.

Happy researching!