The purpose of this article is to raise consciousness about fundamental problems with traditional focus groups used for product development. Even experienced researchers often ignore, dismiss, or are sometimes even unaware of these issues and how they may be impacting the quality of their research.
1. There is a gap between what focus group participants say and what they do.
Traditional focus groups do not provide an accurate read on customer behavior.
If you are relying on focus groups to get a direct read on customer behavior or purchase intent, you are most likely using the wrong method. Moderators who tell you they can provide you with “insights” on these topics may be well intentioned, but are likely overconfident of their abilities or lacking serious understanding of the wealth of research that's been done in this area.
As Harvard Business School professor Gerald Zaltman wrote in his influential book How Customers Think, “The correlation between stated intent and actual behavior is usually low and negative.” Eighty percent of new products or services fail within six months of their launch, often after they have been vetted through focus groups. In Hollywood, for example, TV pilots and motion pictures are often screened via focus groups for audience reactions. Countless products “pass” in focus groups, but fail when they reach the market, and the gap between what participants say vs. what they do is likely the reason.
Another problem with focus groups is they often ask people for their quick, topical reaction to products they have not seen, used, or experienced. The problem is that everyone has an opinion. And the more detail they give you about their opinion, the more likely you (the focus group observer) are to think you are hearing the customer’s voice.
In general, the more abstracted an expressed opinion is from reality (i.e., the context of actual usage of the actual service or product), the more likely it is that the research participant’s opinion will be incorrect, inaccurate, and misleading.
What does this mean exactly?
If you show people a product, they will give you an opinion or reaction. If you have them use or interact with the product, their feedback will be based on their actual experience, which is far superior to their initial, hypothetical thoughts.
For example, if you ask a participant what they think of Android devices, they will offer an opinion. If they do not have an Android device, or have never used one, their opinion will be based on their attitudes toward Google or Android, what they have read, and what their friends have told them. But it will not be based on their actual experience.
If you have them use the device instead and then ask for their feedback, now you’re dealing with their actual experience. And feedback based on actual experience occurs at a much deeper level of cognitive involvement, meaning the participants will give you better, more accurate feedback.
This is a critical distinction. The deeper the level of involvement with the product, the better the insights customers will be able to offer about their usage of the product.
Traditional focus groups miss the mark when they focus on hypothetical choices rather than actual experiences. As Zaltman says, “Contrary to conventional wisdom, [focus groups] are not effective when developing and evaluating new product ideas, testing ads, or evaluating brand images.”
The solution is to create a research plan that focuses on actual user experience rather than hypothetical questions or scenarios. This will look like a naturalistic usability test, or a hybrid usability test that might involve having users use the product individually before returning to the group. In that manner, they will have the individual experience of using the product and can have a group discussion about the experience.
2. Small local samples cannot be generalized.
Focus groups tend to work with small numbers of people. The most common arrangement is to conduct 2-3 groups with 6-8 participants per group in several cities over the course of a week or so.
There are three factors here that affect your research:
- Small numbers – a small sample size is often useful when you want to probe people’s attitudes in greater depth and try to get into their minds. Qualitative research can help flesh out themes that can be used for follow-up in quantitative methods such as surveys.
- Sample – the sample used in this type of research is not representative of the larger population. Will the findings be generalizable from what a few people expressed as their opinions in Los Angeles? Are you willing to gamble with the direction of your product and/or company on the basis of what a few people said during a focus group? Obviously there is great danger here.
- Generalizability – there is a problem generalizing small sample sizes because they are not representative of the underlying population. This is a basic concept in inferential statistics and probability.
Do not use focus groups or research with small samples to assess purchase intent. This type of finding is not generalizable from a small sample.
If your research questions pertain to user experience, have the participants use the product and then probe the experience. A common mistake is to provide a guided walkthrough of the product. As mentioned above, you will obtain far deeper insights from actual product usage vs. hypothetical experience. People are able to give you better input when they are not relying on their memory and they are also more cognitively involved in the experience so you will obtain better insights.
3. The group leader affects what is, or is not, said during the group, and the group leader is not the moderator!
Have you ever heard the term alter leader? I’ve met very few people who are aware of the concept of the alter leader and how it impacts observed behavior and comments in focus groups.
Sharon Livingstone, a well-known market researcher, has written extensively about this type of group member. The concept is that seating position in a traditional group, and the group dynamics surrounding that, affect what occurs in the group. This notion is rooted in group psychotherapy and I have observed these phenomena in my previous work as a clinical psychologist, before I started working in the UX field.
The alter leader can impact focus groups in the following ways:
- Impact on positive and productive focus group - In a focus group, or traditional market research, the natural leader or alter leader helps the moderator. He often physically sits in the chair opposite to the moderator. He is often the first to speak and may be enthusiastic about the topic, thus influencing what others say (or what they don't).
- Impact on a nonproductive or negatively focused group - The alter leader is vocal, but he is negative or resistant, and fights for control with the moderator. For example, he may object to the questions the moderator is trying to ask, or may answer questions that were not asked. In essence, this person's behavior is driven by his need to be argumentative and competitive, and he is in competition with the moderator to control the group.
The alter leader may be the first one to speak negatively about the topic at hand and question the tone, focus, or direction of what is being evaluated. He might say, “Well that might be good for other people, but it's too basic for me” And viola, suddenly you have a group that wants to be seen as more experienced, advanced, or knowledgeable than they actually are about the topic at hand. Or the alter leader might comment about the privacy concerns of the topic, and suddenly he has touched a hot button that will cause the group members to voice concerns as part of the group that they probably would not bring up if they were being interviewed individually.
I was involved in focus groups many years ago for a prominent online authentication system. At the time, it was the largest system of its kind, used by tens of millions of people on a daily basis. During a round of focus groups in Los Angeles, participants were asked for their input on a certain topic. One dominant group member raised the issue of security. Suddenly, security emerged as a concern for the group and the tone was that people were “very concerned” about security. Contrast this finding with reality that tens of millions of users were actually using that exact system on a daily basis that the focus group was discussing hypothetically.
Were the group’s concerns reasonable or legitimate? Certainly. But these concerns appeared larger than life in that 90-minute session (this relates to the vividness effect discussed below). Undoubtedly some of the same people in the group who expressed these security concerns during the focus group went home and actually used that authentication system the same day without even giving it a moment’s thought!
The best solution to completely eliminate the effect of group dynamics is to conduct the groups online. Online research eliminates the positive and negative impact of the alter leader entirely. Depending on the specific research platform that is used, participants can be in a group but have a one-to-one dialog with the moderator that is uncontaminated by group dynamics. For example, participants can be required to type their response before they see others’ responses, so they are not influenced by others as they would be in a real-life focus group. By having participants offer their own opinions before they hear or see other people's opinions, you ensure that they are:
- being more thoughtful in their responses by having a chance to think them through
- not being influenced by others, especially extroverted and dominant group members
- prevent group dynamics from biasing the results and conclusions of the group
Even the most skilled and experienced focus group moderator cannot achieve this outcome in all groups. It is simply impossible to control these variables in traditional, in-person focus groups.
4. Introverts lose their voice during focus groups.
If you've observed a focus group recently, try to reflect on members in the group you observed. Here are a few questions to stimulate your memory:
- How many people were in the group?
- Do a few of the group members stand out in your mind's eye?
- Do you remember what all of the people said, or just some of them?
- As you think about what you observed and what you concluded, were your conclusions skewed toward the verbalizations of the more extroverted and dominant group members?
- Can you remember any of the more introverted group members?
- Do you recall what the introverts said during the group?
If you're like most people I've tried this exercise with, you'll find it nearly impossible to recall much of what the introverts said! This phenomenon is another hidden dynamic that occurs in nearly every focus group.
Distinguishing introverts and extroverts
Researchers have found that introversion/extraversion is one of five fundamental personality variables that have been shown to be consistent across many studies, across many cultures and countries, and across an individual’s lifespan. In fact, if you’ve ever taken a personality test such as the Myers-Briggs or the NEO-PI, this is one of the core dimensions.
There’s a fairly easy way to determine whether someone is an introvert or an extrovert: just ask! But instead of asking someone if he’s an introvert or an extrovert, ask if he gets his energy from being around other people, or if he likes to be alone.
An extrovert will tell you he likes interacting with people and that it “charges his batteries,” while an introvert will say people can be overwhelming and draining at times and he likes to be alone to recharge his batteries. People who are more in the middle will offer responses that show more shades of gray and indicate that sometimes they like to socialize and sometimes they like to be alone.
In group settings, extroverts love the attention and energy of the group. They will be “charged up” by it and it will bring out that energy.
They will often be talking quickly and more loudly, and will take up a greater percentage of the “air time” in the group, which subtly or not-so-subtly affects the group agenda. If the focus group is 60 minutes, you can be assured that the extroverts will dominate most of the hour.
Why is this personality factor so important for in-person focus groups?
It matters because you’re not hearing the voice of the introverts, who are likely half of your customer base since the personality dimension tends to be normally distributed.
In an in-person focus group, you are unlikely to obtain insights into the perspectives of your introverted participants to the same degree as you do with the extroverts. They often have equally interesting and compelling things to say, but it takes them longer to articulate those thoughts. In reality, your customers are a mix of introverts and extroverts, but if you are doing traditional focus groups, the group interactions and findings and conclusions are biased by the effect the extroverts have on the group. Extroverts prevent the introverts from having a say, and that affects the group discussion and your research conclusions.
This personality dimension can become more polarized depending on who we are with
Here's a simple rule-of-thumb: our personalities become more polarized and extreme based on who is in our environment. Introverts tend to be quiet by nature. When they are surrounded by extroverts, they typically become even more introverted.
And since extroverts love the energy of others and being in groups, they tend to become more energized by the focus group. This has the effect of making the extroverts appear even more extroverted and loud. This, in turn, makes the introverts even more passive and introverted.
In summary, introverted group members tend to be heard less during in person focus groups while extroverts tend to be heard more. This personality variable impacts what occurs during the focus group in an uncontrollable way, and that means what you’re hearing is more biased toward what the introverted participants have to say.
And this personality dimension also ties in with the next item, the vividness effect.
5. The Vividness Effect
The vividness effect is a well-researched phenomenon in cognitive psychology research that explains how vivid events (those that are highly graphic or dramatic) affect an individual’s perception and recall of a situation, event, or memory.
Because of the way our brains process cognitive and emotional information, we are more likely to be impacted by, and remember, highly emotionally charged information. Vivid pictures, movies, or things we’ve observed tend to stick out in our memory more than the mundane ones do. And since vivid events are more easily remembered, they may receive a disproportionate share of our memory, focus, and attention, and this tends to make our recall and experience biased.
Specifically, this means that the vividness effect often impacts our own recall of situations and events, and this effects they way we assess current or future situations.
The vividness effect impacts focus group observations and conclusions
How does this apply to consumer research and focus groups? The vividness effect impacts both the focus group participants and the observers.
When research is observed in person, the vividness effect causes observers to automatically be drawn toward sensational, vivid, and extreme examples. If a particular person in the focus group is animated and dominates the discussion (common with dominant, extroverted group members), what he says is more likely to dominate the conversation and be remembered.
After the focus group, when having a team debrief about what was observed, the observers are more likely to remember the more extreme and vivid comments.
Imagine a focus group exploring people’s attitudes about an airline’s customer service.
One of the participants, Helen, was an extroverted woman who was the dominant group member. Helen talked at length about the problem she experienced with sitting on the tarmac for four hours during the flight delays during the summer of 2007. She told the whole group in a very colorful, dramatic manner about how the flight attendants would not even serve her food as she sat for the four hours! The plane sat on the tarmac in the Florida heat, and began to smell like a high school locker room. She was drinking coffee before the flight and she had to go to the bathroom so badly that she nearly peed her pants. And the flight attendants, according to Helen, were “Nazi-like” and spoke over the intercom that no one would be allowed to get up out of their seats.
Do you have a good picture of Helen and how she feels about the airline? What is your conclusion—did the airline do a good job with customer service?
Actually, the answer is that we do not know since Helen is a made up example to illustrate the vividness effect at work and the impact it can have both on the group and those observing the group. This example is so vivid that it will likely obscure the ability to hear what and recall what the others are saying. If the other seven group members all had positive experiences with the airline, you are still likely to remember Helen’s comments because they will stand out in your brain due to the vividness effect. And despite knowing this is a fictitious example, it will still stand out in your mind—that is how powerful the vividness effect is! Quite clearly, the vividness effect biases what occurs during the group and also what observers observe and recall after the group, and that impacts the validity of the research.