Recruiting participants is rightfully considered to be one of the biggest struggles in the world of UX research. From finding enough participants who qualify for your study to screening them and ensuring that they actually show up for the test – there’s a lot to handle.
At UXtweak, we recently conducted a community survey called Research Recruities, aiming to gather some insightful recruiting stories and statistics on the most common research recruiting mistakes. Apart from that, we’ve reached out to some seasoned UX professionals, asking them to provide tips and advice on how to combat those mistakes.
The survey is all wrapped up now, and we are ready to present you with the results. In this article, you’ll find out what recruiting mistakes UXers encounter most often, which ones they consider to be the most detrimental, and how you can prevent those mistakes from happening in your next study.
7 Biggest Recruiting Mistakes + Tips to Overcome Them
When it comes to research recruiting mistakes, we asked 2 main questions:
Question #1 –
“Which of the following mistakes have you observed or experienced in research recruiting? (select all you experienced)”
This question is designed to uncover frequent missteps in the research recruitment process from the perspective of the respondents. By presenting a series of potential mistakes, it aims to shed light on which aspects of recruitment are most error-prone, gauge the prevalence of these issues, and identify key areas for improvement based on their frequency.
Question #2 –
“From the list, which mistake do you consider the most critical or detrimental in research recruiting?”
This question aimed to pinpoint which mistake is perceived by participants as the most damaging or impactful in hindering the research recruitment process. The respondents were asked to highlight the most severe mistake from their previous selections, aiming to rank the problems in order of their potential to be detrimental to the research process.
The second question was followed by the open-ended question:
“How would you handle or overcome this mistake?”,
offering the respondents a chance to propose their own solutions and share ways in which they recommend combating the mistake they identified as the most critical.
Based on the Research Recruities survey results, the most commonly experienced research recruiting mistake is failing to properly screen the participants, which was the option chosen by 17.6 % of our respondents.
Following closely is the mistake of rushing the recruitment process (13.4 %) and not defining clear targeting criteria (10.9%).
The answers to the second question, however, painted a slightly different picture, clearly pointing out the 8 most detrimental UX research recruiting mistakes, according to our participants. Below, we’ll take a look at each of them and their nuances and include tips from our respondents and UX experts for handling those mistakes.
1. Not defining clear targeting criteria
This mistake was selected as the most challenging by most of the respondents – 37,46%. In essence, it refers to the failure to establish precise and specific criteria for selecting study participants.
In many cases, this means including individuals who may not fully represent the actual characteristics, behaviors, or needs of the future use group. This mistake often leads to incorrect assumptions about user behavior and waste of resources on irrelevant research.
This problem often arises when the project scope is broad, or when there’s a rush to start the research without adequate planning. Additionally, it may stem from an insufficient knowledge of the intended user group or poorly defined research study goals.
How to combat this mistake?
One of the most common responses was defining the goals & hypothesis of the research clearly, talking to stakeholders, and, finally, “spending a lot of time creating a good screening questionnaire”.
Here is an example of how NOT to approach the process of recruiting a particular user group and what to do instead, from Parker Sorensen, Associate Director, Conversion Optimization.
“I believe this comes down to good test planning. I have seen (and in my early days or research did this myself) where researchers recruit a general audience and try to have participants imagine a scenario (ie. “imagine you are looking to purchase a home in the next 1-3 months”). In my experience, it is difficult to impossible for participants to put themselves fully in the scenario – because they are not actually there. A better way is to recruit the exact audience (ie. screen for “which of the following are you planning to do in the next 1-3 months”, give 5-10 options, and make “purchase a home” required.”
For those at the early stages of their research, here’s a great tip for creating specific targeting criteria from Blanche Letakis, Associate Product Manager & UX Researcher:
“I start research studies with a slightly broader target and as sessions go on, I learn more about our target users, and tweak the criteria, including screener surveys.”
2. Not Screening Participants Properly
Screening is undeniably one of the most problematic aspects of UX research recruiting. It is also, according to our data, the most prevalent recruiting mistake.
Unfortunately, ineffective screening is a common problem because it is linked to a lack of a thorough screening procedure and poor screening criteria compared to the study’s goals.
Poor screening harms your research study in many ways, including biasing study results, wasting resources, and allowing unqualified or “fake” participants to enter your study.
How to combat this mistake?
Here’s what our respondents suggest for fine-tunning your screening strategy:
“If possible, test before launching to a wider audience, peer-review and opt for manual reviewing of the participants before getting the confidence that the screening process works as it should.” – Andreea Dalia Lazar, PhD, Senior Researcher
“Using a research panel that allows UXers to filter for specific criteria, is the first step I try to take. Additionally, we use a variety of screener tactics to help ensure the people signing up or taking our unmoderated tests are the right people.” – Kelly Meredith, Senior Design Researcher
“Ask questions regarding the product but don’t be too straightforward to minimise the amount of “professional respondents”” – Alyona Poluiko, UX Researcher
Nikki Anderson, Founder @ User Research Academy, also recommends crafting as detailed a screening questionnaire as possible:
“Think through every piece of information you need from the participant and craft that into a screener question – it might mean the screener is a bit longer (try not to go more than seven questions) but it will enable you to get the right people for your study!”
3. Over-reliance on Convenience Sampling
Convenience sampling is the selection of participants based on their ease of access or availability and not an accurate sampling of the target population.
The high dependency on this type of recruiting takes place when decision-makers select only convenient ways of getting access to participants. Such an approach results in no need to recruit people to take part in the study and instead use the services of colleagues, friends, and family members.
However, even though this method seems acceptable in some cases (for example, when you are recruiting from the general population), most of the time it still results in biased insights.
How to combat this mistake?
To counter this issue our respondents recommend:
- Defining the recruitment criteria and following it strictly during the recruiting process,
- Including the criteria in your research plan, ensuring everyone has access to it
- Crafting a good screener to eliminate irrelevant people
- Spending more time on the recruiting process, ensuring you have the time to recruit a representative sample
- Growing a sample database of users over time
- Using user research tools with participant recruitment capabilities
Keeping a database of your past testers allows you to recruit them again and again for another study, and with that, you don’t have to invest much time and effort in recruitment. It’s a great alternative to convenience sampling that can actually provide useful insights.
With that being said, be sure to provide fair compensation, because it can directly affect the participants’ willingness to participate in your next study.
“Compensate in time! This is something my company is currently struggling with, but what I’ve found (from my past company) is that when we pay the participants on time, they are more willing to participate again and even refer their network to us for future studies.” – Karen Mirmanas, UX, Inclusive Design and A11y ResearchOps Coordinator.
Here’s another great tip for growing your database of respondents from Caitlin Sullivan, Product Discovery and User Research Expert:
“Ask every participant you find who is a really good fit if they know someone who might be up for joining your research. It can make recruiting the right kind of people much faster!”
4. Ignoring diversity
The term diversity in this context not only means demographic data such as the users’ age, gender, and origin but also a variety of user habits, backgrounds, and access requirements.
When ignoring diversity you’re risking that your product will be only catered to a narrow user base, neglecting the needs of others who didn’t fit the initial participant profile.
Of course, it’s crucial to remain focused on the specific user group your product or service is intended for. However, it’s also critical to balance the importance of not overlooking diversity, while adhering to your targeting criteria.
How to combat this mistake?
To ensure you’re looking at a variety of customer types, Kelly Meredith, Senior Design Researcher, recommends:
“When looking to talk to a variety of customer types, rather than sending one invite link and hoping for the best, we use a few methods to ensure we are talking to the right mix of customers. We use Calendly routing forms, which have been a huge help! We create a different event type per type of participant and use the routing form (screener) to filter prospective participants to the right event.
For example, we can chat with 5 large customers, 5 small customers, and 5 non-customers in the most efficient and effective way. Before this feature or for those without Calendly Pro, we use Alchemer to create a screener and use the logic to direct potential participants to different sign up links.”
5. Not Validating Participant Information
If you don’t carefully verify who’s joining the study, you risk having scammers complete the study and skew the results. These individuals, often looking to game the system for a monetary reward, can slip through if we’re not filtered out during the screening stage of the study.
Their presence isn’t just a numbers issue; it distorts the data, leading to insights that don’t truly reflect our target audience’s perspectives and behaviors. Filtering them out is crucial to ensure the integrity of the research data.
How to combat this mistake?
To safeguard your research from being compromised, it’s essential to verify participant information upfront. This step ensures that only qualified participants contribute to your study, maintaining the reliability of your findings.
“Do not allow participants to schedule study sessions directly once they’ve qualified through the screener. Teams of scammers will sometimes re-take screeners until they’ve “passed,” then use those answers to qualify under other additional identities or share the answers with other scammers.
Inviting participants to schedule their sessions takes longer, yet it offers additional benefits. It enables a secondary screening process, provides a chance to confirm the accuracy of the contact information they’ve provided, and allows you to cross-reference responses to identify potential duplicates among the screeners.” – Michele Ronsen, Author, Researcher, Educator, Founder, UX Coach.
Here’s how another respondent, a UX researcher with 5-10 years of experience recommends approaching that problem:
“Write better screener questions (no yes/nos, etc.), and when things are tricky – call to verify.” – UX Researcher, 5-10 years of experience
6. Not Checking for Previous Participation
Not checking whether participants have been part of similar studies before can also impact the integrity of your research.
People who’ve done this kind of testing/interview before might not respond like new users would. They’ve got their own expectations and are too used to the process, which can color their feedback.
So, without the previous participation check, you’re missing out on fresh, unbiased insights, which are gold for getting a real understanding of what your target audience thinks or feels.
How to combat this mistake?
To dodge this issue, you can set up a system to track and screen out the participants who’ve been in similar studies before. This could mean keeping a record of past participants or using specific questions during the screening process to spot anyone who’s already taken part. This way, you ensure the feedback you’re getting is fresh and representative of new users.
If you recruit from a user panel of a UX research tool such as UXtweak, they conduct this check themselves, so you don’t have to worry about it.
7. Rushing the recruitment process
When project deadlines are looming and budgets are tight, there’s a temptation to speed up the participant recruitment process to stay on track.
This pressure often comes from stakeholders as the main decision-makers. However, rushing the recruiting and just grabbing whoever’s available without thorough screening can backfire. This approach often results in a participant group that’s not diverse enough or truly reflective of the intended user base.
The irony is, while rushing might seem like a shortcut, it can actually lead to more work later. If the initial recruitment isn’t done right, you might end up having to redo parts of the study or need to conduct more research to compensate for the skewed results, eating up more time and resources than if you’d taken the time to recruit properly from the start.
How to combat this mistake?
Jasmine (Lo) Winata, Research Operations Specialist, emphasizes the importance of informing stakeholders and communicating the need for a longer and more structured recruiting process:
“Pushback on stakeholders who are rushing recruitment and lay down the realities and expectations regarding recruitment, especially with a new pool (looking for brand new users vs. using a pool we already have access to). Let them know that recruitment will still happen, but the desired number and quality of users may not always meet expectations.” – Jasmine (Lo) Winata, Research Operations Specialist
What’s next?
Even though the advice above already seems like a load of insights, we’ve got a lot more of professional tips during the time of the Research Recruities project. Recruiting is a wide and complex topic, and there is so much more to discuss and learn!
Apart from the insights on the biggest research recruiting mistakes, we also collected hilarious and strange recruiting stories from researchers around the world, gathered general recruiting tips and strategies from UX pros like Stephanie Walter, Debbie Levitt, Nikki Anderson, Caitlin Sullivan, Michele Ronsen, Larry Marine and others!
➡️ Read the stories: Hilarious & Strange: UXR Participant Recruiting Stories that You Need to Hear
✅Discover a collection of recruiting tips and strategies from UX experts: UX Research Recruiting Tips from 19 UX Experts [+ Checklist]
