We stand with Ukraine and our team members from Ukraine.

The Community Of Over 578,000

Home ›› Analytics and Tracking ›› Six Myths about Data-Driven Design

Six Myths about Data-Driven Design

by Pamela Pavliscak
Share this post on


Beyond algorithms, automation, A/B testing, and analytics, the goal of data-driven design is to develop a better understanding of everyday experience.

There is a lot of buzz about data-driven design, but very little agreement about what data-driven design really means. Even deciding how to define data is difficult for teams with spotty access to data within their organizations, uneven understanding, and little shared language. For any site or app, it’s standard practice to have analytics, A/B tests, surveys, intercepts, benchmarks, scores of usability tests, ethnographic studies, and interviews. So what counts as data? And more importantly, what will inform design in a meaningful way?

In trying to understand what data-driven design is, we can start with what data-driven design is not. Then maybe we can work toward how data might actually help make user experience better.

Myth 1: Data Means Numbers

A lot of the data about a site or app flows in from analytics, and analytics are basically tallies of who has come to your website, how they got there, how long they stayed, and what they clicked or tapped. Other data may come from how many clicked A and how many clicked B. More tallies. Then there are intercepts and surveys where scored responses are counted. Still more tallies.

Numbers represent the actions of real people with complicated lives. But rolling the behavior of millions of people into a single number is not always useful, or reliable. Even the most organized sets of numbers don’t answer a lot of the questions we still have about user experience, like why people take action or why they don’t, or how they felt about it, or what expectations they bring to the experience. Interviews, ethnographic studies, and usability tests fill in the gaps left by numeric data. But because qualitative insights are not numeric, they are often not considered data.

In other fields, like the social sciences and medicine, there is no question that qualitative is still data. Small numbers, or thick data, still count whether they remain as narrative or are quantified. This makes sense for data-informed design too.

The true defining characteristic of data that can be used to inform design, are the traces left behind by people, no matter the source.

Myth 2: Data Is the Objective Truth

Quantitative data typically tallies completed actions, and usually those tallies are compiled by software rather than humans. This makes quantitative data seem like hard fact.

Even if data is big, it does not mean it is objective. Bias is inherent in any dataset. Datasets are created by humans, who interpret them and assign meaning, even if a machine runs the numbers.

Signal bias—or the bias of omission, inclusion, and emphasis—is one common bias, especially for big data. For example, social media data shows only one segment of the population—those who are on Twitter or Facebook using a certain hashtag. Likewise, the subset of people who agree to be observed or participate in a study represents only a very small subset of the general population.

Another bias is something like the Heisenberg uncertainty principle, where results are changed just by observing. No matter how neutral the environment and the demeanor of the moderator in a lab usability test, behavior, once observed, is different. Very few studies, even contextual or ethnographic studies, leave it totally up to the user. This is the case with quantitative studies too, like surveys or intercepts, where we frame the experience in a certain way just by asking questions about it.

Designing with data has to go beyond algorithms, automation, A/B testing, and analytics

Further, when we compare online studies with no human contact, and in-person studies where the user is interacting with a moderator or with another user, we see a substantial positive effect on how people perceive a site or app when there is another human being involved. This extends to what they say, how they rate, and what they do or try. Call it the nice factor.

Big or small, no data is perfect. There are limitations and bias present in every type of data. Good data describes its biases, and always provides context.

Myth 3: Bigger Is Always Better

The hype around Big Data (note the CAPS) suggests it has the power to reveal all the secrets of humanity and accurately predict the future. And OK, sometimes bigger is better when it comes to data. If you are measuring something subjective, like emotional response based on self-reported ratings, more responses give you a greater confidence level in the results.

When we think bigger, we tend to think about tallies: the volume and velocity part of the big data equation. But big data is also about variety, and that means diverse sources. We can’t expect analytics to tell us everything we need to know about user behavior any more than we can expect a usability test to address every possible user behavior We have to get our data working together in a way that isn’t all about back-end integration. It’s more about creating meaningful categories—also known as metrics—to evaluate, understand, and keep track.

Data from multiple sources creates a more nuanced picture and, in the end, an actionable outcome. Broader, not bigger, is better.

Myth 4: Data Is for Managers, Not Designers

Data is often used to pass judgment on a site or app (e.g. “The data says that conversions are down after the latest redesign”). Of course, this strikes experience design practitioners as reductionist, opportunistic, or just plain wrong. Certainly, it’s tempting to look for data that proves the wisdom of a decision, whether to solve internal battles, counter a gut-feel approach, or just prove return on investment. Proving a point is only one part of the data story though.

If you are using data to inform design, you have three ways to look at things: proving, improving, and discovering. Using data to improve often means the data-informed equivalent of iteration—tracking across time and across versions or even against competitors. Using data to discover is about looking at data in relation to other data (big or thick) to explore patterns and trends.

Part of the issue here is that data use is siloed—the business leadership is working with one type of data and the UX team another. Each team has a different frame of reference, and may not be aware of, or simply discount, the data of the other.

Data is not just about proving who is right or wrong, it is about making improvements and discovering new possibilities. It’s another way to tell the story of the real people using technology.

Myth 5: Data Kills Innovation

Data is seen as the antithesis of innovation, in all sorts of ways. Well, actually in three ways:

  1. Most data, whether analytics, survey data, or customer service data, is backward-looking. Although we can discover patterns and trends, it is not easy to make predictions based off of those discoveries.
  2. Data is tactical rather than strategic. Think of Google’s 41 shades of blue testing. Because data-informed design is associated with A/B testing, it seems like a good way to tweak a design element but it’s not so great for creating an amazing experience.
  3. Data, especially analytics, seems to skim the surface. Seeing what people clicked, how much they scrolled, or where they lingered can work well to form a picture about how to market a product. It does not work so well for informing design, because it lacks information about motivations, expectations, perceptions, or emotions.

There is some truth to all of these observations, of course. The core problem is not in the data itself, but in how it’s being used.

For any data to inform design it has to be calibrated to embrace complexity. The user experience is not a funnel.

Myth 6: There Is a Right Way to Use Data to Inform Design

So far, there isn’t one canonical way that works for every team in every organization. There are a few guidelines to start with though.

  • Use data from a variety of sources to inform your design: analytics, A/B tests, social media sentiment, customer service logs, sales data, surveys, interviews, usability tests, contextual research, and other studies.
  • Include numbers and context. Whether you call them quantitative and qualitative, studies and non-studies, or big data and thick data, you need the numbers and the context to tell the real story.
  • Make sure data is sensitive to the complexity of the human experience. Use averages sparingly, infer with caution, corroborate liberally.
  • Use data to track changes over time, explore new patterns, and dig deeper on problems, rather than just to prove who’s right or wrong.
  • Decide on meaningful categories that let you make sense of the data and tell a story about the experience.
  • Develop a way to share and discuss data in your organization, and start by defining the basics together.

Designing with data has to go beyond algorithms, automation, A/B testing, and analytics. Rather, the goal is to use all the data to develop a better understanding of everyday experience.


Illustration of user’s brain courtesy Shutterstock.

post authorPamela Pavliscak

Pamela Pavliscak, This user does not have bio yet.


Related Articles

The overview of current challenges and opportunities design faces and how BBC team can help designers out to enhance digital experience and understanding of the world.

The Evolution of Experience Design
  • Dan Ramadan, Creative Director for UX Architecture & Content Design at BBC, tells about the current challenges and opportunities design faces by describing 3 stages of ‘the web’:
    • Challenges of the past (document retrieval)
    • Challenges of the present (control and contribution)
    • Challenges of the future (pervasive and ubiquitous)
  • Technology is as capable of solving problems as it is of creating them.
  • The team at BBC can explore how digital experience can enhance our understanding of the world, develop empathy for others, instill pride and commitment to the importance of the individual and the inherent value of shared values and cooperative society.
Share:The Evolution of Experience Design
5 min read

As organizations grow in their conversational maturity, there’s an increasing demand for conversation designers. Explore 7 skills to learn for conversation designers in 2022.


7 new skills to learn for conversation designers in 2022
  • Conversational design requires far more than having got all your convo design courses nailed, completed all the challenges on VUI-challenge and finished re-reading Pearl, Evanhoe & Deibel and Cohen, Giangola & Balogh for the umptieth time.
  • There is a number of new skills that can up your career as a conversation designer:
    • NLU
    • Entities
    • Entities on steroids: ontologies and graphs
    • Building conversational teams
    • Open source movement
    • Mastering conversational AI platforms
  • It’s time for conversation designers to develop t-shaped profile: specialize in one or two particular conversation design skills and systems , and mastering skills that allow us to connect with people from neighbouring disciplines.
Share:7 new skills to learn for conversation designers in 2022
7 min read
7 new skills to learn for conversation designers in 2022

There are different ways participatory design can influence young people with cognitive disability. Read reflections on co-design methodology and ways it can help.

Reflections: Co-Designing with Young People with Cognitive Disability
  • In this article, Jax Wechsler, Principal Designer at Sticky Design Studio, shares:
    • Facts about Young People with a cognitive disability that may be useful when working with this group
    • Discussion and reflections about her methodological choices
  • Things to know about Young People with cognitive disabilities:
    • Ability levels can be very nuanced, every Young Person is different
    • Life Tasting not Life Wasting!
    • The level of advocacy of parents impacts experiences and opportunities for Young People with Cognitive Disabilities
    • Social inclusion and relationships are keys to wellbeing
  • Reflections on Co-Design Methodology:
    • Recruitment is hard!
    • Using referrals and ‘snowball sampling’
    • It’s important to build rapport
    • Understanding ability and research design
    • Cultural probes/diary studies are gold
    • Parents mediating participation
    • Flexibility is key
    • Value in participation
Share:Reflections: Co-Designing with Young People with Cognitive Disability
10 min read
Reflections : Co-Designing with Young People with Cognitive Disability

This website uses cookies to ensure you get the best experience on our website. Check our privacy policy and