Case Study—This case study provides an inside look at what occurs when methods from the data science and ethnographic fields are mixed to solve perennial customer service problems within the call center and cruise industries. The paper details this particular blend of ethnographic practitioners with a data scientist resulted in changes to design approaches, debunking myths about qualitative and quantitative research methods being at odds and altering team member perspectives about the value of both. The project also led to the creation of innovative blended design research and data science methods to discover and leverage the right customer data to the benefit of both the customer and the call center agents who serve them. This paper offers insight into the untold value design teams can unlock when data scientists and ethnographers work together to solve a problem. The result was a design solution that gives a top-performing company an edge to grow even better by leveraging the millions of data records housed in its warehouse to the benefit of its customers.
BACKGROUND AND CONTEXT
Anyone who has ever called a call center more than once knows the process can be painful. Customers call with a quick question only to be forced to provide a slew of non-relevant information. Most often they’re forced to give information to automated systems. Once their question goes unanswered; only then do they get to talk to a person. But by then they’re forced to repeat the irrelevant information to a call center agent before they can actually ask a question.
Even getting a human agent doesn’t guarantee the question gets answered if the information needed isn’t at the agent’s fingertips. Good luck if that call gets interrupted. The customer calls back and the frustrating process begins again. The process can be equally unpleasant for call center employees. Call center workers routinely complain of customer aggression and one study shows 1 out of 5 calls to contact employees are from angry customers, an average of 10 calls a day. The aggression from customers has risen over the years, so much so that the turnover rate within call centers has exploded from 19% in 2008 to 24% and rising today. (Dixon, Ponomareff & Turner 2017)
For every call center interaction between an agent and a customer, there’s an equal and corresponding data creation and storage action. Each phone call a customer makes. Every email an agent sends to a customer. Each time a customer goes to a website and enters the search words of some product they want or exotic local they want to visit data collected and often stored. All of these actions create data points. These data points, or digital bits as they’re known by the industry, make up what’s called the digital universe – all the bits of data that are created, replicated, and consumed by people and businesses. (Gert & Reinsel 2013) But even though all this data available, call centers can’t seem to recognize who is calling, why they’re calling or how to more quickly solve their problem. This not only frustrates customers but annoys agents as well. It’s like there’s data everywhere but not a drop that’s useful. It’s a common problem across industries. A lot of data is being created, collected and stored but that data is incredibly difficult to track back to individuals. There’s many reasons for this, but one of the central reasons is that realistically, 91 percent of companies globally report they have inaccurate data, according to a survey of more than 1,200 global companies on data quality commissioned by Experian. (SourceMedia Research 2018) The most common ingredients that lead to data inaccuracies include incomplete or missing data, outdated data and wrong data.
In addition to the data being notoriously inaccurate, data on individuals are collected through multiple channels that are rarely connected. A company may have data on one individual collected from at least three different channels, including email, a mobile application, a phone call or a text. A customer’s information is split piece by piece into three, four, five, even 10 different databases, never giving companies a true “360-degree” view of the customer.
Cruise Company’s Customer Service Woes
This challenge of having lots of information but not really knowing how to use it, is one many companies face, including, recently, a major international cruise line. Like many companies, the cruise line had collected millions of data records through multiple channels on customers and potential customers. Yet despite this, customers were still experiencing immense frustrations when booking a cruise. Though wildly successful, the cruise line’s parent company’s new CEO wasn’t entirely happy. Rookie cruisers set on having fun in exotic locales, were finding booking cruises extremely difficult and not fun at all. The company’s CEO decided to conduct some “mystery shopping,” calls to his company’s call center. He didn’t like what he experienced. So, the CEO of a large cruise company issued a challenge to the largest of its operating cruise lines – improve the customer service. The design challenge was to improve the cruise line’s customer service so that the booking process mirrored the fun cruisers had onboard the ships.
The company believed the solution was to be found in their call center. There was a list of metrics they wanted to improve including average call time, handle time, how many repeat calls etc. Fix that, they said and the problem goes away. There was also a cursory interest in using customer data to help improve the customer experience. The company had several work streams focused on familiar territory of trying to piece together a “360-degree view,” of their customers. In an effort to improve its customer service efforts the cruise company hired the global design firm IDEO.
IDEO teams working on the project would soon discover that to truly solve the problem, to make sure customers felt heard, understood and confident when buying a cruise, the solution lie not just with data, but with people as well. During the first of several research phases, the team discovered that possessing data alone, it seems, couldn’t solve the very human desire to be heard. A challenge most companies face is how to surface the right data, to the right person at the right time. That’s why companies crave the consult of a data scientist. They seek someone to take largely unorganized and unruly data sets, wrangle them and come up with some salient insights that will give them a business edge. But the team discovered rather quickly that data wrangling wasn’t the answer; at least not at first.
Some Level-Setting Definitions
Before explaining the methods used during this particular design project it’s helpful to do some level-setting definitions. For this paper, let’s define data science and data scientist as it is practiced at IDEO. Data scientists at IDEO are adept at shaping data as a resource for human-centered design. They sketch in pencil and in code, shape and reshape data, know what data can and can’t tell us, help design feedback mechanisms, and prototype machine learning algorithms. They do all this to improve human experiences through the design of intelligent products, services, and systems. Data is the paint they use in the art of creating intelligent products.
The most easily understandable example of how data scientists and designers come together to create products is The Nest. This intelligent thermostat system developed by famed Apple designer Tony Fadell, and now owned by Google, is a smart device. The Nest processes data from a number of places including the Internet, the body heat of people inside a room, and adjusts a home’s temperature accordingly. To create such smart devices, a company needs people who understand and can develop devices that use data to sense, listen (adjust) and then act. These people are data scientists. Generally, data scientists are seen as people who can wrangle large data sets. But data scientists can also be designers, people who use data as their art to create a different, more improved world. And yet, like most designers, data scientists often find it can be difficult to know where to start, because without context, millions of data bytes sit dormant unable to be rendered useful. That’s where ethnographic research can illuminate pathways forward.
For this paper let’s define ethnographic research, in the context of design research. In this context, ethnographic research is the study of how people live their lives in order to better understand their behavior, motivations, needs and aspirational wants to inspire new design. This approach is comprised of various methods including interviews, observations, role-playing games and journey mapping. This paper will show that using ethnographic research at the very beginning of the project, transformed the data scientist’s initial assumptions about how to use data to solve the problem.
METHODOLOGY
To better prepare the team for upcoming field research, the design researcher on the project did a literature review about the cruising industry and cruiser. Using that information, she created an empathy exercise to help the team understand the reality of people who booked cruises over the phone.
Only two out of the team’s six members had ever been on a cruise. And no team member had gone through the entire cruise booking process. Among the team, and in the broader media, the typical persona of someone who goes on cruises is a silver-haired, often retired, affluent couple looking to relax. A quick literature review of the cruise industry painted a very different picture what team members had envisioned. The average cruiser is a 49-year-old employed, married, college graduate with a six-figure income. (Cruise Lines International Association 2017) In addition, “Generation Xers,” and “Millennials,” were fast outpacing Baby Boomers as cruisers. In one study, Millennials were twice as likely to have taken a cruise than their Baby Boomer parents. (Sheivachman 2018)
Repeatedly, in articles about cruising the idea of seeing multiple locations without ever having to unpack a bag, made cruising appealing to all age groups. Exploring the data collected by the client really couldn’t get to the central question, why was booking a cruise so darn difficult in the first place? Once that question was answered, the team felt confident they could design a satisfactory solution.
IDEO conducted two large-scale projects, Project Traveler* and Project Board* (not the projects real names), in multiple phases and with multiple design teams, to diagnose, prototype and design solutions to create a better cruise booking experience. There was 27 weeks of work though the projects spanned more than a year. Only two designers remained constant throughout all phases – the data scientist and the design researcher. The design teams used iterative research and design approaches that included a variety of methods. The project essentially had two user groups: customers who were booking cruises and call center agents who were helping them to book the cruise. The complexity and multi-variant challenge to the problem required an extensive array of design and data science research methods to illuminate a path toward design solutions.
First Phase: Diagnosing the Problem
In the first phase, which lasted five months, the team conducted in-depth interviews with first-time cruisers, cruise-curious people, repeat cruisers in their homes as well as talked to cruisers onboard a cruise to understand their pain points and challenges as well as aspirations when it came to booking a cruise. These interviews were conducted in three rounds before, during and after concept design. During the interviews, team members used card sorting, and role-playing games to help gaining a better understanding of customers’ beliefs, motivations and aspirational needs before, during and after the booking process. The number of customers and experts interviewed during this phase totaled 20.
For its second user group, call center agents, the team not only conducted in-depth interviews but also did on-site observations and multiple co-design sessions with agents. For the on-site observations, teams split off into twos, and sat with agents at their desks while they took phone calls. Using headphones specially made for listening to agents while on calls, team members were able to hear both customers calling in and agents’ responses. The team made multiple trips to the client’s call center to listen in on phone calls. Interviews with agents had to be kept short, because every minute agents were taken from the floor to talk to the team was a minute they lost the opportunity to make money or serve a customer. Because the team didn’t want to inconvenience agents or disrupt their job, the team leaned heavily on observation to learn what was needed.
In addition, the team conducted several co-design sessions one of which included creating a gallery walk of proposed design concepts that was installed on-site at the call center for a weeklong feedback session. Agents were able to view design concepts at their leisure and vote on the ones they liked the most. The team interviewed 40 agents in two rounds of on-site observations and interviews. To help client executives and agents better understand insights gathered through field research with consumers the team also conducted several role-playing and empathy exercises.
Second Phase: Testing Solution Prototypes
Once design direction became more concrete (read about how and why in the Findings section below) the team also conducted two live prototyping sessions. In the first phase, the team and the client designed a two-week prototype which altered the organizational structure of the call center to better meet customers’ needs. In the second phase the team conducted a large-scale usability test to help solidify the desire for a propose software solution. The team tested a proposed new software application live with 11 agents using anonymized customer data. To create ensure that our usability test with agents were as real as possible the team used real customer data that had all personal identifying details expunged but still had real information that agents would recognized. The team created various behavior typologies that we reconstructed from real customer calls and hired four actors from the Chicago-based Second City Comedy Troupe to embodies those behavioral typologies and into the call center to book a cruise.
Agents knew they were testing a new software application but they had no idea the calls weren’t real until after the “customers,” hung up. The actors didn’t receive scripts, rather the team used various behavioral data points to create behavioral frameworks for actors to embody when they made the phone call. Four behavioral frameworks were constructed from anonymized customer data the team collected and analyzed from the client. The behavioral frameworks used during the agent user testing phase stemmed from a new mixed-method approach developed by the data scientist and the ethnographers on the team.
Mix Method Creations: Data/Human Journey Mapping
It is a customary method for data scientists to map where data is housed and how it flows within throughout systems. (Loukides 2018) This is done for a variety of reasons but most acutely so data scientists can pinpoint how data is created and transformed as it is moves from one system to another. Mapping data flows helps to pinpoint flaws, biases and errors in data. It also can yield opportunities for creating new models that help to take what is known as unstructured data, data that is notoriously difficult to analyze and make sense of, and turn that data into something useful for an intelligent model. Plus, it’s just basic to know where data comes from before a data scientists starts massaging and working on it.
For example, a data scientist may map a system and see that a company may not keep track of customer complaints using software but does tape all its phone calls. Locating the audio recordings within the company’s systems is paramount for a data scientist who wants to detect patterns and anomalies in customer speech from those phone calls. While the data scientist may request all the audio recordings, who exactly stores and archives them may be a missing point of information. Often who enters and extracts data is left out of the data mapping process.
During this project, the team decided not only to map the data, but map who created, transferred, transformed and extracted the data. In addition, the team decided to map at which point in the data flow, data was created, transferred or transformed and for what purpose and, of course, by whom. This method was dubbed “A Data/Human Journey Map.” This detailed mapping not only told the team what data was being stored, but where it came from and how it was used by agents. It also allowed the design team to pinpoint the exact data that was most important to both customers and agents, the systems where that data was housed and became a key part of the resulting design of a new software application. Data mapping becomes an essential ethical exercise when you pair this flow mapping with people. A data/human journey map is simply a journey map that shows the input, transference and transforming of data throughout a company’s information technology systems and who creates, touches, transfers or transforms that data as it flows through the system. Creating a data/human data flow journey map from the user’s perspective is best when ethnographic methods such as observations and user interviews are combined with data exploration. While journey maps have always included people and systems, this map included people, systems and data, actual information points created, stored and retrieved by customers and agents as part of the journey.
For example, when a customer called and gave their name and where they were from, it was mapped to the database or system that information was transferred to or retrieved from by an agent. These systems roundtable interviews were also key as they helped the team to determine what data was stored where within the company. Those roundtable discussions gave the team insight into the current data and systems the client used. It also allowed the team to find ways to leverage the client’s existing stored data and systems in a new and innovative way without significant expense or technology investment. This would prove key in getting buy in from the client on design recommendations.
“Having a data scientist on the team, as a form of research with the understanding and the ability to leverage what data exists was huge,” the project leader said. “As designers, we always think to start from scratch. But the [data scientists] pushed us to think about what data we could lever to make the experience better. She pushed us to think what are the systems that our client is using, how do they operate. Normally we wouldn’t consider that until later on.”
In all, the team spent the better part of 18 months researching, concepting, prototyping and user testing more than 21 different design concepts. Teams spent interviewed or observed more than 70 stakeholders, customers and agents in homes, offices and onboard a cruise.
GENERAL FINDINGS
The team’s in-depth interviews quickly lead them to discover a universal truth–no one cruises alone. The complexity of the booking processes compounds this truth. Even if a person is going on the cruise solo, he or she usually has to consult someone else to make the decision. One person may be on the phone, but to make the final decision it could be more than 10 people involved. The team imagined the cruise booking process was simpler than it actually was. After conducting interviews with both first-time cruisers and those who had not been on a cruise yet but were actively looking, it became apparent that cruising ain’t easy. Take Michelle (not her real name). Michelle is a 37-year-old mother of two who lives off the Florida coast. The team interviewed her at her home just as she was in the midst of booking her next cruise. Even though she has been cruising since she was a teen-ager, Michelle says she’s still a novice when it comes to booking a cruise.
“While I’m booking [a cruise], I’m stressed,” she said softly as the team sat at her dining room table in rattan chairs. The stress of the process weighing on her face as she dictated all the decisions she had to make. “I need to figure out the date, the room type, do they have it, is it the right port that I want to go to, so it’s kind of stressful. “I’m not an expert,” she continued. “I’ve done a lot of cruising, but not so much recently. I know about the tipping procedures and what you can bring on board and what you can’t. I mean, I think I do. Who knows? It might have changed since then.”
Even though Michelle had been through the booking process repeatedly, she had been on nine cruises, she still felt like a novice. Her insight that the booking process is ever changing making it difficult for anyone rookie or not to navigate was particularly inspiring to the team. The team sat in Michelle’s living room, outfitted with rattan chairs and a beautiful glass table, covered with a white sheet to protect its shine, as she made a phone call to a cruise line. She said she wanted to call two different cruise lines to get pricing and availability for her next cruise. On the day the team went to interview Michelle, she called the client’s cruise line. (Incidentally, researchers knew about the call before and asked Michelle’s permission to listen to it. It was totally coincidental that Michelle called the client’s cruise line.) The call was painful. Cruise line stakeholders visibly cringed as they listened to the Michelle struggle to get her questions answered. She had to repeat her destination desires multiple times and never really did get the answers she was seeking. Shortly after that call, she called another cruise line, the client’s competition. The call went better, Michelle thought, but she was still unsatisfied.
“I feel okay. I don’t feel fantastic with either of them. I guess I feel better from this call. The other guy seemed just kind of like, you know, wanting to talk about his own trip. … But both of them, I don’t have all my questions answered, so…” and her voice trailed off in soft disappointment.
This article is an excerpt of a paper written by Ovetta Sampson’s for EPIC.
EPIC is a members-and-sponsors-driven nonprofit focused on advancing ethnographic principles in business, bringing conceptual leadership and practical skills to the problems and opportunities faced by organizations, markets, and society.
To learn more about EPIC and to read the full version of this paper, visit https://.epicpeople.org/
*The information included is in this paper is accurate but does not represent the official case study about the project from IDEO.
REFERENCES
Cruise Lines International Association
2017 “Cruise Travel Report.” Annual Survey. 2017.
Matthew Dixon, Lara Ponomareff, Scott Turner, and Rick DeLisi
2018 “Kickass Customer Service.” 1 Jan-Feb 2017. HBR.org. 25 September 2018. <https://hbr.org/2017/01/kick-ass-customer-service>.
Gert, John and David Reinsel.
2012 The Digital Universe IN 2020: Big Data, Bigger Digital Shadows and the Biggest Growth in the Far East – United States. 2012 IDC Country Brief . Study. IDC. Framingham: EMC, 2012.
Loukides, Mike.
2018 The Ethics of Data Flow. 11 September 2018. 18 September 2018. <https://.oreilly.com/ideas/the-ethics-of-data-flow>.
Neff, Jack.
2018 Ad Age. 11 July 2018. 2018 September 2018. https://adage.com/article/news/ai-models-real-consumers-reveal-research-answers/314137/ 2018.
Sheivachman, Andrew.
2018 Millennials Now Enjoy Cruising More than Boomers. 8 May 2017. 8 July 2018. <https://skift.com/2017/05/08/millennials-now-enjoy-cruising-more-than-boomers/>.
SourceMedia Research.
2018 The State of Data Quality in Enterprise 2018. New York, 1 January 2018. Report.
