Flag

We stand with Ukraine and our team members from Ukraine. Here are ways you can help

Home ›› Business Value and ROI ›› 6 Key Questions to Guide International UX Research ›› Moneyball 2: How to Sweep the UX World Series

Moneyball 2: How to Sweep the UX World Series

by Steve Tengler
7 min read
Share this post on
Tweet
Share
Post
Share
Email
Print

Save

Using statistical data wisely, like the characters in the baseball movie Moneyball, UX teams can leverage strikes one and two to get on base.

When I worked for a Japanese automotive original equipment manufacturer (OEM), they had a saying, which may or may not be a national cliché: “Free is oft the most expensive price.”

Huh? Let me translate. What they’re saying is something slightly beyond the American cliché: “There’s no such thing as a free lunch.” You must spend some money wisely to save money in the long run.

To illustrate that slightly further, let’s look at the movie Moneyball: the story of a financial underdog—the Oakland A’s—keeping pace with the big boys by reevaluating the value of statistics and applying their money efficiently. I know what you’re thinking: What does that have to do with user experience? Everything. Rethinking the process based on goal-outcome-oriented data can help the A’s of the world beat the Yankees and win the UX World Series.

The UX Equivalent to “On-Base Percentage”

In the movie, a statistician (played by Jonah Hill) convinces the general manager of the small-budget Oakland A’s (Brad Pitt) that on-base percentage (OBP) is the key to evaluating a player’s success: not how often they hit (a.k.a. batting average, the old school way of evaluating a player), but essentially how often they don’t strike out. Staying with our analogy, if we consider launching a lousy design as the equivalent of striking out, then the obvious desire is to avoid final failure completely despite learning through a few early strikes.

And, sure enough, there’s data to support such a claim within the UX world. As infamously inscribed in Gilb’s 1998 study, “The rule of thumb in many usability-aware organizations is that the cost-benefit ratio for usability is $1:$10-$100. Once a system is in development, correcting a problem costs 10 times as much as fixing the same problem in design. If the system has been released, it costs 100 times as much relative to fixing in design.”

[google_ad:WITHINARTICLE_1_468X60]

These problems may be simple bugs (i.e. a few lines of code) or as severe as a flawed business assumption that requires pivoting the company’s direction (i.e. millions of lines of code). As Eric Reis says in Lean Startup, “We have successfully, faithfully, rigorously, executed a plan to achieve failure [early].” In support of Gilb’s study, Reis shows examples where minimizing the time to create those pivots—characterized by building and testing the minimal model of the idea(s)—helps to reduce significant cost downstream for a flawed feature or product. As Reis said in his speech at Google, “We all know. Software companies. We can build anything we can imagine. Think about that for a second. The dominant question of our time is not ‘Can it be built?’ but ‘Should it be built?’” Finding failures early in the process and pivoting can reduce the cost of fixing problems later.

Moneyball translation: Don’t worry too much about strike #1; worry a lot about not learning early and waiting to fail by strike #3.

Value Equals Cost Divided By Runs

In a critical scene in the movie, Brad Pitt’s character corners his eventual statistician wanting to know why a baseball trade conversation went the way it did. The statistician rants: “[Johnny Damon] has a great glove. He’s a decent leadoff hitter. He can steal bases. But is he worth the $7.5M a year that the Boston Red Sox are paying him? No … no. Baseball thinking is medieval. They are asking all the wrong questions.”

The measure that the characters in the film create is essentially the value of a player based on the cost of buying the runs he creates by getting on base. An MBA knows this as return on investment (ROI).

I’ve been working for years in the automotive industry, so allow me to use the design and development of infotainment systems to illustrate these notions in action. The thinking surrounding infotainment systems is just as medieval as baseball scouting at the turn of the 21st century. Snazzy looks and differentiating features define success, but meanwhile two well-known OEMs have created new platforms in the past couple of years that unofficially cost each corporation $250M to develop.

As Brad Pitt says in the movie, “When your enemy’s making mistakes, don’t interrupt him.”

What was the ROI? No one knows because that isn’t even close to being measured. The only measurables are spending vs. budget, development time vs. budget, and quality scores vs. goals. If the automotive companies paid for the infotainment-equivalent of Johnny Damon and delivered on-time without too many errors, it’s deemed a success without understanding if it provided competitive value.

So let’s create a hypothetical-yet-typical scenario to compare ROI’s of two different processes. Let’s imagine two corporations—Yankees Inc. and A’s Inc.—that both need three iterations to have a successful product due to firstly a product-direction error and secondly multiple mistakes in the user interface (UI) that need correction. They will use two different processes to get to the million lines of code that have made their way to production in the past two years. The Yankees will use what I call the Velocity Method (since it supposedly “goes quickly” in one direction) and A’s will use the Modeling Method.

“If we try to play like the Yankees in here, we will lose to the Yankees out there.”

THE FIRST PIVOT: Let’s start Yankees Inc. out by creating a written specification of what will be competitively outsourced. This will require 12 months for a team of 10 graphical designers at $120/hour fully burdened along with another thirty engineers writing requirements at $150/hour, fully burdened. It will then take six months for three Purchasing folks at $150/hour fully burdened to source the project and another nine months at $150-200/hour to have the first prototype working on hardware. Ignoring the costs for the usability testing (we’ll assume it’s the same for both Yankees Inc. and A’s Inc.), Yankees finds out the business assumptions were flawed after spending 27 months and $15M.

Ouch. Maybe velocity in one direction doesn’t equate to speed.

A’s Inc. decides to do some quick graphical prototypes (models) with a $50,000 software tool that permits code generation for an existing set of hardware (touchscreen, processors, voice engine) that’s close enough to production hardware to simulate the experience. Per Abowd et al (2004), this development process will be 75% more efficient and can be done coincident with a portion of the requirements (e.g. the interface matches to a specified API), thereby causing the whole operation to only require three graphical designers and 10 engineers for seven months along with the same three purchasing agents for six months to the tune of $2.5M in seven seven months. Per Conklin (1991), a one-quarter delay in speed to market results in a 50% loss in profit. Assuming the three million vehicles are priced at $30,000 for an 11% profit, the 20 month delay for Yankees to establish that the business direction was flawed adds in even more cost … but we’re not done yet.

THE SECOND PIVOT: We’ll be kind to Yankees Inc. and say the flawed business assumptions only created a 50% rework in the user interface and/or associated logic. Despite the coding being outsourced overseas with cheap(er) labor, the supplier now is fully into “Change Request Mode.” This requires reworking of the requirements using the same team of designers and engineers, and working through Purchasing to the supplier.

to make things easier, we’ll simply split the previous numbers in half to 13 months and $7.5M, despite the likelihood that the supplier will seek higher margins at this juncture. To be tremendously generous in this comparison, let’s assume getting A’s Inc. to the Second Pivot took as much time and effort as the First Pivot: seven months and $2.5M. Per Wixon & Jones (1994), the speed at which A’s Inc. has working prototypes means more upfront usability testing is possible, which can result in 60-80% higher revenues, but we can ignore that for now as well.

THE THIRD LANDING: Per Landauer (1995) the average UI has forty flaws, so upon reaching the second pivot, both companies recognize the need for a 10% rework of the upper layers of the software. Applying the exact ratio to Yankees Inc.’s costs, it would result in another three months and $1.5M. For A’s Inc., there’s no need to go through Purchasing again since the code exported from the modeling software can simply be handed to the supplier, but we can generously apply the 10% number to realize one additional month and $.25M.

[google_ad:WITHINARTICLE_1_234X60_ALL]

The Return on Investment

Yankees spent a whopping total of 43 months and $24M on just the user interface portion of the infotainment system. Interestingly, Yankees Inc. may be pleased by this, since it met all of the corporate budget and quality goals while avoiding a delay in shipping any given vehicle line. Yes, J.D. Powers numbers suggest a 24% decrease in overall quality due to infotainment flaws or complaints, but … well … that’s just the customer talking.

A’s Inc, on the other hand, eradicated that medieval thinking, spent $50,000 upfront and only chipped in an additional 15 months and $5.25M to get to the same development spot, thereby saving 28 calendar months and ~$19M. If the infotainment system could ship independently, Conklin’s (1991) data suggests A’s Inc. would increase revenue in the billions, but let’s simply assume that A’s Inc.’s early usage of reference hardware allowed “right-sizing” of the infotainment’s UI processor, thereby decreasing costs $1 per vehicle. That’s another $3M per year over the lifespan of the infotainment platform, which is typically 3 years or $9M. Therefore, A’s Inc.’s results from changing their way of thinking was ~$28M on a $50k investment or a 55,900% ROI.

You would think the two, real-world automotive OEMs who have stumbled upon this radical, Moneyball-esque thinking would be bragging loudly of their supreme intelligence but, as Brad Pitt says in the movie, “When your enemy’s making mistakes, don’t interrupt him.”

post authorSteve Tengler

Steve Tengler, Steve Tengler is a Senior Director with the Honeywell Connected Vehicle group, where he oversees a global team of engineers, data scientists and designers working on vehicle monitoring solutions such as cybersecurity, prognostics, etc. Steve is a proven expert in the field of the connectivity and user experience with over twenty-five years of experience on some of the country's top automotive teams, such as OnStar, Nissan, Ford and Honeywell.

Tweet
Share
Post
Share
Email
Print

Related Articles

What do Architecture, Computer Science, Agile, and Design Systems have in common?

Article by Kevin Muldoon
A Pattern Language
  • The article explores Christopher Alexander’s impact on diverse fields, from architecture to software development, introducing the concept of design patterns and their influence on methodologies like Agile and the evolution of Design Systems.
Share:A Pattern Language
7 min read

In this article, I’ll share seven incredibly useful design resources that can elevate your skills as a UX designer in 2024 and make you a well-rounded designer overall. These resources not only help you work more efficiently but also automate repetitive tasks and enable you to create outstanding user experiences. So, let’s dive into this without further delay.

Article by Abhi Chatterjee
7 Must-Have Resources for UX Designers in 2024
  • The article explores seven indispensable resources for UX designers in 2024, offering insights into design methodologies, case studies, UX laws, and practical challenges to elevate designers’ skills and efficiency.
Share:7 Must-Have Resources for UX Designers in 2024
3 min read
Article by Eleanor Hecks
8 Key Metrics to Measure and Analyze in UX Research
  • The article outlines eight essential metrics for effective UX research, ranging from time on page to social media saturation
  • The author emphasizes the significance of these metrics in enhancing user experience and boosting brand growth.

Share:8 Key Metrics to Measure and Analyze in UX Research
6 min read

Did you know UX Magazine hosts the most popular podcast about conversational AI?

Listen to Invisible Machines

This website uses cookies to ensure you get the best experience on our website. Check our privacy policy and