When I worked for a Japanese automotive original equipment manufacturer (OEM), they had a saying, which may or may not be a national cliché: “Free is oft the most expensive price.”
Huh? Let me translate. What they’re saying is something slightly beyond the American cliché: “There’s no such thing as a free lunch.” You must spend some money wisely to save money in the long run.
To illustrate that slightly further, let’s look at the movie Moneyball: the story of a financial underdog—the Oakland A’s—keeping pace with the big boys by reevaluating the value of statistics and applying their money efficiently. I know what you’re thinking: What does that have to do with user experience? Everything. Rethinking the process based on goal-outcome-oriented data can help the A’s of the world beat the Yankees and win the UX World Series.
The UX Equivalent to “On-Base Percentage”
In the movie, a statistician (played by Jonah Hill) convinces the general manager of the small-budget Oakland A’s (Brad Pitt) that on-base percentage (OBP) is the key to evaluating a player’s success: not how often they hit (a.k.a. batting average, the old school way of evaluating a player), but essentially how often they don’t strike out. Staying with our analogy, if we consider launching a lousy design as the equivalent of striking out, then the obvious desire is to avoid final failure completely despite learning through a few early strikes.
And, sure enough, there’s data to support such a claim within the UX world. As infamously inscribed in Gilb’s 1998 study, “The rule of thumb in many usability-aware organizations is that the cost-benefit ratio for usability is $1:$10-$100. Once a system is in development, correcting a problem costs 10 times as much as fixing the same problem in design. If the system has been released, it costs 100 times as much relative to fixing in design.”
[google_ad:WITHINARTICLE_1_468X60]
These problems may be simple bugs (i.e. a few lines of code) or as severe as a flawed business assumption that requires pivoting the company’s direction (i.e. millions of lines of code). As Eric Reis says in Lean Startup, “We have successfully, faithfully, rigorously, executed a plan to achieve failure [early].” In support of Gilb’s study, Reis shows examples where minimizing the time to create those pivots—characterized by building and testing the minimal model of the idea(s)—helps to reduce significant cost downstream for a flawed feature or product. As Reis said in his speech at Google, “We all know. Software companies. We can build anything we can imagine. Think about that for a second. The dominant question of our time is not ‘Can it be built?’ but ‘Should it be built?’” Finding failures early in the process and pivoting can reduce the cost of fixing problems later.
Moneyball translation: Don’t worry too much about strike #1; worry a lot about not learning early and waiting to fail by strike #3.
Value Equals Cost Divided By Runs
In a critical scene in the movie, Brad Pitt’s character corners his eventual statistician wanting to know why a baseball trade conversation went the way it did. The statistician rants: “[Johnny Damon] has a great glove. He’s a decent leadoff hitter. He can steal bases. But is he worth the $7.5M a year that the Boston Red Sox are paying him? No … no. Baseball thinking is medieval. They are asking all the wrong questions.”
The measure that the characters in the film create is essentially the value of a player based on the cost of buying the runs he creates by getting on base. An MBA knows this as return on investment (ROI).
I’ve been working for years in the automotive industry, so allow me to use the design and development of infotainment systems to illustrate these notions in action. The thinking surrounding infotainment systems is just as medieval as baseball scouting at the turn of the 21st century. Snazzy looks and differentiating features define success, but meanwhile two well-known OEMs have created new platforms in the past couple of years that unofficially cost each corporation $250M to develop.
As Brad Pitt says in the movie, “When your enemy’s making mistakes, don’t interrupt him.”
What was the ROI? No one knows because that isn’t even close to being measured. The only measurables are spending vs. budget, development time vs. budget, and quality scores vs. goals. If the automotive companies paid for the infotainment-equivalent of Johnny Damon and delivered on-time without too many errors, it’s deemed a success without understanding if it provided competitive value.
So let’s create a hypothetical-yet-typical scenario to compare ROI’s of two different processes. Let’s imagine two corporations—Yankees Inc. and A’s Inc.—that both need three iterations to have a successful product due to firstly a product-direction error and secondly multiple mistakes in the user interface (UI) that need correction. They will use two different processes to get to the million lines of code that have made their way to production in the past two years. The Yankees will use what I call the Velocity Method (since it supposedly “goes quickly” in one direction) and A’s will use the Modeling Method.
“If we try to play like the Yankees in here, we will lose to the Yankees out there.”
THE FIRST PIVOT: Let’s start Yankees Inc. out by creating a written specification of what will be competitively outsourced. This will require 12 months for a team of 10 graphical designers at $120/hour fully burdened along with another thirty engineers writing requirements at $150/hour, fully burdened. It will then take six months for three Purchasing folks at $150/hour fully burdened to source the project and another nine months at $150-200/hour to have the first prototype working on hardware. Ignoring the costs for the usability testing (we’ll assume it’s the same for both Yankees Inc. and A’s Inc.), Yankees finds out the business assumptions were flawed after spending 27 months and $15M.
Ouch. Maybe velocity in one direction doesn’t equate to speed.
A’s Inc. decides to do some quick graphical prototypes (models) with a $50,000 software tool that permits code generation for an existing set of hardware (touchscreen, processors, voice engine) that’s close enough to production hardware to simulate the experience. Per Abowd et al (2004), this development process will be 75% more efficient and can be done coincident with a portion of the requirements (e.g. the interface matches to a specified API), thereby causing the whole operation to only require three graphical designers and 10 engineers for seven months along with the same three purchasing agents for six months to the tune of $2.5M in seven seven months. Per Conklin (1991), a one-quarter delay in speed to market results in a 50% loss in profit. Assuming the three million vehicles are priced at $30,000 for an 11% profit, the 20 month delay for Yankees to establish that the business direction was flawed adds in even more cost … but we’re not done yet.
THE SECOND PIVOT: We’ll be kind to Yankees Inc. and say the flawed business assumptions only created a 50% rework in the user interface and/or associated logic. Despite the coding being outsourced overseas with cheap(er) labor, the supplier now is fully into “Change Request Mode.” This requires reworking of the requirements using the same team of designers and engineers, and working through Purchasing to the supplier.
to make things easier, we’ll simply split the previous numbers in half to 13 months and $7.5M, despite the likelihood that the supplier will seek higher margins at this juncture. To be tremendously generous in this comparison, let’s assume getting A’s Inc. to the Second Pivot took as much time and effort as the First Pivot: seven months and $2.5M. Per Wixon & Jones (1994), the speed at which A’s Inc. has working prototypes means more upfront usability testing is possible, which can result in 60-80% higher revenues, but we can ignore that for now as well.
THE THIRD LANDING: Per Landauer (1995) the average UI has forty flaws, so upon reaching the second pivot, both companies recognize the need for a 10% rework of the upper layers of the software. Applying the exact ratio to Yankees Inc.’s costs, it would result in another three months and $1.5M. For A’s Inc., there’s no need to go through Purchasing again since the code exported from the modeling software can simply be handed to the supplier, but we can generously apply the 10% number to realize one additional month and $.25M.
[google_ad:WITHINARTICLE_1_234X60_ALL]
The Return on Investment
Yankees spent a whopping total of 43 months and $24M on just the user interface portion of the infotainment system. Interestingly, Yankees Inc. may be pleased by this, since it met all of the corporate budget and quality goals while avoiding a delay in shipping any given vehicle line. Yes, J.D. Powers numbers suggest a 24% decrease in overall quality due to infotainment flaws or complaints, but … well … that’s just the customer talking.
A’s Inc, on the other hand, eradicated that medieval thinking, spent $50,000 upfront and only chipped in an additional 15 months and $5.25M to get to the same development spot, thereby saving 28 calendar months and ~$19M. If the infotainment system could ship independently, Conklin’s (1991) data suggests A’s Inc. would increase revenue in the billions, but let’s simply assume that A’s Inc.’s early usage of reference hardware allowed “right-sizing” of the infotainment’s UI processor, thereby decreasing costs $1 per vehicle. That’s another $3M per year over the lifespan of the infotainment platform, which is typically 3 years or $9M. Therefore, A’s Inc.’s results from changing their way of thinking was ~$28M on a $50k investment or a 55,900% ROI.
You would think the two, real-world automotive OEMs who have stumbled upon this radical, Moneyball-esque thinking would be bragging loudly of their supreme intelligence but, as Brad Pitt says in the movie, “When your enemy’s making mistakes, don’t interrupt him.”
