“Failing fast” means getting putting applications out in the wild as soon as possible to learn whether they will succeed. This gives you access to early user feedback to quickly weed out ideas and methods that don’t work. Failing fast is a good thing—or, at least, it’s preferable to failing slowly and spending too much time, effort and money developing a product that should have been put to rest earlier. Money and time you save by cutting off unsuccessful projects quickly will mean you have more money and time for the successful ones. The concept of failing fast can help businesspeople and stakeholders reduce the riskiness of launching products by letting real users and the marketplace dictate their product choices.
This article features a conversation among thought leaders from eBay, Google, Adobe Systems Inc. and Discovery Channel during a conference session held at O’Reilly Media‘s Web 2.0 Expo Conference & Expo in San Francisco. The panel discussion was moderated by Tony Hillerson, a software architect at EffectiveUI.
Panelists
Alan Lewis, Product Manager, eBay Desktop, eBay Inc.
Randy Rieland, Senior Vice President, Digital Media, Discovery Channel
Scott Green, Site Director, Google’s Boulder, Colo. Engineering Office
Ryan Stewart, RIA Evangelist, Adobe Systems Inc.
Panelist Discussion
- Moderator: What do we mean when we talk about failure today? Not technical failure, but failure to gain user adoption. Without user adoption—the fundamental measure of any project’s success—there is no commercial payoff or return on investment. Today we’re going to explore the process of integrating user feedback into the evolution of a product, with the end goal of encouraging users to stick with your product.
To start with, how do you know when to add a feature based on user input? For example, Twitter started out as a simple application and now it’s evolved based on the way people want to use it. How do you find the tipping point to make changes to an application?
- Alan Lewis, eBay: I believe in the organic approach of putting something out there and seeing if it gets traction. From there, support what is happening organically and make it official if it makes sense. This is definitely not a conservative or cautious approach, but it lets the community define the application.
- Scott Green, Google: You never really do know when to invest time and money into developing new features and functionality. It’s more important to do user testing during early stages. Remember that listening to user feedback is an active, not a passive, thing. I can rarely think of an instance where I wish we had done things slower. It’s just not the way things work.
- Moderator: Let me ask whether it’s better to stick with proven technology or risk facing the bleeding edge. The bleeding edge can be a dangerous place, but also alluring because there is always emerging technology that lets you do things that couldn’t be done before—easier, faster, cheaper, better.
- Alan Lewis, eBay: Leveraging new technology is one way to put new things out there. But you can also try a new business idea. It just might not be a good idea to do both at the same time. There would be multiple levels of risk. For example, eBay Desktop was not a new idea. It was something users wanted for a long time, but it wasn’t a feasible project until new technology—Adobe AIR—came along. That’s a case of leveraging new technology to bring an existing idea to market.
- Ryan Stewart, Adobe: The best time to use a new technology is when it is going to give you a competitive advantage. I cover a lot of new uses of Flash and Flex, good and bad. The trick is to use the technology in a good way to differentiate your application.
- Moderator: What kind of users do you get when you embrace a new platform? A friend told me about a great new site about wine and that it was a Rails application. The experience turned out to be more about the technology than the wine.
- Alan Lewis, eBay: When we released eBay Desktop, it was one of the first AIR applications built. We had no marketing budget, so we focused our outreach on the fact that it was such a new technology. No surprise, we got users who were very technology-centric. Now we look at the data and see that the most popular thing people are buying on eBay Desktop is video games—which is the number four product on the ebay.com. It’s important to act on user feedback when and where it makes the most sense for different parts of our product line. If you’re trying to get coverage on TechCrunch, then you’re going to get users who follow TechCrunch, and they are a very specific kind of consumer.
- Moderator: Should we be using technology to attract users to the technology itself?
- Randy Rieland, Discovery: No. We are more interested in the direction of the content for the mass audience that shows loyalty for the Discovery brand than we are about the technology behind the content. For example, years ago we had an application called Earth Alert. It was very basic, had a simple interface, and was very successful. It allowed people to track environmental stories around the planet — represented by pinpoints on a globe. We wanted to bring back that basic concept in a more meaningful way, and this year we launched Discovery Earth Live. It uses Flash and Flex and it wraps satellite animations around a 3D globe, and also presents environmental stories. Users can layer animations, as well. While Discovery Earth Live is a complex technological achievement, it’s really very simple to use. We want our users to interact with the content and the brand more than we want them to focus on the technology itself.
- Moderator: How do you protect your brand in the face of both positive and negative user feedback?
- Randy Rieland, Discovery: In TV, everything has to be totally perfect before it airs. On the Web, however, negative feedback is inherent to the medium. At Discovery, we focus on two things. First, we help the television people understand that the Web is a more organic medium than TV, that things don’t always work perfectly out of the gate, and that there’s a constant tweaking process to improve Web offerings. We fight the notion of “you can put meatloaf on the Web,” and change that thinking to be about understanding that the Web is really never a done deal. It constantly evolves. Second, when we invest in a large application, we want a sponsor for it. TV sponsors are accustomed to seeing exactly how things work, so you can’t pitch a TV sponsor with high concepts. Sponsors want to know how it’s going to work, even though the Web audience is pretty forgiving. When we roll out a new Web app, we want user input and we want to improve the product. TV sponsors are just beginning to understand that.
- Moderator: Might there ever be a situation where the openness of Web presence is a bad thing?
- Ryan Stewart, Adobe: No. Look at what Twitter did with the @ sign—simple and open. The more transparent you are with your process, the better users like it and the more easily you can adapt your application to what users want.
- Scott Green, Google: The Google brand is built on aggression and innovation, so we don’t have the same brand issues that, say, Discovery has. But you don’t want to focus on beta aspects all the time. You do want to enhance applications with new features in a faster, rather than a slower, way.
- Moderator: How do sales organizations cope with the uncertainty of failing fast? Do salespeople have to buy into something concrete to get behind it, and how can that work with an open process?
- Alan Lewis, eBay: The open process definitely makes for an interesting sales dynamic. We have to convince them that a product is worthwhile and why, in order for them to sell it.
- Randy Rieland, Discovery: It’s a push-pull situation. People want to be associated with something innovative, while at the same time protect the brand, achieve metrics and deliver results. Before we invest in an application, we have to show its ROI. It may be fiction, but we have to do it. We map out products in phases and define the expectations of each phase. It’s important to get people thinking that way, so they know software is a process and not something that sits on a shelf and never changes.
- Moderator: How do you handle buzz? Is buzz ever bad?
- Ryan Stewart, Adobe: No. Buzz brings the kind of user that will contribute to your application, provide valuable feedback and help you incorporate feedback into your project.
- Scott Green, Google: You learn a lot from buzz. Amplified viral marketing helps you listen better.
Q&A with the Web 2.0 audience
- Q: When you’re working in an experimental environment, how do you manage teams when there is a run of failures and un-met expectations? How do you manage the process of losing confidence?
- Alan Lewis, eBay: In our disruptive innovation group, part of our charter is to experiment. Like venture capitalists, we invest in a number of “startups” expecting that a percentage of them will pan out. The makeup of our team is such that we move quickly from idea to idea. For example, we built and launched an eBay Facebook application, but it didn’t attract users. It’s not the implementation that was bad, it was just the wrong platform—commerce and Facebook don’t mix. But we knew going in that our success criteria is our learning process. We were not measured on the success of the product, but on the success of innovation itself. We were not measured on the quality of the answer, but rather how we framed the question.
- Scott Green, Google: What’s the alternative? Work on the same bad idea for two years or six months? What if the satellite companies invested billions into networks and no one used their cell phones? Not funny. But by failing fast, you minimize your investment. Dealing with reality helps you make fast course corrections.
- Alan Lewis, eBay: Failure comes when you don’t listen. You can’t put something out there and assume it’s great. It’s up to us to make sure we’re listening to improve our chances for success—if not this time, next time.
- Q: From a marketing perspective, how do you cope with failure? How do you recover publicly?
- Scott Green, Google: Transparency is a good starting point. If you fail, so be it. Just let people know.
- Q: Can you talk about how you handle user feedback internally among design and development teams? What’s the process?
- Alan Lewis, eBay: Remove the layers between the people who are making the decisions and the people who are telling what they should be doing. I get emails directly from users, outside the customer support team, and I share information appropriately with designers and developers.
- Scott Green, Google: We encourage our designers and developers to attend our worldwide user conference so they can talk directly with people and deal with reality.
- Ryan Stewart, Adobe: As an evangelist, I attend conferences like this one to get information I can feed that information back to developers. It’s important to fit into their workflow, get into their bug-fixing and feature-enhancing processes. That really benefits them as well as the users. Even when feedback is negative, it’s helpful. Generally, developers really want to make cool things.
- Scott Green, Google: Conversely, sometimes developers don’t like what they are building and like to hear that users agree with them.
- Q: How do you really know when to pull features? And when do you call it quits?
- Alan Lewis, eBay: Numbers are a big factor. Also, it’s good to have as many users as possible participate in the evolution of a product. If it gets to a point when users are simply done with an app, then you’re done with it. When we invest heavily into an application, I get questions from above—and sometimes pulling the plug is valid. You have to educate the people above you to understand that not everything gets traction. Sometimes we’ve been really surprised to find that an idea takes hold later when we least expect it. Also, we have a wealth of technology that helps us gather the data we need to make a decision to kill something or let it live. Also, today it’s less expensive than ever to experiment, test and try to find sparks of user interest.
- Q: How do the big companies decide to scoop up startups and bring emerging technologies to market?
- Ryan Stewart, Adobe: We don’t want to compete with startups that use our platform. We make sure we give them access to latest technologies. We try to bring them in and support them. At the end of the day, the best ideas win.
- Alan Lewis, eBay: Most larger companies looking for acquisitions seek specific expertise that they don’t already have—for example a untapped user segment or a business concept that aligns itself somehow with the larger company’s core technology.
- Randy Rieland, Discovery: We don’t have an internal innovation team, so all innovation we get comes from outside.
- Scott Green, Google: We look to the big world of independent businesses for innovation.
- Q: How do you determine the scale of a launch—when to do invitation beta or wider? And how do you decide something is ready for launch?
- Alan Lewis, eBay: One important consideration is to determine what are you trying to learn from the beta. Then estimate how many users you need to get that question answered. With our products, when we stop being able to answer our own questions, we know we need others to tell us what we need. We basically propose a hypothesis and see if others can disprove it.
- Q: Scott, what’s it like to be acquired by a big company and bring innovation in from the outside?
- [Google acquired @Last Software and SketchUp in March of 2006.]
- Scott Green, Google: From my perspective, our culture was very similar to Google’s. We moved fast, did not have a top-down decision-making process. There were no ivory towers. And at Google, each product team is responsible for its own success.
- Randy Rieland, Discovery: I’ve been with digital arm of Discovery since it started. Back then, all the money was in the TV group where nobody knew anything about the Web. Now there are a lot of people with a little bit of knowledge about the Web and come at us saying “do this and do that.” The key is that I understand the brand and how to present it in this different medium. If other divisions think we’re doing crazy things on the Web, they will impose their will on you. They need to know that we understand the Discovery brand, and know how to extend it in meaningful ways and not segregate it.
- Alan Lewis, eBay: At eBay, even though innovation is my group’s charter, if we are the only group from which innovation is emanating, we consider that a failure. So we do a lot of grunt work to showcase innovation coming from other groups. We foster and encourage it.
- Q: Is the beta process or the launch process any different when an application deals with users’ money?
- Alan Lewis, eBay: Absolutely, it makes a huge difference. When we launched PayPal, we were told in no uncertain terms by the CEO that it’s not ok to throw something out there for beta unless the core technology really works.
- Moderator: Thank you, panelists, for your participation today. And thanks to the audience for being with us.
About Alan Lewis
Alan is the product manager and creator of eBay Desktop application. He worked at eBay as a product manager, technical evangelist and technical writer. Alan is a regular on the conference circuit, and speaks frequently about rich Internet applications and Web services.
About Scott Green
Scott Green leads Google’s Boulder, Colo., engineering office. Google acquired @Last Software (SketchUp) in March of 2006. At that time, Scott was the VP of engineering (Scott joined the SketchUp team in October of 2003.) Prior to joining Google, Scott had been the director of technical operations at ReturnPath. Earlier, he was the director of engineering at E-mail Publishing / MessageMedia, which was acquired by DoubleClick. Scott graduated from Iowa State University in 1982 with a BS in mechanical engineering.
About Ryan Stewart
After a couple years of doing ColdFusion and Flex development for the Learning Lab group at The Wharton School, Ryan worked with WorldClass Strategy, an interactive advertising agency in Seattle. After a stint with Threecast, a small startup using Flash, he joined Adobe as a rich Internet application evangelist.
About Randy Rieland
Randy Rieland has been with Discovery.com since it was created by Discovery Communications in May, 1995. Discovery.com includes the websites for all of the Discovery Communications networks, including Discovery Channel, TLC, Animal Planet, Travel Channel, and Discovery Health, plus the company’s various digital networks, including Discovery Kids, Science and Discovery Civilization. Prior to moving to Discovery in 1995, Randy had been a print journalist for almost 25 years.
This article was originally published on the User Interface Resource Center (UIRC).
Special thanks to Alan Gray of NewsBlaze for contributing to this article.
