Why PBS Uses Data to Drive Continuous Design

At PBS, we strive to provide viewers with their favorite content wherever and whenever they want, which is becoming an increasingly hard problem to solve, given the frequency at which new platforms emerge. In 2010, PBS had a website and an iPad app to deliver video, our primary content offering. By the beginning of 2014, there were eight platforms. This year will see that number double.

For that reason we need to be able to quickly identify the biggest opportunities and deliver new applications, while retaining our brand presence on diverse platforms. PBS has long been an agile development shop—it offers the flexibility to incrementally and simultaneously build for multiple platforms, while tracking and prioritizing cross-team dependencies. We further refined the standard agile processes by enacting Kanban, a system of continuous code deployment. To allow us to double our platform offerings in the space of a year, we have extended the agile process through the principles of data-driven and continuous design.

With Continuous Design, You’re in the Business of Change

Continuous design has two main principles. The first requires you to abandon the long-held concept of “final” designs. Designers must be involved throughout the entire product development process, not just at the beginning. The second mandates designers create reusable front-end code, or interactive prototypes, instead of document-based deliverables like PSDs.

To facilitate the continuous design process, PBS Digital created a cross-product design team of UX, UI, and front-end designers who are responsible for delivering reusable HTML/CSS/JS designs. On our native mobile app projects, the design team takes advantage of tools such as Flinto to deliver interactive prototypes that can be installed on actual mobile devices. The prototypes include important design details such as screen transitions, which are difficult to illustrate in a static document. In both scenarios, the design team isn’t just producing documents to “throw over the wall”—instead, the team remains actively engaged throughout the entire build cycle and is able to respond quickly to changes.

Holding Yourself Accountable to Data

Before designers design, they need a direction. Data-driven design provides the orientation for creating and evolving products based on collected data about how current designs are performing in the market. To paint the full picture, we treat each major release as an experiment and collect both quantitative and qualitative data. Our analytic arsenal contains a swath of tools we find indispensable, including Splunk, which measures video streaming data from the server’s point of view; Google Analytics, for mass anonymous usage data; ForeSee, for customer satisfaction surveys; UserTesting.com, to conduct user tests; App Store reviews; Twitter buzz; and a few others.

Collecting data is not hard; collecting the right data—relevant data—and drawing the necessary insights is. But even good data and good analysis does not ensure correct decisions. That requires the institutional holds itself accountable to the data. For example, PBS recently decided to move away from building HTML5-hybrid apps after analyzing data collected from an experiment with our iPhone app.

Collecting data is not hard; collecting relevant data and drawing the necessary insights is

Beginning in the summer 2013, PBS experimented with a hybrid iPhone app, releasing a native “wrapper” for our responsive video portal. By all measures, quantitative and qualitative, the hybrid approach left PBS consumers less satisfied than the previous native app experience. There was a quantitative drop in app engagement, measured by video streams, and qualitative drop in user satisfaction, as noted in app store reviews and users surveys.

The experiment, on a lesser-used platform, led PBS to conclude that hybrid (or wrapped) apps were not a viable long-term strategy. (We are not the only ones to have reached this conclusion.) It is not easy to design a single site that satisfies both app and web users across all mobile and desktop platforms. Despite the investment of time and design resources in pursuing the hybrid strategy, the data showed it was not the correct direction. Objectively reviewing the data, and holding ourselves accountable, was an important step in the right direction, and serves as an example of incremental, data-driven design.

Tools Are Not Solutions

Data-driven design and continuous design are not universal solutions. Understanding their value led PBS Digital’s product team to fully integrate the principles into standard workflows. As with any tool or process, however, you need to make sure it is the correct one for the job. Regardless of your current design process, there are elements of data-driven and continuous design from which you may benefit.