Lean Learning Loops

Experimentation is at the heart of the Lean Startup approach. Lean Startup practitioners design and execute business experiments in the same way that laboratory scientists design scientific experiments—by framing hypotheses and creating experiments to test them. The Lean Learning Loop (LLL) is the name Lean Startup uses for a cycle of experimentation. Lean Learning Loops make up the basic construct of the Lean Startup method (see Figure 2.1).

In a single Lean Learning Loop, an innovation team:

1. Frames a hypothesis
2. Designs an experiment to test the hypothesis
3. Conducts the experiment, and then
4. Reviews the results to asses whether they confirm or disconfirm the hypothesis and decide on the next steps.

A Lean Learning Loop can be conducted to learn about almost anything important to the business—the attractiveness of particular features of a product, an estimation of the value created, or the most efficient price point. An experiment can also be designed to test the viability of a channel or the attractiveness of a customer segment; it can be used to understand the costs of delivering a solution or the potential issues that might arise in working with a partner. As each loop delivers new information, the team can zero in on the product design and the business model. In short, Lean Learning Loops are the building blocks of the Lean Startup method.

A team I led at Goodyear, for example, designed an experiment to test the viability of a concept for a “green tire”—one that included significantly more recycled material than a typical tire. The concept would be costly to deliver, but it addressed a known desire among an important customer segment. The question the team wanted to answer was whether people would pay enough to make the concept viable. An experiment was designed to answer this question.

The hypothesis was that customers who were concerned about the environment would pay more for an eco-friendly tire. To test this hypothesis, we selected two Goodyear tire stores in California as test locations. These locations were selected because of their demographics; if the test was not positive there, the team reasoned, it would not be positive anywhere.

The tires did not yet exist (though we had conceived of a means of delivering them), so we created prototypes—new tires with a “New Earth” logo engraved on them. We developed displays and marketing materials for the tires and set them up in the selected stores. Team members manned the stores, and we monitored traffic and customer response over a two-week period. When a customer decided to buy the tires, we informed them that they were part of a research experiment and offered them a discount coupon for their tire purchase.

The experiment disproved our hypothesis. Some customers found the concept attractive, but they would not accept any compromise in tire performance or pay any premium for such a tire. In fact, the customers expected a discount on the tires. This information was a key factor in the decision to discontinue the project. The experiment was a success because it saved significant R&D investment and redirected the innovation team to another concept.

In another example, Pitney Bowes used a business experiment to test the channel for a new postage metering solution. The new concept was novel and addressed a new segment for Pitney Bowes: Very small businesses and individuals who were currently using stamps. The concept we proposed included the ability for customers to customize the images on the stamps the meter printed.

The team hypothesized that the meter could be sold to customers through the company’s telemarketing operation. We set up an experiment in which agents were taught to “drop down” to the new offering if the customer decided not to purchase the company’s traditional meter. Customers were directed to a web page that showed the meter and its features, although the meter did not yet exist. We measured the rate at which customers placed an order for the concept at different price points for both inbound and outbound telemarketing. The results were very promising. They enabled the innovation team to create a fact-based business case that led to a significant investment in product development.

A Lean Learning Loop is generally a quick, relatively inexpensive, targeted endeavor designed to yield answers to a specific question quickly. Designing a successful learning loop can take some skill. One useful approach is to consider several alternative designs and have others on the team challenge them. The challenges usually center on what can be done to create a “good enough” simulation faster or more inexpensively. More detail on the design of business experiments can be found in “Conducting Business Experiments,” by Ganguly and Euchner, excerpts of which are included in Chapter 4.

Minimum Viable Product

A business experiment generally requires some form of a prototype (see Figure 2.2). Teams must have something—whether it is a web page describing features or a mocked-up tire—to show customers and test hypotheses. The Lean Startup method, which emerged in the software industry, coined the term Minimum Viable Product to describe the prototype used for testing. An MVP is a product that has a very limited (minimal) scope but is useful to and usable by (viable for) some set of customers. The MVP is designed to prove that the product concept is attractive to at least some customers and can succeed in the market.

As Lean Startup has been applied to physical products, innovation teams have borrowed other types of prototypes from the world of design, which has developed a more diverse set of practices for developing and testing prototypes prior to market entry. Many of these kinds of prototypes are created to test a concept or feature before a product is marketed. Thus, in the physical product world, prototypes may include:

• Probes—Low-fidelity, nonfunctioning prototypes, such as story­boards or foam-core models, designed to provoke a response to an idea; the idea of a probe is to validate or invalidate the existence of a hypothesized need
• Technical prototypes—Prototypes designed to demonstrate specific functional capabilities
• Concept prototypes—Nonfunctioning or “Wizard of Oz” prototypes designed to assess whether the proposed solution would work in the customer’s world
• Business prototypes—Small-scale tests of an element of the business model designed to assess and understand how to manage a particular element of risk in the business model
• Minimum Viable Products—Prototypes that provide the minimum functionality that is usable by a set of customers.

Most of these prototypes are, strictly speaking, not MVPs because they are not functional products, even at a minimal level. They are not products that are tested in the market; rather they are prototypes that are tested with the market. I call them mvps (minimum viable prototypes).

In Goodyear’s eco-tire experiment, the prototype was a mock tire engraved with the “New Earth” logo, together with professional-looking marketing materials. In Pitney Bowes’s small-scale metering machine experiment, the prototype was a web landing page with images showing the new device’s capabilities.

In developing an MVP or an mvp, the prototype must be reduced to the minimal artifact necessary to conduct the experiment. The prototype itself does not need to meet any cost targets or requirements for performance or scalability or maintainability; it simply needs to support whatever experiment is being conducted.

Pivot or Persist

After each experiment, the team must make a decision to pivot or persist (see Figure 2.3). To persist means to continue learning along the lines of the current plan; to pivot means to rethink a major element of the plan, such as the channel or an important aspect of the product. This discipline—to let data drive the pivot-or-persist decision—is at the heart of the Lean Startup method.

After the “New Earth” tire experiment, the team chose to pivot—in fact, to discontinue the initiative. In the case of Amita (the personal postage meter with custom images), the team used the experimental data to support a decision to persist—to proceed to product development.

There is no science to the pivot-or-persist decision; in the end, it is a judgment call, albeit one based on data. Some variant of the most recent experiment that might overcome the negative results is always conceivable. The discipline of quick, well-considered experiments that yield objective data, however, generally leads to a consensus when the evidence is reviewed. That consensus may be a decision to stop an initiative, prompt the team to rethink several elements of the business model, or drive significant new investment to move the concept forward.

The Integrative Principle: Innovation Accounting

A Lean Startup team is focused on what needs to be true for a product to be successful in the market. Lean Learning Loops test the key assumptions about the business—one at a time. As the loops accumulate, however, the results of the business experiments must be tracked so that the team can hone its hypotheses and avoid repeating work. This tracking is what is meant by innovation accounting, and it is essential to progress. Innovation accounting provides a way of keeping track of progress when traditional measures like Stage-Gates or project milestones are not reliable indicators.

Innovation accounting may be carried out using a variety of tools. A Kanban board is a good way to track the status of the baseline assumptions (see Figure 2.4). Organizing the Kanban around the categories of Osterwalder’s Business Model Canvas can be helpful, as well, to assure that you do not become too focused on one part of the business [Osterwalder and Pigneur, 2010] (see Figure 2.5). A portfolio view can help to track progress across initiatives. Whatever the structure, the tool must clearly associate the evidence gathered from an experiment with the hypothesis (or hypotheses) that it validates or invalidates. The team should meet regularly to review the status of the various experiments, make decisions to pivot or persist, and prioritize the next set of experiments.

Maintaining the discipline required for innovation accounting can be difficult. With so much going on, it can seem a distraction—and even a waste of time—to write everything down and to organize it in a Kanban or other tool. But innovation accounting is critical. If the practice is not followed, the team soon finds itself drifting into the urgencies of the day. The learning agenda is buried, and the team is no longer practicing Lean Startup.

The Value Hypothesis

The Value Hypothesis captures the fit between the product or service concept and the market. The process of testing the Value Hypothesis goes by several names: Steve Blank calls the iterative process of creating the offering “customer development” [Blank, 2013], the Lean Product community talks about product-market fit [Olson, 2015], Eric Ries uses the term Value Hypothesis [Ries, 2011], in the design community, the term of art is “customer value proposition” [Lanning, 1998].

Whatever it is called, testing the Value Hypothesis (or crafting product-market fit) is very customer-intensive. It begins with a hypothesis about a customer need—what Clay [Christensen and Raynor, 2003] call a Job-to-be-Done (JTBD).

The validity of a JTBD is tested with a series of prototypes. The initial prototype is usually very simple—a storyboard, for example.

Later versions may be more complex and complete, at least from the customer’s perspective.

A “Wizard of Oz” prototype, for example, appears to the user as a functioning product but is actually manipulated from behind the scenes. Ultimately, the test may require a prototype that the user can actually use—the MVP, as defined by Lean Startup.

Testing prototypes with customers and iteration over time assures that a customer need really exists and that the product concept could fulfill it—that is, that the product provides customer value. Geoffrey Moore provides a useful structure for defining the customer value proposition [Moore, 1991]:

• For <a customer segment>
• Who <experiences a specific problem (has a JTBD)>
• We offer <description of proposed solution>
• Which delivers <specific benefits>
• Unlike competitive offers, our offer <delivers important differentiators>.

This framing of the value proposition forces detailed consideration of all its key elements. Too often, a team can create a statement of the value proposition that may be very specific about the offering but is not as clear about the problem it solves or the benefits it delivers.

An example of a strong Value Hypothesis from the trucking industry, written in Moore’s basic format, is:

For critical cargo fleets

Who experience significant penalties from downtime due to roadside tire failure

We offer intelligent monitoring of tires and predictive analytics

Which deliver a reduction of over 80% in roadside failures due to poor tire maintenance

Unlike competitive offers, our offer (a) provides actionable warnings and (b) is integrated with the service network.

Note that this description is very specific with regard to the target customer, the issue to be resolved, and the way the customer’s world will change as a result.

Another example, from the postal world:

For small businesses and individuals who currently use stamps

Who seek to differentiate themselves in the eyes of their customers

We offer a personal postage meter that can print custom stamp images

Which distinguishes the customer for branding or for special occasions

Unlike other offers, our offer lets you create your own, personalized stamp image.

Again, the description is specific to each of the key elements. The specificity allows each of the elements to be tested before a major commitment is made to full development.

Any offering must create real customer value, tangible or intangible. It must be worth more to customers than it costs to produce and sell. If the equation doesn’t balance properly, the product may be very desirable to customers but not economically feasible for the producer. Understanding the sources and magnitude of this value in very specific terms is part of developing the Value Hypothesis.

The Value Hypothesis is usually developed and tested using design methods. These methods are based on observational research (on-site insight), rapid prototyping of potential concepts, and iteration with potential users. Tom Kelley’s books, The Art of Innovation and The Ten Faces of Innovation, provide a primer on design methods [Kelley, 2002, 2005].

The Business Model Hypothesis

The Business Model Hypothesis is focused on how to capture value from a business concept. For a startup, creating a business model means starting from scratch, but for established companies, business model considerations can be more complex. In addition to being effective in the marketplace, the business model for the new concept must co-exist with the company’s existing business model. Unfortunately, a truly new business idea will often not fit nicely into the existing model, and it cannot be shoehorned into it. Trying to force-fit a new concept into the dominant business model is a common recipe for failure—either failure to create a new business or failure to capture a fair share of the value created.

Getting the business model right is very important. Adrian Slywotzky notes that the difference in value capture for two firms meeting the same customer need but pursuing different business models can be as much as a factor of five [Slywotzky and Euchner, 2015]. I have found this to be true. One startup I worked with had a great offering but was selling it on a tiered subscription basis. When we did the math, we found that they were capturing only 3% of the value they created for their best customer. Refocusing the pricing around a gain-sharing model permitted a tripling in revenue.

Startups often use the Business Model Canvas to develop their business models [Osterwalder and Pigneur, 2010]. The canvas is useful in opening minds to new possibilities and for representing the elements of the business model. It gets good conversations going. But the Business Model Canvas cannot do several important things. First, because the elements are brainstormed independently, it does not assure the coherence of the business model—that is, it does not guarantee that the elements of the business model will synergistically support one another. Second, the canvas is not quantitative; the elements are there, but the canvas does not help to understand the viability or dynamics of the business model. Finally, competitors are off the canvas; as a result, the canvas does not deal with creating a competitive advantage, which is a key element of any business model.

The first business model that occurs to you—the obvious one—may very well not be the best one. Seeking alternatives and understanding the pros and cons of all the possibilities is important. I recommend that new ventures explore at least three alternative business models.

Business model archetypes are an excellent place to start. Entrepreneurs and researchers have identified several dozen archetypes, each of which assembles the elements in a coherent way and—if properly executed—creates economic leverage. In The Art of Profitability, Slywotzky explores the dynamics of 23 business model archetypes [Slywotzky, 2002]. Slywotzky has also written about Profit Patterns and Value Migration as industries move from one archetype to another. Oliver Gassmann and colleagues, in The Business Model Navigator, identified 54 quasi-archetypes [Gassmann et al., 2020]. Learning archetypal business models is extremely useful in selecting and adapting a model for your new venture.

As an example, Goodyear developed a technology that could use real-time data and predictive analytics to prevent over 80% of roadside failures due to tire maintenance issues in truck fleets (see above). In creating the Business Model Hypothesis for the project, the team looked at three alternatives. One business model bundled the service with tire sales and generated profit via pull-through of a product; another model sold the component parts of the system to fleets so that they could implement the system themselves; a third charged for the service on a subscription model and provided the service for both Goodyear and competitor tires.

We analyzed the risks of each model and ultimately chose the third, which represented a marked departure from the product-centric business model of the core business. This model required a different approach to selling, a different revenue model, and different metrics than the company was accustomed to. Pull-through of tires was a consequence but not the driver. When we analyzed the economics and the risks, it was the only model that we believed would be sustainable. This model has enabled the business to grow and has been adapted to markets throughout the world.

The Growth Hypothesis

The Growth Hypothesis is a theory for how the business will scale. In startups, scale often happens organically: As sales increase, the startup reconfigures itself to manage them and seeks investment to accelerate growth. This often requires a significant infusion of cash to properly fund the elements of the business necessary for growth. Investments in the sales force, in infrastructure, in operations personnel, and in technology are often required. Moving too slowly or investing too little may mean that the venture misses its window of opportunity.

In established corporations, the Growth Hypothesis is often developed and validated in a distinct step: Incubation. Incubation has the advantage of testing the whole business proposition at a small scale and low risk. It validates first and foremost that customers will buy the offering at a price sufficient to sustain the business and, therefore, that the business can be made profitable.

Once profitability has been demonstrated, many options are available for achieving scale. In a large company, the options include investment for organic growth, reorganization of parts of the existing business to serve as a basis for the new venture, and acquisition—or a combination of all of these. The business-building strategy is, in essence, the validated Growth Hypothesis (see Figure 2.6).

For example, a company that had developed a new document management system for hospitals considered three options to drive growth. The first two envisioned building the business organically but with different levels of investment. In the first, a plan was developed to maximize value, which included substantial investment. In the second, the team sought to minimize cash-flow requirements and to grow incrementally. The third option was to build the business by acquiring another business with complementary capabilities. The acquired business would provide access to a customer base and a means of making the acquired business more profitable. The innovation team developed full financial projections for each option, as well as an analysis of the risks of each. These projections provided the basis for the scale decision.

The assumptions inherent in a Growth Hypothesis need to be tested with customers, just as the elements of the business model are tested. In the end, however, the decision about growth is constrained by two things: The economic dynamics of the business and the leadership team’s risk profile. Many internal ventures succeed or fail based on the willingness of executives to invest sufficiently in growth.

The Lean Startup is a system. Each of the elements is attractive as a stand-alone practice, but the value for new business innovation comes from adopting Lean Startup in its entirety. Developing MVPs without systematically testing them through Lean Learning Loops misses critical customer feedback. Developing radically new value propositions and then forcing them through an existing business model can make a viable business fail. Doing the work without a learning-based management system devolves into assessment by more traditional metrics and can lead to shortsighted decisions. Although the Lean Startup tools are each useful in thinking about innovation, it is the integration of them that leads to success.