Chapter 7. Creating Prototypes for Experiments
“The premature outlay of huge amounts of money in pursuit of the wrong strategy is the thing to avoid. You need to have an experimental mindset.”[47]
—CLAYTON CHRISTENSEN
THE PREMISE OF LEAN STARTUP IS TO GET FEEDBACK EARLY AND OFTEN in order to validate that you are on the right path, which is also the fondation of Tenet 3.[48] Eric Ries and Steve Blank insist that it’s important to run experiments on your product as soon as possible. There is even a spinoff movement called Lean Startup Machine (see Figure 7-1) that holds worldwide events in which startups and product makers learn how to design, build, and test a Minimum Viable Product (MVP) experiment over a weekend.
A successful UX strategy also needs this continuous testing to ensure that your product will deliver a solution that people will really, really want. Thus, you need to jump from your storyboard to an MVP or prototype of your product. You’ll then take it into small, structured, lean experiments to learn as soon as possible if your team’s latest assumptions are on the right track and to force you to confront the reality of what it would actually take to make your business model work in the real world. This kick-starts the process of blending the ingredients that make up all four tenets of my UX strategy framework, as presented in Figure 7-2.
Giving It Your Best Shot
Years before my dad bought the doomed hot dog stand I discuss in Chapter 4, I watched my mom start and run a successful business out of her bedroom closet of our San Fernando Valley home. It was the early 1970s, and my mom, age 35, fell hard for the sport of tennis. This was the decade of tennis in the United States. Its popularity was fueled by the televised championships at Wimbledon as well as the US Open and the French Open, with competitors like John Newcombe, Ken Rosewall, and Chris Evert. Backyard tennis courts, country clubs with strong tennis programs, and tournaments popped up all across the country. In sunny southern California, tennis became an integral part of the upper-middle-class lifestyle. While I attended elementary school, my mom dragged my little brother in his playpen to the local park where she took lessons. She was a natural who developed a powerful slice return, and six months after starting lessons, she won her first ladies double tournament trophy (see Figure 7-3).
It didn’t take long before my mom began to look beyond the game. One day, while having lunch with her doubles partner, Lea Kramer, inspiration struck. The friends loved all aspects about the sport—on and off the court—but both agreed that it was almost impossible to find reasonably priced tennis clothes. A competitive audit of discount tennis clothing stores in Los Angeles assured the women they had stumbled on a blue ocean. They had found themselves a value proposition.
They each had $500 to invest. Neither of them had any retail experience. Lea knew how to do bookkeeping. My mom had only worked as a legal secretary and never attended college. But my mother was a hustler. She suggested that they experiment with their value proposition by doing a trial run. She reached out to a family friend to see if he could help them acquire some product. He worked in the “schmatta” (rag) business and knew his way around downtown Los Angeles clothing manufacturers. He also knew the movie star Elke Sommer, who agreed to sell the new business partners some of her new tennis clothing line at just over cost. With 4 dresses, 10 pastel skirts, and 12 bras, my mom and Lea just needed to find some customers.
They first started trying to acquire customers at the tennis courts, where they showed off their wares from the trunk of my mom’s Chevy Nova. But people really needed a private place to try things on. So they obtained local club lists with names and numbers of players they could schmooze into coming over to their homes. I remember coming home from school to find dozens of half-naked women parading around my mom’s bedroom while she urged them to try on different styles. Then, my mom and Lea were even invited to be part of a charity (donating 10 percent of sales) in a Beverly Hills home. They trucked their clothes to this wealthy customer segment, where they sold the product in a “pop-up” dressing room in the mansion’s garage. These experiments eventually made them a rousing success. Their entrepreneurial scheme was even featured as one of the “Ten Biggest Bargains in L.A” in LA Magazine.[49] Within their first year, they built up $10,000 of inventory and a customer base from all over, including Beverly Hills, Ojai, and Pacific Palisades. It was time to scale from their bedroom closets to a brick-and-mortar location on Ventura Boulevard. Love Match Tennis Shop (see Figure 7-4) was officially born!
The business was a break-even from the start, and my mom and Lea typically reinvested any profits back into purchasing more inventory. They enjoyed the discretionary income and the flexible schedule that let them raise their young children and also play tennis. After three years, they moved the store into a space three times the size in a new shopping mall. Love Match Tennis Shop continued its success, but after almost 10 years, my mom decided to move on. Lea bought out her share for an amount that they both agreed was fair. In accordance with the etiquette demonstrated at the end of a proper tennis match, they shook hands and thus concluded their business partnership.
How I Became an Experiment Addict
When Marc Andreessen coined the term “Product/Market Fit” back in 2007,[50] he said, “In a great market—a market with lots of real potential customers—the market pulls product out of the startup.” This aligns with Steve Blank’s Customer Development methodology, in which he insists that product makers begin building by understanding their customers’ problems, and then tweaking the solution to the customers’ needs. In a UX strategy, you do that by running experiments.
First, though, we need to define what exactly an experiment is. An experiment is a test of a hypothesis. The goal is to discover whether your hypothesis is right or wrong based on measurable results. After the experiment, you should be able to evaluate your results and accept or reject your original hypothesis.
Experiments are diverse and can be run in a laboratory or in the field. They can be controlled (run with a control group for comparison) or natural (run with no control groups). No matter what type, though, experiments are all about testing a variable in order to falsify a hypothesis. This variable is any item, factor, or condition that you can control or change. In observing the variables in the experiments, you look for a cause-and-effect relationship, and you conduct the experiment for a finite amount of time, because you want to measure and empirically capture observable evidence about what happens when the variable changes. It’s that simple. Or so you might initially think.
In early 2011, I was working full-time as an offsite UX strategy consultant for Cisco Systems. At the same time, I was looking for smaller, get-your-hands-dirty opportunities with local tech startups in Los Angeles. In March, I met Jared Krause, a brash, hilarious, articulate entrepreneur, and a fellow NYU alumnus to boot. He had big plans to make a fully featured online platform that enabled people to easily trade all types of goods and services, and he also had initial private investment lined up. Sure, there were other bartering/trading platforms out there, but none with a sophisticated mechanism for matching users based on common interests and geolocation. Other trading platforms such as BarterQuest and Swap.com had clunky interfaces with the type of inventory you might find at a yard sale. Jared wanted to create something groundbreaking.
I started working on the project immediately in addition to my full-time enterprise job, going through a divorce, and my son starting kindergarten. About six months later, we had completed the business requirements, a project roadmap, information architecture, and about 50 percent of the UX wireframes. Jared put together an impressive team including a specialist in artificial intelligence. Our value proposition was basically that it would be the “OkCupid for barter,” or in Jared’s words, “a dating site for your shit.” The big idea was that customers could make lists of all the things they had to offer and all the things they wanted. Then, the back-end algorithm would match up the appropriate people.
The project was ambitious and complicated. Figure 7-5 demonstrates that the transaction flow alone was complex.
Then one day, at the beginning of a wireframe review call, Jared told me I had to stop working on the UX. Instead I had to go read the New York Times best seller The Lean Startup. Over two days I listened to the audio version while hiking the Arroyo Seco trails in Pasadena. The concepts in the book jolted me awake to two scary realities. The first was that Jared and my course of action for the product drastically needed to change. This meant I had to throw out or put aside a lot of hard UX work. The second reality was that my practice of UX strategy and design methodology based on the traditional “waterfall” software development model was now outdated.
The rules had completely changed:
No more working on UX strategy phases geared toward a big 1.0 “launch.” Now, I needed to plan for making small incremental prereleases (MVPs) that articulated different aspects of the UX.
No more working in isolation and then just handing off documentation to the team (stakeholders, developers, designers). Now, I needed to continuously collaborate and strategize with them to ensure that the product was released as quickly as possible.
No more crossing my fingers and hoping that customers would love the product after it was built. Now, I needed to insist that my clients let me conduct experiments along the way that tested the UX and the value proposition.
Changing our process mid-project was very stressful. Plus, we were running out of funding, so Jared needed to bring in another round of investment. The investors wanted real evidence of a solid rationale for the potential success of our product. The big question Jared kept getting was, “Why would someone want to wait for a worthwhile trade for their old laptop when they could just sell it on Craig’s List and then spend the cash on whatever they wanted?” Jared and I needed to begin experimenting immediately with an MVP to show that our utopian vision wasn’t from Planet Utopia.
The challenge was to isolate a slice of the UX that would truly test the essence of the value proposition. We also needed something technically feasible that the developers could build in less than one month. Jared was confident that his own extreme marketing abilities would drive enough people to the landing page. Still, we needed a place for them to land that would make immediate sense and not feel like an empty shopping mall.
Because we didn’t want to wait for a month to learn if people followed through with the actual transaction in the offline world (which was outside of our online transactional funnel), the MVP also needed to yield immediate results for any successful trades. We needed to learn a lot from this experiment. The most important things we needed to learn was whether trades were more likely to happen if the person who wanted to trade something was open to a variety of commensurable offers versus a specific “want” item. Jared had a hunch that if we spotlighted just one trade per day and made it for something highly attractive to our hypothesized customer segments, we would have a higher guarantee of a successful transaction (a trade actually happening).
The most difficult part to determine was how the experiment would work without any backend engineering. It required us to focus on the key experience that provided the most innovation. For TradeYa, it was the capability of being able to get something you wanted without having to pay money for it. We needed people to want to engage in the act of trading with strangers. To ensure that all trades actually went through without a hitch, Jared intended to manually facilitate them. This could be as easy as helping both parties with the swap logistics over email or coordinating a time for them to meet up in front of a 7-Eleven with Jared as a mediator. Over one weekend, Jared and I sat side by side and knocked out the all the necessary UX documentation for the developers. In less than one month, “Trade of the Day” was born.
Figure 7-6 depicts the application maps of the before and after of the original TradeYa vision and the first MVP.
Figure 7-7 shows the home page wireframes of the before and after of the original TradeYa vision and the first MVP.
It’s pretty clear from just a quick glance at the UX documentation how much the product shrank. We basically had to figure out a way to enable a rudimentary trade on the frontend (the user interface) without requiring the developers to do any backend engineering (code and database). Our thought was that if Craig’s List had done perfectly fine without requiring users to even have accounts, the same could be true for the first version of TradeYa. As a result, I chopped out all the personalization and transactional wireframes—no user profiles, shopping carts, or user reviews. The MVP’s experience simply didn’t need it.
The product went from fat to lean; it was going to do a lot less but do it well. That was crucial, because we didn’t have the time or the resources to build out the original feature-heavy version. We also had the common two-sided marketplace issue of the chicken and the egg. Without users to trade “stuff,” who would come to trade with them? Thus, the age-old question: what came first, the trader or the tradee?
Here’s where Jared made the experiment more intense. He insisted that everybody on the team (investors, developers, designers, all of us) had to test this by coming up with goods or services to trade until each of us had completed a successful trade. I had not planned on getting my hands this dirty! I didn’t have any old couches or personal computers that I wanted (or needed) to trade. So I chose to trade on my UX skills. (See Figure 7-8.) My trade-of-the-day offer was two hours of my UX consulting time over Skype in exchange for either a) open to offers or b) the highly specific task of converting some old Flash animations of mine into YouTube videos.
It was scary yet fun. Moreover, it was truly like our original value proposition, a dating site for your shit. Within 24 hours, I accepted a trade offer from Edward, a digital consultant out of Portland. This was when I wholeheartedly connected the dots. I had firsthand experience that the value proposition and UX weren’t quite like eBay; rather, it was much closer to OkCupid. The trade was successful. Edward put my animated cartoon series on YouTube, and I schooled him on how to begin obtaining UX work in Portland. I even hooked him up with a job interview. The entire user journey was pretty magical. Our trade was worth more than paper money because we both got so much out of the exchange of our traded skills without either of us having to file a W9. The current and potential investors liked what they saw, and more funding was raised so that we could continue experimenting and learning.
As you can see from this case study, there are lots of different types of experiments that you might run and design to determine product/market fit—ones that don’t involve writing a single line of code. Here’s two popular methods:
- Online campaigns
Jared was confident in his ability to draw people to our new landing page through marketing, which meant that he would need to advertise TradeYa. The primary purpose of any advertisement, concierge MVP, explainer video, or Kickstarter campaign, is to measure if potential customers take action. They need to respond with that tentative first click. A positive response such as a click can track specific actions with bite-sized measurements called metrics. Metrics can tell us the following:
How many people landed on the YouTube video page?
Of those, how many people watched the entire video?
How much traffic was driven from the video page to the product site?
How many people submitted their email address for more information?
How many people made it down the entire sales funnel and signed up for a monthly subscription?
A truly successful conversion will engage people with your product’s value proposition and eventually convert them into customers. Metrics have the potential to tell us everything from the click-through rate (CTR) of an online campaign to customer satisfaction levels. (I discuss this in greater detail in Chapter 9.)
But online campaigns can also be used to pitch the MVP’s value proposition to users to see if they’ll “buy in” to the concept. This is usually a short-format “explainer” video or animation that explains the benefits of the product. You can find them on web pages, YouTube, or crowdfunding platforms such as Kickstarter and Indiegogo; and they are meant to draw investors, raise funds, test product appeal, and acquire users. Successful buy-in is indicated by customer suspects who provide their email address or other personal information. This type of feedback data is considered “currency.”
One of the most famous examples occurred in 2008 when Drew Houston released a three-minute screencast recording of a product he was working on called Dropbox. The screencast showed a simulation of the product’s value proposition: its functionality, ease of use, and benefits. The video went viral and almost overnight garnered Houston more than 75,000 signups[51] from early adopters eager for a product that had not even been built. Now, that’s some successful conversion results that show people want your product! This advertisement or “concierge MVP” (as Eric Ries called it in Lean Startup) primed the pump for startups to begin going ballistic running experiments using explainer videos. (See Mint, Crazy Egg, Dollar Shave Club, Groupon, and even Airbnb.) Figure 7-9 shows the explainer video that Jared produced for TradeYa.
Figure 7-9. Jared’s explainer video of the TradeYa value proposition (https://www.youtube.com/watch?v=ENBGDRHAJN4)- Concierge MVP
“Concierge” is a French word that essentially means doorkeeper. The job of a concierge in a hotel or residential complex is to ensure that the experience of its customers (tenants, guests, whoever) is frictionless from the moment the person enters the building (or service or product). When I speak about Concierge MVPs, I’m referring to an attempt to simulate an aspect of the customer’s experience without the interface and to do it with as little friction as possible. You are simply de-risking your product by simulating as many of the frontend key experiences as possible without the interface backend to see what goes wrong and what goes right. When you don’t have the time or resources to build a backend, creating a Concierge MVP is the next best thing.
This is exactly what Jared did while we worked on cutting off a lot of feature weight. He personally facilitated trades between users—by phone, by email, and in person—without the use of the backend to do it automatically. There are many other examples and articles that describe twenty-first-century concierge experiments concocted to test value propositions of products such as Car’s Direct, Zappos, Food on the Table, and even Airbnb. My take is that early adopters are simply fascinated by technological innovation and open to experiencing something new, even if there is some trickery involved.
But this isn’t anything new; innovators having been “faking it,” and users have been “falling for it” for a long time. In 1769, an inventor named Wolfgang von Kempelen performed a demonstration of his automaton chess-playing machine for the empress of Austria. The Mechanical Turk, or “The Turk,” toured Europe for decades playing chess and defeating challengers including statesmen such as Napoleon Bonaparte and Benjamin Franklin.[52] Even Edger Allen Poe published a theory speculating that the chess moves were actually determined by a legless war veteran who could fit in the tiny space inside the cabinet. What users didn’t know was that it actually was a chess master hidden inside the machine, but in a much larger secret compartment concealed by folding partitions (see Figure 7-10). Users were playing him rather than it.
But the illusion really got people to think. It got mathematicians thinking about mechanical computers. It also gave audiences at the dawn of the industrial revolution a taste of the value proposition of the supercomputer. Today, it’s feasible for any aspiring entrepreneur to manually simulate a complex digital product. Amazon’s Mechanical Turk[53] offers a crowd-sourcing marketplace with which businesses can outsource microtasks to individuals. The platform provides on-demand access to a diverse and scalable workforce that takes advantage of human intelligence to mimic backend technology. This is the same as how Jared personally enabled the transactions behind the scenes of TradeYa before the larger product was reworked to align with our new vision. Whatever magic you decide to perform, concierge experiments are less risky than building out the entire product and then seeing what happens.
Testing Product/Market Fit by Using Prototypes
TradeYa already had a digital presence when Jared and I decided to backtrack and build an MVP. But, what if you don’t have an actual website yet? What if you just have a storyboard and an idea? That’s when a prototype comes in handy. The goal of a prototype is to avoid coding and designing until you have true validation that your solution is desired and can be sustained by your hypothetical customer segment.
A prototype is anything that serves to familiarize the user with the ultimate experience you are trying to create. These can be low-fidelity paper prototypes or high-fidelity mockups. In the tech industry today, prototyping is a big deal. Digital teams create highly detailed ones in programs such as Axure or OmniGraffle. These prototypes can be useful for tactical usability testing and conveying functionality to the development teams. Conversely, they can easily become resource overkill just making something “clickable” to convey a strategic concept. In fact, you might have noticed how I haven’t mentioned wireframing (the typical UX deliverable) at all as something to do for strategy. Most customers and stakeholders can’t look at a wireframe and truly “get” the experience. If you leave important ideas up to their imaginations, you risk some customers’ disapproval because they can’t understand it.
I believe a prototype that you don’t learn from is a waste of time. In Chapter 6, I talk about the reasons you needed to focus on key experiences. Now, you’ll use your storyboard as the starting point for creating a solution prototype so that you can run an experiment. You will use it to show proof-of-concept to users for when your team goes out to do guerrilla user research, which we look at in the Chapter 8.
Three Steps to Design Hacking the Solution Prototype
I used to call a solution prototype the five-screen MVP, but I have moved away from that because, honestly, a prototype is certainly not “viable” or a “product.” Its sole purpose is to force your team to create the minimum amount of screens necessary to demonstrate the key experience of the interface, the value proposition, and a sneak peek of the potential business model of the product. It pushes your team to take your innovative storyboard to the next level.
However, like the storyboards, solution prototypes are not meant to be pixel-perfect for development. The interfaces and artwork can be cut and pasted from other existing websites, thus requiring minor typesetting of the content. You are using the output to conduct an experiment to get customer validation. The prototype is not the final product, although it might inform, inspire, or annoy the future designers of it.
Here is the general framework of the solution prototype screens and their content. The order and amount of the middle screens is flexible based on how many key experiences you need to show.
Setup. Typically the landing page or user dashboard.
Key UX 1. Typically in one to three screens, shows the crucial interactions that show value innovation.
Key UX 2. Typically in one to three screens, shows the crucial interactions that show value innovation.
Value proposition. This is the final result of a successful transaction.
Pricing strategy (if applicable). This shows the cost of the app, monthly fees, package costs, and so on. If the revenue stream of the product is advertising, then you should consider putting examples of how ads might look on the prior screens.
Let’s return to Bita and Ena, my students whom I introduced in Chapter 3, to see how to make a successful solution prototype.
Step 1
Write a simple list or an outline of the screens you are going to show. It’s going to potentially include one or two key experiences. As with the storyboard, you want to really distill out the details that you need to show.
Bita and Ena wanted to show the screens from their storyboard but with more details. This was the list they wrote:
The landing page with the user’s query entered
The result set with the listing and the maps and the filters exposed
A filtered result set with the listing and the maps and the filters closed
A result detailed screen
A photo gallery featuring images of a home, specific to the wedding
The tour setup screen
The confirmation screen with final pricing, including cost of service
Notice how all the steps of the solution prototype refer to the experience online. That’s because this is what Bita and Ena need to test: does the digital product solve the customer’s problem?
Step 2
Start creating mashing up the images that tell the story.
Chapter 6 points out that there are lots of well-thought-out UX and UI design patterns that you can safely reference at this stage in your process instead of building them from scratch. However, try as hard as you can to make your prototype seem real, which as I mentioned earlier with the Mechanical Turk is key to making a simulation of a machine pass muster.
Figure 7-11 through Figure 7-18 show what Bita and Ena’s ultimate solution prototype looked like. If you look closely, you will see that most of the ideas were borrowed from competitors and influencers.
Screens 1 through 5 are Airbnb concepts reworked with content.
Screen 6 is the DIRECTV package retooled for a wedding package.
Screen 7 is the appointment booking system lifted from Apple’s Genius bar.
Screen 8 was custom designed.
Feel free to create the first version using whatever tool (pencil, whiteboard, Photoshop, or whatever works for you) will help you and your team work the fastest. When you know what’s going on in the entire solution prototype, it’s time for you or the visual designers to get to work.
Step 3
Paste all the screenshots into the presentation tool.
If I am working on my own, I do mockups in Photoshop. If I am working with a team of designers, I advocate building the prototype by using Google Presentation. This helps the team easily collaborate as we build out different screens. The Google Presentation prototype also outputs perfectly to PDF format, as well, so it’s easy to distribute. Bita and Ena decided to build their linear solution prototype in Photoshop and then exported the final product to PDF.
This is an important step, because the point of the presentation isn’t to show the solution prototype to stakeholders. It’s to show it to customers. The users need to feel like they are using the interface even if it isn’t interactive. Having a PDF on a pretty color screen like an iPad can be very effective in guerrilla user research (I discuss this in more detail in Chapter 8). The participant can swipe the screen at her own pace. She can move forward and back so she understands the narrative linearity of the key experience. Based on this minimal interaction, you will get great feedback. For instance, when Bita showed the solution prototype to a bride-to-be and her groom, they loved the idea of being able to have a fancy wedding in an affordable home. They had used Airbnb and understood the way private subletting worked. However, they noticed that they couldn’t set up a tour in the home through the solution prototype interface. Bita and Ena hadn’t thought of that. Consequently, they adjusted it to include a calendar function with which users could pick tour dates, as illustrated in Figure 7-17. They then ran another round of experiments to show their new prototype to five more targeted customers to validate that the feature was more than just a “nice to have” add.
Solution Prototype Reality Check: Why User Experiences and Business Models Must Go Hand in Hand
Remember the Business Model Canvas in Chapter 2? Let’s zoom into the left side of the canvas and take a look at Figure 7-19. We need to consider how those components—key partnerships, key activities, and key resources—might affect the composition of your solution prototype.
You need to keep your team honest about the logistics behind your prototype concept. If this is the simulated digital solution of your interface, are you hyperaware of all the things that need to make it work behind the scenes? Moreover, are those things feasible and sustainable? In the case of TradeYa, obviously our concierge model was not sustainable. If we wanted to scale, we couldn’t rely on Jared personally facilitating each and every transaction. He couldn’t hand-hold every new user through the anxiety of making their first successful trade. So, we had to come up with a better option. (More on that in Chapter 9.)
In the case of Bita and Ena, before they showed their prototype to real users, they created a list of questions and issues that were specific to each step. Among the questions they asked themselves were who the key partners are. What are the key activities? And what and from where would they get the key resources? To get those answers, they asked these questions:
- Screens 1 through 5
Where would we get the initial inventory of big houses to make it work? Can we initially build our inventory of houses by contacting people on Airbnb or other sublet listing sites?
- Screen 5
How are we going to get the photography good enough to entice customers to book a private home? Can we (like Airbnb) hire a photographer to shoot the backyards and rooms?
- Screen 6
How are we going to select the party vendors for the food, valet, and flowers? Should we go with fewer than a dozen to begin? How will we establish those partnerships? Is it better to focus on local, small franchises who already cater to specific areas?
- Screen 7
What are the most likely time-slot breakdowns that hosts and guests will choose for private tours? Is what is shown the best breakdown or can it be limited to weekends?
- Screen 8
Is it even possible to generate this type of total cost for a wedding package? Is putting the word “estimate” in there a better safeguard or will it turn off the guests who have a certain budget? What is the reality of creating packages with so many variables?
- Overall prototype questions
What would make this solution easier to trial? What if we zoomed into a specific segment of the market, such as just Santa Monica or beach areas in Los Angeles? How many people would it take to truly organize this major event when it actually occurs? Do we potentially need a wedding planner who will orchestrate everything? Or, is this platform to cut out the middle man (wedding planners) and to create a turnkey solutions to save costs to the brides?
The answers to these questions change over time. For instance, where do Bita and Ena get the homes to showcase on the site? Initially, they would probably need to solicit hosts personally—much as my mother and her partner Lea did for their first clientele. As they begin to scale, Bita and Ena would need to use online advertising to bring users to the first iteration of their digital interface. They would need to sell those users the value proposition, which would cause them to join the service.
Before I get into conversion (Chapter 9), it’s time to learn how to test drive your solution prototype by running face-to-face experiments on customers. It’s time to engage your potential users in a guerrilla user research attack.
Recap
The moral of this chapter is: don’t burn your time, money, or efforts on a product no one may want to use. Instead, concoct ways to develop and run small, structured experiments to gain validated user learnings. Do something to concierge the experience of a new product, even if it means selling it out of your own bedroom closet.
Most experiments fail, so just think through the outcomes and focus on the worthwhile takeaways. Sometimes, the results will not be black and white. Thus, you will need to interpret them by having collaborative team-debriefing discussions following each experiment.
[48] Ries, Eric. Lean Startup. HarperBusiness, 2011.
[49] Rena LeBlanc, “The Ten Biggest Bargains in L.A.” LA Magazine, July 1973
[51] Ries, Eric. Lean Startup. HarperBusiness, 2011.
[52] Standage, Tom. The Mechanical Turk. The Penguin Press, 2002.
[54] Kelley, J.F. “An empirical methodology for writing user-friendly natural language computer applications.” Proceedings of ACM SIG-CHI ’83 Human Factors in Computing Systems (Boston, December 12-15, 1983). New York; ACM: pp 193-6.
[55] Baum, L. Frank. The Wonderful Wizard of Oz. George M. Hill Company, 1900.