Booking.com’s most consequential product lesson was not how to ship more. It was how to decide whether shipping mattered. By making experimentation a shared decision system rather than a specialist ritual, the company shifted roadmap authority from hierarchy to customer behavior.
What Booking.com changed
Lukas Vermeer joined Booking.com in 2013 as a data scientist. His task was to improve products: generate ideas, develop new features, create variations of existing functions and test them. The company’s extensive use of A/B testing soon revealed an uncomfortable truth. Many of the features Vermeer built had no meaningful effect on customer behavior.
That insight remained central throughout his eight years at the company. His team grew from three to thirty people, and he became Director of Experimentation. But the organizational achievement was more important than the team’s growth: Booking.com turned testing into a way of deciding what should reach customers.
Harvard professor Stefan Thomke describes how broadly that principle applied. Anyone could start an experiment without management approval, while anyone in the company could stop an experiment started by someone else. The organization called this “pushing the nuclear button.”
The arrangement shifted the roadmap’s center of gravity. A senior executive’s support, an urgent request or a confident presentation could no longer settle the matter by themselves. The decisive question became: does this solve a customer problem and change customer behavior?
The roadmap zoo
Most organizations begin with a different question: which solution should we build? That question invites a familiar cast of characters.
HiPPO, or Highest Paid Person’s Opinion, gives the highest-paid person in the room the final say. RHiNO, or Really High Value, New Opportunity, favors the loudest, most urgent idea, often before its claimed value has been checked. ZEBRA, or Zero Evidence But Really Arrogant, treats conviction as a substitute for evidence.
The consequence is a roadmap zoo in which hierarchy, volume and confidence matter more than demonstrated customer benefit. Velocity, story points, the number of delivered features and roadmap completion become practical definitions of success. The team can look productive while customers receive little benefit.
A 2019 analysis by analytics provider Pendo estimated that up to 80 percent of features developed for SaaS applications are rarely or never used by end users. The estimate highlights the expensive gap between development activity and customer value.
Agile methods can organize changing work, but delivery activity alone does not prove that the work deserves to exist. The problem is not necessarily the framework. It is the substitution of output measures for outcomes.
Define value before choosing the solution
The word “value” creates another layer of confusion. Leaders often mean business value such as revenue. Product teams may mean customer value, but interpret successful releases as customer benefit. Customer service, IT, security and HR can mean reduced complaints or fewer critical incidents.
Those definitions may look separate, but they begin in behavior. Customers book more often, stay longer, return or recommend the service again. Those actions produce commercial consequences. If a product makes customers more successful, it creates value.
Jeff Gothelf and Josh Seiden capture this in their book “Who Does What By How Much?” The framework asks three questions: Which customer are we considering? What observable behavior do we want to influence? By how much should that behavior change?
This sharpens roadmap discussions. “Are users satisfied?” is too vague. “Do more customers complete a booking, return within a defined period or use the new function again?” can become a testable hypothesis. It also clarifies which customer matters: a hesitant first-time buyer and an existing customer considering another provider require different interventions.
Internal products follow the same logic. A tool used by operations teams should change how colleagues work, not merely reach its rollout deadline.
What experimentation changes
Experimentation does more than generate evidence. It changes who loses the right to control the roadmap.
The loudest idea does not automatically win, but neither does every proposal receive equal attention. Leaders still set strategy; customers reveal needs; data tests assumptions. The change is that evidence must mediate the final decision.
The trade-off is equally important. Testing before a full rollout may delay or prevent a release, but it can also protect development capacity from a permanent product commitment. A well-run experiment that disproves a hypothesis produces useful information. It should not be treated as failed work.
Stopping experiments also needs clear criteria. Without them, “decentralization” can become another way for powerful people to intervene. The relevant questions concern customer harm, evidence quality, technical risk and what the organization can learn—not who shouted loudest.
A decision checklist for roadmap reviews
Before approving a feature, require answers to the following:
- Which specific customer problem are we addressing?
- Which customer segment is experiencing that problem?
- What observable behavior should change?
- How large must that change be to matter?
- What evidence will show whether the change occurred?
- What is the smallest responsible test?
- Who may review, revise or stop the test, and against which criteria?
- What will we do if the evidence contradicts the hypothesis?
This checklist does not prescribe one experiment format for every context. It forces the roadmap discussion to connect a proposed solution with observable value before delivery consumes resources.
Change the planning conversation
If value appears in behavior, planning must begin with the intended outcome rather than the desired feature. This shifts the order of the conversation: identify the customer problem, state the expected behavior, determine the evidence, then select the solution and test.
The practical lesson from Booking.com is not that every organization needs an experimentation department. It is that shipping should never be confused with value.
Watch whether roadmap decisions increasingly include a hypothesis and a measurable behavior. Watch whether teams can stop weak ideas without political damage. Watch whether leaders reward learning as well as delivery. And watch what happens when delivery and customer outcomes point in different directions.
If teams are producing more features but customers are not behaving differently, the bottleneck is not velocity. It is evidence.
Key takeaways
- Define value as a change in customer behavior, not a feature release or roadmap completion.
- Separate outcome measures from delivery metrics so speed cannot disguise ineffective work.
- Give teams clear criteria for starting, reviewing and stopping experiments.
- Treat a well-designed negative result as useful learning rather than team failure.
- Require a hypothesis and a test before committing major delivery capacity.
Frequently asked questions
Is A/B testing required for every agile team?
No. The broader principle is to test whether an investment changes the intended customer behavior. The method should fit the problem, the available evidence and the risk of the decision.
How do you measure customer value without creating too many metrics?
Start with the customer problem and the behavior the team wants to influence. Choose measures that are specific enough to guide a decision rather than collecting every available data point.
What if a senior stakeholder rejects an experiment?
Make the decision criteria explicit before testing and discuss the evidence, risks and expected learning. This allows authority to be challenged without turning every experiment into a political contest.
Does customer value experimentation mean less agile project management?
No. It gives agile delivery a stronger purpose. Teams still plan, build, test and adapt, but their work is judged by the outcome it produces rather than output alone.