The Best Marketers Don’t Guess, They Test
by Stefan Maritz, from CXL |
Marketing isn’t magic - it’s a disciplined science of experimentation. It’s easy to think of it as a creative pursuit, full of gut feelings and bold ideas. But at its core, marketing is a series of educated hypotheses put to the test. It’s a process of systematically uncovering what works to move the needle. Or, at least, when it's done well, it is.
The basics of marketing won’t win the race. It might get you to the starting line - maybe even through the first lap. But it's not enough to carry you to victory. No two businesses, industries, or brands are the same. There’s no universal strategy, no one-size-fits-all playbook that guarantees success.
What works for one company in one industry can completely fail for another. Even among competitors in the same market, subtle differences—like target audience behavior, positioning, or brand perception - can drastically change what drives results.
That’s why the best marketers test everything. They don’t rely on templates, trends, or intuition. They experiment relentlessly because the only way to know what works for your business is to test it. |
|
|
Guesswork is Expensive Guessing, and making calls without data is like throwing spaghetti against a wall in a dark room. You'll have no idea why something worked - or why it didn’t.
Thinking or assuming something might work is a gamble, and if it doesn't work it can be an expensive mistake. It has cost even some of the biggest brands in the world a lot of wasted time and resources.
To point out some examples here:
Dell launched a new website in 2017. Looked great, but revenue per user dropped by 33%. They had to roll back and start over. Losing on both the time and cost of the project, and the revenue. User testing could have indicated problems before going live, allowing them to fix it and avoid losses.
Pepsi’s infamous “Live for Now” ad campaign, which sparked backlash for being tone-deaf. The company spent millions producing and distributing the ad, only to pull it after public outrage. Proper audience testing could have flagged the negative reception before the campaign launched, saving Pepsi both money and reputation damage.
When you do research and test, It helps you understand what’s happening, where your efforts are landing, and what to do. It’s how you move beyond gut feelings and anecdotal evidence into measurable, repeatable growth. It also helps you avoid spending time and effort on what's very clearly not going to work.
|
Testing Applies to Every Part of Marketing Experimentation isn’t limited to A/B testing website changes or ad campaigns - it applies across the entire marketing funnel. Here are a few areas where testing can have a significant impact: - Traffic Generation:
Which channels bring in the most qualified leads or sales? Testing platforms like Google Ads, Facebook Ads, or LinkedIn can help you allocate your budget more effectively, increasing ROAS.
- Messaging and Creative:
Does your audience respond better to playful or professional copy? Testing can reveal tone and language preferences on your content and website.
-
Pricing:
Are you charging the optimal amount? Testing pricing tiers, bundles, and discounts can uncover what maximizes both revenue and conversions.
- Retention Strategies:
What keeps customers coming back? Testing onboarding sequences, loyalty programs, or reactivation campaigns can enhance lifetime value.
- Product Positioning:
What resonates most with your audience - features, benefits, or values? Testing different angles can sharpen your message.
The key to success in all these areas is a mindset of curiosity and a willingness to challenge assumptions. What works today may not work tomorrow, and what works for others may not work for you. Testing is how you uncover the unique levers that drive growth for your business. |
Growth is Built on Experimentation At its core, growth experimentation is about asking the right questions: - What gets customers to your website?
- What makes them stay long enough to explore?
- What convinces them to give you their email address?
- What motivates them to buy?
- What keeps them coming back?
The answers to these questions aren’t obvious. And even if they were, they’re constantly shifting as markets evolve and competitors adapt. Testing gives you a way to stay ahead. It’s not about being right all the time; it’s about learning quickly and applying those insights to drive better results.
Amazon’s recommendation engine is powered by ongoing testing and optimization. By experimenting with where and how recommendations are displayed - whether on product pages, during checkout, or in follow-up emails—the company has driven billions in incremental revenue. A/B tests have shown that even small tweaks, like emphasizing "frequently bought together" items, can significantly boost average order value and customer satisfaction.
Airbnb significantly improved host sign-ups by testing its onboarding process. A key experiment involved simplifying the initial steps, such as reducing the information required upfront and providing clearer instructions. This small but meaningful change resulted in a noticeable increase in new hosts listing their properties. It demonstrated how understanding and testing user behavior can eliminate friction and drive growth.
Spotify tested the effectiveness of personalized playlists like "Discover Weekly" by comparing user engagement metrics before and after its introduction. Experiments showed that personalized playlists increased listening time and reduced churn, encouraging Spotify to roll out even more tailored listening experiences. This data-driven approach to enhancing user retention has been central to their success in the competitive streaming market.
But experimentation isn’t just about running a bunch of A/B tests. It’s a methodical process that requires focus and discipline. Here’s what it takes to experiment well: -
Do your research: Run surveys, talk to customers and lost deals. You need to understand what your audience and customers want and need. Get that data from them directly. Get it in qualitative and quantitative formats. This will also help you formulate better hypos and smarter tests.
-
Define clear hypotheses: Before you run any test, you need to know what you’re trying to learn. What are you testing, and what do you expect to happen? A clear hypothesis keeps your experiments focused and actionable.
-
Design smart tests: Not all tests are created equal. Some experiments will yield clearer, more meaningful results than others. Learning how to design experiments that minimize bias and maximize insights is a skill in itself.
-
Measure what matters: It’s easy to get distracted by vanity metrics - impressions, likes, shares - that don’t drive real business outcomes. Effective testing focuses on KPIs that align with your goals, whether that’s revenue, conversions, or retention.
-
Learn from the data: Testing without analysis is a waste of everyone's time. The goal isn’t just to collect data; it’s to turn that data into actionable insights. What worked? What didn’t? Why? Use insights to inform your next move, not noodles.
|
Testing Requires a Shift in Mindset
One of the biggest hurdles to effective testing is buy-in and culture. Many organizations struggle to create an environment where experimentation is valued. Teams often feel pressure to deliver immediate results, which can make testing seem like a risk or distraction.
But the most successful marketers - and companies—aren’t the ones who avoid uncertainty; they’re the ones who prioritize a culture of learning and embrace data-driven decision-making. They encourage curiosity, reward learning, and give employees the freedom to experiment without fear of blame.
When everyone understands the value of experimentation and sees it as an investment rather than a cost, the entire organization becomes more agile and effective. |
The ROI of Testing
When done well, testing doesn’t just improve individual campaigns; it compounds over time. Each experiment builds on the last, creating a snowball effect of insights and improvements. Over weeks, months, and years, the cumulative impact of testing can be transformative.
Take Booking.com as an example, which runs over 25,000 tests annually. They optimize everything, constantly, continuously enhancing their user experience. While individual experiments may result in small gains, the cumulative effect has helped Booking.com drive higher conversions and sustained growth over the years.
Netflix uses testing to fine-tune its personalization strategies, from algorithm tweaks to the thumbnail images it displays. By ensuring users spend less time deciding and more time watching, Netflix has significantly improved user engagement and retention, reinforcing its dominance in the streaming space.
The alternative to testing? Guessing, copying competitors, or chasing trends. These strategies might occasionally produce wins, but they’re not sustainable. For instance, Gap’s 2010 logo redesign was launched without adequate audience input or testing. The backlash was immediate and severe, forcing the company to revert to its original logo within a week. This misstep resulted in wasted resources and damaged the brand’s reputation - an outcome that proper testing could have prevented.
At the very least it can help you generate more revenue from your current traffic and ad spend, but there's so much more to it. Testing provides something far more valuable than quick wins: it generates actionable insights that help businesses make smarter decisions, reduce inefficiencies, and reduce waste.
Companies that prioritize testing build a competitive advantage, learning from every iteration and compounding those insights into long-term success. |
The Bottom Line: The Best Marketers Don't Guess, They Test
Marketing is a moving target. What works today won’t necessarily work tomorrow. Trends fade, competitors adapt, consumer behaviors change, and sometimes, entire platforms evolve or disappear. Andrew Chen’s Law of Shitty Clickthroughs makes this clear: over time, the effectiveness of any tactic—no matter how innovative - declines as it becomes overused and saturated. The only way to stay ahead is to keep testing.
But testing isn’t just about keeping up—it’s about taking control. The marketers who consistently win understand this. They know that relying on yesterday’s best practices is not enough. Instead, they continuously adapt, experiment, and uncover new ways to drive growth. Testing allows you to break free from the cycle of diminishing returns and uncover fresh opportunities that others miss.
The marketers who excel - the really great ones - are the ones who commit to relentless experimentation. They understand that growth isn’t about being perfect; it’s about being curious, methodical, and persistent.
It's science, after all.
Regards,
Stefan |
A collection of deep, practical marketing courses to help you drive constant innovation to stay ahead of the market. - Practical training by industry-leading practitioners
- Learn how to scale growth through experimentation
- Learn what’s working with data & analytics
- Learn everything about running and scaling successful CRO programs
Learn more >
|
|
|
|
_01JGXFD8ZTK0X07F4P710PGXW3.png)