As marketers, one of the most humbling lessons experimentation teaches us is that no matter how much knowledge or experience we have, data can always surprise us—and sometimes prove our assumptions wrong. However, these experiences also help us evolve, pushing us to refine our strategies, challenge our thinking, and overcome impostor syndrome that often holds back bold ideas.
For many CRO specialists, the path to mastering experimentation is paved with transformative “A-HA” moments and hard-earned lessons. They’ve learned that the key to growth isn’t just about knowing what works, but about continuously testing, learning, and adapting. No matter how far you’ve come, the learning never stops.
To inspire and guide you on your own CRO journey, we asked several leading experts to share their most valuable lessons by asking these 3 questions and share them with you on this post:
- What was the biggest A-HA moment in your experimentation journey?
- Which experiment did you/ your team run that made your impostor syndrome disappear?
- When have you started your journey in experimentation?
Let’s dip our toes on the journey of the greatest experimentation specialists.
Ronny Kohavi: When We Realized that Over 50% of Our Experiments Failed to Improve the Metrics
What was the biggest A-HA moment in your experimentation journey?
Undoubtedly, it was when we aggregated the results of multiple experiments at Amazon, and realized that over 50% of our experiments failed to improve the metrics they were designed to improve. The results were humbling, and when I mentioned that statistic when I moved to Microsoft as a reason to build an experimentation platform, the typical answer was: we have better PMs. That denial is very common. After leading Microsoft’s experimentation platform, the overall success rate of experiments at Microsoft was about 33%, and at Bing it was about 15%
Which experiment did you/ your team run that made your impostor syndrome disappear?
The opening experiment in my book and in my online class is about a small UI change that resulted in a 12% increase to revenue. After multiple replications and validations, and financial audits, we realized that the results are indeed real and we need to shift our thinking about the value of UI changes. When have you started your journey in experimentation? I ran some email experiments, but the real usage of online controlled experiments, or A/B tests, started at Amazon to try personalization ideas and home-page ideas.
Fosca Fimiani: When I Got That My Idea for a Better Site Doesn’t Really Align With What Users Want
What was the biggest A-HA moment in your experimentation journey?
Sometimes you realize that your idea for a better site doesn’t really align with what users want. So, you need to set aside your wonderful castle and perhaps opt for a straw house that’s better suited to the purpose
Which experiment did you/ your team run that made your impostor syndrome disappear?
One day, we proposed a test to one of our clients: moving a carousel from the middle of a PLP (Product Listing Page) because, based on our ideas and benchmarks, we believed it was blocking the view of the products and that removing it might increase the conversion rate (CR). We set up an A/B test, removing the banner and leaving the PLP with only the products. What we discovered was the exact opposite— the conversion rate decreased! We reverted the page to its original design, but it demonstrated to both us and our client how crucial it is to test an idea before implementing it in production, as the results can sometimes be the opposite of what you expect!
When have you started your journey in experimentation?
Kelly Wortham: WYSIWYG Testing Actually Caused a Lot of Bad Decision-Making
What was the biggest A-HA moment in your experimentation journey?
The moment I realized that “making it easier” for companies to make decisions with WYSIWYG testing actually caused a lot of bad decision-making – because a lot of folks didn’t understand the underlying stats and were making BAD decisions.
Which experiment did you/ your team run that made your impostor syndrome disappear?
Using meta-analysis – and triangulation – we had tons of research (and previous test results from elsewhere in the business) that told us that we should absolutely run this test on the homepage now. but – home page real estate is shared between lots of teams – and no one wanted to fight that battle. so – I did. and fought hard. And it won. And they still wouldn’t believe it. So we ran it again. And again. And again. It won EVERY TIME. And in every country. So, they finally implemented it – and saw HUGE UPLIFTS.
When have you started your journey in experimentation?
Well, I was doing A/B testing OFFLINE in academia before entering the digital space. But my first digital test was in the early 2000s with an email subject line test. I lucked into it – I ran an email for that company and I had a history of teaching quat stats in college and knew about these things called t-tests we could do to find out if group a performed better than group b… and I thought… why not do that with subject lines??? We analyzed the data in Excel.
Brian Massey: It was the first time I've subjected one of my most brilliant design changes to an A/B test.
Brian Massey is the founder of Conversion Sciences and the author of the book Your Customer Creation Equation. Since 2007, he has worked with hundreds of companies to improve their online business. He is a sought-after speaker, presenting at IBM, Inbound, LeadsCon, Content Marketing World, Affiliate Summit, and others.
What was the biggest A-HA moment in your experimentation journey?
Easily it was the first time I'd subjected one of my most brilliant design changes to an A/B test. The new design was sparkling with my brilliance, dripping with data, a sure win! And then it didn't beat the control. This was the first of many humbling A/B tests, each teaching me a lesson in trusting the science.
Which experiment did you/ your team run that made your impostor syndrome disappear?
We had been working with Galeton.com for almost six years when it became apparent that the site needed a complete re-templating to support wider and smaller screens natively. Obviously, we were terrified that changing so much would reverse some of the gains we'd made. So, we implemented a complete site redesign in the A/B testing tool and A/B tested the entire redesign. While we would have been happy with an inconclusive result, we saw a 13% increase in revenue per visit. We were peacocking for a while after that.
When have you started your journey in experimentation?
It was a seminar at the enigmatic business school "The Wizard Academy" in 2006 that introduced me to the concepts of applying behavioral science to websites. As a computer programmer turned corporate salesperson turned marketing person turned entrepreneur, I couldn't think of anything I was more qualified for than "Conversion Scientist®". It was a long process of educating the market before CRO became cool. I'm glad I waited it out.
Ole Gregersen: When I Discovered That it's Not About the HiPP, But all About the O! Opinion
What was the biggest A-HA moment in your experimentation journey?
When I discovered that it's not about the HiPP, but all about the O! Opinion, or knowing better or thinking that one can predict outcomes aka the future, is the real issue here. Our tradition, experience, culture, educational system, and how we organize businesses - it's a lot about opinion. But when we optimize, opinion must step back to data and evidence.
Which experiment did you/ your team run that made your impostor syndrome disappear?
All tests - done right - deliver only positive results. Whether you "win", lose, or get an insignificant result, it is valuable for your business. No need for imposter syndrome at all then. Just trust the process, as long as you respect the ground rules. The core principle of optimisation is so strong and so self-explanatory, that the most important thing is to stand tall a remember what this game is about. The rest is mechanics, methods, skills, and so forth, which you are rarely in charge of anyways.
When have you started your journey in experimentation?
I was part of the whole WWW thing when it started. Learning HMTL, burning CD-ROMS, and animating in Flash. The CRO journey started in government big tech software development. It was back when user testing was new and UX was a hip thing somewhere in the States.
We fought the same battles back then, just for user involvement and user testing. History repeats. Then I moved to a travel agency and suddenly success was a direct result of user behavior. This opened my eyes to the importance of optimizing for the buying journey and thus for human decision-making.
To me, understanding humans and why they do what they do, opened the world of experimentation. You can say it was a welcome add-on to my existing skill set. One day we started calling it Conversion Optimisation, roughly around 2012.
Carlos Trujillo: I Realized MOST of the Product Changes Don't Really "Move the Needle"
What was the biggest A-HA moment in your experimentation journey?
After I started testing actively and consistently (through my agency role), and seeing the results of hundreds of experiments, I realized MOST of the product changes don't really "move the needle'. Most tests will end with inconclusive results when you do it with professional standards. That's been one of the biggest A-HA moments in my career.Before being exposed to so many controlled experiments, I thought everything was relevant and every minor change would make a difference. When you're able to see real data you learn that's far from true. This is one of the biggest reasons why we should test!
Which experiment did you/ your team run that made your impostor syndrome disappear?
An ecomm client was driving a ton of paid traffic to a landing page with a very specific offer. After doing some research, we learned a big chunk of these visitors were eventually returning to the site and got frustrated because they could no longer see the offer they were previously exposed to. We then introduced a treatment to help those returning customers find the offer they were previously exposed to in an easy way and it made a huge impact on the business. Definitely a good one!
When have you started your journey in experimentation?
I was running a web development agency focused on Shopify projects and had a top-level understanding of what CRO was. Then, when looking to upskill, I enrolled at cxl.com as a student and got so crazy with the content that I completed all the pieces of training available on the platform in 11 months. This eventually got me hired by Speero (a sister company) where I've been doing experimentation full-time since 2021.
Nicolai Høg: Even a Failed Test is Valuable Because it Teaches You Something New
What was the biggest A-HA moment in your experimentation journey?
My biggest A-HA moment came from working in CRO and Experimentation. Many just run tests without structure, hoping something sticks. It’s like comparing a precise sniper to someone wildly firing a machine gun. Sure, both might hit the target, but one wastes far more time and resources with misfires.
CROs, on the other hand, approach testing with discipline. They don’t just test to win; they test to learn. Even a failed test is valuable because it teaches something new and informs the next move, increasing the chances of success down the line.
Which experiment did you/ your team run that made your impostor syndrome disappear?
A client had a strategy for improving their mobile performance. And we got involved as part of that. It was your average case of a website that was built for desktop and jammed into a smaller screen. The client consists of 5 product verticals, and the hero section was displaying a campaign for one of them.
The navigation had all of the product verticals easily accessible, so positioning one of them wasn’t necessarily a bad thing. If a visitor was motivated towards another vertical than the promoted one, it was just 1 click away. On desktop. On mobile, however, the entire first screen had this hero section with 1 vertical promotion. And the easily accessible navigation had now turned into a burger menu of madness.
Disabling even the highest motivated visitors to interact. What we did was minimize the hero section on mobile and create a module with links to all 5 vertical’s front pages. Giving visitors the opportunity to design their journey from the homepage on mobile. And the results of this test was beyond incredible. While we had softer metrics than revenue as our primary focus, like bounce/exit rate and avg. number of pages seen, we saw a positive impact throughout the entire journey. And because the test was coded with easy implementation in mind, the solution was up and running in no time on the actual website after the test ended.
Our obsession with understanding the key problem paid off. And it’s this obsession alongside the methodology of testing that has made me lose any imposter syndrome years ago.
When have you started your journey in experimentation?
I started as an intern in 2015 in the same team I am now managing. Back in 2015, I had no idea what CRO was, but from studying a Master in Communication and Tech, I had sniffed up stuff about Analytics, User Experience, and Web Development. Not knowing what I wanted to do as an internship, which was a mandatory part of my education,
I wrote to GroupM in Denmark.
I remember I wrote something like: “This is what I can do, and what I’ve studied. Where can you use me?” Thankfully they found a chair and desk for me. A new team was created as an extension of the SEO team: CRO. And everything just clicked for me. In a good way. Suddenly I knew what I wanted to work with. What my career should be. After my internship, I worked in a couple of media, marketing, and web agencies then returned to GroupM 6 years ago.
To lead the team I had left as an intern. Now 6 years later we are more than 20 people working in a mix of CRO, SEO, UX, and dynamic creatives.
Luka Nikolić: Realizing That Conducting Usability Tests can be Useful
What was the biggest A-HA moment in your experimentation journey?
Realizing that conducting usability tests can be so useful! Even talking to a few actual users of a website can yield so many quality optimization ideas. And the best part is – the simplest of tweaks based on these ideas can result in the biggest performance uplifts, you just have to make sure you’re testing the right things!
Which experiment did you/ your team run that made your impostor syndrome disappear?
I’m currently running an experiment in the cart stage of an ecommerce website, based on an insight I got from one of the usability testing sessions mentioned above. The test is still running but the revenue per visitor uplift so far has been over 20% and with statistical confidence – hoping that it stays that way all the way through! The funniest thing is the change which is a simple rewording of an upsell offer – now that people understand what it is, they’re adding it to their cart much more frequently!
When have you started your journey in experimentation?
By accident, just like everybody else haha! I was at first attracted by UX design, trying my hand with that before getting more interested in analytics and copywriting. I somehow stumbled into a CRO role in a Danish iGaming company called Better Collective, and the rest is history, as they say 🙂 It was the perfect marriage of all my interests, and the first role where being a generalist was actually a requirement to be able to do the job well. I consider myself something of a jack of all trades so that was the perfect job fit for my ADHD brain!
Guido X Jansen: I Should Have Spent Much More Time on Internal Stakeholders Management and Promoting CRO
What was the biggest A-HA moment in your experimentation journey?
I once started working at a company in a new role aligning global CRO programs. We made great progress aligned with my assignment: centralized tooling and documentation, greatly increased the amount and alignment of A/B tests, and collaboration between 15+ countries… only to be let go after 11 months. My assumption was that since I was hired to do this, I could just focus on the execution. A critical mistake it turns out. I should have spent much more time on internal stakeholders management and promoting CRO and our results towards higher management. Lesson learned.
When have you started your journey in experimentation?
In 2005 at the University of Utrecht, first introduction on how to create good experimental setups and validating hypotheses.
Michael Aagaard: Basing Experiments on Real User Research and Actual Hypotheses
What was the biggest A-HA moment in your experimentation journey?
When I went from testing “good ideas” to basing experiments on real user research and actual hypotheses. It completely transformed my work and not least the results I was able to generate for clients (both in terms of learning and business impact).
Which experiment did you/ your team run that made your impostor syndrome disappear?
Back in 2013, I was working for a client with a subscription-based business model. I led a project where we did in-depth user research that resulted in radically different product pages that were focused on answering questions, addressing barriers, and reinforcing motivation.
It was a bold project and the treatment pages were radically different from anything the company had ever done. We ran a series of experiments to validate our hypotheses and saw impressive results that had a profound, positive impact on the business. Probably one of the proudest moments of my CRO career.
Gabriel Marçal: Conversion Rate is Not Tied Directly to Sending More Customers Directly to the Conversion Path
What was the biggest A-HA moment in your experimentation journey?
My A-HA moment was when I understood that conversion rate is not tied directly to sending more customers directly to the conversion path when I was a rookie. In the case of products/services that involve more complexity, it’s important that you can make them aware of what your business can offer them, build trust by doing so, and then show them the way how they can purchase/sign with your service.
When have you started your journey in experimentation?
I wanted to pay for a course, I was in university, and I started looking for internships. Found this opportunity to be a developer for a big agency in Brazil, and passed the interview, on my first day, they said I could choose to be a developer or be their first CRO intern. Since 2021 I’ve been working with CRO, and it has been quite fun.
Martin Greif: Go Big or Go Home!
What was the biggest A-HA moment in your experimentation journey?
Go big or go home! For years, we concentrated on granular, incremental tests, tracking as many goals as possible. However, our biggest A-HA moment came when we shifted to a more strategic, big-picture approach. Today, when clients approach us for conversion rate optimization (CRO), we start by looking at the entire customer journey from a much broader perspective. In about 70% of new clients, we’ve discovered that there’s a greater opportunity for transformational leaps in conversion rates rather than just incremental gains. Instead of focusing on individual elements, we now combine multiple improvements into one comprehensive test.
Which experiment did you/ your team run that made your impostor syndrome disappear?
We were confident we had identified a “guaranteed win” for a client, but they pushed back on several aspects of the test, citing “brand discrepancies.” We made numerous edits and concessions walking back our improvements until the once-promising test felt watered down. Begrudgingly, we moved forward, skeptical of the outcome. To our surprise, this “watered-down” test ended up being an astounding success, generating an additional $181,239 in revenue from a single-page change. The results were undeniable, and the client quickly relaxed their rigid stance on ‘brand discrepancies’.
When have you started your journey in experimentation?
In essence, our collective journey has been about learning to connect with audiences at a deeper level, translating that knowledge into strategies that convert users into loyal customers. Regardless of our starting point, it all comes down to mastering how to create meaningful and compelling user experiences online.
Tom Van Den Berg: If you Want to Get People Excited, it Helps to Let Them Come up With the First Ideas Themselves
What was the biggest A-HA moment in your experimentation journey?
In my opinion, UX’ers or product owners did not understand how to make a good design for experiments. After all, as a CRO specialist, I knew what worked and what made an impact. This arrogant attitude does not work towards colleagues, especially not if you want them to embrace experimenting. If you want to get people excited, it helps to let them come up with the first ideas themselves. Execute these experiments and then have a conversation about the results.
Which experiment did you/ your team run that made your impostor syndrome disappear?
Multiple analysis show that removing elements has better results than adding elements. I have often seen that we add elements rather than remove them because you think you are adding value. If you want to start experimenting cheaply and easily, experiment with removing existing elements one by one on different pages. This way you immediately know what value each element has. Of course not the crucial elements such as buttons or products.
When have you started your journey in experimentation?
In 2013, while working for T-Mobile in the Netherlands as a web analyst, I was introduced to A/B testing tools. I made some changes using the WYSIWYG editor and put my first experiment live. It did not work in all browsers, and I made more rookie mistakes. For example, I had not checked in advance whether we could even test on that page and had no idea how long the test needed to run. Still, it was a lot of fun.
Chris Marsh: Any Figure that Looks Interesting or Different is Usually Wrong
What was the biggest A-HA moment in your experimentation journey?
My biggest a-ha moment was learning of Twyman’s law, which states: “Any figure that looks interesting or different is usually wrong”. This has helped me identify issues in tracking and test data early, and get them resolved. It also helps prevent human error when processing data and reporting.
Which experiment did you/ your team run that made your impostor syndrome disappear?
Early on in my career, we tested removing the Quantity selector from a Product Page, as heatmaps had shown there was almost no engagement with it. This produced a statistically significant positive result and showed how an easy-to-find insight and test can improve UX and business revenue.
When have you started your journey in experimentation?
In 2017 I started at a digital agency and was fortunate enough to run CRO and UX projects for the UK’s largest university. Funnily enough, I was looking for a UX job at the time – and thankfully, I moved into a CRO role, as CRO merges the world of UX and creativity with Data. CRO is exciting, rewarding and impactful for my clients.
Valentin Radu: When I realized that 65% of visitors didn't know that they could buy from our website😮
What was the biggest A-HA moment in your experimentation journey?
My "aha moment" with CRO? It was back in 2010 while working with an online car insurance company. We were struggling with a 1.2% conversion rate, barely breaking even. It felt like conversion rate was stuck there, like a fixed law of nature. To figure out why visitors weren’t buying, I ran exit-intent surveys and learned that 65% of visitors didn't trust the company, and many thought the site was just for information. That insight was my first big breakthrough.
Which experiment did you/ your team run that made your impostor syndrome disappear?
I got obsessed with experimentation. We tested a lot of things: Real testimonials with fake stock photos. Personalizing the homepage with photos from the main square of the city that the visitor was in. Real-time changing the main banner with the car model that our visitors selected. A lot. After a year of A/B testing, we achieved a 60% uplift in conversions, reaching 1.8%, which allowed us to grow and outspend the competition.
This experience inspired me to create Omniconvert, to help others experiment, learn, and grow the way we did. It also taught me the importance of questioning assumptions and staying curious.
When have you started your journey in experimentation? It was back in 1997.
I wanted to go to the seaside with my girfriend.
So, I got a temp job, selling cookbooks in the street.
I discovered after a lot of rejections that I must change variables.
The location: from everywhere to bus stations (nobody could told me that they were in a hurry anymore)
The target: from everyone to old ladies with their nephews
The offer: from "do you want a cookbook" to "what if your nephews will love you even more because you'll cook even better?"
That led me to change my mindset and understand that how you frame reality to others changed the reality for you.
Rosario Toscano: Any Experimental Program Could Get Stuck
What was the biggest A-HA moment in your experimentation journey?
I realized that most of the time we focus a lot on hard skills (which make sense) because to make the experiment possible you need to do research, have a good hypothesis, no flickering, the right configuration, and so on. But even if everything is perfect, any experimental program could get stuck. You may have done an amazing job on the research phase, but people (all stakeholders, internal and external) need to believe in it. They need to be involved. That has a huge impact on the velocity of the experimentation journey, which means results are faster.
When have you started your journey in experimentation?
6 years of accounting and PM for a small redesign agency, taking care also of web analytics, landing pages, and email marketing and I managed SEO and Digital advertising projects with an external agency. After that experience, I was ready to approach CRO. I developed a CRO unit for a digital marketing agency focused on performance for 8 years.
Will Laurenson: The Concept of Usability, Anxiety, and Motivation.
What was the biggest A-HA moment in your experimentation journey?
The concept of Usability, Anxiety, and Motivation as the 3 key pillars of website optimization. We’d been running tests in all 3 of these as an agency, as well as myself in-house for years. Everything is about making it easier for the user to do something, handling all their questions & concerns as quickly as possible in the user experience, and then making sure they actually feel motivated to buy.
Which experiment did you/ your team run that made your impostor syndrome disappear?
We ran some customer research and found that people really liked the founder story, but it just wasn’t that visible on the website, and wasn’t actually related to the products at all. So we took this research and thought about how we apply this to the product page and we ended up creating a kind of letter from the founder, of the PDPs explaining why their products were so great, and the process that went into making them. I hadn’t been so convinced because I tend to be quite direct and found the brand story stuff just a bit fluffy, but when it was implemented it worked fantastically.
When have you started your journey in experimentation?
I started way back in about 2014 at a startup which was a bit like a Spotify for magazines, so a digital product, very data heavy. The advertising side of things was running smoothly, but we had a nightmare with conversion and retention. I took it upon myself to do the research, run some tests, and really try to understand what it was that people cared about and was going to get them not just to convert, but to then stay as a subscriber.
‘All Roads Lead to Rome’
All these experimentation stories and paths are truly inspirational and motivating.
On one hand, you’ll notice that while every CRO specialist in this post has had a unique journey toward expertise, the challenges and ‘aha’ moments they’ve experienced are quite similar.
If I had to choose one key takeaway or phrase to remember in every experiment, A/B test, or conversion strategy after reading all these stories, it would be this:
Success in experimentation is not about always getting it right but learning through trial, error, and adaptation.
