👋 Welcome back!
It’s been a while. Partly because I’ve been heads-down building, and partly because I’ve just spent three weeks travelling, talking with experimenters all around the world.
Sydney → Singapore → Amsterdam → Berlin → London → Helsinki → New York → Austin → San Francisco → Sydney 🤯
It was simultaneously incredible, exhausting, full of wonderful people, and a lot of missing my family.
The catalyst was an invite to run the first workshop at Circus Austin, Speero’s experimentation conference. Around it, I squeezed in as many coffees, lunches and pints as I could with product managers, data scientists, growth engineers, execs, heads of experimentation, and even leaders from competing platforms (damn it if they aren’t all so nice!).
So this edition is a bit different. Part travel log, part what I learned about where experimentation is heading. And I can share a bit more about what I’m building.
A few highlights
Amsterdam: my journey into experimentation at scale really started here. I dropped into the Booking.com offices for lunch with old friends and to discuss how massive-scale platforms are changing.
London: another of my former homes, and I hit the sweet spot of one pub per day (a very important metric to track!). Nothing like a pint to go along with some of the best industry discussions I’ve had in a while.
Austin: running the first Circus workshop on building stats tools with AI. Huge thanks to Ben Labay and the Speero team for the invite, and to everyone who came along.
A big highlight was meeting so many people I’d only ever known online. I was told I was “famous” after trying to introduce myself, got to hang out with “a fanboy,” and generally had plenty of people recognise me.
It was sometimes funny, sometimes embarrassing, but always meant a lot to me. It meant more than people probably realised, because my wife and I decided to move back to Australia for personal reasons (and it’s been the right call for our family). But I miss the wider global community, and I put real work into staying connected from afar. So many other passionate people in the community are also juggling work, having kids, finding new roles, moving countries. We’re all just people living our lives.
So to everyone who said hi: thank you!
One industry, very different realities
OK, let’s talk shop.
Something that stood out was how different everyone’s day-to-day looks.
At one end, experimentation is becoming continuous and largely automated. It was best described at Circus by Alexey Komissarouk, who spoke about Anthropic’s growth system, which is putting a lot of the quick-win style experimentation work on auto-pilot (freeing up growth teams to focus on “big game hunting”). In these loops, agents find opportunities, pitch tests, build them and run them, and people largely just approve. It’s the future and something we’re helping our customers build (more on Cherto at the end).
At the other end, I met a large retailer that ran around 17 tests last year. I met teams using their testing tool purely to randomise, then doing every bit of analysis by hand in the warehouse. And I met leaders still fighting for the buy-in to get testing off the ground at all.
There has always been a gap between the frontier and others, but but it feels like it’s widening faster than ever.
AI is moving the bottleneck, but the big problems remain
We have the frontier pushing towards autonomous growth loops. It’s awesome. But it’s still not solving some of the hardest problems. Coming up with good ideas and prioritising among remains a challenging problem. Elsewhere, I heard about:
A data science leader whose biggest worry about non-technical teams running tests was more about the quality of hypotheses than the stats.
A program running thousands of experiments a year, where every team measures its own metrics and nobody measures anyone else’s.
A business trying to align around shared guardrails, only to find everyone added 15+ metrics and then just ignored them because it was too confusing.
An insurance business where churn only shows up a month or more after a test ends, with almost nothing to steer by in between.
A data team at an AI company, confident building pipelines but nervous about the statistics.
One person acting as the experimentation help desk for an entire company.
None of these around about building tests. Building and running tests is just getting cheaper and cheaper. The hard parts are scaling a useful and trustworthy learning system: deciding what’s worth testing, how to measure it, when to stop, what the result means, and where to go next.
Marty Cagan named the bigger version of this at Circus. He called it the AI productivity paradox: we’re shipping more than ever, but few teams have seen outcomes improve to match. The hard part now is knowing what to build, and how to test and learn our way there faster.
Why I’m excited for experimenters
For some, there’s an understandable unease about the state of the world. Especially hearing things like “agents run the tests now”. I get it. Some of the stories from frontier teams sound like the experimenter has been automated out of the room.
That’s not what I saw.
At the frontier, the role is changing. There’s less hands-on test running and more designing the system the tests run in: the defaults, the metrics, the decision rules, the guardrails. Someone has to decide what good looks like before an agent can do it a thousand times.
Everywhere else, demand is going up. More companies are trying experimentation, across a much wider spread of maturity. And nearly every team I spoke with placed as much value on expert support as on the software.
More experimentation, and more sophisticated experimentation, is a net win for our industry, for businesses, and for the world.
Unordered ideas I enjoyed sparring on
There’s no way I can condense all the interesting ideas from this trip into a single post. But here are some I enjoyed sparring on with fellow experts (in no particular order):
Match the threshold to the decision. Some argued that much looser significance thresholds can be profit-maximising for low-risk ship calls, while high-stakes decisions keep the tight ones.
Frequentist to ship, Bayesian for holdouts. A ship call is a high-frequency decision. A holdout is a one-off estimate of how much we think all that work was worth?
Deferred readouts and rollbacks. Conclude and ship tests on fast signals, but keep the flag live and read lagging metrics later. If lagging metrics show harm, roll back.
Experiments as code. Hypotheses, protocols and experiment definitions as files in git, where engineers and their agents already work.
Experiment platforms owning the comms. What changed, when it launched, what happened, and why we decided what we decided, all in one place.
Betting on coding agents (or not). Several platforms are betting on owning the whole growth loop, coding agent included. My team and I (more below) have made the opposite call, for now: be an exceptional component in that loop. Let frontier coding agents do what they do best (ideas, building, shipping), while we do what we do best (deciding what’s worth running, running it well, and remembering what it taught the team). It’s a strategic bet, and time will tell which one pays off.
What I’m building: Cherto
OK, so the most common question I heard: “What are you building, Simon? What is Cherto?”
We’ve been intentionally quiet, but it was time to reveal enough to make sense, and our site at getcherto.com has been updated to show it:
Cherto is a new kind of experimentation platform. It’s what I’ve always imagined an experimentation platform should be and do, to help people (and agents) test fast, win fast, and ultimately achieve their goals with data.
We’ve been building it since day one with this future in mind: server-side and warehouse-native by default, agents doing more of the work, and the learning system becoming the hard part. We’re excited to build for where things are heading, and to help as many people and businesses as we can get there along the way.
We’re still early and keeping the secret sauce stuff behind closed doors. But if you’re considering a move to a warehouse-native platform, or starting to wonder whether your current setup is holding you back, I’d love to talk. Message me directly via getcherto.com, hit reply, or DM me on LinkedIn.
Oh, and the second-most common question I got: “What does Cherto mean?”
The name Cherto comes from the Italian/Latin word certo (pronounced roughly “Cherto”), meaning certain or sure. It’s also found in concerto, where one voice confidently leads an ensemble. That’s the idea behind Cherto: moving from isolated tests to orchestrated experimentation programs, run at scale with confidence and clarity.
What’s next
Well, after a round the world trip like that, I was first and foremost, excited to get home to my family and my own bed!
Then, to get back to my team. They’re amazing, and we all share a crazy passion for this space. Everything from this trip is fuelling us to keep building the best, most exciting experimentation and decision engine we can.
But that’s the round-up. Three weeks, nine cities, a lot of coffee, a few pints, and one takeaway:
Holy crap, the future of experimentation is exciting.
If you ever want to talk experimentation, get some advice, or share your opinions, I’d love to hear from you!
Until next time 🙌
Simon
Founder, Cherto
Web: getcherto.com
LinkedIn: linkedin.com/in/drsimonj