👋 Welcome back!

There’s an anxiety creeping through experimentation teams right now. Nobody’s saying it out loud, but I hear it between the lines in almost every conversation I have with experimentation leaders:

If engineering can ship this fast, does anyone still need us?

AI-assisted development is accelerating how quickly teams can build and deploy product changes. What used to take a sprint takes a day. What used to take a quarter is becoming a month. And when shipping gets that fast, the instinct is to wonder whether experimentation — with its sample sizes, its run times, its careful measurement — becomes the bottleneck everyone routes around.

I think that instinct is wrong. But not for the reason you might expect.

So, as someone watching the shift that AI is creating play out in real time, I wanted to share my thoughts here on how our industry needs to evolve over the next few years.

Learning doesn’t lose value. It changes shape.

The core promise of experimentation has never been about slowing things down. It’s about learning: understanding what works, what doesn’t, and why. That value doesn’t disappear when shipping gets faster. If anything, it increases.

But here’s what does change: what you’re learning from, what you’re learning about, and why you’re learning it.

When a team can spin up 50 new ideas in a week instead of five, the volume of decisions that need evidence behind them multiplies. The question isn’t whether to learn. It’s whether your program can keep up with the pace of decisions that now need to be made.

And for most teams, it can’t. But not because experimentation is slow.

Because ownership is too narrow.

The real bottleneck isn’t speed. It’s scope.

Here’s the pattern I see: a team owns a specific feature or surface. They ship a change, set up an experiment, and then… wait. They need data. They need time for signal to accumulate. You can’t manufacture true evidence: that has a velocity cap no amount of AI can remove.

So the team is stuck. They either:

• Wait: killing the speed advantage they just gained.

• Skip the experiment: shipping fast but learning nothing.

• Peek early: making decisions on noise.

None of these are good. And the root cause isn’t that experimentation takes time. It’s that the team has nowhere else to go while they wait.

It reminds me of the old elevator problem. When tenants complained their building’s elevators were too slow, the solution wasn’t faster motors. It was installing mirrors next to the doors. People stopped noticing the wait because they had something to look at. The real issue was never the speed of the elevator. It was the emptiness of the wait.

Experimentation has the same problem. You can’t make learning faster. But you can make sure teams aren’t standing around staring at the doors.

This is the narrow-ownership trap. When a team’s world is one page, one feature, one surface, there’s no room to rotate. Speed without scope creates a queue, not a learning engine.

The rotation model

Now picture a different setup.

A team owns customer acquisition as an outcome, not a specific button or page. Their territory spans value perception, onboarding flows, pricing clarity, and payment experience.

With AI-assisted engineering, they spin up 50–100 ideas focused on the onboarding funnel. Those experiments go into the program: designed, prioritised, launched. While they’re running and collecting signal, the team shifts focus to the payment flow. They build there, spin up the next wave of experiments, and send those off too.

By the time the onboarding results come back, the team has been productive the entire time. They haven’t waited. They haven’t skipped measurement. They’ve rotated.

This is what I think high-performing experimentation looks like in an AI-accelerated world: teams managing many concurrent threads across a wide enough ownership space that they can build in one area while learning in another.

Not slower. Not faster without guardrails. Just structured differently.

Why this runs against the grain

For the past decade, the trend in product organisations, especially at large companies, has been to narrow ownership. Smaller teams. More specific surfaces. Tighter scope.

In theory, this creates clarity and accountability. In practice, I’ve seen it create something else entirely.

I worked in a tech company where teams ended up with ownership scoped to very specific features. Worse, they were KPI’d on adoption and usage of those features. The result? Teams pushing their own things, boosting their own metrics — often at the cost of cannibalising others — without moving the needle on the outcomes that actually mattered, like acquisition or retention.

That model was designed for a world where shipping was expensive and slow. When building a feature was a quarter-long investment, it made sense to give a team singular focus on it.

In a world where building is fast and cheap, that logic breaks down.

Ownership will expand

In my view, the micro-team, single-feature ownership model is ending.

Not because it was wrong. It was right for its era. But the constraint it was designed around (shipping is expensive) is dissolving. And the constraint that remains (learning is slow) demands a different structure.

Teams of the future will own broader territories. They’ll be aligned to higher-level outcomes like acquisition, retention, and activation rather than individual surfaces. The number of surfaces and options they experiment across will increase, because they’ll have the building capacity to explore more of the space.

This maps naturally to what’s often called the “empowered product team” model, where teams are accountable for outcomes, not output. That concept has existed for a while. What changes now is the scale at which it needs to operate.

An empowered product team with AI-assisted engineering and a wide ownership scope can generate ideas, build, and experiment at a rate that would have been unimaginable two years ago. But only if the experimentation program underneath them can support it.

What needs to change in experimentation programs

This is where things get honest.

Most experimentation programs and the tools that support them were built for a world of sequential experiments with narrow scope. One team, at most a few tests and a few decisions at a time.

The rotation model demands something different. Teams switching in and out of problem spaces need to be able to pick up where they left off. They need memory: a clear record of what they’ve tested, what they’ve learned, and where the compounding insights are forming.

Without that, the rotation model collapses into something much worse: lots of threads, no continuity. Teams forget what they’ve done. Learnings evaporate between rotations. The idea backlog becomes a graveyard of disconnected, low-impact tests instead of a structured engine for compounding knowledge.

And then there’s the operational side. The things that are painful today like figuring out statistical power, selecting the right test design, sequencing experiments in parallel, making clear decisions from ambiguous results. All of these become more painful when volume increases. If these aren’t effectively on autopilot, the program becomes a bottleneck regardless of how the teams are structured.

In short, the future needs two things from experimentation programs:

1. Program-level tracking: the ability to manage structured programs of work across multiple problem spaces, so teams can switch context without losing the thread.

2. Operational autopilot: the statistical and design mechanics that currently require specialist effort need to be automated, so that increased experiment volume doesn’t require proportionally increased human overhead.

Neither of these exist well today. And until they do, even the best-structured teams will struggle to realise the full potential of AI-accelerated shipping.

The bottom line

AI isn’t going to kill experimentation. But it is going to expose the structural limitations of how most teams are set up to learn.

The teams that thrive will be the ones who recognise that the bottleneck was never speed. It was scope. They’ll widen their ownership, rotate across problem spaces, and invest in the program-level infrastructure that makes compounding learning possible at a pace that matches their building capacity.

Speed is being solved. Learning isn’t. Not yet. In my view, the organisations that figure out how to learn as fast as they can ship will be the ones that pull ahead. This is the problem I’m focussed on solving.

This is a prediction, not a report from the field. I haven’t seen this fully play out yet. But the logic feels inevitable to me: code velocity is accelerating past a point where narrow ownership and manual experimentation workflows can keep up. Something has to give, and I think it’s the ownership model.

I wrote this piece because I genuinely believe this is where things are headed. But I’m one person with one lens.

I want to hear your perspective. Do you see the same shift coming? Are you already feeling the tension between shipping speed and learning speed? Or do you think I’ve got this wrong?

Reply to this email or, better yet, leave a comment on the post so we can get a real discussion going.

Until next time 🙌

Simon linkedin.com/in/drsimonj

Some of my resources you might find useful:

• 🧮 The Experimenter’s Calculator: a tool to plan high-quality experiments

• 📊 Intro to A/B Test Statistics: a free webinar for practitioners

👋 Welcome back.

Most teams I work with aren’t short on test ideas.

They have backlogs. Hypotheses. Roadmaps full of experiments.
They’re shipping A/B tests every week.

And yet — when you zoom out — it’s hard to answer a simple question:

What are we actually learning?

That’s the gap.

Running tests isn’t the same as running a program.

Tests answer isolated questions.
Programs build structured learning that compounds into strategic advantage.

If your experiments don’t clearly connect all the way up to P&L impact, you don’t have a program.

You have activity.

In this newsletter, I’ll share the simple mental model I use to turn disconnected tests into a coherent experimentation program: one where every experiment is part of a larger decision thread that actually shapes the business.

The Learning Spine

When I join a team, regardless of how mature they are, the first thing I look for is what I call The Learning Spine.

The Learning Spine is the chain of logic that connects every single experiment all the way up to the P&L.

It looks like this:

1× P&L → 1× Scoped Objective → 2–3 Problem Spaces → Many Solutions → Many Tests

Simple.

But most teams don’t actually operate this way.

Let’s break it down.

1️⃣ Start with the P&L

At the very top, there should be one clear thing you’re trying to move.

Revenue.
Gross margin.
Customer lifetime value.

Not five things. One.

If you can’t clearly articulate what line on the P&L you’re trying to influence, your experimentation program is already drifting.

This is where many teams go wrong. They start with “ideas” instead of outcomes.

2️⃣ Define a Scoped Objective

Next, define the part of the system you’re going to work on.

For example:

• Customer acquisition

• Activation

• Subscription conversion

• Retention

This is usually where your North Star or primary metric lives.

It’s specific enough to focus effort but clearly connected upward to the P&L.

Without this layer, teams jump straight from “revenue” to “let’s test button colours.” The gap in between is usually too wide to bridge and where most programs fall apart.

3️⃣ Prioritise 2–3 Problem Spaces

Now it gets interesting.

Within your scoped objective, you identify a small number of meaningful problems or opportunities to work on.

For example:

• We believe we’re attracting low-intent traffic through paid channels.

• We believe customers are confused in the payment flow.

• We believe first-time users don’t experience value quickly enough.

These are not solutions. They’re hypotheses about where value is leaking.

This step forces alignment and focus.

It also makes it easy to say no to random ideas that don’t clearly address one of these spaces.

You’re not saying “no” to everything else forever. Just creating focus right now.

4️⃣ Generate Many Solutions

Only now do typical “experiment ideas” enter the conversation.

For each problem space, you generate multiple potential solutions.

For example:

“Making the subscribe button more visible will reduce confusion and improve conversion.”

Crucially, every solution should explicitly link upward:

Doing [solution] will address [problem], which will improve [scoped objective], which will move [P&L metric].

When that chain isn’t clear, the test probably shouldn’t exist.

5️⃣ Run Many Tests

Finally, you implement and test those solutions.

Often, one solution can generate multiple tests:

• Larger button

• Different colour contrast

• Alternate copy

• Different placement

Now something powerful happens.

Tests stop being isolated events.

They become evidence against a clearly defined problem space.

And that’s where compounding begins.

The Learning Spine structure does three things at once:

• It accelerates ideation (because the boundaries are clear)

• It simplifies reporting (because every test has context)

• It enables strategic learning (because failure tells you when to pivot up a level)

Without it, you just have tests.

With it, you have a program.

Leveraging The Learning Spine

Structure is useful.

But its real power shows up over time.

Let’s take a concrete example.

Imagine your P&L focus is Revenue.

You’ve scoped your objective to Subscription Conversion.

You’ve identified two priority problem spaces:

1. Customers are confused in the payment flow

2. Customers don’t perceive enough value before being asked to subscribe

Let’s focus on the first one.

Step 1: Multiple Tests Within One Solution

You hypothesise that customers are confused because the “Subscribe” button isn’t clear enough.

So you test:

• Making the button larger

• Increasing contrast

• Changing the copy

• Repositioning it

Three or four tests later… nothing meaningful moves.

This is where most teams either:

• Keep tweaking micro-details forever

• Or abandon experimentation entirely

But with The Learning Spine in place, the response is obvious:

It’s probably not a button problem.

You pivot to a new solution inside the same problem space.

Step 2: Multiple Solutions Within One Problem

Next, you test:

• Simplifying the pricing explanation

• Reducing the number of plan options

• Adding testimonials

• Introducing a free trial

If three to five well-thought-out solutions fail to meaningfully move conversion…

Now the evidence is stronger.

It’s probably not a “payment confusion” problem.

You move up a level.

Step 3: Pivot the Problem Space

You shift focus to the second problem:

Customers don’t experience value early enough.

Now you test onboarding changes instead.

This is where structure turns experimentation into a program.

Because failure isn’t random.

It’s directional.

It tells you when to pivot up a level.

The 3–5 Rule

I often suggest teams think in terms of small clusters:

• 3–5 tests fail → change solution

• 3–5 solutions fail → change problem

• 3–5 problems fail → reconsider your objective

This doesn’t need to be rigid.

It just forces you to ask:

Are we learning at the right level?

On the flip side:

• If 3–5 tests within a solution succeed → double down

• If a problem space consistently produces wins → prioritise it harder

That’s how advantage compounds.

Why The Learning Spine Matters

Without structure:

• Tests fail randomly

• Wins feel isolated

• Reporting feels scattered

• Learning doesn’t accumulate

You get motion but not momentum.

With The Learning Spine in place:

• Failure becomes directional

• Success becomes scalable

• Decisions get easier

• Reporting becomes strategic

• Leaders see progress, not just activity

Most importantly, your testing becomes a structured program generating strategic advantage.

A Simple Starting Point

As a quick exercise, take your last 10 experiments.

Can you clearly map each one up through:

• A defined solution

• A clear problem space

• A scoped objective

• A single P&L driver

If not, that’s your starting point.

Because when every test sits on the Learning Spine, experimentation stops being random.

It becomes directional.
It becomes cumulative.
It becomes a program.

Until next time 🙌

Simon
linkedin.com/in/drsimonj

Some of my other resources you might find useful:

• 🧮 The Experimenter’s Calculator: a tool to plan high-quality experiments

• 📊 Intro to A/B Test Statistics: a free webinar for practitioners

👋 Welcome back! In this newsletter I’ll be tackling a debate I hear teams having far too often: whether an experiment should run at 10% or 50%.

It usually sounds like a discussion about risk.

In reality, it’s a muddled conversation hiding two very different decisions, neither of which is being made explicitly.

Before we get into details, here’s the mental model I use to cut through the noise:

The core idea (read this even if you skim the rest)

Every experiment design choice involves two inputs:

1. How much of the total population is included in the experiment (0–100%)

2. How the included users are split between control and test (100% control to 100% test)

Inside that space, different outcomes emerge:

• Faster or slower learning

• More or less business impact while learning

Most teams collapse all of this into a single “10% vs 50%” debate.

That’s the mistake.

ℹ️ To make this material as accessible as possible, I’m assuming a typical A/B experiment setup where the two conditions are control and test. The same key takeaways apply as you expand to multiple variants.

Decision 1: Learning efficiency (power & speed)

This visual below represents how well and quickly you can learn from various combinations of these two factors: population included (x-axis) and split between control and test (y-axis).

This visual captures something many teams feel but struggle to articulate:

• Learning accelerates as more users are included (moving left to right)

• Learning is maximised near balanced allocation (moving top/bottom to centre)

• Unequal splits slow learning in both directions

Decision 2: Business impact while learning

Now for the dimension most experiment discussions completely ignore.

This visual below represents how much impact an experiment can have on customers and your business depending on these two factors: population included (x-axis) and split between control and test (y-axis).

This visual makes something uncomfortable — but important — very clear:

• Experiments create real business impact while they’re running

• That impact scales with exposure

• Low exposure caps both upside and downside

• Higher exposure accelerates both value creation and risk

  ---

It’s All a Tradeoff

At this point, the 10% vs 50% debate should look a little different.

You’re not choosing between “safe” and “risky.”

You’re choosing where to sit on a surface that trades off:

• Learning speed & statistical robustness

• Against realised business impact or downside exposure

There isn’t a single correct percentage.

But there are clearly better and worse places to sit depending on what you’re optimising for.

Let’s make that concrete.

Three Common Choices (Side by Side)

Imagine these three common choices teams might make: A, B, and C.

🟢 Choice A: Low inclusion, 50/50 split

This often feels responsible.

• Balanced allocation → decent learning per included user

• Limited exposure → capped downside

• Low business impact during the test

But here’s the tradeoff:

• Slower learning overall

• Longer time-to-decision

• Delayed value realisation (or even extended downside)

This is fine for catching catastrophic issues early.

It’s not great if you care about speed.

🟢 Choice B: High inclusion, 50/50 split

This is what the math — and real-world experience — tends to favour.

• Fastest learning (and fastest to find and kill harm)

• Strongest statistical robustness

• Clear, defensible decisions

• Meaningful business impact while the experiment runs

The tradeoff?

• You’re exposed while uncertain

• You need good monitoring and rollback discipline

But in stable environments, this is usually the most sensible default.

There’s a reason 50/50 at high inclusion sits on the “ridge” of the learning surface.

🟢 Choice C: High inclusion, 90/10 split

This one is popular.

It feels cautious:

• “Let’s give most users the safe version.”

• “We’ll still learn something.”

But the surface tells a different story.

• Learning slows compared to 50/50

• Variance increases

• Trust in results drops

• You gain little real protection relative to what you sacrifice

In theory, unequal splits can offer minor statistical benefits.

In practice, theory and production systems rarely align. Or, simply put by Kohavi, Deng, and Vermeer (2022), as:

“Beware of Unequal Variants“

Unless you have a compelling reason, this is often the worst of both worlds.

How This Plays Out in Ramp and Rollout

The surface becomes even more useful when you think in terms of movement and not just static points.

Let’s compare these two patterns.

✅ Path A: The High-Performance Path

1. Low inclusion, 50/50
   Catch egregious issues quickly.

2. Rapid move to high inclusion, 50/50
   Maximise learning speed and decision clarity.

3. Move vertically to 100% treatment
   Commit once confident.

Visually, that’s:

Left-middle → Right-middle → Top-right

This path:

• Minimises time spent uncertain

• Preserves statistical robustness

• Accelerates realised business impact

❌ Path B: The Common but Suboptimal Path

1. Low inclusion, 10/90

2. Slow creep upward and right

3. Eventual rollout

This path often:

• Prolongs uncertainty

• Encourages peeking and reinterpretation

• Delays value capture

• Reduces trust in early results

It feels safer.

But it’s often slower and less clear.

The Real Punchline

There’s a reason high-inclusion, 50/50 sits at the sweet spot of the learning surface.

It maximises:

• Speed

• Trust

• Decision clarity

You can deviate from it.

But if you do, be explicit about what you’re trading away.

Default to clarity.

Shift away in a considered fashion.

And remember:

If your exposure decision takes 30 minutes to debate, it probably wasn’t a statistical decision to begin with.

Try it for yourself

If you’d like to explore how these tradeoffs play out in your own context, I built a free power calculator where you can:

• See how sample size affects time-to-decision

• Explore minimum detectable effects across traffic levels

• Understand how allocation choices influence statistical clarity

• Play with different setups and see how the numbers respond

👉 You can try it here:

calculator.drsimonj.com

It’s a simple way to turn intuition into evidence, and to make exposure decisions based on tradeoffs, not instinct.

Until next time 🙌

Simon
linkedin.com/in/drsimonj

Some of my other resources you might find useful:

• 🧮 The Experimenter’s Calculator: a tool to plan high-quality experiments

• 📊 Intro to A/B Test Statistics: a free webinar for practitioners

✋ “Umm, what’s an AA test?”

This was THE FIRST question I ever asked as a data scientist.

Ever since, I’ve been constantly explaining what AAs are and recommending that product and growth teams to run more of them!

Classic example:

So today I’m going to get it all out and, hopefully, convince you too.

Because, as you’ll learn here, when you start to run AA tests, the foundations of your experiment programs go from quick sand to concrete.

So, let’s dive into this often under appreciated topic.

What is an AA test?

The “AA” is a play on “AB tests.”

Most readers will know this but just in case: The “A” and “B” in AB testing refers to two different versions of a product being shown to different groups of customers.

So an example of an AB test (not an AA) from me doing a search on Google using (A) my logged-in account and (B) an incognito window:

Side by side, you can see that the company icons are being rendered larger and in a higher-contrast way in the version on the right.

So if “A” and “B” are different, then “A” and “A” are… the same!

An AA test is like an AB test in every way except customers split in each group see the exact same thing. No change at all!

Seems boring, right?

Time to dive into why I care so much about them!

Reason 1 to run more: Making sure things work

AB testing is pretty complicated. Seriously.

It seems easy at a high level to “just give different customers different versions.” But the technology that powers online AB testing is complicated. It’s why I’ve worked with teams ranging from 20 to 200 highly-skilled engineers (depending on the company) to build, maintain, and improve AB testing platforms.

So how do we know that something so complicated is working?
We run the damn thing!

But there’s a problem. When you run an AB test, you have no idea what the impact of going from A to B on customers is 😬

This brings us to the first major reason to use AA tests: Because we know exactly what’s going on (nothing), we know what results to expect - nothing 😜

Running AA tests is the safest and fastest way to check all sorts of things are working with your experiment software such as:

• Am I starting to enrol participants as expected?

• Are there any signs of sample ratio mismatch?

• Are the statistical results being computed correctly?

• Over time, do I see a false positive rate (significant results) as often as I’d expect (e.g., 5% or 10%)?

• etc!

Perfect example from my time at Meta: we had constant AA tests running to monitor the false positive rate of or testing. One day we noticed a massive spike, meaning all running experiment results were rendered invalid - oh crap! Turns out it was caused by a different platform team introducing a breaking change to a service we depended on. Anyway, monitoring this enabled us to notify all experimenting teams about the problem before they all got too excited about their positive results (like baby Yoda down here). We then go to work identifying and resolving the issue.

The point is, running AA test is often your best way to make sure everything is working before you start running AB tests! And because customers aren’t getting anything different, if something goes wrong, you know there’s a problem with your experiment tech and nothing else.

Reason 2 to run more: Baselines for Planning

OK, you’ve now run a couple AA tests and know everything is working. Happy days!

Time to forget about them?

NOOOoooooo!

Knowing things are working is just table stakes. Now you get to focus on the juicy stuff – running great experiments!

But what’s one of the hardest things about running good experiments?

Experiment Design.

The fastest way to throw AB testing money down the drain is to just run experiments without any thought or planning to ensure you can detect the signals you care about.

To design a great experiment you need to plan for things like:

• How many participants will I have available?

• How do participants enrol over time?

• What are the baseline metric scores I should expect?

• etc

Ideally you do this with a power calculator (like my free and obviously amazing one at calculator.drsimonj.com).

But you still need to get some numbers from somewhere! You could go and get a data person to go scraping your web data, spending hours mindlessly bringing it together, and trying to figure out the answers to these questions.

OR – better idea – if you’d run an AA test in the past, you’ll have this all just waiting for you!

But don’t stop there...

Answers to these questions are usually highly dependent on WHERE you’re running experiments, WHO you’re running experiments on, even WHEN you run them.

So, one of the best things you can do…

Run AA tests everywhere, all the time!

OK, maybe I’m going a bit overboard BUT the point is this: regularly running AA tests in many areas of your product will mean you constantly have access to the data you need to plan great experiments.

No more guessing. No more data people tantrums. No more uncertainty. Just a nice, hot, bowl of fresh data and experiment planning confidence.

Reason 3 to run more: Validate Internal Features

My third and final reason came from a reminder by Bertil’s comment on my recent post:

It’s common when you start to run larger experiment programs that you want to release new features... and I don’t mean new customer-facing features.

I mean new features for your experiment platform and program to help teams experiment better and faster. For example, new statistical techniques, results UIs, new metrics, and so on.

Similar to there being various gates to safely release code to end users of a product (with AB testing typically being the final one), you can use AA tests to validate the behaviour and quality of features such as new experiment metrics.

For example, eight years ago I introduced CUPED at Booking.com as a statistical technique to boost power and help run smaller, faster experiments. I did so by creating a new CUPED-adjusted metric that could be added to experiments. Now, everything panned out in my analyses but you never really know until something hits production. So I added the new metrics to a bunch of AA tests and was able to monitor it on live data. Similar to above, I could now answer questions like:

• Did the metric display in an understandable way on the platform?

• Was it producing data in a way I expected?

• Did it maintain an expected false positive rate?

• Did (as we hoped in that context), it yield 30-40% lower variance than it’s unadjusted version?

• etc

These are critical tests that can be done before releasing new experiment features like metrics that can make their way into everyone’s hands and have an amplified impact (for positive or negative).

Reason 4 to run more: Build a Stronger Culture

This one has been inspired by this post written by Ron Kohavi:

His points 1 and 2, to me, are both excellent examples of how AA tests can help build a culture of more trustworthy and valuable experimentation.

The idea here is that you can “secretly” use AA tests to share around scorecards from experiments and ask people what they’d do.

Often people will give all sorts of explanations, decisions, opinions, etc. BUT, once they realise they’re looking at results from an AA test, reality starts to sink in…

People realise from AA experiments that results can be noisy; that they fluctuate; that our great psychological intellect brings hidden trade-offs and biases like apophenia. So the more people are exposed to AA test results, they more they come to appreciate how rigorous processes like runtime planning, not peeking, replication, etc etc – all which intuitively feel painful – carry great value.

Bonus Tip: When & Where to run AA tests

Now, one final tip that came out of a LinkedIn post I recently shared on how starting to run A/A tests can rapidly increase team velocity (find the post here). There I got this great question from Patrick Kitchen: “Do you recommend isolating the always-on A/A users from other experiments?”

Here’s the answer I gave:

Patrick Kitchen nope. Actually, I’d actively encourage overlapping. This gives you a few advantages:

1. You confirm your randomisation engine is working and actively making different experiments independent of each other.

2. Given above, it’s useful evidence to help others feel safer about overlapping (many people are scared of it).

3. It gives you a more accurate sense of experimentation in the wild so more realistic data on metrics variances, user overlaps, etc etc

Run more AA tests. You won’t regret it.

Well, this has been a long one today but I hope you now know exactly what AA tests are and why they’re so useful.

So get out there and start running heaps of AA tests... You won’t regret it!

And if this sparked a thought, or describes something you’ve seen in your own team, hit reply. I always love hearing from fellow experimenters!

Until next time 🙌

— Simon
Helping businesses scale high-impact experimentation
drsimonj.com

👋 Welcome back!

A couple of weeks ago I ran this LinkedIn poll asking what experimentation skill people would most like to learn live. The clear winner? Measuring Business Impact.

It was a clear winner and it makes sense: this is the single most common challenge I see with clients. Teams run good experiments, see strong local wins… and then leadership asks: “So why isn’t revenue moving?”

The newspaper problem

Here’s the visceral analogy I like to use: imagine you’re running experiments on a daily newspaper. Remember those? 😉

You might test different layouts, typography, or headlines. You might make the articles more engaging, and improve how many people read past page three. You might even boost retention among loyal readers.

All good things. But will you stop the overall decline of print media? Of course not. That’s a global trend, driven by forces much bigger than your page layout. At best, your work slows the decline.

The same is true for digital products or any business:

• Experiments isolate the effect of a specific change.

• That’s their superpower. They strip out noise so you can ask: “Did this change cause that effect?”

• But that local effect is usually too small to show up in topline KPIs like revenue or churn, especially when dozens of other changes and external forces are at play.

So ironically, the thing that makes experimentation amazing—isolating cause and effect—is also why you can’t expect one test to “move the needle” at the macro level.

Why this disconnect matters

Where teams go wrong is in how they interpret or communicate this gap:

1. Over-promising: Telling leadership that a test will move revenue, and then struggling when it doesn’t.

2. Under-valuing: Discarding valid learnings because topline metrics stayed flat.

3. Frustrating execs: Leaving leaders questioning whether experimentation really matters.

This isn’t a problem with the experiments themselves. It’s a problem with expectations and translation.

The three layers of Value

High-impact teams look at experiment outcomes on three levels:

1. Local value: The isolated test result. Did this change drives a specific, desirable behaviour or outcome (e.g. onboarding completion, checkout conversions)?

2. Portfolio value: How the accumulation of changes across many experiments impact overall business performance. No single article saves the newspaper, but 100 small improvements compound.

3. Strategic value: The learnings that shape long-term decisions (to GO and to NO-GO. Even a “neutral” result can prevent costly mistakes or reframe how a business approaches growth.

If you only look at topline metrics, you’ll miss most of the value experimentation creates.

How to show business impact the right way

Here’s what I advise teams to do instead:

✅ Use proxy metrics

Find leading indicators that link to long-term business goals (activation, engagement, funnel conversion). Show how they connect to longer-term retention or revenue.

✅ Roll up portfolio impact

Communicate results as part of a broader portfolio. “These 10 onboarding tests together lifted activation by 7%, which equates to X new actives a month.”
Side note: there is an art and science to this which I’m teaching live (see here).

✅ Track decision follow-through

The biggest value killer is wins that never get shipped. Measure implementation rates. Value isn’t realised until decisions change.

✅ Translate results into exec language

Frame outcomes as business enablers: “This change unblocked the sales team” or “This reduced risk on a $20m product launch.”

✅ Translate results into exec language

Frame outcomes as business enablers: “This change unblocked the sales team” or “This reduced risk on a $20m product launch.”

✅ Bring experimentation into strategic decisions

Experimentation shouldn’t just optimise buttons or layouts. The real value is when it informs strategic choices, like whether to pivot from print to digital or launch a new pricing model.

Engaging executives (and knowing when not to)

One of the most practical levers for turning experiment results into business impact has nothing to do with metrics or methods. It’s about engagement.

If execs don’t understand the disconnect between local experiment results and topline metrics, you have to invest time in educating them.

• Share analogies like the newspaper example.

• Walk through concrete examples of how experiments provided evidence that guided the business.

• Be clear about both the power and the limits of experimentation.

• And, when you can, show them how experimentation can help answer the bigger strategic questions: Should we shift from print to digital? Should we double down on self-serve or sales-led?

These conversations build trust and position experimentation as a strategic asset, not just a tactical testing function.

But here’s the kicker. After doing this across companies, industries, and exec teams, I’ve seen a consistent truth:

👉 Some executives lean in. They believe in being evidence-based, and they’ll take the time to listen, learn, and apply.
👉 Others don’t. It’s their way or the highway. No amount of data or analogies will change their mind.

When you run into the second type, it’s important to recognise the reality: even with enormous effort, you’re unlikely to shift them. At that point, you have to ask yourself a different question:

Do I want to keep fighting this fight here, or would my time and energy be better spent with leaders and teams who are eager to learn, test, and grow?

Because in my experience, the world is full of the first type. And those are the businesses and people that run better experiments, make better decisions, and grow faster.

Back to NAV

If you’re new here, NAV is the framework I use to define and diagnose high-impact experimentation. I introduced it in my first edition, which you can read here.

NAV breaks impact into three levers:

• Number: how much you’re testing

• Accuracy: how much you can trust the results

• Value: whether those results are actually shaping outcomes and strategy

And of the three, Value is the hardest—but most important—to move. Because it’s where experimentation either proves itself as a growth engine, or stalls out as a reporting function.

The newspaper may still be in decline. But if your experiments are improving the product, unblocking teams, and shaping smarter decisions, you’re doing the work that slows the fall and sets you up for the next growth curve.

What’s next

Given how strongly this topic resonated in the poll, I’m running some live workshops on Measuring Business Impact. Specifically, techniques for translating experiment results into business-metric-level measures (like showing how an uplift in onboarding translates into increased revenue by X%).

If that sounds useful, checkout availability at drsimonj.com/impact-cohort 🚀

And in the meantime, I’d love to hear from you:
👉 What’s the biggest challenge your team faces in proving experimentation’s business impact?

Until next time 🙌

— Simon
Helping businesses scale high-impact experimentation
drsimonj.com

In the last edition, I introduced NAV: my framework for defining and diagnosing high-impact experimentation.

Since then, many of you have already taken the self-serve NAV audit (if you haven’t, it’s here). An emerging theme I’ve seen time and time again: Accuracy is perhaps the most common, and misunderstood, failure point.

And it’s not just a failure of omission.

Teams fall short when Accuracy is ignored AND when it’s overcorrected.

In this edition, we’ll walk through what those two extremes look like, how they impact the rest of your experimentation program, and what high-impact teams do to strike the right balance.

When Accuracy is off: The two extremes

1. Accuracy is underrated

You’ve seen this version before. It’s the team shipping lots of experiments with great intentions but shaky foundations:

• Using low-signal behavioural metrics because they’re easy to measure

• Skipping power analysis entirely (or not knowing what it is)

• Peeking at results, changing variants mid-test, slicing post-hoc to find "wins"

On the surface, these teams look productive. Lots of motion, lots of charts, lots of dashboards.

But the learning? Weak. Direction? Absent.

These teams sacrifice direction for motion.

My former colleague and head of Experimentation, Lukas Vermeer, explains it perfectly in his short blog, “Speed is not velocity.”

It’s usually not the result of laziness. It’s a mix of inexperience, unclear expectations, and external pressure to deliver something. But the result is the same: erosion of trust. Teams can’t tell which results to believe, stakeholders stop listening, and soon experimentation is seen as "nice to have" instead of mission-critical.

2. Accuracy is overrated

At the other extreme are teams with gold-plated pipelines and rigorous scientific processes. The kind that won't test a copy change without a perfectly powered sample size and a six-week revenue holdout.

Sound familiar?

• Waiting a month (or more) to measure long-term metrics like cancellations

• Demanding overly precise revenue attribution when directional signal is what’s needed.

• Blocking tests because power calculations aren’t precise enough

These teams are technically right but, in my opinion, strategically stuck.

They sacrifice speed for the illusion of precision.

It often comes from good intentions. Maybe the org overcorrected after past mistakes. Maybe leadership demanded absolute certainty. Maybe the team just loves doing science. But at some point, the pursuit of perfect undermines the ability to ship, learn, and adapt.

Instead of unblocking roadmaps or accelerating decisions, the experimentation program becomes a bottleneck. Valuable, but slow. Accurate, but irrelevant.

How Accuracy trade-offs affect NAV

When we put the above sections together, I hope it becomes clear that one of the key reasons Accuracy matters is because of how it affects the other NAV levers:

⚙️ Number

• Underrated Accuracy can inflate test velocity while hiding the fact that many experiments aren’t worth running or learning from.

• Overrated Accuracy reduces throughput by adding unnecessary friction and slowing test cycles.

📈 Value

• Underrated Accuracy undermines trust, meaning results don’t shape decisions, and value is never realised.

• Overrated Accuracy delays or discourages experimentation in key areas, missing out on potential wins and compounding insight.

If you want to scale experimentation, you can’t just optimise one lever. You need to manage the interplay.

That’s why high-impact teams don’t treat Accuracy as an absolute. They treat it as a strategic trade-off.

How to strike the right balance

The goal isn’t perfection. It’s trusted, decision-ready direction, achieved quickly enough to matter, and carefully enough to guide.

Here’s what high-impact teams tend to do:

✅ Choose smart proxies

You don’t always need to wait for long-term revenue or retention. Instead, find early indicators that strongly correlate with business outcomes (e.g., activation, engagement, funnel completion).

These signals help you move faster and stay directionally aligned.

🔧 Use tools, not rules

Power calculators (like my Experimenter’s Calculator) are a great way to make trade-offs explicit. But they’re not commandments. Use them to understand risk, not to block experimentation altogether.

💬 Build a culture of clarity

Every test doesn’t need to be flawless, but it does need to be understood.

Flag when tests are exploratory or underpowered. Be explicit about uncertainty and risks. Sacrificing precision for speed when the context demands is fine, but be open about it. Focus on decisions, not just results.

Back to my definition in NAV:

Accuracy = How trustworthy, interpretable, and aligned experiment results are.

Accuracy = How trustworthy, interpretable, and aligned results are.

Not statistically perfect and not philosophically pure. While both may be important inputs, they’re rarely achievable in practical situations. It’s about being solid, transparent, and decision-ready.

Because when Accuracy becomes a blocker, you lose momentum. But when it's neglected, you lose trust. And either way, you lose impact.

What’s next

This is the second edition of my freshly rebooted newsletter, and part of a mini-series unpacking the NAV framework.

In upcoming posts, I’ll dive into:

• Why Value is the ultimate test of your experimentation program

• How to use NAV results to prioritise and unlock change

• Real-world case studies from teams scaling experimentation the right way

In the meantime, if you haven’t taken the NAV self-assessment yet, you can do that here. It only takes 3–5 minutes.

And if this sparked a thought, or describes something you’ve seen in your own team, hit reply. I always love hearing from fellow experimenters!

Until next time 🙌

— Simon
Helping businesses scale high-impact experimentation
drsimonj.com

👋 Hey there, welcome (or welcome back)!

This newsletter isn’t entirely new but it is freshly rebooted. What started as a space for data science and analytics posts now has a much sharper focus:

How to scale experimentation into a high-impact, growth engine.

Since my time at Meta, Booking.com, and Canva, I’ve been helping APAC businesses do exactly that: building experimentation programs that don’t just run more tests or ship prettier dashboards, but deliver real strategic value.

If that’s something you’re working toward, you’re in the right place.

This phrase, “high-impact experimentation”, is central to what I do. But what does it actually mean?

In this first edition, I’ll share how I define it, how I measure it, and why most teams struggle to achieve it.

Why most teams fall short of high-impact experimentation

Early on, things feel good:

• Tests are getting shipped.

• A few big wins pop up.

• Stakeholders lean in.

Then the vibe shifts:

• Velocity drops.

• Wins get smaller or harder to interpret.

• The backlog grows, but nothing feels urgent.

The team’s still working hard. But impact? Unclear.
It all starts to feel… flat.

Why it happens: teams focus on the wrong things

When experimentation programs plateau, it’s rarely due to lack of interest or effort. It’s usually because teams are focusing on the wrong things or solving problems that aren’t actually theirs to solve (yet).

1. They focus on solutions, not problems

I get asked all the time about advanced experimentation techniques like CUPED adjustments, sequential testing, real-time pipelines. All great tools. All used by world-class teams at scale.

But what most people don’t see is the foundation those tools sit on:

• A culture of innovation

• Empowered cross-functional teams

• Clear problem ownership

• Executive sponsorship

Without that, adding complex techniques is like installing turbochargers in a car with no wheels.

Many frameworks out there—including some good ones—focus on maturity checklists: what tech stack you should have, how teams are structured, what features to build. That’s useful if you’re already scaling. But for most teams, these are solutions to the wrong problems.

High-impact experimentation starts with problems worth solving, not features worth building.

2. They struggle to communicate the right value to the right people

Even teams running genuinely good tests often struggle to win the support they need to grow. Why? Because they frame experimentation in their language, not their stakeholders’.

Experimenters talk about:

• Improved decision quality

• Avoided negative launches

• Nuanced customer insights

Executives, meanwhile, are asking:

• "Can this help us go faster with less risk?"

• "Can this unblock our teams or priorities?"

• "Can this scale my influence or align the org?"

High-impact programs don’t just generate value, they translate it.

That brings us to the framework I use to define, evaluate, and scale high-impact experimentation.

Defining high-impact experimentation: NAV

To make experimentation work at scale, you need a clear view of what "good" looks like. That's why I developed NAV: a simple but powerful framework to evaluate the overall impact of your experimentation program and uncover the right levers to improve it.

NAV breaks the impact of an experimentation program into three key drivers:

Number, Accuracy, and Value

Each lever represents one of the core drivers of experimentation impact. Pull them all in the right direction, and you get a compounding, organisation-wide learning engine.

Let’s break them down.

Number

How frequently, broadly, and operationally experiments are run: the pace and pervasiveness of learning across the business.

Number reflects how much useful experimentation is actually happening across teams, domains, and time. It covers factors like how quickly tests move from idea to launch, how much of the product surface is being explored, and how often customers experience change.

This matters because experimentation is a compounding process. The more frequently and broadly you run good tests, the faster your organisation learns and adapts. If testing is slow, siloed, or limited to narrow areas, growth stalls; not because of poor decisions, but because too few decisions are being tested in the first place.

Accuracy

How trustworthy, interpretable, and aligned experiment results are: ensuring that decisions based on experiments are sound.

Accuracy reflects how well your experiments are designed, analysed, and interpreted. It captures the clarity and rigour of your process from hypothesis to stats, to communication, and how well that process aligns with business reality.

This matters because poor Accuracy erodes trust and misleads decision-makers. Even with high test volume, if the underlying insights are shaky or misread, the program becomes noise or, worse, dangerous. High-accuracy programs don’t just produce “results”; they produce decisions people can stand behind.

Value

The extent to which experimentation drives real outcomes, shapes decisions, and compounds learning over time.

Value reflects the impact experimentation has on the business and not just whether results exist, but whether they’re used, whether they change outcomes, and whether they shape strategy over time.

This matters because testing is a means to an end. If nobody’s using the insights, or the business isn’t changing as a result, you don’t have an experimentation program. You have a reporting engine. High-Value programs are the ones that drive decisions, spark investment, and create momentum at every level of the organisation.

Want to know where your program stands?

You can now take a quick self-serve NAV assessment on my site:
👉 drsimonj.com/nav-survey

It only takes 3–5 minutes to complete.

Once you’re done, I’ll send you a personalised NAV assessment — including where you sit compared to world-class teams, and where the biggest opportunities for growth might be.

If you’re serious about scaling impact, this is a great place to start.

What’s next

NAV isn’t just a framework. It’s how I define high-impact experimentation and how I guide teams from “more testing” to real business impact.

So you can expect this newsletter to be packed with:

• Deep dives into moving each NAV lever and its sub-facets

• Real-world examples of high-impact programs (and how they got there)

• A growing library of tools to plan, run, and scale better experiments

• Tactical guides on moving from “test team” to “growth engine”

• Behind-the-scenes breakdowns of how I assess and advise teams using NAV

• And much more!

If this resonated, please share it with a team that could use a better compass.

Until next time 🙌

– Simon
Helping businesses scale high-impact experimentation
www.drsimonj.com

This week reminded me of something I care about: the value of helping others.

The catalyst was this LinkedIn post where I gave a shout out to 11 amazing data people. I did it because I’ve been learning from them and felt that others in my community might too! What I didn’t expect was what you see below: a massive spike in new followers.

In this newsletter edition I’ll share stories of how I’ve helped data peers and, in return, accelerated my career. I’ll also share advice for those of you who feel they’re not able to help others.

Why is this important

People have discouraged me from helping others in the past.

When I was an early-career academic, a professor told me to stop teaching other students how to code. Why? Because they said I was wiping out my advantage and wasting time I could have been using to write more papers. In other words, I should have been competing with my peers, not collaborating.

It was one of a few experiences that led me away from academia. I don’t see the world as a zero-sum game. One person’s success doesn’t need to be another’s failure. I have always enjoyed helping others because everyone benefits.

What landed me a job as a Data Scientist

As I was wrapping up in academia and learning more about data science, I was so passionate. I was learning all this new amazing stuff but had no one to talk to. So I turned to the internet, and I shared.

I started posting learnings on Twitter. I wrote and contributed to open-source code on GitHub. I started a blog (not this one). I even spent a few minutes a day answering questions on StackOverflow.

People started writing to me. I’d get follow up questions and also a lot of thanks. I’d even get other students and academics come up to me at events and say they read my blog or something and it helped them. It was, and continues to be, a motivating thing for me to help others in this way.

What I didn’t realise is that my internet presence had become my portfolio. I’d never had a job as a data scientist and I was failing to get anywhere applying with my CV. Until, one day, a recruiter contacted me. How?? They found me as a top-rated candidate via StackOverflow. I ended up taking the job,

Helping others launched my career in data science.

What got me promoted to Senior Data Scientist

To be a Senior Data Scientist you must be a craft expert who uplifts the skills and value of peers in your company.

When I started out as a data scientist, I noticed that a lot of my peers were using R but struggling. I dove in to help. I setup a training course and took my data peers through rigorous upskilling. I also turned a lot of my routine code into packages that became default inclusions for all R users at the company.

A peer shared feedback that work that often took them over 20 minutes now took them 2 minutes thanks to my help. That’s a 10x speed up. When you've got over 100 data scientists using R, that’s a significant uplift in the pace and quality of data work!

If I’m being honest, I never set out with that in mind. I enjoyed helping others learn. Seeing them do better was very rewarding for me. I also became better myself. Teaching solidified patterns, exposed me to interesting questions, and improved my public speaking.

This ended up being a defining factor in my promotion to Senior Data Scientist.

What got me promoted to Senior Data Science Manager and beyond

I hope that the value is clear to you by now. Similar stories litter my journey to data science manager, promo to Senior, and now PM.

So here’s why I see helping others as a career accelerant. By helping others, you:

• Scale yourself to solve more and bigger problems than you can alone

• Enhance your existing skills as you’re forced to articulate them

• Expand your skills as you learn to share and new things from those you help

• Interact with and become visible to others

There’s more but these are core ones for me.

I don’t think I can help others. What do I do?

Mental blocks to helping others is common (I’ve had them myself). Feelings of imposter syndrome or comparing yourself to more advanced peers. Let me try to persuade you to ignore these thoughts.

1. We all have something to share and people to help. Every time I’ve learned something, there’s plenty of people a few steps behind me I can pull up by sharing. They mightn’t be right in front of me but they’re out there.

2. We don’t know what other people find valuable. I’ve often stopped myself from sharing because I thought “oh, that’s stupid, of course everyone knows that.” Only to later find out that a lot of people would have found it useful to hear.

3. There are many ways to help people beyond sharing your knowledge. You can take notes, give feedback, repost content, prepare slides, whatever.

If this hits home, I have a challenge for you. Next week, block 15 minutes in your calendar to try one small act of helping a peer. Share a paragraph about something you did or learned, offer to help someone do their work, or give a shout out.

It will amaze you how something so simple can put so much value into the world. Plus, if you need someone to encourage you, reach out. You can tell me what you'd like to do and report back how it went. I promise to be encouraging and helpful!

You got this 👊

Turning data into a valuable business asset is hard.

For starters, we have a complex solution space: data warehouses, BI tools, stats, ML, and more. There are countless PhDs dedicated to niche details behind each of these! Being able to ‘do’ data science is hard.

Then, even when a company has all the best data people and tech, there's still a tremendous amount of waste! Dashboards with 0 views, unqueried data tables, analyses no one acts on, or ML models waiting to go into prod.

Failed projects litter the world of data.

It pains me to feel it’s more common than not. I’ve seen more data projects fail to be useful to a business than I’ve seen projects that make an impact.

So why is this??

Why are all these amazing projects, from intelligent people, failing to be of value? Well, I alluded to why in the second sentence. Can you spot it?

Problems vs Solutions

Here’s a simple diagram I drew when I first became a data product manager.

This diagram tries to represent that:

• There are problems to be solved, and solutions that can solve them

• Some problems can be solved by many solutions

• Some solutions can solve many problems

• Some problems can’t be solved

• Some solutions don’t solve any of your problems

Now, I see everything in the world of data on the right: as solutions.

• Statistics helps solve the problem of making decisions under uncertainty.

• Machine learning helps solve the problem of scaling complex behaviour.

• Data warehouses help solve the problem of needing to store masses of data for other work.

As a data specialist, this is an important thing to be aware of: you, and the skills you bring to a business, are solutions.

This begs the question: what the hell are the problems?

The Problem-Solution Gap in Data

This is where our challenge starts: a gap between business problems and data solutions.

The problems come from business leaders. Problems like how to increase market share? How to accelerate the growth of a product vertical? Or how to prioritise among initiatives given many possible trade-offs?

These problems are valid and it seems intuitive that data can help. It is one of the greatest pitfalls of the data world: the perception that data can solve almost anything. Contrary to this, data can't easily solve these problems. At least, not when they're articulated this way. Which now brings us to the heart of why data projects fail.

The Skills Gap in Data

At the heart of why most data projects fail is a skills gap:

• Most people don’t know enough about data to define problems in a way that data experts can solve them. OR...

• Data specialists don’t know enough about business to connect their solutions to the right problems.

And so what do we end up with?

We get business owners asking data specialists to solve any and all problems. Data specialists who fail to deliver on those. A breakdown of trust and interaction. Data specialists losing a seat at the table. Business owners operating without data again. Data specialists trying to add value by building solutions to problems that don’t exist.

Closing the Gap

Closing the gap requires closing the skills gap. It needs people to connect high-level business problems with data solutions. Here are a few options to think about.

One option is to insert people who can be the bridge. Sometimes they come from data. E.g., it's something I do as a Data Product Manager. They can also come from the business side. For example, business leaders or finance experts who moved into and up-skilled in data.

Another option is to invest in education. Data teams need to help educate leaders on the sorts of problems they can solve. Vice versa, leaders need to educate their data teams about business problems. This two-way education helps brings both sides a bit closer.

The final option I’ll suggest here is to acknowledge the challenge. Be open that you might be on the problem or solution side. Admit there’s a gap and ask for patience as you build the bridge. Collaborate in a very iterative way to maximise continuous alignment.

Give me the summary, Simon!

Alright, summary time.

Data Science and Analytics is a complex field but it’s one of “solutions.” The “problems” to be solved lie somewhere else. There can be a vast gap between business problems and data solutions. To compound the problem, very few people have the skills to bridge this. This is the #1 reason why I believe most data projects fail.

The result? Business leaders and data teams drift apart. The promise of data-accelerated business growth diminishes. Impact becomes something that data teams need to fight for.

If you want to land more impact with data, look at the problem-solution connection. Who is defining the problems and how are they articulated? Can a bridge be built from them to a meaningful data solution? Can the solution be delivered in a way that solves the problem? You can connect them but don't underestimate the challenge.

I’ve managed and worked with a lot of data specialists. I’ve spent time talking to them about their futures and what sort of careers interest them. I’ve gone on to try many paths myself and seen others try even more. Something I’ve discovered is that most data specialists only get exposed to a narrow set of paths.

So, in this edition, I want to break down seven career paths to consider if you’re a data specialist. I’ve written it from the perspective of a “Data Scientist,” but it applies to pretty much any technical data role. It’s also not an exhaustive list. They're the seven most-common paths I've seen myself or others go through. My hope is that this expands your consideration set and gets you excited for the future. Let’s go from what I consider the least to most change in your career.

1/ Stay Put

I’m a Data Scientist today and I’d like to be…
…a Data Scientist tomorrow.

What? Wanting to stay in your current role was a path that was foreign to me at first. I've learned that it’s a very desirable option for many. Sure, you might change team or something but we’re talking no promotions, no major changes.

Who? Meeting data specialists who wanted this was an eye-opening experience for me. You're out of the “rat race.” You are content with your work and salary. You enjoy having extra time and energy to spend on other parts of your life such as family.

Pro tip? It’s worth mentioning that this isn’t always an option. Some companies expect you to work towards a promotion over time. So be mindful. If it’s something you want, you’re going to have to find a company that respects that wish and values your ongoing work.

2/ Get Promoted

I’m a Data Scientist today and I’d like to be…
…a Senior Data Scientist tomorrow.

What? The most common (and very reasonable) option I hear is to get a promotion. E.g., to go from junior up to full, up to senior, up to principle, and so on.

Who? This is a great option for folks who enjoy what they do and want to keep developing their core, technical skill set. Take what you have, and make it deeper and broader to develop more business value.

Pro tip? It’s valuable to remember that a promotion is often the most visible and clearest path ahead. Your manager might emphasise this option because it’s a common and easier path to work through. So, while it could definitely be right for you, don’t forget to consider your other options.

3/ Be More Technical

I’m a Data Scientist today and I’d like to be…
…a Software Engineer tomorrow.

What? There is a natural pull for many data specialists to step into a related technical role. For example, I’ve seen Data Scientists move to Data Engineer or ML Engineer. It’s also not uncommon to see shifts into various software engineering roles.

Who? This suits people who find tremendous joy getting technically deep in their work. Especially when it comes to coding. It’s a way to push those skills into new domains and shift your focus on expected work and customers.

Pro tip? Done right, you can usually take this path and pretty easily maintain your salary as well. This is because you can often start to mimic the role you want quite closely before making the switch.

4/ Be a Manager

I’m a Data Scientist today and I’d like to be…
…a Data Science Manager tomorrow.

What? After a certain time in data, the manager path will become ever-visible. Still, I’ve put it at number (4), after promotions and technical pivots, because it’s a very different path. It brings a new and very real responsibility: the success of others.

Who? Do you enjoy collaborating with peers, strategising projects, and providing guidance? If you like to balance data work with organising and helping others, Management might be for you.

Pro Tip? All new managers go through a shared experience: Shock. Do not underestimate the shift from technical work to people. Test your fit by finding opportunities to lead small projects or mentor juniors.

5/ Be Less Technical

I’m a Data Scientist today and I’d like to be…
…a Product Manager tomorrow.

What? Another viable path is shifting away from technical specialisation. I took on the Product Manager role. I've also seen others move into roles like Technical Program Managers.

Who? For me, and others I've known, there's been a drive to tackle a different problem in a broader business context. Like management, it also often accompanies a people-centric skill set. It's worth a think if you enjoy collaboration and creating clarity for others.

Pro Tip? This path often means a pretty significant change in the role and type of work you’ll be doing. So find ways to minimise any other change. For example, find roles in your existing company and in the data or domain you're used to already. This gives you a chance to learn a whole new skill set but lean on your existing strengths.

6/ Try Consulting

I’m a Data Scientist today and I’d like to be…
…a Consultant tomorrow.

What? Transitioning to be a consultant, particularly independent consultant/freelancer, is exciting and scary. There is a freedom that comes with it but also a safety net that disappears. I’ve seen data specialists love and hate it.

Who? I've seen it work for individuals who adapt to new environments. They know how to provide engaging and actionable recommendations. It also requires a fair amount of sales and relationship management. If you’re pragmatic, enjoy networking, and working fast across diverse teams, this could be for you!

Pro Tip? Putting the data aspect aside, consulting is a unique skill in itself. There's a lot of unexpected work such as finding and retaining clients that may challenge you. If freelance is your goal, it can help to first work with a boutique consultancy or mentor to learn the ropes.

7/ Be a Founder

I’m a Data Scientist today and I’d like to be…
…a Founder tomorrow.

What? The world of data is full of smart and driven people. So it hasn’t surprised me to see peers go on to build something of their own. Sometimes alone, sometimes with co-founders, but always with conviction.

Who? I’ve seen this appeal to data specialists who first have a strong vision and passion for a specific idea. They also have a drive to see it come to life. This takes determination to navigate the challenges and uncertainties that come with entrepreneurship.

Pro Tip? I haven't done it but I’ve seen many data specialists turned founder later return to their previous work. This shouldn’t deter you but failure is a common reality. Give it your all but don't burn bridges on your way and consider when and if you're ready to take the risk.

Give me the summary, Simon!

Alright, summary time.

A career in data can translate to many paths. I’ve already tried a few myself and seen many others push themselves to new places. In this edition we broke down seven common paths that data specialists take in the hope of inspiring you to follow your best path.

To finish up, here’s the complete list. “Today I’m a data scientist/analyst/etc and tomorrow I’d like to…”

• Stay put

• Get promoted

• Be more technical

• Be a manager

• Be less technical

• Try consulting

• Be a founder

Whatever is right for you, go for it and good luck! Comment below or contact me if you think I can help give you a boost :)

I recently posted about this topic on LinkedIn and it seemed to strike a chord with many of you. So I thought it useful to go deeper into what you can do! Let’s dive in.

When I started as a data science and analytics manager I became swamped by stakeholder requests. I’d stepped up to lead the data teams in a department of over 200 people. Many people wanted regular data support but there was about 1 data scientist for every 20 others. And everything started being thrown at me.

I’d be lying if I didn’t share that it quickly became overwhelming. Stakeholders grew impatient as I wasn’t responding to their requests and I started to crumble.

I’ll soon share a solution that the VP of the department helped with me all those years ago and that I’ve re-used many times over. But, before we skip ahead, it’s useful to talk about the problem and how this (or other solutions) help relieve the pressure.

The Problem of Multiple Data Stakeholders

What causes all this pressure? I break it into three key drivers:

Scale

Say it takes 30min a week to gather and prioritise data needs from one stakeholder. Manageable.

New stakeholders and more requests add increasing complexity (more below). Say an extra 15min per stakeholder: +45min for Stakeholder 2, +60min for Stakeholder 3, etc.

Dealing with 10 stakeholders means spending 30+45+60+75+90+105+120+135+150+165min
= over 16 hours, two work days, or 40% of your time, just gathering and prioritising!

To deal with multiple stakeholders, you need a process to manage the sheer volume and complexity.

Competing priorities

What’s with all that extra time for each new stakeholder?

Most of it comes from having to deal with competing priorities. Each stakeholder has their own goals and agenda. Requests from a new stakeholder aren’t just extra items on your backlog. There’s a whole extra set of priorities you need to incorporate.

The hardest part: you need to convince folks who’s work isn’t prioritised that some other stakeholder had higher priority asks. Good luck.

You need a system for standardising request priority in a way that stakeholders can agree on.

Lack of visibility

Worst of all: all that pressure is invisible to everyone but you.

Each stakeholder is going to ask and push as much as they can because… why not? From their perspective, they need a job done and you can do it.

This is where the pain really compounds. You’re under pressure, you’re on edge, and it feels like people just keep piling on the work with complete disregard for your wellbeing.

You need a way to make your workload visible and help your stakeholders empathise with your situation.

Overcoming the pressure

I’m going to share an approach a VP helped me with a long time ago and that I’ve re-used in various forms:

1. Gather requests in a single space that everyone can access

2. Include a transparent scoring system to stack-rank all requests

3. Use senior leadership to break ties or handle conflicts

4. Track progress and record outcomes in the same space

5. Do this on a regular cadence that everyone is aware of (e.g., monthly)

To make it a little more concrete, check out or copy this spreadsheet template.

Get Template

How it relieves ‘Lack of visibility’

The first great thing about this approach is that it makes your pressure visible. Any stakeholder adding a request has to see everything else on your plate. It’s an amazing way to quickly build that empathy and understanding.

By later recording progress and outcomes, stakeholders can also see how requests are translating into business impact. It keeps everyone holding a high bar for entering meaningful asks.

How it relieves ‘Competing priorities’

Focussing stakeholders on the impact of you doing the work is the first important thing. It funnels everyone into thinking the same way and being mindful of the value you can bring.

Then, a single and transparent scoring system does most of the heavy lifting. Everyone can see and understand how you’ve made your prioritisation calls. If someone doesn’t agree, you can discuss and adjust as needed. Finally, in the case conflicts arise, have senior leadership on standby to break ties.

How it relieves ‘Scale’

Using some sort of tech (a spreadsheet is a fine place to start) means you can scale out request gathering. Having a regular cadence then time-boxes everything and lets you focus. For example, you send out the sheet last week of the month to all stakeholders with a reminder: “Please add your requests for next month in by end of the week. We’ll lock in our priorities the week after.”

Don’t sit still

Take this advice as a starting point but every business and context brings unique challenges. So expect to change things. Furthermore, if you only work like this, there’s a risk of becoming a centralised data service that loses business context and deep relationships.

What matters is that you are able to relieve the pressure and free yourself to do more impactful work.

Give me the summary, Simon!

Alright, summary time.

Are you feeling the pressure of too many data stakeholders? It’s likely you’re facing problems of scale, competing priorities, and a lack of visibility.

Get on top of these with a clear and transparent prioritisation system (here's that template again). Iteratively improve it be sure to use what relief you get to plan a better approach for the future!

I’ve had a few readers reach out and organise a chat. I’ve loved meeting these people, learning about their careers, and hearing what they’d value reading about in this newsletter.

So I thought I’d ask if you would like to meet as well? I’ve setup some regular times that we can talk via the link below (my evening in AEST).

Find a time to talk

If those times don’t work or talking isn’t your thing, feel free to email me at simon@drsimonj.com and we can go from there.

Or maybe we already know each other? No worries. Find some time and it would be great to reconnect!

If not, no worries at all. Your involvement with this newsletter and community is appreciated 🙏

Few things have a step change impact on your data science and analytics career more than clear communication.

Sadly, most data folks speak a special dialect. They use words people know like “power,“ “regression,” or “feature,“ but it’s all jargon. Most people hear gibberish and they tune out.

And I get it! Data science and analytics is a highly technical field. Many of us in this field come from even more technical backgrounds. Not that it’s needed but I and plenty of my peers have a PhD. We’re trained and live in a world full of jargon.

So how do we pull ourselves out of the jargon hole? A hole we don’t even know we’re in?! Because once we do, it opens up doors to influencing others, promotions, and roles in leadership.

My Secret #1: Playbacks

Here’s my first secret weapon: ask someone to playback what I said.

That’s it! Here’s an example of a playback conversation:

[me] “We didn’t have sufficient statistical power.”

[colleague] “Errr, ok.”

[me] “Can I just check I’m being clear, what do you think I meant by that?”

[colleague] “Sure! Ummm. There was something wrong with our statistics? I’m not really sure what the problem is though.”

[me] “Ah! Ok, sort of. I meant we haven’t got enough data yet to be confident in the results. So we probably need to gather more. Is that a bit clearer?”

[colleague] “Yep, much clearer! Thanks!”

Clear communication means easily getting a message into someone else’s head. The best way to know if you’ve done it is just ask :)

My Secret #2: Data Translation Dictionary

Playbacks are a great way to solve an in-the-moment comms problem but if you really want to become a great communicator, you need to learn from those experiences.

This is where my data translation dictionary comes in. Each time you do a play back, take note of the data words/phrases you started with along side the plain-language words/phrases that were better understood.

If you build it, improve it, and note what works for different audiences, overtime you’ll build up an amazing list and ability to communicate with anyone!

Want to get started? Here’s a selection of my data translations:

• Power -> Enough information to be confident

• Modelled -> Learned from

• Decision tree, Support Vector Machine, Neural Network, etc -> algorithm

• Variance -> Noise

• Null hypothesis -> Safe assumption

• Alpha criterion -> Decision Threshold

• Valid -> Measures what we think

• Probability -> Chance

• Segment -> Slice and dice

• Correlate -> Tend to move together

Give me the summary, Simon!

Alright, summary time.

Worried you’re not communicating clearly enough? Ask people to playback what they think you said, then note down what works in your own data dictionary!

Overtime you’ll become a far better communicator and create step-change opportunities for your data career.

Read time: 4 minutes

I vividly recall my first analysis as a data scientist. I was brimming with pride. I had gathered masses of data, processed it, and presented a set of amazing insights.

However, I soon realised the real challenge. It was not just producing great analyses but inspiring action based on those insights. To make a difference, I had to evolve my approach and transform data into actionable information.

In this newsletter I'll share the pivotal moments that shaped my journey and present a three-step plan to deliver impactful data insights.

What is “actionable” data?

Now, before we dive into the three steps, it’s probably worth getting on the same page in terms of what I mean by “actionable” data. Here’s my take:

Information that someone uses to take one course of action over others.

There are two subtle but important points to note:

1. “information that someone uses.” I consider data to be actionable if it is actually used or actioned. Yes yes, I can make all sorts of recommendations to decision makers but the data is useless unless they care about the problem enough to seek guidance.

2. “take a course of action.“ Behind this one is that the data must help decision makers navigate between actions they can truly take. Recommending “double investment” might be completely impossible.

With this out of the way, let’s walk through the three steps..

1. Find the Right Problems

Rather than jumping straight into data analysis, I learned the importance of engaging decision makers and understanding the problems they cared deeply about solving. I began asking questions such as:

• What challenges are you currently facing?

• What specific problems are you grappling with?

• How does the outcome of these challenges impact you?

• Do you have all the necessary resources, or could you use some help?

• Would additional information be valuable to you?

By aligning the data analysis with the decision makers' interests and pain points, I ensured that the insights provided would be relevant and compelling.

2. Explore the Options

In the past, I made the mistake of assuming that data would reveal the right course of action. However, I came to understand that it's crucial to consider the available options and potential actions before diving into analysis.

I started asking decision makers questions like:

• What potential solutions have you been considering?

• What are the current options on the table?

• Are there any actions you've already ruled out?

• Are there any desirable actions that you can't take due to constraints?

Understanding the decision makers' available choices allowed me to frame the data analysis in a more relevant and practical context.

3. Connect Data to Actions

With a clear understanding of the problem and the available actions, I delved into the data. Instead of just presenting insights, I focused on how each possible action could address the problem at hand. The result was a compelling summary that decision makers could readily act upon, such as:

• We're aiming to solve [insert problem here].

• After considering [insert possible actions here], our data suggests that [action X] is likely to be the most successful approach.

• This analysis dives into why.

By diving into a problem that mattered, then connecting the data directly to feasible actions, decision makers were equipped (and inspired) to take informed steps.

Give me the summary, Simon!

Over time I’ve learned that delivering impactful insights involves more than just presenting data. It requires (1) understanding the specific problems that decision makers are invested in, (2) exploring the options, and (3) connecting the data to actionable steps. Here’s a small graphic if you want a quick reminder:

This three-step plan can help data specialists to provide insights tha have a tangible impact instead of just presenting numbers on a page. I hope this approach helps to elevate your data analyses to new heights of effectiveness and usefulness!

Thanks for reading!

Thank you for reading my newsletter and I hope this was useful! Subscribe and comment below to hear more tips like these. You’re participation is what grows this community and helps me keep writing 🎉

Subscribe now

Subscribed? Share with someone else and let’s keep this community growing 🙌

Share

Beginnings are hard.

This is my first newsletter here. The beginning of what I expect to be a hard but very rewarding activity. It felt right to share why I started and what my plans are.

1. Why this newsletter?

For many years I put myself into the public sphere by blogging, presenting at conferences, etc. I loved it. I was part of a great community, I built a wonderful network, and it launched my career in Data Science.

Until, one day, I just stopped. It was when I moved from Data Scientist to Data Science Manager. I moved into a very different job. One where my focus shifted from numbers to people. I had little idea of what I was doing, much less what to write about.

My writing habit slowly died for the next 3-4 years :(

However, in that time I’ve learned a lot outside the “technical” skills that first got me into Data Science and got me writing. I’ve managed teams of over 20 and am now a Data Product lead.

I’ve learned a lot more “non-technical” skills. You might say “leadership,” “people,” or even “soft” skills. I like a label from a data legend of mine, Cassie Kozyrkov: “hardest to automate.” I’ve found these skills can be taught, learned, and constantly improved. I’ve greatly enjoyed growing and applying these skills to supercharge the careers of others and the capabilities of businesses in ways I could never do as a mostly technical individual contributor.

2. Why now?

So this year I set myself a challenge to write and share more. To build another wonderful community and network of people interested in this complex world of hard-to-automate skills. To learn from others and uplift folks who I might be able to help.

I decided to start with LinkedIn posts. I struggled to post just once a week but I built up the habit to now finding it easy to post 3x/week. I also started without focus, talking about anything and everything. I now feel the topics I care to talk about and connect with others on is clearer.

I haven’t figured it all out but I’ve found a groove. Finally, after all these years, I thought: “I’m ready to get back into longer-form content and share my thoughts and experiences with the world in more detail.”

3. What kind of community am I hoping to build?

I’m really hoping to build a community of folks who work in or with technical disciplines (mostly data science and analytics) and who want to grow as people leaders. A community who are excited to learn about problem discovery, communication, collaboration, leadership, motivation, and so on. Skills that help us stand out and create value in a world full of people, not machines.

4. What can you expect from me?

I’m aiming to publish one article per week — completely free. I’m thinking on a Saturday (but TBD). As with my LinkedIn posting, I expect it’ll take me some time to warm up. So please bear with me as I ramp up!

There’s still plenty I haven’t figured out. A catchy newsletter name for starters. But that’s all part of it. I’m going to lean into publishing without trying to be perfect. Then listen to what you and the community find valuable, and try do more of that.

5. How can you get the most value?

Participation is the best way to get more value. It starts by subscribing. Here’s a nifty button ;)

Subscribe now

Then by engaging. Likes are great. Comments are even better. Share your problems, things you want to learn, and even help by responding to other people. Comments are going to make you a part of the community. It’ll help you go beyond the information in this newsletter to solving problems specific to you and building your own network.

Finally, all this depends on us building a community. So please consider sharing with at least one other person to get started. Here’s another nifty button ;)

Share drsimonj's Newsletter

6. Let’s Go!

There’s not much more to say right now. Let’s go and build this community together. See you round :)