Hi Simon! Thanks for sharing your thoughts on this, its highly appreciated.
I agree with the general view, but I think most of what you are "seeing" now it was also true before AI. The cost of "building" things has been trending downwards for many years now. So the "bottleneck" was always on the "soft" side of things: Management, research, prioritization, business context, etc.
With AI that "cost" now became even less very rapidly, which makes the constrains in all the other areas to become more obvious. And on some of those AI could also help, like summarizing big volumes of research data, or speeding up reportings and dashboards and the likes for results presentations and so on.
The part I dont agree on is this:
Operational autopilot: the statistical and design mechanics that currently require specialist effort need to be automated, so that increased experiment volume doesn’t require proportionally increased human overhead.
Proper research, prioritization given business and customer context, and statistical design are all high editorial tasks, where AI usually falls short. Can it help making it faster? Sure, but automated? Right now at least I dont think so or even if it could, cant see why someone would want that.
What I can see happening in the short term is that with the cost of building becoming MUCH lower thanks to AI, exploratory "tests" becomes much more lucrative. Approaches like "2 Stage" one that we researched on, or any explore - exploit method is has probably better payoffs now, since for each new Hypothesis you can now maybe come up with 10 or 20 "treatments" and pick up real potential candidates for a subsequent confirmatory test much sooner.
So before some hypothesis maybe required 2, 3, 4 “confirmatory” iterations with "bigger" samples until getting a consistent or good enough winner, you can now do a “2 stage test” in the same or even less time.
I really like this rapid speed through smaller/incremental tests towards big ideas take. It makes a tonne of sense. I also agree that "Proper research, prioritization given business and customer context" won't be going anywhere and not on auto-pilot. BUT, I think the level at which it happens will be changing (though maybe you feel this way too?). Eg, we don't need to prioritise now between working on the layout of the homepage or the ranking of the search results. Hell, we don't even really need to prioritise between building some entirely new products. But we do need to prioritise who we're serving, why, what success means, etc. So the human thinking will gradually be more and more in the strategic space.
Either way, I'm excited to see where it takes us and thanks for sharing!
This is an interesting perspective—seems very much like a potentially positive way for things to play out.
There does seem like there's a *little* bit of an echo of the early days of GUI-driven testing platforms (read: Optimizely) when there was a frenetic pace of "just do stuff and test everything". It took years to start to get past that mentality so that there would be some measure of deliberation before testing something. Question #1: do you see any risk of this? Basically, "We're going to be waiting for test results, so we need to just kick up the volume of 'things' we're testing (across former siloes) so that, once we get a bunch rolling, we'll have a steady stream of results flowing in." Basically, it's so fast to build that thoughtful progress goes out the window.
Question #2: do you see this envisioned future having any impact on the various challenges of tests that overlap? I feel like a case could be made that this could get worse, but maybe it's actually no different than the current (and, with properly designed tooling and processes, it's not an issue?).
Hey Tim and good take! I think the world being unlocked by AI means lots of stuff is going to get thrown out there regardless. So my hope is that testing still plays a strong role helping to filter out the crap (but it'll have to keep up). I believe the "hard thinking" will inevitably move up the decision stack though. More to goals, metrics, strategy, etc, and less on the specifics of what to build and test.
Re Question #2, I am definitely thinking about this! I think people over-hype the overlap problem today anyway, so we've got room to move. However, there's a limit to everything. So overlapping probably will become a challenge as testing scales rapidly. I believe there's room for more intelligent handling of this. Something we'll have to chat about some time :)
Hi Simon! Thanks for sharing your thoughts on this, its highly appreciated.
I agree with the general view, but I think most of what you are "seeing" now it was also true before AI. The cost of "building" things has been trending downwards for many years now. So the "bottleneck" was always on the "soft" side of things: Management, research, prioritization, business context, etc.
With AI that "cost" now became even less very rapidly, which makes the constrains in all the other areas to become more obvious. And on some of those AI could also help, like summarizing big volumes of research data, or speeding up reportings and dashboards and the likes for results presentations and so on.
The part I dont agree on is this:
Operational autopilot: the statistical and design mechanics that currently require specialist effort need to be automated, so that increased experiment volume doesn’t require proportionally increased human overhead.
Proper research, prioritization given business and customer context, and statistical design are all high editorial tasks, where AI usually falls short. Can it help making it faster? Sure, but automated? Right now at least I dont think so or even if it could, cant see why someone would want that.
What I can see happening in the short term is that with the cost of building becoming MUCH lower thanks to AI, exploratory "tests" becomes much more lucrative. Approaches like "2 Stage" one that we researched on, or any explore - exploit method is has probably better payoffs now, since for each new Hypothesis you can now maybe come up with 10 or 20 "treatments" and pick up real potential candidates for a subsequent confirmatory test much sooner.
So before some hypothesis maybe required 2, 3, 4 “confirmatory” iterations with "bigger" samples until getting a consistent or good enough winner, you can now do a “2 stage test” in the same or even less time.
I really like this rapid speed through smaller/incremental tests towards big ideas take. It makes a tonne of sense. I also agree that "Proper research, prioritization given business and customer context" won't be going anywhere and not on auto-pilot. BUT, I think the level at which it happens will be changing (though maybe you feel this way too?). Eg, we don't need to prioritise now between working on the layout of the homepage or the ranking of the search results. Hell, we don't even really need to prioritise between building some entirely new products. But we do need to prioritise who we're serving, why, what success means, etc. So the human thinking will gradually be more and more in the strategic space.
Either way, I'm excited to see where it takes us and thanks for sharing!
This is an interesting perspective—seems very much like a potentially positive way for things to play out.
There does seem like there's a *little* bit of an echo of the early days of GUI-driven testing platforms (read: Optimizely) when there was a frenetic pace of "just do stuff and test everything". It took years to start to get past that mentality so that there would be some measure of deliberation before testing something. Question #1: do you see any risk of this? Basically, "We're going to be waiting for test results, so we need to just kick up the volume of 'things' we're testing (across former siloes) so that, once we get a bunch rolling, we'll have a steady stream of results flowing in." Basically, it's so fast to build that thoughtful progress goes out the window.
Question #2: do you see this envisioned future having any impact on the various challenges of tests that overlap? I feel like a case could be made that this could get worse, but maybe it's actually no different than the current (and, with properly designed tooling and processes, it's not an issue?).
Hey Tim and good take! I think the world being unlocked by AI means lots of stuff is going to get thrown out there regardless. So my hope is that testing still plays a strong role helping to filter out the crap (but it'll have to keep up). I believe the "hard thinking" will inevitably move up the decision stack though. More to goals, metrics, strategy, etc, and less on the specifics of what to build and test.
Re Question #2, I am definitely thinking about this! I think people over-hype the overlap problem today anyway, so we've got room to move. However, there's a limit to everything. So overlapping probably will become a challenge as testing scales rapidly. I believe there's room for more intelligent handling of this. Something we'll have to chat about some time :)