What goes wrong when AI agents run growth experiments?

Three failures recur. Experiment soup, where the portfolio drifts toward trivial tests because nobody is choosing problems anymore, only approving tests. Metric gaming, where the agent optimizes the number it was given rather than the outcome the number represented, at scale and sincerely. And an accountability gap, because the agent shipped it has never been an acceptable answer when revenue dips.

What should you instrument before letting agents run the growth loop?

Provenance first: every variant, allocation decision, and shipped change needs a record of which agent produced it, from what input, against which metric, so the revenue question takes minutes rather than a forensic week. Then a weekly portfolio review that looks at what the loop chose to work on rather than how the tests performed.

The Agentic Growth Team: When Agents Run the Loop

Q: What does an agentic growth team look like?

Fewer operators and more editors. Agents run the mechanical middle of the experiment loop: mining event data for drop-offs, generating variants, writing test configurations, allocating traffic toward winners, and watching guardrails. Humans keep 4 things: choosing which problems deserve experiments at all, writing metric definitions precise enough that satisfying them means real user value, setting guardrails as hard limits enforced at execution time rather than dashboards reviewed weekly, and making the decision to kill a whole line of optimization.

What does an agentic growth team look like? Fewer operators, more editors. Agents run the mechanical middle of the experiment loop: mining event data for drop-offs, generating variants, writing test configurations, shifting traffic toward winners, watching guardrails. That part is solved.

Humans keep 4 things, and those 4 are the job now. Choosing which problems deserve experiments at all. Writing metric definitions precise enough that satisfying them means real user value. Setting guardrails as hard limits at execution time rather than dashboards reviewed on Fridays. And deciding when a whole line of optimization is finished. Remove the capacity constraint on experimentation and the bottleneck moves to judgment, which is the part nobody staffed.

Growth teams spent a decade automating the funnel. What is being automated now is the loop itself: the cycle of analyze, hypothesize, test, and ship that used to be the team's entire job. Agents already run most of that loop well. The interesting question is what happens to the parts they run badly.

What do agents actually do well?

The middle of the experiment loop is mechanical, and agents are good at mechanical. Mining event data for drop-off anomalies. Generating variants of copy, layouts, and offers. Writing test configurations. Allocating traffic toward winners while the test is still running. Watching guardrail metrics and flagging movement. A loop that took a growth team a week now runs continuously, in parallel, without a standup.

This is current reality, not a forecast. Experimentation platforms ship agents that set up a test from a prompt. Decisioning systems reallocate traffic in real time. A team of 2 with agents can run more experiments than a team of 10 could in 2022. The capacity constraint on experimentation is gone.

What happens when the capacity constraint disappears?

Remove a constraint and the bottleneck moves. In most teams it moves to judgment, and the failure has 3 recognizable shapes.

Experiment soup. Agents optimize what is measurable this week. Button copy, subject lines, banner placement, send times. Each test is individually reasonable. The portfolio drifts toward the trivial, because nobody is choosing problems anymore, only approving tests. A quarter later the team has run 400 experiments and cannot name the 3 that mattered.

Metric gaming. An agent optimizes exactly the number it is given, which is a different thing from the outcome the number was meant to represent. Define activation as first prompt submitted, and the agent will get everyone to hit submit. Define retention as a week-4 login, and it will manufacture logins. Humans game metrics too, but slowly, and with some awareness of what they are doing. An agent does it at scale, sincerely, and reports it as a win. This gets considerably more expensive when the metric is also the invoice, which is the situation I described in When Resolved Becomes a Price.

The accountability gap. An experiment goes out, revenue dips, and someone senior asks what happened. "The agent shipped it" has never once been an acceptable answer to that question, and it never will be. Autonomy without a named owner is a resignation letter on a delay. The same gap sinks agent deployments outside growth, which is most of what I found in Production Is an Org Problem.

What do the humans keep?

The teams handling this well look different from the growth pods of 5 years ago. Fewer operators. More editors. The humans keep 4 things, and the 4 things are the job now.

Problem selection. Deciding which part of the funnel deserves 1,000 experiments and which deserves 0 is strategy, and agents are poor at it precisely because it depends on context that never made it into the event stream: the board conversation, the competitive move, the thing 5 churned customers said on calls last month.

Metric definitions. This is now the highest-leverage artifact on the team. A definition loose enough to game is a production bug, because something tireless will be optimizing against it by tomorrow morning. Writing activation, retention, and expansion definitions that cannot be satisfied by anything other than real user value used to be analytics hygiene. It is now the spec for the machine that runs your growth, and the standard for what makes a definition worth having is the one I set out in Metrics That Move Teams.

Guardrails as constraints, not dashboards. A guardrail metric a human reviews weekly is a suggestion. Agents need hard limits enforced at execution time: floors on revenue and activation, caps on how much traffic any experiment touches, categories of change that always require a human approval. The difference between a dashboard and a constraint is the difference between noticing damage and preventing it.

The kill decision. Agents are systematically bad at stopping. They will keep squeezing a local maximum long after the honest conclusion is that the flow needs rethinking, because rethinking is outside the loop. Deciding that a whole line of optimization is exhausted stays human work.

What should you instrument before handing over the loop?

Provenance first. Every variant, every allocation decision, and every shipped change needs a record of which agent produced it, from what input, against which metric. When the revenue question comes, the answer has to take minutes, not a forensic week.

Then a portfolio review with a fixed cadence. Once a week, a human looks at what the loop chose to work on, not how the tests performed. Performance review is the agent's job. Portfolio review is yours: is the machine still pointed at the right problems, or has it spent 6 days perfecting a button nobody's revenue depends on.

If the agents are also touching the product surface rather than only the funnel, the release path needs the same treatment, which is the argument in Evals Are the New QA.

The uncomfortable symmetry

Growth teams spent years telling the rest of the company that intuition should be replaced by experiments. The loop is now doing to growth teams what growth teams did to everyone else, and the discomfort is identical. The honest response is also identical: keep the judgment, delegate the mechanics.

Running the loop stopped being a scarce skill the day agents learned to run it. What stays scarce is choosing what the loop should chase, and writing definitions precise enough that chasing them is the same thing as growing the product.