Review Is the Capacity Plan for Coding Agents

One of the better talks I’ve seen on coding agents this year is Spotify’s Rewriting All of Spotify’s Code Base, All the Time, from QCon London. Jo Kelly-Fenton and Aleksandar Mitic walk through Honk, their background coding agent, and somewhere in the middle Mitic says this:

We were so happy to announce that we had 1,000 merged PRs in 3 months. We were over the top. Today, we do this in 10 days, and this number keeps going up. What is the result of this? It’s a bunch of PRs to review. This is becoming the new bottleneck.

That’s pretty much the argument of this post. Once agents write the changes, what runs short is the CI that checks a change and the person who has to say it’s safe to merge, so those two need a capacity plan. I’d size them before I worried much about the token bill.

The first queue is CI#

Before Honk hit the review problem it hit a verification problem, and the way they solved it is the part I’d copy.

Early on, Honk built and tested code wherever it happened to run. That was fine on an engineer’s laptop and fell apart once it ran in a Linux pod across the fleet. Kelly-Fenton describes rolling it out to hundreds of repos and seeing hundreds of failures, missing permissions, Docker not loading, iOS builds that simply can’t run on Linux, “nothing to do with the generated code, but they’re very much to do with the infrastructure around the code.” They were rebuilding their CI inside the agent runtime, one repo quirk at a time.

What changed it, as Kelly-Fenton puts it: “if we could separate out our verification runtime from our agent runtime, then we could leverage the pre-existing CI system that’s already set up.” The agent still runs quick local checks, but before it opens a PR it pushes a branch and a verification service kicks off the real CI build, waits, summarizes the failures, and hands them back to the agent. Only a green build turns into a PR. “We don’t want to create garbage PRs.”

I like this a lot because it’s a platform answer. The CI system already knows how to build every repo in the company, and now the agent just calls it. The catch is that every agent attempt is now CI load. A migration across a couple of thousand repos is a couple of thousand builds, plus retries, and that lands on the same runners your engineers are waiting on. That’s the first number I’d want on a dashboard if I ran a platform with agents on it: CI minutes spent on agent changes, and how many of those attempts never became a PR.

DoorDash’s feature flag cleanup has the same shape, going by InfoQ’s write-up (I couldn’t get DoorDash’s own post to load, so treat the details as InfoQ’s account). Agents run builds, tests and static analysis, and “a pull request is opened only after the validation checks pass.” In their evaluation of 50 stale flags, InfoQ reports usable PRs for 45 of them.

The second queue is people#

Requiring green CI before a PR exists keeps garbage out of the review queue, but every one of those correct PRs still needs a code owner to look at it and decide, and Mitic is honest that they haven’t cracked it: “I would be lying if I said we’ve solved all of this. We haven’t.”

What they’re doing about it reads like a capacity plan to me, even though nobody in the talk calls it that:

• Expectations on review timeframes, so a PR author knows when to expect an answer.
• Being deliberate about which PRs shouldn’t be reviewed at all. “Stale PRs get closed. We will have to try again.”
• Letting the person driving a migration approve it in some cases, since the owner of the library knows best whether the change to it is correct.
• One PR inbox instead of review requests scattered across Slack, mail and GitHub Enterprise.
• Working out which PRs can be auto-merged. They float docs changes, or internal systems where a bad change just gets reverted.

Then Kelly-Fenton’s section on standardization, which I think is the most interesting bit. Review is slow partly because every PR is different, and every PR is different because the codebase is. Their migrations used to stall at 70 or 80 percent of the fleet, which left two ways of doing everything, and both prompts and reviews have to account for both. Killing that long tail makes the agent’s output more predictable, which makes it faster to review, so some of their reviewer throughput gets bought with migration work done ahead of time.

Even the automated reviewer has a quota. Microsoft’s Copilot code review for Azure Repos documents five concurrent reviews per organization, two per user, one per pull request. Fine as a limit, but if you plan to put an AI reviewer in front of agent-written PRs, the reviewer is a rationed resource too, and someone has to size it.

What I’d actually measure#

None of this needs new tooling to start. I’d put these next to whatever token dashboard already exists:

• Open PRs authored by agents, and how old they are.
• Time from PR opened to first human review, split by agent and human authors.
• Agent PRs closed without merging, and why.
• CI minutes per merged agent change.

If the token spend goes up and merged changes don’t, the gap is sitting in one of those four, and I’d bet it’s usually the second.

Where this loses#

Scale is the obvious one. Spotify has thousands of repos and a fleet management system that already opened PRs automatically before LLMs came along. A team of eight with one repo is more likely stuck on the agent getting things wrong, and for them generation quality really is the bottleneck. I wouldn’t tell that team to go build a verification service.

It’s also mostly one company’s story. The talk is a first-hand account with real numbers, but it’s two engineers describing their own system, and the review-side fixes are, by their own say, still in progress. The DoorDash numbers are one evaluation of 50 flags as reported secondhand.

And the bottleneck might move again. If auto-merge rules get good enough that most low-risk agent changes never need a human, the scarce thing becomes deciding what counts as low-risk, which is a policy problem more than a capacity one. I don’t think anyone’s there yet, but the heuristics Spotify mentions point that way.

The takeaway#

When agents write the changes, the thing that runs out first is the ability to check them. Spotify’s first fix was architectural, moving verification out of the agent’s sandbox and into the CI that already knew how to build everything, and their second is still being worked out in review policy and standardization. If you run the platform those agents land on, plan CI and review capacity the way you’d plan any other queue. That’s where the agent’s output is sitting.

I write about agents as a workload on real platforms: where they run, what they cost, and which queue they end up stuck in. More about me.