Capacity Planning for Agents: Measure Yield, Not Tokens

Every capacity conversation about agents I’ve been in this year starts with token demand. Goldman put out a 24x growth figure by 2030. The counter-argument is that 24x is conservative: run ten parallel agents on the same feature and you’re at 240x for that workflow alone, before the orchestration layer spends anything deciding which of the ten won.

I believe the forecast. I just can’t do anything with it. A demand curve tells you how many tokens the industry will burn, and nothing at all about whether burning them was worth it, and in practice it’s the second question that changes what you build.

The more interesting half of that debate is the part that gets quoted less: the best CTOs report that most parallel-agent code is never committed. The right unit isn’t tokens produced, it’s tokens that converted. OpenAI’s CFO published an AI ROI scorecard last week built on exactly that instinct, cost per successful task and return on compute. That’s the right instinct, and it’s a lot harder to compute than it looks.

A pipeline that knows its own yield#

Take a concrete example. A daily curation pipeline, three stages, the shape most scheduled agent jobs end up with:

# fetch (deterministic, network)  →  reason (agent, offline, Read/Write only)  →  deliver

Fetch is dumb code hitting a set of feeds. Reason is a single agent call with no network, told to curate rather than dump and to report what it dropped. Deliver writes the result somewhere a human will read it.

That middle instruction is the whole point. The output opens with a line like:

*<N> kept from <M> swept · <date>*

Across fifteen runs of one such pipeline, the funnel looks like this:

A 31.5% keep rate from candidate to published, which is the number the pipeline reports about itself. But the number that matters is at the other end. Fifteen days of a system running every single morning, and three items changed what anyone did.

That isn’t a criticism of the job. Three good redirects in three weeks is a fine return for something nobody has to operate. The point is that the last number exists at all, and it exists for a boring reason: the final stage is a checkbox sitting next to the proposal, in the same file, ticked or not ticked days later.

- [x] **Trim the prompts before Thursday:** Anthropic cut Claude Code's system
  prompt by 80% on the logic that smarter models need fewer examples, so audit
  the skills for example bloat while the week is quiet. ✅

The agent proposed. A human accepted. The acceptance got written down next to the proposal. Nothing about that is clever, and it’s more than most coding agents can tell you.

Acceptance happens after the telemetry stops#

The reason the number is missing elsewhere has very little to do with difficulty.

Every agent harness measures what happens inside its own process: tokens in, tokens out, tool calls, wall clock, sometimes a dollar figure. All of it is available the moment the run ends, so that’s where instrumentation stops.

Acceptance is on the other side of that boundary and it arrives late. A coding agent’s output becomes valuable when a diff survives review and lands, which happens minutes or days later, in a different system, decided by someone who isn’t thinking about telemetry. The harness reports the cost of the attempt and never learns the outcome. You get perfect accounting for the numerator and nothing at all for the denominator.

The curation pipeline dodges this by accident. Its accept event lives in the same artifact as its proposal, so the ledger and the work are one document.

What the metric requires#

Three things, and only the third is annoying:

1. A unit of work with a stable ID
2. Cost attached to that ID at run time
3. An acceptance event that resolves against that ID later

For coding agents the unit is the branch, so the ID already exists. Cost you take from whatever the harness reports. The lagging resolution is the piece you build, and for git-based work it’s a post-merge hook and a few lines of append-only JSON:

#!/usr/bin/env bash
# Resolve an agent run against the merge that accepted it.
branch="$(git reflog -1 --format=%gs | sed -n 's/.*merge \(.*\):.*/\1/p')"
[ -n "$branch" ] || exit 0

jq -c --arg sha "$(git rev-parse HEAD)" \
      --arg diff "$(git diff --shortstat HEAD@{1} HEAD)" \
   '. + {accepted: true, merge_sha: $sha, diff: $diff}' \
   "runs/$branch.json" >> runs/ledger.jsonl 2>/dev/null || true

Runs that never merge never get a line, which is the correct accounting. Sweep for stale branches, record them as rejected, and the ledger closes. The metric is then a one-liner over ledger.jsonl.

It has to be built deliberately, because no vendor will ship it for you. The harness vendors can’t see your merges, and it isn’t in their interest to publish the conversion rate on the tokens they sell you.

What it changes#

Model selection stops being vibes. A migration writeup reported 2.2x faster and 27% cheaper after moving a production agent to GPT-5.6. Cheaper per what, though? If the cheaper model needs three attempts where the expensive one needed one, per-token savings are a loss and the invoice still looks like a win. Only cost per accepted change catches that inversion, and it’s exactly where cheap models bite you.

Parallel agents become a bet you can settle. The 240x multiplier assumes running ten versions and picking the best is worth ten times the spend. Maybe it is, for a hard feature. Almost certainly it isn’t for a config change. A yield ledger makes that an empirical question per task class instead of an article of faith.

It’s the only lever you own. Look at the last two weeks. OpenAI cut Codex’s context window from 372k to 272k, visible in a PR diff. Moonshot paused new Kimi K3 subscriptions after demand hit their compute ceiling inside 48 hours. Anthropic reshuffled which Fable 5 tier gets what fraction of the limits, twice, in a week. You cannot forecast supply that moves like that, and anyone selling you a capacity plan built on reserved-instance portfolios is planning around a variable the vendor changes on a Tuesday. What you control is how much useful output you get from whatever capacity you’re granted this week.

Where this is weak#

Acceptance is a lagging, noisy signal. People merge things they shouldn’t and reject things they should have kept, so the ledger measures reviewer judgment as much as agent quality. Over a big enough sample that’s fine. On a slow week it’s mostly noise and you shouldn’t tune anything on it.

Some work has no discrete accept event. Exploration, research, the reading that changes what you build three weeks later. A curation pipeline is largely that, which is why three acted-on items out of fifteen runs undersells it. Optimize a research pipeline for acceptance rate and you’ll destroy it.

The metric is also gameable in the obvious direction. Reward acceptance and you get agents proposing small safe diffs, because those merge. The counterweight is weighting by size or by what shipped rather than by count, which is more bookkeeping than most teams will do.

I’d still rather have a noisy number on the right quantity than a precise one on the wrong quantity, which is what token spend is.

The takeaway#

The forecasts are probably right and they’re the wrong instrument. 24x or 240x tells you what the industry will burn, not what you should. It’s a supply-side number dressed up as a planning input, and it arrives at the moment you have least control over supply.

Build the ledger instead. Give every agent run an ID, attach its cost, and record what happened to it after the run ended. The gap between what your agents produce and what anyone accepts is your real capacity number, and right now almost nobody can see theirs.

I’m a platform/SRE engineer writing about making agentic AI reliable in production. If you’re running agents at any scale and can’t answer what a merged change costs you, get in touch.