← All Writing
· 10 min read

Scale to Zero Is a Queue Feature

Kubernetes v1.37 shipped scale-to-zero autoscaling as a default-on beta, and the release post names its own precondition in one sentence: this works when the work can wait in a durable queue. An agent parked on a human approval has no queue holding it, so the saving has to be bought somewhere else. Three teams have shipped three different answers, and only one of them has a number attached.

A 3D wireframe diagram drawn in thin glowing white lines on a solid pure black background, with no other colour anywhere. The frame holds four objects arranged in two rows, all seen in three-quarter perspective. The upper row has two objects joined together. On the left is one long open wireframe cylinder lying on its side, built from evenly spaced circular ribs joined by straight lines so the interior is completely see-through, and resting inside it is a row of small wireframe cubes stacked against one another. A single straight thin white line runs from the open right end of that cylinder across a wide black gap to the second object, a low flat wireframe tray drawn as an open rectangular frame with cross-bracing and nothing resting on it. The lower row repeats the tray but has no cylinder at all. The entire left half of the lower row is empty black space containing nothing. In the middle of that emptiness one single small wireframe cube floats alone, with nothing beneath it, nothing around it, and no line touching it on either side. To its right sits the second low flat wireframe tray, identical to the first and also empty, separated from the floating cube by a clear black gap with no connection between them. Wide bands of empty black space surround both rows.

Kubernetes v1.37 shipped horizontal autoscaling down to zero replicas. It is beta and enabled by default, which means a capability that used to need an add-on or an alpha gate now sits in every cluster that upgrades.

The release post is unusually honest about what it costs, and it puts the whole trade in two sentences. The first:

The trade-off is cold-start time: the HPA must observe the metric, schedule a Pod, and start the application. This works well when work can wait in a durable queue.

The second:

Kubernetes Services do not buffer requests while no Pods are ready, so HTTP and other request-driven workloads need a separate buffering layer.

Read those together and the precondition is doing all the work. The queue in front of the autoscaler is what made scaling to zero safe, and v1.37 lets the HPA take advantage of something the queue was already doing for free.

My claim is that “can I scale this to zero” is a custody question. Something has to hold the pending work while nothing is running, and for a queue consumer the queue does that job without anyone deciding it should. An agent parked on a human approval gate has no owner for that job by default. That is why “just scale the idle agents to zero” turns into months of platform work for anyone whose agents wait on people.

What the queue is actually doing#

The queue looks like one component and it is doing three jobs.

It holds the work item durably, so the item survives the absence of any consumer. It exposes depth as a number, which is exactly the object metric or external metric the HPA needs to make a decision. And depth crossing zero is the wake signal that brings a Pod back. You get custody, the metric and the trigger from one piece of infrastructure you were already running for unrelated reasons.

A Deployment behind a Service has none of the three. A request arrives, no Pod is ready, and nothing anywhere is holding that request. This is why the release note sends you off to build a buffering layer, which is a much bigger job than that clause suggests.

The agent shape#

I argued a couple of weeks ago that the pod is the wrong unit for an agent because an agent is three things with three different lifetimes. I set the utilization argument aside there, since idle cost is the symptom everyone notices first and I wanted the structural problem underneath it. This post is the symptom, taken seriously.

An agent run executes for eight seconds, proposes a change, and then waits on a person who is in a meeting or asleep. The wake signal is a human clicking approve, possibly on Monday.

Ask the three questions of that gap. There is no queue depth to read, because the pending thing is not a message in a broker. Nothing holds the run durably either, since it lives in the memory of a process that is about to be scaled away. And the wake signal is an inbound HTTP request from a browser, which is precisely the case the release note says Services will not buffer.

The custody problem also gets harder in proportion to how much state the parked thing owns. A queue consumer parks holding nothing at all, which is why its story is so clean. An agent parks holding a conversation log, a working filesystem, a checkout, and credentials that someone scoped to this run, so the thing needing custody is closer to a machine than to a message.

Three answers, and what each one costs#

Make the parked run a row#

The cleanest version I have read recently is an InfoQ article on running durable workflows directly on Postgres, with no external orchestrator. The mechanism is an ordinary database table:

FOR UPDATE SKIP LOCKED turns an ordinary Postgres table into a concurrent work queue where each row is claimed by exactly one worker.

Crash recovery is a lease and a sweeper, where workers heartbeat the rows they own and a periodic query re-enqueues any execution whose lease expired. And the number that matters for the idle question:

No matter how long a wait or approval window is, whether it is two hours or two weeks, the cost is the same: just one row.

That is custody solved for the price of a row in a database you already run, and the article is specific that it covers approvals: waiting for a human to approve from Slack is durable because the state lives in Postgres rather than in a thread in memory.

What it costs is subtler than the row. Your agent’s parked state has to be expressible as workflow state, which is easy for an approval and awkward for a half-built working directory. And you have taken ownership of a sweeper, which is a control loop with a nasty failure mode. When a sweeper stops running it does not throw. It stops resuming things, and you find out from the person who has been waiting since Monday.

Make the substrate resumable#

The other direction is to stop trying to have nothing running, and make a parked machine nearly free instead. Felipe Huici, CEO of Unikraft, demonstrated the extreme version: over a million scale-to-zero virtual machines on one box, with a few kilobytes of metadata per sleeping VM, a controller using around 12GB to manage the fleet, and startup he describes as “about 10 milliseconds” and “fairly constant as you ramp things up”. Getting to a million snapshots takes roughly 12TB with compression and differential snapshots. The capability he names for it, stateful scale to zero, is the honest label for what agent platforms actually want, because the machine resumes exactly where it left off.

Fly’s Sprites make the same bet commercially. Their pitch is that sprites “appear instantly, always include durable filesystems, and cost practically nothing when idle.”

Both are worth reading and neither is evidence yet. The Unikraft demonstration is a vendor on a conference stage scaling nginx, a workload with no session, no credentials and nothing to diverge across restores. Fly says “practically nothing”, and for a claim whose whole value is a cost comparison, no figure appears anywhere in the post.

Keep a pool warm and stop calling it zero#

The Kubernetes Agent Sandbox project takes the position that idle should be cheap rather than absent. Its overview lists deep hibernation, “saving state to persistent storage and potentially archiving the Sandbox object”, automatic resume, “resuming a sandbox on network connection”, and a SandboxWarmPool that “manages a pool of pre-warmed Sandboxes that can be quickly allocated to users”.

Automatic resume on network connection is the buffering layer wearing a different hat. Somebody still has to hold that connection open while the sandbox comes back.

The warm pool is the one option here with a price you can compute before committing to it, because the price is simply the pool. I suspect that is why it keeps winning arguments. The alternatives are cheaper in a way nobody has published, and a saving you cannot put on a slide loses to a cost you can.

The shape that does fit#

Custody is easy when the wake time is known in advance. AWS wrote up scheduling email campaigns with EventBridge Scheduler, creating one schedule per recipient rather than a cron job that sweeps a table looking for work:

Amazon EventBridge Scheduler holds that intent with zero compute cost until the moment arrives, then triggers the scheduled action.

The service supports billions of schedules against a default quota of 10 million. It works because the wake signal is a clock, and a clock fires without needing a consumer or a depth metric. Agent work that runs on a schedule inherits all of this and should take it. The awkward cases are the runs with no clock, where the trigger is a person.

Where this loses#

A lot of agent work genuinely is queue-shaped, and for that work the queue answer is correct. If a run starts from a ticket, a webhook, a nightly sweep or a merged pull request, there is a durable queue in the picture already and scale to zero does exactly what it says on the label. That describes most teams’ first production agents. The argument here only bites for interactive and approval-gated runs, and a platform team that applies it everywhere will have built custody machinery for workloads that never needed any.

“You need a separate buffering layer” overstates the work. For HTTP this has been a solved product for years. Knative’s activator buffers requests for a scaled-to-zero revision: when a request is routed to a Revision without available capacity, the activator “delays the request and signals to the autoscaler that additional capacity is needed”, then forwards it once pods are ready. If the only thing parked is an HTTP endpoint, this is a dependency you adopt in an afternoon. The custody problem stays hard when the parked thing owns a filesystem, and gets much easier when it does not.

And I cannot price the comparison, which is the honest limit of everything above. There is no published figure for what an idle agent sandbox costs per hour against the row that could have replaced it. The best numbers in this post come from a vendor demonstration of nginx. I believe the direction because the mechanics are clear and the custody argument holds whatever the price turns out to be, though believing a direction is a weaker position than reading a bill. Anyone rebuilding an agent platform around this should measure their own idle fraction first, which is cheap to do and which nobody I have asked has done.

The takeaway#

Before anyone sets minReplicas: 0 on something agent-shaped, answer two questions in writing. What holds the run while nothing is running, and what wakes it up.

If the answer to the first is “the pod”, scale to zero is not available to you yet, whatever the autoscaler claims to support. If the answer to the second is “a person”, you also need something holding their connection when they come back, and that thing is a component with an owner and an on-call rotation rather than a checkbox in a manifest.

One real agent run is enough to size the work. Find its longest gap, then work out what would have to change for that gap to cost a single row. The distance between where you are now and that answer is the actual project, and it is usually not the autoscaler.

Working out which component is holding the thing that must not be lost is the same job whether the workload is a payment, a Flink pipeline or an agent waiting on a human, and agents have made a lot of teams ask it for the first time. If that is the sort of work your team is hiring for, here is what I do.

Related

All Writing →
Review Is the Capacity Plan for Coding Agents

Review Is the Capacity Plan for Coding Agents

Spotify's coding agent went from 1,000 merged PRs in three months to the same number in ten days, and the team's own talk names what happened next: a pile of PRs nobody had time to review. Once agents write the changes, the capacity worth planning is CI and human review, and the first fix is moving verification out of the agent's sandbox.
The Validator Is the Platform

The Validator Is the Platform

An InfoQ playbook cut an LLM system's hallucination rate from 15% to 1.5% without touching the model, and the article credits most of that to plain code that checks the output before anything uses it. That code is the part a platform team should own, and what it's worth depends on whether it checks the model's word or checks the world.
Proxy Variables Are Not an Egress Policy

Proxy Variables Are Not an Egress Policy

Injecting HTTPS_PROXY into every pod at admission works fine for apps that cooperate, and an agent is a workload you can't count on to cooperate. Two recent OpenAI incident reports show an agent getting told no and going looking for another way out, so the egress decision has to be enforced below the workload by something it can't read or route around.