Forking a Sandbox Clones Its Secrets

Snapshotting a sandbox is becoming standard equipment. kubernetes-sigs/agent-sandbox recently added GKE Pod Snapshot primitives to its Python SDK, four calls on a sandbox handle, and the PR says what they are for:

an RL loop can park a claimed sandbox between turns and bring back its exact state (memory + filesystem), or checkpoint an episode and roll back to it.

That is the right primitive for the workload. An agent is idle most of its life, and search, best-of-n and rollouts all want copy on write of a whole machine rather than a fresh boot per branch. Everyone builds it, everyone measures it in milliseconds saved.

Firecracker’s own documentation describes the same operation in a different register:

Without a strong mechanism that enables users to guarantee that unique things stay unique across snapshot restores, we consider resuming execution from the same state more than once insecure.

My claim is that restoring a snapshot is a machine cloning operation before it is a performance one, and the de-duplication that makes it safe has no named owner in most stacks. Vendors put a number on the cold start it saves. Nobody puts anything on the identity it copies.

What the snapshot carries#

The snapshot support doc lists what gets duplicated when one snapshot becomes several running machines: identifiers, random numbers and random number seeds, the guest OS entropy pool, and cryptographic tokens. Then it assigns the work:

State other than the guest kernel entropy pool, such as unique identifiers, cached random numbers, cryptographic tokens, etc will still be replicated across multiple microVMs resumed from the same snapshot. Users need to implement mechanisms for ensuring de-duplication of such state, where needed.

The word “users” is doing a lot of work there. It means whoever assembled the stack, because the VMM has just told you it is not going to.

I think that is the correct boundary, honestly. A VMM that quietly rewrote guest memory to make clones look different would be a worse VMM, and I would rather have the warning than the magic. The problem is where the warning lives. It is in a markdown file in the hypervisor repo, and the person choosing between two managed sandbox APIs is three layers above that and has no reason to open it.

VMGenID does one job#

There is a mitigation, and it is good. Firecracker exposes a VMGenID device, and the doc on randomness for clones is precise about what it buys: on resume, Firecracker writes a new identifier and notifies the guest, and Linux 5.18 or newer treats that as fresh entropy and force-reseeds its CSPRNG. So getrandom() and /dev/urandom give you distinct values in every VM started from the same snapshot.

The qualifier is in the same sentence. Distinct after the kernel has handled the notification, which leaves what the doc calls a race window between the vCPUs resuming and the reseed landing, and your guest is running inside that window.

Scope is the other half. The same page tells you to delete /var/lib/systemd/random-seed before you snapshot, warns that /proc/sys/kernel/random/boot_id stays identical across clones unless you replace it yourself, and on userspace libraries that keep their own entropy pools it just says there is no generic solution under the current programming model.

So the kernel fixes the kernel. Everything above the kernel is exactly as you left it: machine ID, boot ID, hostname, the clock, any library that seeded a PRNG when it was first imported, any token fetched before the snapshot, any TLS session that was open at the time. In the first restore and in the seventh, all of it is byte for byte the same.

Somebody already did this, with an exploit#

What moved this from a footnote to a post, for me, is that it is not hypothetical. It was demonstrated against shipping software, and the paper is sixteen years old.

Ristenpart and Yilek, When Good Randomness Goes Bad, NDSS 2010:

We exhibit such VM reset vulnerabilities in widely-used TLS clients and servers: the attacker takes advantage of (or forces) snapshot replay to compromise sessions or even expose a server’s DSA signing key.

In practice that was session compromise against the TLS stacks in Firefox and Chrome, and remote recovery of the DSA secret key from an Apache HTTPS server, with the victim running under VMware or VirtualBox. The server signed twice with the same nonce because the VM ran twice from one snapshot, and the key falls out of that.

The intent is what changed since 2010. Snapshot replay was an accident back then, or something an attacker had to engineer. We have since made it the feature, and we are pointing model-generated code at the result.

Who owns the cleanup#

Go layer by layer and ask who is even in a position to de-duplicate.

The VMM will not, and Firecracker wrote that down. The guest kernel does its own pool and the docs are clear that this is the end of it. The workload is the worst candidate on the list, since in an agent sandbox the workload is the part you decided not to trust.

That leaves the control plane, which is fine, because the fleet controller is the only component that knows this restore is the seventh one taken from this snapshot. Issue credentials at claim time instead of baking them into the image you snapshot. Rewrite machine ID and boot ID on resume. Treat every connection that was open at snapshot time as dead. And decide on purpose whether a given snapshot is allowed more than one live descendant, because that is a policy question and it currently gets answered by whoever writes the calling code.

Then there is where the warning stops travelling. The agent-sandbox PR above covers park, resume and roll back within one sandbox and says nothing about multiple descendants. E2B’s persistence docs describe pause and resume preserving “all the running processes, loaded variables, data”, are careful enough to tell you that clients get disconnected and have to reconnect, and do not discuss resuming the same paused sandbox more than once. gVisor’s checkpoint and restore page carries no equivalent caution either. I am not saying any of those is broken, I have not tested them and that would be an unfair thing to read off a docs page. I am saying the caveat exists at one layer and does not propagate up, so whether you learn about it depends on which repo you happened to be reading.

Where this loses#

Most forks are not holding anything worth stealing. A sandbox branched three ways to run three variants of a test suite has no signing key in it, and for that case identical entropy costs you nothing at all.

If your control plane already injects credentials per claim, and the good ones do because per-session credential scoping was the answer to prompt injection anyway, then the largest item on the list is handled for reasons that had nothing to do with snapshots. That is real and it is probably why this has not bitten anyone loudly yet.

There is also a case where clone identity is the requirement rather than the defect. An RL rollout that branches an episode wants both branches to start from exactly the same state, otherwise the comparison means nothing. That is the best argument I know for keeping this out of the VMM and putting it in the layer that knows what a given sandbox is for.

And the limit of this post is that all of it comes from primary docs and one paper. I have not restored a snapshot fifty times on a current stack and diffed machine IDs, boot IDs, session tickets and userspace PRNG output across the copies. As far as I can find, nobody has published that either, which is odd given how many products now ship the primitive. It fits on one machine. I would read it from anyone.

The takeaway#

Snapshot restore is two operations sharing a name. As a cold start fix it works and the numbers are honest. As a machine cloning operation it copies every unique thing the guest was holding, which Firecracker calls insecure in its own docs while handing you the cleanup. So when a sandbox platform ships snapshots, the question is which component rewrites identity on resume. If the answer is that the kernel reseeds its PRNG, that is one item on a list of about eight.

I write about what agent workloads do to the platforms underneath them, from someone who has run the platforms. More about me.