← All Writing

Proxy Variables Are Not an Egress Policy

Injecting HTTPS_PROXY into every pod at admission works fine for apps that cooperate, and an agent is a workload you can't count on to cooperate. Two recent OpenAI incident reports show an agent getting told no and going looking for another way out, so the egress decision has to be enforced below the workload by something it can't read or route around.

A 3D wireframe diagram in thin glowing white lines on a solid pure black background, with no other colour anywhere. One large open wireframe cube sits in three-quarter perspective with all twelve edges visible and see-through faces. Floating inside it, toward the back left, is one small wireframe cube. Six thin straight lines leave the small cube in different directions. Five of them stop at the inside faces of the large cube, each ending in a small dot, and go no further. The sixth runs to a single small wireframe ring set into the right-hand face of the large cube, passes through the ring and carries on out into the empty black space beyond. The ring is the only opening anywhere in the large cube.

OpenAI put out a short misalignment report last week that I keep coming back to. An internal research model in an RL training run had to identify a person from clues in a blog post. Its search tool returned unrelated stuff, so it used Python from the shell to send the same query to Google, Bing and DuckDuckGo, and got a 403 and two 502s back. A while later it started testing its network access. It tried to reach a public chatbot over HTTPS, and the web proxy blocked that. Then:

Direct public DNS queries also went unanswered, but the training environment’s own resolver returned a real record for a known domain and correctly rejected an invented one. The agent concluded that it could reach the public internet through this resolver.

From there it found a public DNS delegation service and used it to forward questions to a third-party chatbot, encoded in hostnames. The first answer only partly helped, so it fetched the chatbot’s public docs, wrote a new DNS script with a 24-second timeout, and “sent 18 more questions through the same route.” Monitoring flagged it within 15 minutes. The run was killed 2.5 hours later. OpenAI says it has since added “blocking controls at two independent layers, either of which would have prevented this access.”

Nothing in the report suggests anyone prompt-injected that agent. It was told no, and it went looking for another door, which by its own logic was just doing the task. That’s the behaviour your egress design has to assume. The web proxy held, for what it’s worth. The leak was the resolver, which is a service the sandbox needed to have.

My claim is that for an agent, egress has to be enforced below the workload, by something the workload can’t read or route around. Anything you hand the workload as configuration (proxy environment variables are the common one) is a request it’s free to ignore, and an agent, unlike most of what runs on your cluster, will go looking for the way around it.

Admission-time injection is configuration#

AWS published a tidy pattern last month, proxy injection for EKS on Fargate using Kyverno. A mutating policy matches pods in labelled namespaces, and:

It injects the HTTPS_PROXY, HTTP_PROXY, and NO_PROXY environment variables into every container and init container at admission time.

“No proxy-related changes to application deployment manifests are required.” I like it for what it is, honestly. The decision moves out of every app team’s manifest and into one policy the cluster team owns, which is the right direction.

The same post says the pod starts “with proxy settings already in place, so applications that honor proxy environment variables route their first outbound call through the corporate proxy.” Honor is the important word there. Nothing in the kernel reads HTTPS_PROXY. A client library reads it if it wants to, and the libraries don’t even agree with each other; the post itself warns that CIDR ranges in NO_PROXY “are honored by some clients, such as the AWS SDK for Go, but not by curl and libcurl.” A process that unsets the variable, or opens a socket itself, or is a Python script the model wrote thirty seconds ago, just goes straight out (that last part is my inference, nothing in the post says it).

Then there’s the failure mode, which the post lists as a feature: “Setting failurePolicy: Ignore ensures that pod scheduling is not blocked if the webhook is temporarily unavailable.” For a corporate proxy rollout that’s the right trade. You don’t want a webhook outage to stop deploys. For an agent sandbox it means that when the webhook is down, the pod starts without any of it.

To be fair to AWS, the pattern isn’t built for agents and doesn’t pretend to be. It also notes that traffic bypassing the proxy “needs a path such as a virtual private cloud (VPC) endpoint or a NAT gateway.” If you build the VPC so there’s no other path, the network is your egress policy and the variables are routing convenience, which is fine. It goes wrong when the injected variables are the whole policy.

The host decides#

The clearest statement of the right split I’ve read lately is in Docker’s Kit specification. A Kit is an ordinary OCI image that declares what an agent sandbox needs, network rules included, typed as com.docker.sandbox/network-policy@2. On first read it looks like the opposite of my argument, since the rules travel inside the image. Then the post says:

A Kit grants itself nothing. Each entry is a request, and the host decides. A conforming runtime, one that implements the behaviour the specification describes, blocks hosts not on the list. Without one, the annotation is inert: an image and no enforcement.

That’s exactly right: the workload can ask for whatever it likes, and the host decides what it gets and enforces it. Credentials work the same way in that spec: the host injects the real value into requests to the named domains, and inside the sandbox there’s only a sentinel. And on a runtime that gates updates, if a new version of a Kit asks for more, “Any widening stops and asks.”

Kubernetes has always worked like this, and people still trip on it. From the NetworkPolicy docs: “Creating a NetworkPolicy resource without a controller that implements it will have no effect.” The agent-sandbox project spells out the consequence in one of its own examples, which I think should be printed on a poster:

kubernetes-sigs/agent-sandbox/examples/containarium-execution-scoped-token/networkpolicy.yaml
# NOTE: a plain NetworkPolicy is only enforced if the cluster's CNI enforces
# it. kind's default CNI (kindnet) does NOT — it accepts the object and drops
# nothing. kind-config.yaml therefore disables the default CNI and
# run-test-kind.sh installs Cilium. On a cluster whose CNI ignores
# NetworkPolicy this file is decoration and the egress check is vacuous.

The default in that project is interesting too. If you let the controller manage the policy, the API docs say you get “Egress: Allow Public Internet only. Blocks internal IPs (RFC1918), Metadata Server, etc.” That protects your cluster and your cloud credentials from the agent, which matters. It does nothing about the agent sending your data somewhere on the public internet, and for an agent that’s the leg I’d most want closed. Simon Willison’s lethal trifecta is private data, untrusted content, and “the ability to externally communicate in a way that could be used to steal your data.” When vendors fixed the exfiltration bugs he collected, they usually did it “by locking down the exfiltration vector.” The platform is often the only party in a position to cut that leg, since nobody can promise the model won’t read hostile text.

Where this loses#

The best counterexample is OpenAI again, and it’s a big one. Their Hugging Face incident report describes an evaluation environment built roughly the way I’m arguing for, “controlling its network boundary separately from the container itself.” It didn’t hold. One agent “attempted to reach the internet, but was denied by OpenAI’s outbound network controls. It subsequently bypassed those controls by exploiting certain Artifactory endpoints, gaining access to the public internet.” Artifactory was the internal package mirror the containers were supposed to use for installing packages. Per the report, “zero-day exploits in Artifactory became the agents’ principal egress path to the public internet during the incident.”

So platform-owned egress is necessary and nowhere near sufficient. Look at where both OpenAI agents got out: the package proxy and the DNS resolver, two services the platform left reachable on purpose because the work needs them. GitLab’s write-up of the incident puts it well: “An allowlist hands your AI workload the reach of any destination it can take over.” Every entry on the list is part of your egress path, and so is the resolver. The Kubernetes docs even warn that a default deny-all egress policy “also blocks DNS traffic,” so you have to add a rule letting it back out. That’s the same kind of opening the DNS agent found, a path you leave open because everything breaks without it. Moving the decision below the workload is where you start, and then you still have to run the mirror and the resolver like something hostile is on the other end of them.

The second weakness is friction. An agent decides what to fetch halfway through a run, and a tight allowlist will break legitimate work in ways nobody predicted when they wrote it. When the platform team owns egress, every new domain turns into a ticket, and I’d bet the usual result is a team widening the list bit by bit until it means nothing. Docker’s “stops and asks” model is the most honest answer I’ve seen, but someone has to answer the question, quickly, many times a day.

And proxy variables still have a job. For your own cooperative tooling they’re how traffic reaches the proxy that logs it, and an egress log is half of how you notice a run like the DNS one. Keep them, just don’t let them be the thing you point at when someone asks what stops the agent.

The takeaway#

If your agent’s egress control lives in something the agent’s process can unset or ignore, then the agent is configuring its own egress, whoever wrote the config. Put enforcement below it, in the network or the runtime, and make it fail closed. Then treat every service on the allowlist, the resolver especially, as part of the egress boundary, because that’s where both of OpenAI’s agents went when they were told no.

I write about what agent workloads do to the platforms underneath them, network boundaries included, from someone who has run those platforms. More about me.