Exit 0 Is Not Success for an Agent Job

Reuters reported this week that the OpenAI agent that chained zero days into Hugging Face production ran for days, and OpenAI did not notice for about a week. Most of the discussion was about the exploit. The part I keep coming back to is duller: for a week, nothing in the operational picture said anything was wrong.

That is the normal failure mode for an agent you leave running unattended, and it is a monitoring problem before it is a security one. I hit a small version of it every morning.

The checks that pass anyway#

I have a cron job on a small box that pulls a set of feeds, hands the results to a Claude Code run with no network access, and writes a curated section into my wiki before I wake up.

30 3 * * * /usr/bin/flock -n /tmp/scan-daily-brief.lock /home/factory/dev/darkfactory/scan/run.sh daily-brief

A green run proves cron fired, the lock was free, the process exited 0, a file got written, and the git push succeeded. Standard batch-job monitoring, and I would wire the same alerts for anything else on that box.

None of it proves the thing in the file is any good.

A run where two of five sources returned nothing exits 0, produces roughly the same file size, and writes exactly the same shape of output as a run where everything worked. The model will happily write ten confident bullets off three sources. It has no reason to mention the two that are missing unless you make it. Exit status and byte count cannot tell those runs apart, and reading the note on my phone three weeks later, neither can I.

Classic monitoring watches the process. The failure lands in the content.

Make the agent report its own coverage#

First rule: the agent gets the fetch log and is told to report on it. The prompt is explicit about it.

PROMPT="... The fetch log is ${LOG}; read it for source health and
report degraded sources honestly. ..."

The skill file makes it non-optional rather than a suggestion:

- If a configured source failed, returned zero unexpectedly, or was disabled, say so
  in one short quality note. Never present a partial sweep as complete coverage.

That costs nothing to add, and over three days it surfaced this:

*Coverage note: Reddit stayed disabled in config, so all six subreddits are
missing, and the labs feeds fetched 24 posts but none survived the recency cut,
so there is no lab coverage today. X delivered 48 tweets but only commentary,
nothing kept. The 30 ranked candidates came from 200 raw items.*

*Coverage note: Reddit was disabled this run and X @DaveBlundin failed on every
instance, so all of Reddit and one X source are missing; the 30 ranked
candidates were distilled from 198 raw items across HN, GitHub, X and labs.*

I would not have known about any of that otherwise. A source disabled in config months ago and forgotten, one X handle 404ing on the only mirror it has, a whole feed fetching cleanly and then losing every item to a recency filter: none of those are visible from outside the job, and all three produce a note that looks completely normal.

Where the line lives matters as much as what it says. It sits at the top of the file I already read every morning, above the content I actually want. Not in a log, not on a dashboard I would open twice and then never again. A health signal that needs its own surface to be noticed does not get noticed.

Then let code decide whether it ships#

The model writes the section. It does not get to decide whether the section ships. In between sits a plain Python validator, and the docstring says why:

"""Validate Claude-generated scan sections before delivery.

The reason stage is probabilistic. Delivery is not.
"""

The checks themselves are boring on purpose:

def validate_daily(text: str, errors: list[str]) -> None:
    validate_common(text, errors)
    if not re.search(r"^## Scan\s*$", text, re.M):
        fail(errors, "missing ## Scan heading")
    if not re.search(r"^### ⚡ Worth acting on\s*$", text, re.M):
        fail(errors, "missing Worth acting on section")
    bullets = re.findall(r"^- ", text, re.M)
    if len(bullets) > 16:
        fail(errors, f"too many total bullets: {len(bullets)}")
    for n, line in enumerate(text.splitlines(), 1):
        if line.startswith("-") and "[link](" not in line:
            fail(errors, f"bullet has no link on line {n}")

Heading present, action section present, bullet counts inside the limits, every bullet carries a link. Wired into the run so a failure stops delivery:

[ -s "$SECTION" ] || { echo "FATAL: no section written to $SECTION"; exit 1; }
python3 "$DIR/lib/validate.py" "$JOB" "$SECTION" --annotate \
  || { echo "FATAL: section empty or human gate broken; nothing delivered"; exit 1; }

If it fails, the job dies and yesterday’s note stays untouched, which beats a broken section overwriting a good one. The probabilistic stage produces, a deterministic stage decides whether it lands. I would put that same gate on any model output that writes to a real location.

Where this is weak#

The health report comes from the same run that might be broken. If the model ignores the instruction or misreads the log, the coverage note is wrong in exactly the way I would not catch. Today I check the fetch log by hand when a note looks off, which is a manual step I have not automated. The stronger version computes the counts in the fetch stage and passes them in as data, so the model reports numbers instead of deriving them.

It also does nothing for the more interesting failure: all five sources fetch cleanly, and the summary of them is confidently wrong. Coverage reporting catches missing inputs and says nothing about bad reasoning over the inputs that arrived.

All of it also assumes someone reads the artifact. This works for a daily note because I read it anyway. Point the same job at output nobody opens and the honest coverage note is just a comment in a file.

The takeaway#

For a job that runs unattended, exit 0 means the process finished and nothing more. Agent output degrades quietly, and a degraded run leaves the same footprint on disk as a good one.

Two boring things cover most of it. Make the agent report what it did not see, inside the artifact a human already reads. Then put deterministic code between the model and delivery, so shipping is a decision your own checks make rather than a side effect of the process not crashing.

I’m a platform/SRE engineer writing about making agentic AI reliable in production. If you’re running scheduled agents and your only health signal is an exit code, get in touch.