The Validator Is the Platform

InfoQ ran a platform engineering playbook for production LLMs by Aditya Mulik this week, and it’s one of the more useful things I’ve read on the subject, mostly because it comes with numbers and a companion repo. The system is a multi-agent recommender for inventory accuracy at a large retailer, and the platform under it now serves dozens of application teams. A month into production, “roughly fifteen percent of agent responses were hallucinations”. Six months later the rate was 1.5 percent, and “the lever that moved the number wasn’t a better foundation model.”

The article covers a lot (a gateway, a prompt registry, cost attribution, evals), but it’s specific about where the drop came from. The retry loop with validation is “the layer where most of the fifteen percent to 1.5 percent hallucination reduction came from”.

My claim is that this is the part of an LLM platform to build first and own centrally: the deterministic code between what the model proposes and anything that acts on it. Per the article, the model in that stack can be swapped by changing a model identifier, so the checks are what the teams actually depend on, and how much a check is worth comes down to what it reads.

What the loop does#

Every call declares a Pydantic schema, and the platform validates the response before handing it back. When validation fails, the platform “re-prompts with the validation error embedded as additional context, up to a configurable retry limit.” Failures get classified before anything is retried, since “collapsing them into one generic retry is the mistake”: a schema violation re-prompts with the error attached, a hallucination signal re-prompts with stronger grounding context, and an infrastructure error backs off exponentially. The cap is three tries, “because a structured-output failure that hasn’t fixed itself by the third try almost never fixes itself on the fourth.”

The rest of the stack follows the same pattern, and I don’t think that’s an accident. Tool authorization is a default-deny YAML policy that each MCP server checks “on every tool call, before the tool runs”. The eval gate is an average trajectory score of 0.8 written into a three-line config file. Even the intent gate in front of the model is described as “a separate, cheap, inspectable gate” (a small trained classifier in production, keyword rules in the repo), and it’s allowed to return “unclassified” rather than push a weak match onto whichever agent scored highest.

AWS published the same shape a few weeks earlier, from the other end of the market: validating multi-agent decisions with Step Functions and AgentCore. Agents propose rebooking options for passengers on a cancelled flight, and “the principle is that agents propose, and deterministic code validates.” The rule they build on is stricter than anything in the playbook:

The key principle: no agent Task state writes to the reservation system or issues a payment. Only deterministic Task states do that, and only after a deterministic validation step has passed.

What the check reads#

The two validators differ in what they look at. The playbook’s schema has the model declare whether it’s grounded. The docs specialist’s DocsOutput carries a grounded boolean, “True if every claim is backed by the fetched docs”, next to a list of sources. The article is careful to say the flag only means something because of the pipeline around it, where tools fetch the evidence and the prompt restricts the model to that evidence. Still, the flag is the model’s own report on its own answer. A validator can reject grounded coming back as the string "yes", a missing sources field, or an honest false. What it can’t catch is a well-formed answer that says true and is wrong. In the repo, “the hallucination counter on the same record is incremented by the schema-validation check,” so that answer wouldn’t be counted either. The production numbers, the article says, come from sampled traffic scored against reference answers, which is better, but it’s a sample.

AWS’s validator looks the proposal up:

# Illustrative - reject any option the agent proposed that is not bookable
def handler(event, context):
    passenger = event["passenger"]
    proposed = event["proposal"]["options"]

    validated = []
    for option in proposed:
        flight = lookup_flight(option["flightId"])
        if flight is None:
            continue  # hallucinated or stale flight, reject
        if flight["seatsAvailable"] < 1:
            continue  # no inventory, reject
        if not fare_rules_allow(passenger["fareClass"], flight):
            continue  # fare rule violation, reject
        if not route_is_valid(passenger["origin"], passenger["destination"], flight):
            continue  # invalid route, reject
        validated.append(option)

    return {
        "passengerId": passenger["id"],
        "validatedOptions": validated,
        "autoConfirmEligible": passenger["loyaltyTier"] == "top" and len(validated) > 0,
    }

Nothing the model says about the flight is taken on trust there. The validator asks the reservation system whether it exists and has a seat, so the model’s confidence in its own answer never comes into it, and that’s the property I’d design for. Checking the shape of the output catches malformed answers, and you need it, but the failure the playbook itself calls the dangerous one is the answer that “passes every traditional health check,” and only a lookup against something outside the model catches that.

For each output type, then, the platform team should ask what the validator could look up. A recommended SKU can be checked against inventory. A cited library can be checked against the tool calls the run actually made, and the playbook already lists “citing a source for a library it was never given” as a hallucination signal, which is exactly that kind of check. The platform’s job is making those lookups cheap enough that every team writes them.

Where it sits#

The playbook argues placement hard in one place. On authorization: “It would be easy to check permissions once at the gateway and trust everything downstream. We don’t.” Each tool server enforces the policy itself, so a bug in the orchestrator can’t expose every connected tool.

The validation loop doesn’t get the same treatment, at least in the repo. It lives in a subclass, CustomLlmAgent, and “the agent wraps its own run loop,” catching the ValidationError and retrying. That is shared platform code written once, which is most of the win, but the guarantee only covers agents built on that class. In the AWS design the agent can’t skip validation because the state machine has no path from an agent step to the booking system. I’d apply the playbook’s own authorization argument to its validator: put the check where agent code can’t route around it, between the proposal and the effect, and let whoever owns the effect own the check. The article doesn’t say how production is wired here, so this is a comment on the reference repo, which may well be simpler than what they run.

Where this loses#

Plenty of outputs have no system of record. A docs answer, a summary, a drafted email: there’s no lookup_flight for prose. For those, the honest validator is the playbook’s, shape plus declared grounding plus offline evals on a sample, and a confident wrong answer in the right shape gets through. A lot of what teams build on LLMs looks more like that than like rebooking a flight.

The numbers are thinner than they look. Fifteen to 1.5 percent is one team reporting on one system, and the article defines its hallucination rate as schema failures plus grounding failures without giving the split, so I can’t tell how much of the drop is the model getting the format right on a retry and how much is it inventing fewer facts. Both count, though they’re different fixes. AWS’s piece is a reference pattern with no production numbers at all.

Validators also cost money and time. Retries multiply token spend (the playbook tracks an “amplification factor” for that reason), and a lookup per proposal puts the system of record in the hot path. The validator is business logic someone has to keep current, and when the rules move and the validator doesn’t, it starts rejecting correct answers. My guess is the keyword gate fails the same way, returning “unclassified” for a perfectly good request because nobody added the word. When the validator ends up holding all the logic, what the model contributes is option generation, which can still be worth paying for, but it’s a smaller thing than the pitch usually makes it sound.

The takeaway#

The playbook’s own summary puts observability first. I’d put the validator first, since it’s the layer the article credits with most of the result. If you’re building the shared layer under your teams’ LLM features, have the check read the world wherever there’s a world to read, and put it somewhere the agent’s code can’t skip on its way to the effect.

I write about running agents and LLM workloads on real platforms, including the plain code that keeps them honest. More about me.