Comprehension Debt Comes Due at Incident Time

Alex Palcuie does AI reliability engineering at Anthropic. In a talk about putting Claude into the incident response path, he asked the question most of the industry is deferring, and then answered it in a way I have not been able to put down:

If AI starts doing this, will we have our skills atrophied? Senior incident responders aren’t smarter…they have the scar tissue.

Scar tissue is an accumulation. You get it by being wrong about a system at three in the morning and then having to find out why, and the finding out is the part that deposits it.

Agent-assisted engineering mostly does not produce bad code, whatever the discourse says. What it does is shorten the process that used to deposit that residue, and on a lot of teams it removes the process entirely. My claim is that this is a debt in the specific sense the word carries in engineering. It is cheap to take on, invisible while you carry it, and it comes due at a moment you do not get to pick. For a system your team did not write line by line, that moment is the incident.

The measurement exists, and it is not about engineers#

Scholars from Stockholm University and the University of Hong Kong tracked more than 26,000 middle and high school students across a county in central China from September 2022 to June 2025. The reported results are the cleanest version of this shape I have seen anywhere.

AI use raised homework scores by 18 percent and cut homework time from 64 minutes to 45. Within six months, monthly exam scores for the same students had dropped by 20 percent. Two years on, the high-stakes results fell harder: 24 percent on the zhongkao, 18 percent on the gaokao.

The size of the drop matters less to me than the inversion underneath it. Homework scores used to predict exam performance, and in this cohort the students scoring highest on homework became more likely to do worse on the exam. The assisted metric went up, the unassisted metric went down, and the assisted metric stopped forecasting the unassisted one.

Read that as a dashboard rather than a school report and it should be uncomfortable. Delivery looks healthy, better than healthy, and it has quietly stopped telling you anything about the thing you actually care about.

The same shape, on engineers#

Alasdair Allan made the career-ladder version of this argument at QCon London, written up by InfoQ. His framing is that the rungs where expertise used to get built are exactly the work agents absorb first:

Junior engineers won’t spend years reading legacy codebases or debugging production incidents at 3 a.m. They’ll just point the AI agent at it and ask it to summarize.

He follows it with the part that makes this a systems problem rather than a talent problem: “The people who can build good contexts for AI are people who have context in their heads through years of craft. The pipeline that produced these people is breaking.”

For the mechanism rather than the measurement, the most useful thing I read this month was a Tell HN post with 53 points on it, which is to say almost nobody saw it:

At first, I would push small features straight to main. But my colleagues were shipping 10x more than me, and I need my job, so I started to push more and rely more on AI.

At some point, I was no longer having mental capacity to understand so much context, so I just started to chose the ‘Recommended’ suggestion by Claude.

One person on a small thread is an anecdote and I am not going to dress it up as anything else. What it gives you is the mechanism, described from the inside. The comprehension was not traded away in a decision anybody made or wrote down. It got competed away, against a peer shipping rate, by someone who wanted to keep their job.

That is why I think engineered comprehension has to be budgeted rather than encouraged. Anything that costs time and carries no number will lose to something that costs no time and carries one, on every team I have watched try it.

Why the bill arrives at the incident#

Palcuie maps LLM incident work onto the OODA loop, and the interesting result is where it lands. Observation goes well: pulling signals out of logs and metrics is close to ideal work for a model that reads fast and can run in parallel. Orientation is where it comes apart.

You add it to CLAUDE.md, and it understands about this situation. It will get wrong correlation versus causation in the other 99 situations.

Correlation against causation, under time pressure, on a system you did not write, is not a retrieval problem. Nothing about a bigger context window fixes it. What that step needs is a model of the system good enough to produce a decent hypothesis about why, and to rank it against three others, in the second minute rather than the fortieth.

That model is exactly what the assisted workflow stopped building. An incident is the one moment where the debt cannot be rolled forward, because there is no version of the on-call rotation where you get to say the code was generated and understood by nobody. You own it at 3am either way.

What paying it down actually costs#

The best writeup of the prescription is InfoQ’s comprehension as an architectural characteristic, by Jacobus Meintjes, Narayana Rengaswamy, Paul Katsande and Sureshbabu Bikki. Their thesis is in the subtitle, that a system which is not understood cannot evolve safely, and the practical move is to treat comprehension like any other architectural characteristic: name it, put fitness functions on it, and check it at design time rather than hoping for it.

The line from it I have actually used in an argument is this one:

The human review at this stage is a comprehension checkpoint, not a quality gate.

That reframing does real work, because it changes what “looks fine to me” means. Approving a diff you could not explain to someone else passes the quality gate and fails the comprehension checkpoint, and only one of those two failures is visible anywhere in your tooling.

Two teams have put a price on this in public, which is rarer than it should be.

Ankur Sethi retypes LLM-generated code by hand rather than letting the assistant edit files, and instead of pretending the practice is free, he states the bill: “Instead of being 10x faster, I’m probably only 2x faster.” I am not recommending that anyone retype their diffs. I am pointing at the fact that he costed it, because a comprehension practice without a stated price is a practice that gets dropped in the first busy quarter.

The other is SpareBank 1 Utvikling, whose senior developers described four months with Claude Code and then a deliberate move back to TDD and mob programming for domain logic in their brownfield systems. They kept the agent for telemetry analysis, dashboards and scripts, which is roughly the split Palcuie’s OODA mapping predicts. Asgaut Mjølne Söderbom put the reason plainly: “The learning part…is much better when you sit with small things and don’t have this interruption.” Several engineers on one problem is the most expensive currency a team has, and they chose to spend it on understanding.

Where this loses#

Four objections, and the first one does not have a clean answer.

The efficiency question is fair and largely unanswered. The top counterargument in the retyping thread (545 points, 444 comments) put it better than I could: “if your workflow is think hard, let LLM write it, read what AI wrote, think hard about what AI wrote, re-type what AI wrote, fix what AI wrote…where in the heck are the efficiency gains?” If comprehension costs you half your throughput, the case for buying it rests entirely on incidents being expensive enough to cover the premium. For plenty of software that calculation genuinely goes the other way. A marketing site does not need scar tissue, and pricing comprehension into everything is how you end up with a process nobody follows.

The study is about teenagers. Twenty-six thousand students in one county, measured on exams, is not twenty-six thousand engineers measured on incidents, and I have not found the engineer-side replication. Homework also differs from production code in a way that cuts against me: nobody keeps the homework running for eighteen months and then pages you about it. I could not confirm the journal from the news coverage either, so treat the numbers as reported rather than as something I have read the methods section of. I am reasoning from a shape, and a shape is weaker than a replication.

Oracle is not the ally it appears to be. Oracle banning AI-generated code from OpenJDK reads like the maximal version of this argument, and I would rather not claim it. The stated reason is the limited time of human reviewers, and the top-voted reading in a 536-point thread was that the policy protects Oracle’s ability to litigate against others over AI-washed proprietary code. A ban with that motivation argues nothing about comprehension, and citing it would be recruiting a source that is not on the topic.

The abstraction objection is the one I take most seriously. Nobody on call understands their kernel scheduler, their TLS implementation or their query planner, and reliability did not collapse. My answer is that those abstractions are stable, widely shared, and documented by people who did understand them, while a service your agents generated last quarter is none of the three. That answer is decent rather than airtight. The strongest form of the counter-position, which swyx has argued as being slop-tolerant beating being anti-slop, says you should build systems that stay safe when nobody comprehends the output. Tight blast radius, real tests, fast rollback, small deploys. That is a serious strategy, it is cheaper than a comprehension budget, and if I could only fund one of the two interventions I am not certain I would fund mine.

The takeaway#

Shipping rate is measured on every team I have worked with. Comprehension is measured on approximately none of them. Put those two facts next to any deadline and the trade happens on its own, with nobody deciding anything and nothing appearing in a review.

The test I would actually run is cheap and takes an afternoon. Take a service your team shipped last quarter with heavy agent assistance, find whoever is on call for it, and ask them to explain without opening the repo what the system does when its main dependency starts returning 30-second latencies. Not what the runbook says to do, but what the code actually does: where the retries queue, what the timeout budget is, which thing falls over first.

The distance between that answer and a good answer is your comprehension debt, quantified. Measuring it on a Tuesday afternoon is considerably cheaper than discovering it at three in the morning, which is the only other time it gets measured.

I run and stay on call for systems that agents helped build, which is a different job from writing about them. If that is the kind of reliability and platform work your team is hiring for, here is what I do.