The validator is the product

Karpathy says to delete your agent harness as models improve. For enterprise data tooling, half of mine is not up for deletion — and knowing which half is the design decision that matters.

Andrej Karpathy’s LOOPS.md — his widely shared field notes on agent systems that run for days — ends with a rule that sounds like heresy to anyone building production tooling: delete the harness. The scaffolding you built around the model exists to compensate for its weaknesses, he argues; as models improve, half of it becomes dead weight. Re-read your harness at every model release and delete what the model now does for free. A harness that only ever grows is a harness you have stopped reading.

He is right. And for Vault-Agent, my open-source agentic Data Vault 2.0 pipeline, I am deliberately not following the rule for half of the system. Working out which half taught me more about enterprise agent design than any other decision in the project.

A test that sorts the harness

Vault-Agent wraps a lot of machinery around its LLM agents: prompt rules steering the modeler, deterministic repairs that fix known LLM mistakes before validation, and a validator with 32 machine-checkable gates that block structurally broken Data Vault models before a line of SQL is generated.

Superficially, all of it “compensates for the model” — every piece exists because an LLM, at some point, got something wrong. But apply one question to each piece and the harness splits cleanly in two:

Would you keep it if the model were perfect?

Take the most instructive example. Data Vault’s multi-active satellites carry a child dependent key — a sub-key like address_type that distinguishes concurrent rows. It is a key column, not payload. The current generation of models reliably lists it in both places, which would emit a duplicate column no warehouse accepts. Steering the model with the error message failed four runs out of four; what fixed it was a prompt rule plus a deterministic pre-validation repair that drops the duplicate.

Would I keep that repair if the model were perfect? No. It exists only because today’s model makes today’s mistake. It is exactly what Karpathy says it is — compensation — and it should be re-tested at every model release and deleted the moment it stops firing.

Now take the validator gate behind it, the one that blocks any satellite whose columns collide. Would I keep that with a perfect model? Yes. Without hesitation. And the reason is the whole point of this article.

What regulated enterprises actually buy

Vault-Agent’s promise to a bank or an insurer is not “an LLM designed your warehouse model.” It is: this model is provably compliant with the Data Vault 2.0 method — independently of which model produced it, and checkable by your auditors. That proof is the deterministic validator: every rule from the canon, encoded as an inspectable gate with a stable error code, run against every generated model, every time.

A perfect model would never trip those gates. It would not make them superfluous — it would make them silent. The same way a good engineering team doesn’t delete its test suite as it gets better: a green run that never fails is not dead code, it is the certificate. In a regulated environment, “the AI is really good now” is not an argument an architect can take to a risk committee. “Every output passes the same 32 deterministic checks, here is the audit trail” is.

So the deletion test yields a two-class harness:

• Model-compensation — prompt steering, pre-validation repairs. Temporary by intention. Instrumented, measured, and deleted when a new model stops needing them.
• The product — the deterministic gates, the human sign-off checkpoint, the decision records. Permanent by design. They are not there because the model is weak; they are there because trust has to be independent of the model.

The twist: the permanent half makes deletion safe

Here is what I find most satisfying about this split, and what I think Karpathy’s rule quietly assumes without saying: you can only afford to delete the compensation because the product half stays.

Deleting a pre-validation repair is a gamble — maybe the new model doesn’t need it, maybe it does and you just haven’t seen the failure yet. With a deterministic gate behind every repair, the gamble becomes a controlled experiment: if the deletion was premature, the gate catches the failure on the very next run, loudly and with a stable error code. Without the gates, “delete the harness” is Russian roulette. With them, it is a measurement.

That is why Vault-Agent’s repairs now announce themselves in the run’s transcript every time they fire, why an ablation runner can re-run the benchmark cases with any single steering rule switched off, and why a steering ledger records — per rule — what it cost to learn, which model needed it, and the evidence for keeping or deleting it. When the next model generation lands, the question “which crutches can go?” gets an empirical answer instead of a vibe.

The takeaway

If you are building agentic tooling for a domain where the output has to be trusted — finance, insurance, pharma, anything an auditor might one day read — don’t treat “delete the harness” as a blanket instruction, and don’t use it as an excuse to keep everything either. Sort your harness with one question: would this survive a perfect model? What wouldn’t, instrument and plan to delete. What would, stop calling a harness — that part is your product.

---

Vault-Agent is open source — the steering ledger, the validator gate catalogue, and the LOOPS.md mapping discussed here are all in the repo. I’m available for mandates around Data Vault and agentic automation — reach out.