AI Agent Verification: Why the Checker Beats the Model

By Harel Asaf · Published September 9, 2026 · Updated September 9, 2026

Short answer: AI agent verification is a mechanical check on an agent's output that runs without a human reading it. Anthropic just shipped a proof Claude produced over 11 unsupervised days, 13 million lines of Lean, and nobody read it. Lean checked it. That check, not the model, is what made six billion tokens of autonomous work into something a professional would sign off on.

TL;DR

• Claude wrote a computer-checked proof of Fermat's Last Theorem in 11 days: 13 million lines of Lean, 30,300 intermediate theorems, roughly six billion output tokens, with only occasional one-line human steering.
• No human read the proof. Lean verified it against three standard axioms. The checker is what made the artifact credible.
• OpenAI disclosed that its research organization now uses 3.1 agent-workdays for every human workday, and that more than half of successful four-to-eight-hour agent tasks still needed a human intervention.
• Meta demonstrated an image-based prompt injection that hits above 80% attack success on open-weight models and 47% on GPT-5.5, and can overwrite an agent's tool manifest in a default deployment.
• Read together: capability is scaling on a visible curve, verification is scaling on a curve almost nobody is funding, and verification is the one that decides whether any of it ships.

The autonomy horizon jumped to eleven days this week. Most teams' verification layer did not move at all.

Andrew Wiles published his proof of Fermat's Last Theorem in 1995. It ran to 129 pages. Several mathematicians spent months checking it, and two months into that effort a reviewer found a gap serious enough that Wiles nearly abandoned the project. This week, Anthropic published a computer-checked proof of the same theorem that Claude produced in 11 days, running largely on its own. It is 13 million lines of Lean code. Nobody read it.

Nobody had to.

What Did the Three Labs Actually Publish?

Three labs published in the same week, and read together they describe one problem from three sides.

According to Anthropic's "Formalizing Fermat's Last Theorem," a team of Claude agents wrote 13 million lines of Lean over 11 days, proving 30,300 intermediate theorems and using 29,500 of them in the final proof. The whole thing consumed roughly six billion output tokens. Human input was thin: occasional one-line steering from researcher Tianyi Peng, things like "push Mazur to be done soon." The finished artifact was checked by Lean using only its three standard axioms, and a separate comparator confirmed the theorem statement matched Mathlib's own.

Kevin Buzzard, who leads the community formalization effort at Imperial College London and reviewed the proof, put the significance where it belongs:

"The techniques will also enable us to rigorously check LLM-generated mathematics, which is currently typically an extremely costly human-led process."

That is the actual result. Not that AI did the math. That a machine checked it.

OpenAI published its own numbers two days later, and they are the most honest disclosure of internal agent economics anyone has shipped. From "Research acceleration: The view inside OpenAI":

"In terms of a standard 8 hour workday, as of mid-August, in total, the research organization uses 3.1 agent-workdays of effort for every workday of human labor."

The median OpenAI researcher now burns more than $600 a day in inference at API prices. The 90th percentile burns more than $7,000. And in the same post, buried under the growth charts, sits the number that matters more: over half of successful four-to-eight-hour agent tasks in the last six months required at least one human intervention. Autonomy at scale still runs on steering.

Then Meta published the attack surface. "Repeat-After-Me: Black-Box Adaptive Visual Prompt Injection" demonstrates an image-based injection that reaches attack success rates above 80% on open-weight models and 47% on GPT-5.5, in a realistic setting where the user's own prompt never authorizes the injected task. The paper's field test is the part that should stop a security review:

"in a default OpenClaw Discord deployment, an untrusted user can use a minimally injected image to overwrite TOOLS.md, enabling future sensitive behaviors like remote code execution and secret exfiltration."

An image. Not a jailbreak prompt. A picture someone posts in a channel.

What Does This Mean for Teams Building With Agents Today?

The three papers share a spine. Agent autonomy is now measured in days, the labs running these agents are honest that it works, and the only place where multi-day unsupervised work is genuinely trustworthy is the one place a machine can check the output.

Lean is not a governance framework. It is a compiler. It does not care that Claude wrote the proof, does not grade effort, and cannot be argued with. It either accepts the artifact or rejects it. That property, and not the model's capability, is what turned 13 million unread lines into something Kevin Buzzard could endorse in public.

Most enterprise AI work has no Lean. A contract summary, a support macro, a pricing recommendation, a marketing brief: none of these have a compiler. So teams substitute human review, and then discover that human review does not scale with a 3.1-to-1 agent-to-human ratio. You cannot read everything an agent produces once the agent produces three days of work per day of yours. This is the actual bottleneck, and it arrived faster than most roadmaps assumed.

The practical move is to stop asking "can the agent do this?" and start asking "what is the cheapest deterministic check on this output?" Sometimes there is one hiding in plain sight. Schema validation on structured extraction. A test suite on generated code. A reconciliation query on a financial figure. A retrieval check that every cited clause actually exists in the source document. These are not glamorous, and they are the difference between an agent you can leave running and a demo you have to babysit.

This is the same conclusion I reached building multi-agent systems that run themselves, and it is why the smartest model is not what makes AI agents reliable.

What Does Verification Still Not Fix?

Formalization worked for FLT because mathematics is the rare domain where truth is fully mechanizable. Most business outputs are not. There is no compiler for "is this the right tone for this customer" or "does this contract position match our risk appetite." Anthropic is careful about this in the post itself, noting that a formalized proof should not replace a human-understandable exposition. Verification and judgment are different jobs.

The Meta paper closes the loop in an uncomfortable way. Once you build the checker, the checker becomes the target. An injected image that rewrites a tool manifest is an attack on the control layer, not the model. Anthropic's own account of the FLT effort includes a detail worth sitting with: early agent runs lost track of project state and stopped collaborating, and those failed efforts still contributed about 7% of the non-boilerplate lines in the final proof. Failed work survives into the artifact. It only stayed safe because Lean checked the whole thing at the end rather than trusting the process that produced it.

That is a design principle, not a warning. Verify the output, not the intent. Intent can be injected.

There is also a human layer this does not touch. A verified output still lands in front of a reviewer who has to decide whether it was the right output to produce at all, and that call runs on domain knowledge no checker holds, which is why domain expertise is what makes AI agents actually work.

The Practitioner Take

I have been building multi-agent systems where the agents write to shared state, and the failure I keep hitting is not that a model is wrong. It is that nothing downstream notices. An agent produces something plausible, another agent consumes it, and by the time a person looks, four steps have compounded on a bad input. Adding a smarter model does not help. Adding a check does.

The FLT result is the clearest proof yet that this is the whole game. Six billion tokens of unsupervised work became credible in a single compile step. Strip out the compile step and it is 13 million lines nobody would ever have read, produced by a process that included failed runs and lost state. Same tokens, completely different asset. The checker is not overhead on the system. The checker is what makes the system an asset.

The thread connecting this week's research: Anthropic showed what autonomy produces when a verifier exists. OpenAI showed what autonomy costs and how often it still needs a human hand on the wheel. Meta showed what happens to the layer you built to keep it honest. Read in sequence, they say the same thing. Capability is scaling on a curve everyone can see. Verification is scaling on a curve almost nobody is funding, and it is the one that determines whether any of this ships.

Key Terms Explained

Term	What it means in plain language
AI agent verification	A mechanical check that decides whether an agent's output is acceptable, without requiring a person to read the output.
Autoformalization	Automatically rewriting a human-written proof or argument into a form a computer can check step by step.
Proof assistant (Lean)	Software that verifies every logical step of a proof and rejects it if any step is unjustified.
Visual prompt injection	Hiding instructions inside an image so that an AI system reading that image follows them as if the user had given them.
Attack success rate (ASR)	The share of attempts in which an attack achieves its intended effect on the target system.
Intervention rate	The share of agent tasks that needed at least one human correction before completing successfully.

How Do I Add Verification to My Own AI Workflows?

Start by writing down, for each AI workflow you run, what the deterministic check on its output actually is. If the honest answer is "a person reads it," you have found the thing that will break first when volume goes up. Pick your highest-volume workflow and build one mechanical check this quarter: a schema validator, a reconciliation query, a citation-existence test, a diff against a known-good template. One check that a machine runs on every output, not a sample.

Then measure your own intervention rate the way OpenAI measured theirs. Count how many agent tasks completed without a human touching them, split by how long the task took. The number will be lower than your leadership deck assumes, and knowing it is worth more than another pilot. It also gives you an honest baseline to improve against.

Treat your tool manifests, system prompts, and configuration files as production security surface, because the Meta paper just showed they are. Any file an agent can write to, an attacker can try to make it write to. Put change detection on those files and require human approval to modify them, the same way you would for an IAM policy.

Finally, when you evaluate a vendor, ask what their agent's output can be checked against rather than what benchmark it scored on. Benchmarks tell you what happened in a lab. A checker tells you what will happen on Monday. If you are formalizing this into a review process, how to evaluate LLM outputs covers the mechanics.

The Bigger Picture

For 350 years, Fermat's Last Theorem was a problem about finding a proof. In 1908 a prize was offered for one, and 621 incorrect attempts arrived in the first year alone. Wiles solved the finding problem in 1995. What Anthropic solved this month is the checking problem, and the checking problem turns out to be the one that governs everything downstream: whether results can be trusted, built upon, and produced at volume without the review burden collapsing.

Every organization deploying agents is now standing exactly where mathematics stood before Lean. Plenty of output. No mechanical way to know which of it is sound.

Capability is the part you can buy. Verification is the part you have to build.

Frequently Asked Questions

What is AI agent verification?

AI agent verification is a mechanical check on an agent's output that runs without a person reading it. A compiler, a schema validator, a test suite, or a reconciliation query all qualify. The defining property is that it accepts or rejects the artifact on its own, so trust does not depend on review capacity.

Why did nobody need to read Claude's Fermat's Last Theorem proof?

Because Lean read it. The proof is 13 million lines of formal code, and Lean verified every step using only its three standard axioms. A separate comparator confirmed the theorem statement matched Mathlib's. Machine checking replaced the months of human review that Wiles's 129-page proof required in 1995.

How long can AI agents now work without human supervision?

Anthropic's FLT effort ran 11 days with only occasional one-line steering from a researcher. That is the current public high-water mark for useful unsupervised agent work. OpenAI's data is more sobering: over half of successful four-to-eight-hour agent tasks still needed at least one human intervention.

What is the 3.1 agent-workdays number from OpenAI?

OpenAI disclosed that as of mid-August 2026, its research organization used 3.1 agent-workdays of effort for every workday of human labor. It is the clearest public measure of how far agent output has outrun human review capacity inside a frontier lab.

What does a visual prompt injection attack do?

It hides instructions inside an image so an AI system that reads the image follows them as if the user had issued them. Meta's Repeat-After-Me paper reached above 80% attack success on open-weight models and 47% on GPT-5.5, and showed an injected image overwriting a tool manifest in a default deployment.

Why is the tool manifest a security surface now?

Because an agent can write to it and an attacker can make the agent write to it. Meta demonstrated an untrusted user overwriting TOOLS.md in a default OpenClaw Discord deployment, opening a path to remote code execution and secret exfiltration. Treat any agent-writable config file like an IAM policy.

What is the cheapest deterministic check for a business AI workflow?

Usually one already available: schema validation on structured extraction, a test suite on generated code, a reconciliation query on a financial figure, or a citation-existence test confirming every quoted clause appears in the source document. Pick the highest-volume workflow and add one machine check that runs on every output.

Does verification replace human judgment?

No. Verification answers whether an output is correct against a rule. Judgment answers whether it is the right output at all. There is no compiler for tone, risk appetite, or strategic fit. Anthropic makes the same point, noting a formalized proof should not replace a human-understandable exposition.

How do I measure my team's agent intervention rate?

Count how many agent tasks completed with no human touching them, split by task duration. Track it over a quarter. OpenAI's own split shows the rate worsens sharply past the four-hour mark, so duration is the variable that matters. The number is usually lower than leadership decks assume.

What should I ask an AI vendor about verification?

Ask what their agent's output can be checked against, not what benchmark it scored on. A benchmark describes lab conditions. A checker describes what happens in your pipeline on Monday. If the only answer is that a human reviews it, you have found the part that breaks when volume rises.

Sources and Credits

All research referenced is the intellectual property of the respective organizations. This article represents Harel Asaf's independent practitioner analysis.

Lab	Publication title	URL	Date
Anthropic	Formalizing Fermat's Last Theorem	https://www.anthropic.com/research/formalizing-fermats-last-theorem	Sep 4, 2026
OpenAI	Research acceleration: The view inside OpenAI	https://openai.com/index/research-acceleration-view-inside-openai/	Sep 6, 2026
Meta AI	Repeat-After-Me: Black-Box Adaptive Visual Prompt Injection	https://ai.meta.com/research/publications/repeat-after-me-black-box-adaptive-visual-prompt-injection/	Sep 7, 2026
OpenAI	GPT-6 Astra: A new generation of intelligence	https://openai.com/index/gpt-6-astra/	Sep 3, 2026
OpenAI	GPT-6 Astra System Card	https://deploymentsafety.openai.com/gpt-6-astra	Sep 3, 2026
Google DeepMind	Designing Proactive Thought Partners for Writing	https://deepmind.google/research/publications/265605	Sep 1, 2026
Meta AI	Alignment-Free Text-Audiobox for Voice Dubbing and Full-Duplex Dialogue Synthesis	https://ai.meta.com/research/publications/alignment-free-text-audiobox-for-voice-dubbing-and-full-duplex-dialogue-synthesis/	Sep 6, 2026

Seven qualifying publications were detected this week. The three carrying the article are the first three listed. The remaining four are credited here for completeness.

---

Harel Asaf is an AI systems builder with a legal background. He focuses on operational AI, the systems that survive Monday morning. Based in Tel Aviv.

type="application/ld+json"

dangerouslySetInnerHTML={{

__html: `{

"@context": "https://schema.org",

"@type": "Article",

"headline": "AI Agent Verification: Why the Checker Beats the Model",

"description": "Claude wrote a 13-million-line proof in 11 days and nobody read it. AI agent verification, not model capability, is what turns autonomous output into an asset.",

"author": { "@type": "Person", "name": "Harel Asaf", "url": "https://harelasaf.com/about" },

"datePublished": "2026-09-09",

"dateModified": "2026-09-09",

"mainEntityOfPage": { "@type": "WebPage", "@id": "https://harelasaf.com/articles/ai-agent-verification" },

"publisher": { "@type": "Person", "name": "Harel Asaf" },

"keywords": "AI agent verification, autonomous AI agents, visual prompt injection, AI oversight for teams"

}`,

}}

/>

type="application/ld+json"

dangerouslySetInnerHTML={{

__html: `{

"@context": "https://schema.org",

"@type": "FAQPage",

"mainEntity": [

{ "@type": "Question", "name": "What is AI agent verification?", "acceptedAnswer": { "@type": "Answer", "text": "AI agent verification is a mechanical check on an agent's output that runs without a person reading it. A compiler, a schema validator, a test suite, or a reconciliation query all qualify. The defining property is that it accepts or rejects the artifact on its own, so trust does not depend on review capacity." } },

{ "@type": "Question", "name": "Why did nobody need to read Claude's Fermat's Last Theorem proof?", "acceptedAnswer": { "@type": "Answer", "text": "Because Lean read it. The proof is 13 million lines of formal code, and Lean verified every step using only its three standard axioms. A separate comparator confirmed the theorem statement matched Mathlib's. Machine checking replaced the months of human review that Wiles's 129-page proof required in 1995." } },

{ "@type": "Question", "name": "How long can AI agents now work without human supervision?", "acceptedAnswer": { "@type": "Answer", "text": "Anthropic's FLT effort ran 11 days with only occasional one-line steering from a researcher. That is the current public high-water mark for useful unsupervised agent work. OpenAI's data is more sobering: over half of successful four-to-eight-hour agent tasks still needed at least one human intervention." } },

{ "@type": "Question", "name": "What is the 3.1 agent-workdays number from OpenAI?", "acceptedAnswer": { "@type": "Answer", "text": "OpenAI disclosed that as of mid-August 2026, its research organization used 3.1 agent-workdays of effort for every workday of human labor. It is the clearest public measure of how far agent output has outrun human review capacity inside a frontier lab." } },

{ "@type": "Question", "name": "What does a visual prompt injection attack do?", "acceptedAnswer": { "@type": "Answer", "text": "It hides instructions inside an image so an AI system that reads the image follows them as if the user had issued them. Meta's Repeat-After-Me paper reached above 80% attack success on open-weight models and 47% on GPT-5.5, and showed an injected image overwriting a tool manifest in a default deployment." } },

{ "@type": "Question", "name": "Why is the tool manifest a security surface now?", "acceptedAnswer": { "@type": "Answer", "text": "Because an agent can write to it and an attacker can make the agent write to it. Meta demonstrated an untrusted user overwriting TOOLS.md in a default OpenClaw Discord deployment, opening a path to remote code execution and secret exfiltration. Treat any agent-writable config file like an IAM policy." } },

{ "@type": "Question", "name": "What is the cheapest deterministic check for a business AI workflow?", "acceptedAnswer": { "@type": "Answer", "text": "Usually one already available: schema validation on structured extraction, a test suite on generated code, a reconciliation query on a financial figure, or a citation-existence test confirming every quoted clause appears in the source document. Pick the highest-volume workflow and add one machine check that runs on every output." } },

{ "@type": "Question", "name": "Does verification replace human judgment?", "acceptedAnswer": { "@type": "Answer", "text": "No. Verification answers whether an output is correct against a rule. Judgment answers whether it is the right output at all. There is no compiler for tone, risk appetite, or strategic fit. Anthropic makes the same point, noting a formalized proof should not replace a human-understandable exposition." } },

{ "@type": "Question", "name": "How do I measure my team's agent intervention rate?", "acceptedAnswer": { "@type": "Answer", "text": "Count how many agent tasks completed with no human touching them, split by task duration. Track it over a quarter. OpenAI's own split shows the rate worsens sharply past the four-hour mark, so duration is the variable that matters. The number is usually lower than leadership decks assume." } },

{ "@type": "Question", "name": "What should I ask an AI vendor about verification?", "acceptedAnswer": { "@type": "Answer", "text": "Ask what their agent's output can be checked against, not what benchmark it scored on. A benchmark describes lab conditions. A checker describes what happens in your pipeline on Monday. If the only answer is that a human reviews it, you have found the part that breaks when volume rises." } }

]

}`,

}}

/>