Epoch Tested Whether AI Can Do Epoch's Job. The Failures Show Where Agents Need a Human Gate

Epoch AI did something most benchmark authors do not do. It gave six models real tasks from its own daily work, graded the results against its own standards, and published that the models cannot yet automate its work.

The useful part is not the ranking. It is the shape of the failures. The models were reliable on well-defined steps and weak on the judgment around them. That is a practical guide for deciding where an agent can own a step and where a person should stay in the loop.

What Epoch tested

On October 8, Epoch published "Can AI automate Epoch?" by Kelly Hong and Greg Burnham. The suite has 11 tasks across five categories: graphic design, Data Insights, Data Explorers, AI data center research, and a research-design task.

Epoch says it gave each model the relevant resources and full permissions in its own workspace and Google account, following the principle of providing as much context as it would to a new hire. The six models were GPT-6 Astra, Claude Fable 5.1, Grok 4.6, Gemini 3.8 Flash, Kimi K3, and Qwen 3.8 Max, each on its own harness at its highest reasoning setting.

Epoch also lists its limits. It ran each model once per task, used a single human grader, and graded against Epoch's own standards. It says the scores are noisy and should not be read as a general measure of what AI can do across work tasks. Epoch says its qualitative findings are more informative than the numbers. AI Insiders makes the same point about the grading.

I did not run these tests myself. Everything below is Epoch's reported result.

Where the frontier models were reliable

Epoch says Fable 5.1 and GPT-6 Astra were broadly tied at the top. Their lead came from reliability on well-defined parts of the tasks, such as coding and computational analysis, where outputs were consistently accurate.

Epoch also reports that computer use was far more capable than it expected. Earlier this year, frontier closed-weight models failed at porting an article to Substack, and newer models succeeded. It removed one graphic design task, restyling a Matplotlib chart to house style, because Fable 5.1 produced an Epoch-quality output. Epoch says it now largely automates that step.

That is the pattern to notice. A step with clear inputs, a clear standard, and a checkable result is a good candidate for an agent.

Where they fell short

Epoch's key findings describe the gaps in open-ended work.

Picking up standards. Models missed implicit conventions such as visual style and what the audience cares about, even with reference material. In one example, Epoch says Fable 5.1 produced a far more information-heavy graphic than its typical work, and that the extra information did not help explain the idea.

Experiment design. Epoch says models can name promising research directions but struggle to design experiments that measure what they claim.

Flawed results treated as findings. Epoch describes a pilot where the model's own token cap of 4,096 cut off 61 of 280 responses before they named an input. The model noticed and raised the budget, but its write-up still framed the effect as "sensitivity to the acquisition budget." Epoch calls that framing misleading, because the models were unable to produce an answer at all.

Convergence. On tasks with a large idea space, Epoch says models often converged on similar ideas. In its polling task, three different models chose the same topic.

Open-weight models. Epoch says they lagged further behind, struggling even on well-defined tasks. In one example, it says Kimi K3 built an insight on a data filtering error and never checked that its filter worked.

A way to split a workflow

Epoch's result suggests a simple split. I would apply it to any agent workflow before deciding how much to hand over.

Step type	Example	Suggested owner
Well-defined, checkable	Restyle a chart to a documented standard	Agent, with an automatic check
Well-defined, silent failure risk	Filter a dataset	Agent, with a sanity check on counts and scope
Judgment about what to measure	Design the experiment	Person, or agent with a person approving the design
Judgment about what a result means	Decide what the pilot proves	Person
Taste and audience fit	Choose style, density, topic	Person, until standards are written down and tested

This table is my own framing, not something Epoch published. It is a way to apply their findings.

Gates that catch these failures

Each failure Epoch describes maps to a cheap gate.

• For silent filter errors: require the agent to report row counts before and after each filter, and to state the population the result covers.
• For flawed setups presented as findings: require a separate section listing known setup problems, and bar the agent from presenting a result as a key finding if a listed problem could explain it.
• For vague experiment designs: ask for the exact procedure and the exact measurement, and have a person check that the measurement answers the question.
• For missing standards: write the implicit rules down. If a human editor would say "we never do that," put it in the instructions and test it on a few examples.
• For convergence: if you want varied ideas, sample several runs and compare, rather than trusting a single run to explore the space.

None of these need a new tool. They are review habits applied at the points where Epoch saw trouble.

What not to conclude

Do not read this as "AI cannot do real work." Epoch itself reports that models reliably handled coding, analysis, and computer use steps, and that one design task is now largely automated in its own workflow.

Do not read it as a general ranking either. Eleven tasks, one run each, one grader, and different harnesses for each model is a small basis. Epoch says the harness may account for some differences.

And do not assume your work looks like Epoch's. Epoch scoped the tasks to its own work and graded them by its own standards. The right move is to run the same kind of test on your workflow: pick tasks you already know how to grade, run them, and see where the output needs a person.

Frequently asked questions

What are Epoch Automation Reports?

A benchmark of realistic, open-ended tasks drawn from Epoch AI's own work. Models produce work outputs, and humans grade them against Epoch's internal standards.

Which models did Epoch test?

GPT-6 Astra, Claude Fable 5.1, Grok 4.6, Gemini 3.8 Flash, Kimi K3, and Qwen 3.8 Max, each on its own harness.

What did Epoch conclude?

That models cannot yet autonomously produce Epoch-quality work. Frontier models were reliable on well-defined tasks and failed in open-ended aspects that need judgment.

How reliable are the scores?

Epoch says they are noisy and subjective. It ran each model once per task with a single grader, and leans on qualitative findings instead.

Does this mean agents are not useful at work?

No. Epoch reports strong results on coding, data analysis, and computer use, and says it now largely automates one design step.

What is the practical takeaway?

Give agents well-defined, checkable steps, and keep a person at the points that need judgment about what to measure, what a result means, and what fits the audience.

Can I rely on this for my own workflow?

Treat it as a pattern to test, not a verdict. Run your own graded tasks before deciding how much to delegate.

Sources

• Epoch AI, Can AI automate Epoch?: the report, task suite, setup, findings, and limitations.
• AI Insiders: independent summary of the report and its grading limits.