Epoch AI tests models on its own real work: Fable 5.1 and GPT-6 Astra lead, but still present flaws in their own experiments as findings

Why it matters

【Key points】

On Oct 8, Epoch AI published "Can AI automate Epoch?" by Kelly Hong and Greg Burnham, launching Epoch Automation Reports. It takes real tasks from Epoch's own work, such as making Epoch-style charts and designing and piloting research experiments, gives them to six models at the highest reasoning settings with no intervention, and grades the outputs by hand against Epoch's employee standards.

Findings: Claude Fable 5.1 and GPT-6 Astra are broadly tied in the lead but fall well short of automating Epoch's work. Open-weight models lag further, struggling even on well-defined tasks frontier models handle reliably. Even with many examples, models miss implicit conventions like visual style and audience interests; Fable 5.1's charts are far denser than Epoch's. Models propose promising directions but design uninformative experiments and present flaws in their own setup as key findings: GPT-6 Astra set agents' token budgets too low and reported "sensitivity to the acquisition budget" as a result. Ideas also converge despite a large idea space. Epoch plans to expand tasks and retire saturated ones.

【Verifiable】

Report: https://epoch.ai/publications/can-ai-automate-epoch

Benchmark page: https://epoch.ai/benchmarks/epoch-automation-reports

Official X: https://x.com/EpochAIResearch/status/2108250259889831979

【Our read】

1. This matches Epoch's InnovationEval (https://agipulse.org/e/712): in research, frontier models lack self-doubt, not ideas. Reporting your own bug as a finding is the most dangerous failure because it looks like output.

2. Mind the limits: tasks from one organization, graded by its own staff, six models. Treat it as structured qualitative evidence, not a general ranking.

3. Teams using agents for analysis should write hypotheses and checks first and require the model to list what could overturn its result. The open-weight gap on well-defined tasks is a real cost for self-hosters.