Epoch AI examines whether AI can automate its work

Epoch AI examines whether AI can automate its work

Epoch AI published this account on October 8, 2026. It describes giving six models real tasks from Epoch, then manually grading their work against employee standards. Epoch says frontier models were reliable on well-defined tasks such as coding and data analysis, but struggled with open-ended judgment: learning implicit workplace standards, designing informative experiments and drawing meaningful conclusions from flawed results. The distinction matters to Epoch’s central question: not just whether a model can complete a bounded task, but whether it can carry work through end to end. Epoch argues that benchmarks focused on easy-to-grade tasks can miss these gaps, while realistic work tasks may reveal them. Its examples include models producing overly information-dense graphics, choosing niche topics, and treating results from a flawed experiment as key findings. Epoch’s conclusion is that AI cannot yet replace workers, at least at Epoch. The evaluation is limited to Epoch-specific tasks and standards; each model was run once per task, grading was partly subjective, and each model used its own harness. Epoch plans to add results as new models are released. How these findings would translate to other workplaces remains unknown.

Publication: 57725a73-5d13-75e9-7bf8-f3eeaada4a1a

attribution — pass: The reading consistently attributes the evaluation, findings, conclusion and plans to Epoch AI. The text is a single Epoch publication with its own methodology and argument, not a listing of other pieces.

We gave six models real work tasks from Epoch, such as graphic generation and research design, to test how close they are to automating our work.

summary — pass: The summary accurately conveys Epoch's method, its findings that models are reliable on well-defined tasks but weak on open-ended judgment, its benchmark critique, examples, the not-yet-replace conclusion and its stated limitations. The glossary defines frontier model and benchmark. Harness is unexplained, but it appears only in a limitation and is not central. The question follows from the stated limitations.

However, they consistently fall short on the judgment that drives real work.

date — pass: The reading states October 8, 2026, which matches the UTC calendar day of evidence.originalPublished. Nothing is presented as happening after that date. The future plan to add results is attributed to Epoch as a plan, not reported as an outcome. The evidence is a publisher page, so no archive disclosure is needed.

We plan to add results as new models are released, informing questions about AI adoption and automation over time.

support — pass: Each claim is source-stated and backed by an exact excerpt. The examples in the summary appear in the source: information-dense graphics, a niche topic choice and flawed experiment results presented as key findings. The limitations, including single runs, subjective grading, Epoch-specific tasks and differing harnesses, are stated by Epoch. The reading adds no unsupported benefits or costs.

Small sample size. We have a limited task suite, and we run each model only once per task.

media — pass: No media is selected, and no verified media candidates were supplied. Choosing none is appropriate.