Epoch AI finds a gap between partial credit and finished work

Epoch AI published results from an effort to evaluate models on work its staff actually performs, including research planning, design, coding, and analysis. The organization assessed outputs against the standards it would apply to employees, rather than relying only on conventional model benchmarks.

The leading models, Claude Fable 5.1 and GPT-6 Astra, met about 65 percent of rubric items on average. Epoch emphasized that this was not equivalent to completing 65 percent of a job or saving 65 percent of its time. The organization also described one simpler charting task that had become useful enough to partly automate.

Partial credit can conceal the expensive part

A task can satisfy many easy requirements and still fail on the condition that determines whether anyone can use it. A report might have clear formatting, appropriate headings, and correct arithmetic while answering the wrong operational question. Counting all of those properties equally would make the output appear closer to completion than it feels to the reviewer.

An internal evaluation should therefore identify blocking requirements before scoring begins. These are the conditions whose failure makes the artifact unusable. A weighted score can still be informative, but the count of accepted deliverables should be reported separately.

Review effort belongs in the same picture. A partially correct result may save time when its errors are obvious and cheap to fix. It can consume time when a specialist must reconstruct the entire analysis to discover whether the conclusions are trustworthy. Those cases should not be treated as equivalent examples of assistance.

Start with tasks whose finish line is clear

The charting example points toward a practical adoption strategy. Break a broad responsibility into components with observable acceptance criteria, then automate the components that consistently meet them. Keep the surrounding judgment and coordination visible instead of assuming that success on one step transfers to the whole role.

A useful pilot can compare three paths: a person working alone, a person reviewing model output, and an agent operating within a tightly bounded workflow. Record elapsed time, correction effort, and final quality. This makes it easier to locate the actual benefit rather than infer it from a model score.

Epoch's results are informative because they connect evaluation to an organization's own standard of usable work. The broader lesson is to ask what can be accepted, what must be repaired, and who performs that repair. Those answers are more valuable to an operating team than an impressive average detached from its daily responsibilities.