Why a safety score is not a deployment forecast

A benchmark score is a measurement under particular conditions. It becomes a forecast only after someone establishes that those conditions are relevant to the work a system will actually do. That second step is easy to omit when comparing AI agents.

In a November 3 OpenAI study, researchers compared six models on a warning-respecting safety evaluation with matched production decisions. They found lower success rates in the evaluation, with gaps ranging from 10.2 to 37.4 percentage points. The direction is notable: the test made these models look less safe than the selected production comparisons.

The researchers also found that explicitly framing the evaluation’s objective could shift behavior. They warn that the production matching was approximate, case mixtures differed, and the sample did not cover every deployment environment. Their result is not a general guarantee that real-world use is safer than testing.

Start with the decision being measured

For a team choosing an agent, a useful first question is what constitutes success. Does the test measure whether a model produces correct text, completes a workflow, respects an authorization boundary, or recovers from a failed tool call? Those are related capabilities, but they can fail independently.

Consider a hypothetical assistant that updates a support knowledge base. One test asks whether it writes a correct article. Another asks whether it waits for the required editorial approval. A third checks that a rejected draft never becomes public. Passing the writing test says little about the other two.

A responsible evaluation report should preserve these distinctions. If a product promises supervised publishing, the approval boundary belongs in the acceptance criteria, even when the generated article is excellent.

Check the denominator before the percentage

Suppose an illustrative trial succeeds on nine of ten tasks. That result sounds like a useful signal. But its interpretation changes if all ten tasks use the same document type, if the tasks were selected after seeing the outputs, or if the most difficult cases were excluded as ambiguous.

The right response is not to dismiss small evaluations. Small, carefully described evaluations can identify useful defects. The response is to write down their scope: the kinds of users represented, the tool access available, the conditions that count as failure, and the cases deliberately left out.

For each result, keep an example of success and an example of failure. A reader can often understand those examples more precisely than an aggregate score. A failure trace can also reveal whether the problem lies in the model, the instructions, the tool interface, or the grading rule.

Preserve the operating context

Our recommendation is to keep an evaluation configuration beside the result. Record the instructions, model identifier, available tools, permission rules, task data, and grading criteria. If any of them changes, the next run is a new experiment rather than a direct repeat.

In the publishing example, a test environment might make every article editable while the live environment includes read-only syndicated material. That difference matters. Likewise, a production approval may be attached to a specific draft revision. An evaluation that represents approval as a permanent boolean would miss the possibility of the agent changing the draft afterward.

Use results to choose the next test

A productive review ends with a question that can be investigated. If the agent misunderstands a warning, test whether clarifying its scope resolves the problem without changing the underlying permission. If the grader rejects a valid approach, repair the grader and rerun both the old and new systems.

After launch, compare real incidents with the cases in the evaluation set. Add representative failures without exposing private user data. Keep a record of why each case exists, so a later maintainer does not remove a difficult test simply because it lowers a score.

The study is a reminder to read benchmark claims as evidence with boundaries. A useful score helps decide what to inspect, which change to make, and what to test next. It should not stand in for those decisions.