ThinkingBox checks the records an agent leaves behind

Microsoft and Hugging Face presented ThinkingBox on October 3, with a benchmark covering 507 stateful business workflows and 20 trials per task. The system evaluates the final backend state and side effects, rather than relying only on an agent’s final message or the apparent validity of its tool calls.

The distinction is easy to recognize in ordinary software. An assistant can say it updated a reservation while changing the wrong record. It can create a support ticket but close it before the issue is resolved. A trace of successful API calls can coexist with an incorrect business outcome.

Write the finish line as a state

For a task that changes records, define which values must be different afterward and which must remain unchanged. A correct appointment update might require the new time, the same attendee list, and no duplicate booking. Those conditions can be checked independently of the model’s explanation.

Include forbidden side effects. If an agent achieves the requested change while deleting an unrelated record, a success-only check can miss the damage. A test should examine the relevant surrounding state as well as the target field.

Reset the environment between trials. Otherwise, an earlier run can make a later task easier, harder, or simply different. Preserve the initial state so a failure can be replayed and investigated without guessing what the agent encountered.

Repeated success is a different metric

ThinkingBox distinguishes average attempt success, whether a task succeeds at least once across repeated attempts, and whether all 20 recorded attempts succeed. Those measures answer different questions. A task that is possible with many tries may still be unsuitable for unattended execution.

Even 20 successful trials do not prove that a system will never fail. They provide stronger evidence under the tested conditions. Production can introduce new records, changed policies, network failures, and users whose requests differ from the benchmark.

Teams can borrow the method without reproducing the full benchmark. Select a few consequential workflows, define executable state checks, and rerun them from clean snapshots. Compare failure patterns across model or prompt changes. This creates a record of whether a modification improved consistency rather than merely producing a more convincing demonstration.

ThinkingBox’s contribution is a concrete way to make the business system answer whether the work was done. That shifts evaluation toward durable evidence: the records that remain, the side effects that occurred, and the consistency with which the same task reaches the correct state.