Build an agent evaluation set before you train

Before tuning an agent, define the work it must complete. Without that foundation, a new prompt or checkpoint can look better simply because it produced a more confident explanation. The goal of an evaluation set is to make a decision reproducible: should this version replace the one currently doing the job?

One practical packaging option is Harbor’s task format, which combines an instruction, an execution environment, and a verification script. Its documentation treats tasks as isolated, reproducible units. The workflow below is an editorial proposal for constructing a useful task collection, not a claim that a particular task count guarantees reliable evaluation.

1. Choose a narrow decision

Write one sentence describing the comparison. For example: “We want to know whether the candidate agent can resolve bounded documentation defects with less review effort while preserving factual accuracy.” This is more useful than asking whether a model is generally intelligent or good at coding.

Define the boundaries beside that sentence. List the files or systems the agent may change, the actions requiring approval, and the resources available during a run. A candidate should not win by using access that the production agent will never have.

2. Collect tasks with known outcomes

Start with a manageable pilot drawn from real categories of work. Include straightforward cases, messy cases, and cases where no change is appropriate. Remove personal data and secrets. Preserve enough context to make the task solvable without private knowledge held only by the evaluator.

For each task, record the initial state and an acceptable outcome. There can be several valid solutions. Avoid assuming that the exact wording or patch from a historical human solution is the only correct answer. The evaluator should check the requirement, not reward imitation.

3. Design the grader before the agent runs

Use deterministic checks for properties that can be measured directly: a document exists, a link resolves inside the fixture, a calculation matches the specified result, or a test suite passes. Use human review for qualities that remain ambiguous, such as whether an explanation accurately represents uncertain evidence.

Here is an illustrative task contract:

• Input: a small repository containing an outdated setup guide and a current configuration file.
• Required outcome: the guide matches the configuration and explains one missing prerequisite.
• Protected behavior: application code is unchanged.
• Verification: check required statements, inspect the changed-file list, and review the final prose.

Try to defeat the grader yourself. Would an empty file pass? Could an answer copy the requirement without completing it? Would a correct alternative fail because of harmless formatting? Fix those weaknesses before treating the score as evidence.

4. Keep acceptance tasks out of the tuning loop

Split tasks by underlying problem family, repository, or source where appropriate. Nearly identical tasks on both sides can produce reassuring scores without demonstrating generalization. Keep the final acceptance set separate from the examples used to adjust prompts, choose training data, or debug the grader.

Version the collection. When a task changes, record why. Repairing a broken fixture is legitimate, but silently changing acceptance criteria after seeing a candidate’s output makes the comparison difficult to trust. Preserve earlier results with the exact task version they used.

5. Report failures in useful categories

At minimum, distinguish an incorrect result, an infrastructure failure, an unavailable dependency, and a run that exceeded its budget. Include the original task, the produced artifact, and the execution trace needed for review. A single success percentage can hide where a team should invest its effort.

Repeat tasks when variability matters, and keep the repeat policy consistent across candidates. Report the number of attempts and any excluded runs. A small pilot is useful for discovering failure modes; it should not be presented as a precise estimate of every production outcome.

6. Make the release decision explicit

Write acceptance criteria before reviewing the final set. Include quality, prohibited side effects, time, and review burden. A gain in one dimension should not silently excuse a serious regression in another.

The output is more than a leaderboard. It is a reusable collection of tasks, environments, graders, and decision records. That collection helps the team compare a new model, a simpler harness change, or a post-trained checkpoint using the same definition of useful work.