Custom Graders & Tasks

When your skill’s success criteria can’t be expressed by the default graders — a domain-specific output format, a service that must end up in a particular state, a scoring rule only you can define — you can bring your own grader, or your own complete Harbor task. Harbor is the open-source agent evaluation framework that executes Tier 3 trials. Scaffolding is keyless and takes one command; running a custom-graded evaluation needs the same Tier 3 stack as any live eval — see Tier 3: Live Evaluation.

Grading modes

Every live evaluation runs in one of three grading modes:

Set the mode with --grading-mode on tier3 evaluate (or on validate together with --tier3), or persist it in evals/config.yml:

1
schema_version: 1
2
grading:
3
  mode: default_plus_custom

CLI flags override the config file. The aliases aces_default and aces_plus_custom are accepted — on the CLI and in evals/config.yml — as spellings of default and default_plus_custom.

Bring your own grader (BYOG)

A BYOG grader is a single evals/grader.py (or evals/grader.sh) that SkillEvaluator stages into every generated task container and runs after the agent finishes.

$
skillevaluator init-custom-grader ./my-skill

$
skillevaluator tier3 evaluate ./my-skill --agents codex --grading-mode default_plus_custom

The reward contract

Inside the container, your grader reads two inputs and writes one result:

1. Read /logs/agent/trajectory.json — the agent’s trajectory.
2. Read /tests/entry.json — the eval case metadata (id, prompt, expected_output, and the rest).
3. Write /logs/verifier/reward.json — your scores, each in the range 0.0–1.0 — and also write the overall score as plain text to /logs/verifier/reward.txt for compatibility. In custom_only mode, a numeric overall in reward.json or a numeric reward.txt is required.

1
{
2
  "overall": 0.85,
3
  "custom_metrics": {
4
    "custom_check": 0.85
5
  },
6
  "details": {
7
    "custom_check": {
8
      "score": 0.85,
9
      "reason": "Replace this with real custom grading logic."
10
    }
11
  }
12
}

Rather than hard-coding the paths, read them from the HARBOR_* environment variables and fall back to the defaults below when a variable is unset — the starter grader already does. Harbor records the interaction history as the Agent Trajectory Interchange Format (ATIF), a standardized JSON trajectory:

Bring your own task (BYOT)

BYOG replaces the scoring; BYOT replaces the whole task — instruction, container environment, and grader — with a native Harbor task you own under evals/harbor/.

$
skillevaluator init-harbor-task ./my-skill

Exact defaults for both scaffolding commands are in the CLI Reference.

The scaffold produces a complete, runnable task source:

Edit instruction.md for the agent-facing task, environment/Dockerfile for the container, and tests/grader.py for the scoring — the same reward contract as BYOG applies.

Two rules keep BYOT predictable:

• Case IDs must match dataset entries. When the grading mode is default or default_plus_custom, each case’s entry id — entry_id under [metadata] in task.toml, falling back to the case directory name — must match an entry id in evals/evals.json, or the run refuses to start. custom_only has no such requirement. See Eval Datasets.
• Your source is never mutated in place. SkillEvaluator stages with-skill and baseline copies of evals/harbor/ under the results directory for each run; the files you author stay untouched.

Validate the contract

Check your evals/ tree and the Harbor task and reward contract before spending compute:

$
skillevaluator tier3 validate ./my-skill --harbor-contract

Add --strict to treat warnings as failures and --json for machine-readable output.

Debug retained jobs

Harbor job artifacts are transient — deleted by default whether the run succeeds or fails. Pass --harbor-keep-jobs to retain them, then browse the trajectories and grader output with Harbor’s own viewer (when a retained jobs directory exists, the Artifacts panel at the end of the run prints the exact harbor-view command to paste):

$
skillevaluator tier3 evaluate ./my-skill --agents codex --harbor-keep-jobs
$
skillevaluator harbor-view <jobs-dir>

Grader stdout, the written reward.json, and the full trajectory are all in the retained job tree — the fastest way to find out why a score isn’t what you expected. Agents & Sandboxes covers the debugging workflow in depth.

What reports show

Custom scores flow into the same results tree as everything else:

• In result.json, each agent carries per-arm custom metric averages (custom_with_skill, custom_without_skill) and a per-metric custom_lift — with-skill minus without-skill, per custom metric. The lift is also written to custom_lift.json next to the standard lift.json.
• In default_plus_custom mode, reports keep the standard metric and dimension view and add your scores as clearly labeled Custom: entries alongside it, each with its own lift; custom metrics never change the standard dimension scores.
• In custom_only mode, your overall reward drives pass@k and the overall lift. The standard dimension rows stay visible but render NO SCORE — unscored metrics are never coerced to 0.0, and your custom metrics are never re-labeled as standard dimensions.

Custom lift needs both arms — a --skip-baseline run reports your custom scores but no lift. Reports & Results covers the full on-disk layout and the machine-readable contract.

Troubleshooting

Next steps