Custom Graders & Tasks

When your skill’s success criteria can’t be expressed by the default graders — a domain-specific output format, a service that must end up in a particular state, a scoring rule only you can define — you can bring your own grader, or your own complete Harbor task. Harbor is the open-source agent evaluation framework that executes Tier 3 trials. Scaffolding is keyless and takes one command; running a custom-graded evaluation needs the same Tier 3 stack as any live eval — see Tier 3: Live Evaluation.

Grading modes

Every live evaluation runs in one of three grading modes:

Set the mode with --grading-mode on tier3 evaluate (or on validate together with --tier3), or persist it in evals/config.yml:

schema_version: 1
grading:
  mode: default_plus_custom

CLI flags override the config file. The aliases aces_default and aces_plus_custom are accepted — on the CLI and in evals/config.yml — as spellings of default and default_plus_custom.

Bring your own grader (BYOG)

A BYOG grader is a single evals/grader.py (or evals/grader.sh) that SkillEvaluator stages into every generated task container and runs after the agent finishes.

skillevaluator init-custom-grader ./my-skill

skillevaluator tier3 evaluate ./my-skill --agents codex --grading-mode default_plus_custom

The reward contract

Inside the container, your grader reads two inputs and writes one result:

1. Read /logs/agent/trajectory.json — the agent’s trajectory.
2. Read /tests/entry.json — the eval case metadata (id, prompt, expected_output, and the rest).
3. Write /logs/verifier/reward.json — your scores, each in the range 0.0–1.0 — and also write the overall score as plain text to /logs/verifier/reward.txt for compatibility. In custom_only mode, a numeric overall in reward.json or a numeric reward.txt is required.

{
  "overall": 0.85,
  "custom_metrics": {
    "custom_check": 0.85
  },
  "details": {
    "custom_check": {
      "score": 0.85,
      "reason": "Replace this with real custom grading logic."
    }
  }
}

Keep each reward to at most 128 custom metrics, with no more than 256 UTF-8 bytes in each printable, whitespace-trimmed metric name. The union across all rewards in one condition must also stay at or below 128 names. Exceeding these limits fails the affected reward or condition before aggregation so the machine-readable summary and HTML report cannot silently disagree. Names that look like credential values or credential fields are omitted without creating a colliding redaction alias; narrow metric-shaped names such as secret_handling and token_efficiency remain supported. Explicit custom-metric entries whose values are booleans, nonnumeric, non-finite, or outside 0.0–1.0 are omitted consistently from the persisted reward, aggregate, and report; they cannot contribute a score or consume the 128-name publication budget.

SkillEvaluator assigns each trial’s case ID from the staged Harbor task, not from an entry_id your grader writes. A trial whose staged identity is missing or ambiguous is kept only as an unscored diagnostic, and that condition’s case coverage fails.

Rather than hard-coding the paths, read them from the HARBOR_* environment variables and fall back to the defaults below when a variable is unset — the starter grader already does. Harbor records the interaction history as the Agent Trajectory Interchange Format (ATIF), a standardized JSON trajectory:

Your grader reads Harbor’s trajectory.json as written. Unlike the standard grader, it does not rebuild a missing or empty trajectory from agent logs, so handle that case explicitly. A step’s message and an observation result’s content can each be a string or a list of content parts (text, image, or audio), so read both forms.

Bring your own task (BYOT)

BYOG replaces the scoring; BYOT replaces the whole task — instruction, container environment, and grader — with a native Harbor task you own under evals/harbor/.

skillevaluator init-harbor-task ./my-skill

Exact defaults for both scaffolding commands are in the CLI Reference.

The scaffold produces a complete, runnable task source:

Edit instruction.md for the agent-facing task, environment/Dockerfile for the container, and tests/grader.py for the scoring — the same reward contract as BYOG applies.

A custom_only task that keeps its own Harbor test script instead of tests/grader.py runs that script unchanged, so the script must follow Harbor’s reward contract. Harbor 0.24 requires every top-level value in its reward.json to be a number, for example {"overall": 1.0, "native_quality": 1.0}; SkillEvaluator reports each numeric key other than overall as a custom metric. Nested objects such as custom_metrics or details make Harbor reject the reward with VerifierOutputParseError, so use tests/grader.py when you need them.

Two rules keep BYOT predictable:

• Case IDs must match dataset entries. When the grading mode is default or default_plus_custom, each case’s entry id — entry_id under [metadata] in task.toml, falling back to the case directory name — must match an entry id in evals/evals.json, or the run refuses to start. custom_only has no such requirement. See Eval Datasets.
• Your source is never mutated in place. SkillEvaluator stages with-skill and baseline copies of evals/harbor/ under the results directory for each run; the files you author stay untouched. In a paired run, the staged copies append -with-skill or -without-skill to each [task] name and record a missing entry_id under [metadata], so Harbor keeps the arms apart; results still resolve to your case IDs.

The staged task also has explicit compatibility boundaries:

• Native task projection is currently POSIX-only. A Windows agent environment, or any task/step verifier whose effective environment is Windows, is rejected before staging in every grading mode.
• default and default_plus_custom keep SkillEvaluator’s standard grader in the shared verifier environment. Effective separate-verifier contexts and non-empty steps/<step-name>/tests/ overlays are rejected. Task or step collect hooks and task-controlled HARBOR_*, executable-search, shell, process-loader, proxy/TLS, discovery, evaluator-provider, or judge-model environment controls are also rejected. An exact ${NAME} assignment for a provider or judge control is accepted in task/step verifier.env only when the operator explicitly staged that same name; unrelated verifier variables remain available. An operator-set LLM_JUDGE_FALLBACK_MODELS value is forwarded only to the standard verifier job, never to the agent environment or a custom_only run. SkillEvaluator’s Python verifier runs in isolated mode from a replaced tests/skill_evaluator/ payload so authored helper modules cannot shadow its imports.
• Use custom_only when the native task must own those surfaces. A shared verifier, or a separate verifier built from environment/, uses a step’s steps/<step-name>/tests/ script when one exists and otherwise falls back to the top-level test. A Dockerfile or docker-compose.yaml in a step or task tests/ directory, or a docker_image in the step’s or task’s [verifier.environment], instead defines a dedicated verifier image: that tests/ directory must contain its own test script, and Harbor uploads no tests to it. A task with tests/grader.py or tests/grader.sh cannot use a dedicated verifier image, because SkillEvaluator’s grader would never reach it. When every native custom_only verifier pass already resolves an authored test script and no native tests/grader.py or tests/grader.sh selects SkillEvaluator’s custom runner, staging preserves an authored tests/skill_evaluator/ directory.
• A paired baseline rejects authored top-level or step healthchecks, steps/<step-name>/workdir/ projections, Harbor’s effective task-shipped prior trajectory (trajectory.json for a single-step task or the first step’s trajectory.json), and skill-owned harbor.pre_agent_setup / setup_commands. Those surfaces can seed or observe agent behavior before the evaluated skill runs. They remain available for a with-skill-only --skip-baseline run.

Validate the contract

Check your evals/ tree and the Harbor task and reward contract before spending compute:

skillevaluator tier3 validate ./my-skill --harbor-contract

Add --strict to treat warnings as failures and --json for machine-readable output.

Debug retained jobs

Harbor job artifacts are transient — deleted by default whether the run succeeds or fails. Pass --harbor-keep-jobs to retain them, then browse the trajectories and grader output with Harbor’s own viewer (when a retained jobs directory exists, the Artifacts panel at the end of the run prints the exact harbor-view command to paste):

skillevaluator tier3 evaluate ./my-skill --agents codex --harbor-keep-jobs
skillevaluator harbor-view <jobs-dir>

Grader stdout, the written reward.json, and the full trajectory are all in the retained job tree — the fastest way to find out why a score isn’t what you expected. Agents & Sandboxes covers the debugging workflow in depth.

What reports show

Custom scores flow into the same results tree as everything else:

• In result.json, each agent carries per-arm custom metric averages (custom_with_skill, custom_without_skill) and a per-metric custom_lift — with-skill minus without-skill, per custom metric. The lift is also written to custom_lift.json next to the standard lift.json.
• In default_plus_custom mode, reports keep the standard metric and dimension view and add your scores as clearly labeled Custom: entries alongside it, each with its own lift; custom metrics never change the standard dimension scores.
• In custom_only mode, your overall reward drives pass@k and the overall lift. The standard dimension rows stay visible but render NO SCORE — unscored metrics are never coerced to 0.0, and your custom metrics are never re-labeled as standard dimensions. An arm that produces no scoreable reward (for example, every attempt fails) still records the custom-only metric set and custom-overall-v1 score policy in its summary and in the run’s attempt policy.

Custom lift needs both arms — a --skip-baseline run reports your custom scores but no lift. Reports & Results covers the full on-disk layout and the machine-readable contract.

Troubleshooting

Next steps