Agents & Sandboxes

Tier 3 runs real agent CLIs against your eval cases, so two choices shape every run: which agents attempt the tasks (--agents) and where the trials execute (--env-mode). This page is the complete matrix for both, plus the resource, safety, and workspace controls around them and the readiness checks that tell you a run will work before you spend compute. The run workflow itself is Tier 3: Live Evaluation.

Harbor is the open-source agent evaluation framework that executes each Tier 3 trial and provides its Docker, local, and cloud environment backends.

Supported agents

Pass agents as a comma-separated list with -a/--agents. Without that option, SkillEvaluator selects the provider-native default: opencode for NVIDIA Build, codex for OpenAI, and claude-code for Anthropic. Three agents are supported:

The pinned CLI releases apply wherever Harbor installs the agent: Docker and the remote environments. Local mode runs the CLI already on your machine instead.

In Docker and Kata, SkillEvaluator reruns a Claude Code or Codex install that fails, for example on a dropped download or a package mirror error. It makes up to three attempts, waiting 2 and then 4 seconds, all within Harbor’s agent setup timeout, so an install that keeps failing still fails the trial. A slow failure is not retried when the attempt took over a third of that timeout, or when another attempt as long would not finish within it, so the trial reports the install error rather than a setup timeout. Each retry logs a warning that ends with the failed command’s output. Claude Code on GKE, and Codex behind an OpenAI-compatible gateway, are retried the same way.

Claude Code runs with background tasks turned off. Every staged task sets CLAUDE_CODE_DISABLE_BACKGROUND_TASKS=1 in its environment, in every env mode, so a shell command finishes in the agent’s turn or is stopped at Claude Code’s Bash timeout (2 minutes unless the agent requests a longer one) instead of moving to the background. A headless Claude Code run waits for its background commands before it exits, so a command that never finishes would otherwise hold a trial that has already answered until the agent timeout. Neither harbor.runtime_env nor a native task’s [environment.env] can turn background tasks back on. If a skill needs a single command to run longer, raise Claude Code’s default in evals/config.yml with harbor.runtime_env: {BASH_DEFAULT_TIMEOUT_MS: "240000"} (milliseconds), and keep it below the trial’s agent timeout: 300 seconds times harbor.timeout_multiplier for generated tasks.

The claude alias is canonicalized to claude-code everywhere an agent name can appear — -a/--agents, --agent-model, and harbor.agents in evals/config.yml — so both spellings always name the same agent. Naming both claude and claude-code in --agent-model or in harbor.agents is rejected as a duplicate; in -a/--agents the repeated agent is simply deduplicated.

Each agent runs on a model, and the evaluator provider’s default model is not automatically valid for every agent. Override the model per agent with --agent-model AGENT=MODEL (repeatable), or globally with --model:

skillevaluator tier3 evaluate ./my-skill --agents codex --env-mode docker \
  --agent-model codex=nvidia/nemotron-3-super-120b-a12b

To make an override persistent, set it in evals/config.yml under harbor.agents.<name>.model — see Eval Datasets.

One NVIDIA Build key for everything

When the evaluator provider is NVIDIA Build (nv_build) and trials run in --env-mode docker, kata, or local, a single NVIDIA_API_KEY powers the evaluator and all three agents:

• OpenCode calls NVIDIA Build directly through its nvidia provider adapter.
• Codex runs against a SkillEvaluator-owned loopback compatibility bridge that translates the OpenAI Responses API into NVIDIA Build Chat Completions calls.
• Claude Code (experimental on this path) runs against an equivalent Messages-to-Chat-Completions bridge.

Both bridges support multi-turn tool-call continuation, so agents complete real tool loops against Build models. The Claude Code bridge drops orchestration and server tools Build cannot execute — WebFetch, WebSearch, Task, and similar — before forwarding a request.

Models on the one-key path

With nv_build and no explicit model override, every supported agent defaults to nvidia/nemotron-3-super-120b-a12b. The provider-native agent selection is OpenCode. Its model is automatically qualified with the nvidia/ adapter namespace, so it renders as nvidia/nvidia/nemotron-3-super-120b-a12b in run output — the doubled prefix is the adapter namespace plus the catalog publisher, not a bug.

An explicit --agent-model value with NVIDIA Build must be a full publisher/model catalog ID; an explicit OpenCode override also carries the adapter namespace (nvidia/publisher/model). Overrides are never rewritten silently, and an unavailable model fails the run rather than falling back.

Key protection

The bridged CLIs — Codex and Claude Code — never see the real Build key:

• Docker mode passes the key to Harbor’s trusted parent process over stdin, then uses a short-lived, container-only stdin handoff for each main-container exec. The handoff is removed before the agent command runs, and values never appear on Docker Compose argv.
• Local mode keeps the real key in Harbor’s trusted parent process; each bridged CLI receives a unique per-trial capability token that only the loopback bridge accepts.

OpenCode is the exception: it calls NVIDIA Build directly, so it necessarily receives the key itself.

NVIDIA Build local agents require network access. When SKILLEVALUATOR_LOCAL_ALLOW_NET is disabled, the run is rejected with NVIDIA Build local agents require network access; unset SKILLEVALUATOR_LOCAL_ALLOW_NET or set it to 1.

Other backends still need native credentials

The bridges exist only in docker, kata, and local mode. On any other Harbor backend, each agent calls its native provider:

Two provider-agent pairings are restricted regardless of NVIDIA Build: a Bedrock evaluator with claude-code requires an explicit AWS_ACCESS_KEY_ID + AWS_SECRET_ACCESS_KEY pair or AWS_BEARER_TOKEN_BEDROCK and does not support local mode at all, and an Anthropic evaluator with opencode also does not support local mode.

Where trials run

SkillEvaluator exposes 27 environment modes: 26 Harbor-native backends, including Docker, plus SkillEvaluator’s own local host-execution mode. Docker is the default. The tier3 extra installs base harbor==0.24.0; install the environment extra shown below when a managed backend needs one.

Installed packages can add more modes, for example an organization’s internal sandbox service. See Custom execution backends.

Harbor 0.24 merged its W&B sandbox into cwsandbox. SkillEvaluator keeps --env-mode wandb and runs it as Harbor’s cwsandbox backend with W&B authentication (auth=wandb), using WANDB_API_KEY or the netrc file that wandb login writes. Because SkillEvaluator sets auth itself, --ek auth=... is rejected for both cwsandbox and wandb.

Harbor 0.24 also defines cua-cloud, opensandbox, hf-sandbox, podman, prime, and smol, but those backends are not exposed until SkillEvaluator can project the complete task bundle — skill, verifier, dependencies, and inputs — through a trusted image or backend-native provisioning path, with credentials and constructor options reviewed per backend. Selecting one of those names is rejected before a run starts. Prime cannot build a Dockerfile-only task, which is how SkillEvaluator stages most tasks, and Smol runs only a published [environment].docker_image, which SkillEvaluator rejects because it bypasses the evaluator-staged image. Exposing either one needs Compose-only staging or a trusted image-publishing step, validated live. Podman also needs a runtime-aware hardened Compose runner: Harbor runs Podman without Compose’s up --wait readiness gate and, for some frontends, without compose cp, which SkillEvaluator’s Docker isolation relies on.

Native backend constructor options

Environment kwargs are operator-only and apply only to Harbor-native non-Docker backends. docker and local reject any environment kwargs, and kata accepts only its two runtime options (see Kata mode). For an eligible backend that needs non-secret constructor options in addition to host credentials, pass each option with the repeatable --environment-kwarg KEY=VALUE / --ek KEY=VALUE flag:

skillevaluator doctor --env-mode gke \
  --ek cluster_name=my-cluster \
  --ek region=us-central1 \
  --ek namespace=skill-evals \
  --ek registry_location=us-central1 \
  --ek registry_name=skill-evals

Values accept Harbor’s JSON-compatible types. Skill-owned evals/config.yml cannot set constructor kwargs because backend endpoints and sandbox controls belong to the operator trust boundary. Never pass secrets through the CLI; backend credentials belong in the operator’s host environment and are kept out of Harbor argv and stored run configuration.

SkillEvaluator also rejects fields reserved for Harbor runtime policy. Do not use environment kwargs for resource overrides (override_cpus, override_gpus, override_memory_mb, override_storage_mb, override_tpu), mount injection (mounts, mounts_json), network policy (network_policy, phase_network_policies, extra_allowed_hosts), extra Compose definitions (extra_docker_compose), lifecycle controls (delete, force_build, keep_containers), Harbor’s output streaming and environment upload controls (stream, enable_environment_dir_upload), the per-trial wiring Harbor supplies itself (such as environment_dir, session_id, and trial_paths), or Kubernetes pod/security overrides. Use SkillEvaluator’s first-class flags and config where available; advanced pod/security overrides are intentionally not exposed through this CLI surface. Equivalent backend-specific aliases are also reserved when they directly bypass those guarantees, including provider volume/resource/network overrides, Kubernetes or cloud workload identities, disabled SSH host verification, and Singularity host mount policy.

The backends with required constructor keys are:

EC2 accepts either Harbor’s explicit ssh_key_path option or the host’s authenticated SSH agent. In the latter case, keep SSH_AUTH_SOCK set when running SkillEvaluator; it is forwarded only to the EC2 backend process. A provided ssh_key_path must resolve to an existing regular file. Ephemeral EC2 with use_public_ip=false also requires a non-empty subnet_id.

Managed-backend readiness is not a connectivity test. EC2 preflight checks the Python extra, the local ssh executable, and the constructor invariants above; it does not contact AWS or prove credentials, AMI/instance access, networking, or quota. GKE preflight checks the required constructor options, local gcloud availability, and that a kubeconfig path exists; it does not contact the GKE cluster or prove authentication, permissions, registry access, or capacity. The first real trial remains the end-to-end infrastructure check.

For ACK, optional context and kubeconfig kwargs select the Kubernetes configuration. Without an available kubeconfig, readiness falls back to the in-cluster service account and performs a bounded, namespaced pod-list probe before launching trials. KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT are forwarded only for this ACK client path.

Harbor 0.24 Modal preflight requires ~/.modal.toml or MODAL_TOKEN_ID plus MODAL_TOKEN_SECRET. SkillEvaluator validates and forwards an explicitly set MODAL_CONFIG_PATH, but Harbor’s child-process preflight does not recognize a custom config path by itself; configure one of Harbor’s supported credential forms before launching trials.

Runta authenticates with RUNTA_TOKEN or the configuration file named by RUNTA_CONFIG (default ~/.config/runta/config.toml), and Harbor’s preflight requires one of the two variables. RUNTA_ENDPOINT selects a non-default API endpoint. A relative RUNTA_CONFIG path is resolved against the directory you run SkillEvaluator from, and a configured file must exist. Runta’s mode kwarg is reserved: mode=direct runs commands in Runta’s base runtime and ignores the task image, while Harbor’s automatic selection builds the task’s Dockerfile or Compose definition inside the runtime.

Mosaic authenticates with MOSAIC_API_TOKEN or the token mos login saves in ~/.config/mosaic-sandbox/config.json; MOSAIC_CONFIG names a different file, which must exist and is resolved against the directory you run SkillEvaluator from when relative. MOSAIC_API_URL and MOSAIC_RETRIES tune the API client, the SDK’s older MAR_API_TOKEN, MAR_ENDPOINT, and MAR_CONFIG names are forwarded too, and MOSAIC_REGISTRY_USERNAME and MOSAIC_REGISTRY_PASSWORD cover private base images. Mosaic’s volume, persist, and enable_ssh kwargs are reserved because they let state or access outlive one isolated trial, and build_args and build_target are reserved because they can skip or alter evaluator-staged Dockerfile layers. Mosaic’s secrets kwarg is rejected because it injects provider secrets into the agent’s sandbox, and a Mosaic token stored only in E2B_API_KEY is rejected because that variable is never forwarded to Mosaic. Mosaic runs one VM per trial, so tasks with an environment/docker-compose.yaml are rejected before the run starts.

Harbor builds each Mosaic task image into a snapshot named from the task and its build context, reuses an existing snapshot with that name in later runs, and never deletes snapshots. Run evaluations in a Mosaic account that only trusted operators can write to, and delete old harbor-* snapshots when you no longer need the staged skills and inputs they contain.

Other eligible, backend-specific constructor options are forwarded after validation. A key that the selected backend’s Harbor 0.24.0 constructor does not accept is rejected before the run starts; for example, Harbor 0.24 removed Modal’s modal_sandbox_v2 option, so drop it from existing --ek flags. ACK scheduling options such as node_selector and tolerations are accepted, and image_pull_secret accepts only the name of an existing Kubernetes Secret object—not credential contents. Credential-bearing options such as LangSmith’s api_key and Vercel’s credential_injection are rejected. Use the same --environment-kwarg values and eligible non-Docker mode with doctor or health-check to preflight the configuration you intend to evaluate.

A few Harbor options contain provider-managed secret references, not secret values. These narrow shapes are allowed; every other credential-shaped kwarg, embedded credential URI, or raw secret value is rejected:

Complex values use JSON syntax in the KEY=VALUE argument. Reference names must be trimmed and bounded; they cannot contain credential-bearing URIs.

Custom execution backends

A Python package can add an --env-mode that runs trials on infrastructure Harbor does not ship, without changing SkillEvaluator. The package registers an entry point in the skillevaluator.env_modes group. The entry-point name becomes the mode name, and its value names an EnvironmentBackendSpec (or a function that returns one):

# pyproject.toml of the backend package
[project.entry-points."skillevaluator.env_modes"]
acme = "skillevaluator_acme.backend:SPEC"

# skillevaluator_acme/backend.py
from skillevaluator.tier3_environments import EnvironmentBackendSpec
SPEC = EnvironmentBackendSpec(
    import_path="skillevaluator_acme.environment:AcmeEnvironment",
    environment_kwargs=frozenset({"profile", "lease_seconds"}),
    required_environment_kwargs=("profile",),
    host_env_vars=frozenset({"ACME_SANDBOX_URL", "ACME_SANDBOX_TOKEN"}),
    prerequisites="skillevaluator_acme.checks:prerequisites",
    install_hint="Install 'skillevaluator-acme' from your package index.",
)

import_path names a Harbor BaseEnvironment subclass. SkillEvaluator passes it to harbor run --env, so the class receives the same evaluator-staged task directory, verifier, and trial paths as a Harbor-native backend.

What a backend must provide

• Support the Harbor release SkillEvaluator pins (harbor==0.24.0), including its environment constructor options.
• Run the evaluator-staged image: build each task’s environment/Dockerfile as written, or an equivalent image with the same layers. Do not substitute a generic image.
• Preserve the task workspace: /workspace with the staged skills under the task’s skills_dir, task input, and the agent working directory.
• Preserve the verifier contract: run tests/ with the verifier environment SkillEvaluator stages, and return /logs/verifier and /logs/agent (including the trajectory) to the host.
• Clean up every trial resource, including after cancellation.

The acceptance test for a backend is a full SkillEvaluator Tier 3 run of a skill with both the baseline and with-skill conditions on that backend. The run must produce the expected scores, retain trajectories and verifier artifacts, and leave no trial resources behind. Compare the results with the same skill on --env-mode docker.

Policy SkillEvaluator applies

A custom execution backend follows the same policy as the Harbor-native backends:

• --ek accepts only the names in environment_kwargs. Names in reserved_environment_kwargs, SkillEvaluator’s runtime-policy kwargs such as network_policy and session_id, and secret-bearing names or values are rejected.
• Only the upper-case variables in host_env_vars reach the Harbor process, in addition to the usual proxy, CA bundle, and locale variables. A skill’s harbor.runtime_env cannot set them. A backend cannot forward agent or provider credentials (for example ANTHROPIC_*, CLAUDE_*, CODEX_*, OPENAI_*, OPENCODE_*, NVIDIA_*, AWS_*, GITHUB_*, GH_*, HF_*), another backend’s credentials, judge settings, or process-control variables, because Harbor’s stock agents read some of them and can copy them into the trial sandbox. KUBECONFIG is allowed. The runner supplies agent credentials per agent as it does for every backend.
• Harbor runs in a private working directory, so list host variables and --ek kwargs that hold file paths in host_path_env_vars and host_path_environment_kwargs; relative values are made absolute first.
• skillevaluator doctor --env-mode <mode> checks required_environment_kwargs, then runs the optional prerequisites function (fn(environment_kwargs) -> list[str]), then the environment class’s Harbor preflight(). Diagnostics are redacted like built-in ones.
• Tasks with a docker-compose.yaml are rejected unless the backend sets supports_docker_compose=True.
• agent_import_paths may map an agent name to a Harbor agent class, for example one that uses a CLI already installed in the backend’s image. Agents without an entry use Harbor’s stock agent. When the evaluator provider needs its own wrapper (for example SkillEvaluatorGatewayCodex for Codex with an OpenAI-compatible gateway), the backend’s class must subclass that wrapper; otherwise the run stops before any trial starts.

A custom execution backend cannot replace a built-in mode name, and a name registered by more than one installed package is refused with every package name until all but one is uninstalled. Listing the available modes reads package metadata only; the backend module is imported when its mode is selected. Published benchmark cards show a custom execution backend as Isolated sandbox and do not name it. Set SKILLEVALUATOR_DISABLE_CUSTOM_BACKENDS=1 to ignore every installed custom execution backend. Backend packages run with your privileges, so install only ones you trust.

Docker mode

The default path needs a running Docker daemon and Docker Compose v2 — skillevaluator doctor --env-mode docker checks both. Each trial runs in its own container, and both arms of the comparison are staged from the same environment definition — the baseline image differs only by not containing the skill — so the skill stays the only variable.

Windows hosts

On native Windows, use Docker Desktop with Linux containers. Keep the Windows system variables available in the shell that runs SkillEvaluator: Harbor inherits PROGRAMFILES, PROGRAMW6432, and PROGRAMDATA so Docker Desktop can discover its Compose plugin.

For a non-UTF-8 Windows locale, enable Python UTF-8 mode in PowerShell before running readiness checks or evaluations:

$env:PYTHONUTF8 = "1"
$env:PYTHONIOENCODING = "utf-8"
skillevaluator doctor --env-mode docker

SkillEvaluator forwards both Python settings to Harbor when they are set; it does not enable them automatically. Native Windows local mode remains unsupported; use Docker or run local mode inside WSL2.

Verifier scripts run inside the Linux container, so they need LF line endings. SkillEvaluator writes generated test.sh files and the init-custom-grader and init-harbor-task grader starters with LF on every host. Keep LF when you edit them: a CRLF grader.sh or test.sh fails in the container before grading starts.

Resource tuning

If an agent’s work is heavier than the defaults allow — large builds, memory-hungry test suites — override the per-container allocation. The flags pass straight through to Harbor:

skillevaluator tier3 evaluate ./my-skill --agents codex --env-mode docker \
  --override-cpus 4 --override-memory-mb 8192 --override-storage-mb 16384

To make the sizing part of the skill’s run policy instead of a flag, set harbor.resources (cpus, memory_mb, storage_mb) in evals/config.yml — see Eval Datasets. A trial that dies mid-run with the agent process killed is the classic out-of-memory symptom; raise --override-memory-mb first.

Custom Dockerfiles

When a skill ships its own evals/environment/Dockerfile, --custom-dockerfile-mode decides how it combines with the eval image:

Environment rules — size limits, auto-generated fallbacks, and sidecar containers — are part of the evals/ contract in Eval Datasets.

Docker Compose project .env files are rejected before startup. Protected main-container and sidecar operations disable Compose’s implicit env-file loading so the same definition cannot start with one interpolation value and later run with another. Put non-secret interpolation values in harbor.runtime_env in evals/config.yml; keep operator credentials in the host environment as described above.

GKE mode

When running with --env-mode gke, SkillEvaluator enforces least-privilege pod identity by default unless you explicitly opt in for trusted skills via SKILLEVALUATOR_GKE_ALLOW_WORKLOAD_IDENTITY=1 or --ek allow_workload_identity=1:

• ServiceAccount and metadata isolation: Pods run with automountServiceAccountToken: false, hostNetwork: false, GCE_METADATA_HOST=127.0.0.1:1, and a pod-scoped egress NetworkPolicy (skillevaluator-block-gce-metadata) that blocks HTTP egress to 169.254.169.254/32 and 169.254.169.252/32 while preserving UDP/TCP port 53 DNS. Kubernetes ServiceAccounts annotated with iam.gke.io/gcp-service-account or iam.gke.io/return-principal-id-as-email=true are rejected.
• Trusted init and companion metadata probes: Single-container evaluation pods include a trusted init container (skillevaluator-metadata-probe-init) that verifies metadata endpoints are unreachable before the main task container starts, plus a companion container (skillevaluator-metadata-probe) that re-verifies isolation before task upload. Set SKILLEVALUATOR_GKE_METADATA_PROBE_IMAGE when your cluster requires pulling the probe image (python:3.12-slim by default) from a private Artifact Registry mirror.
• GKE Autopilot ephemeral storage: On GKE Autopilot (auto-detected via node inspection or gcloud, or set explicitly with SKILLEVALUATOR_GKE_AUTOPILOT / --ek autopilot=<bool>), general-purpose pods have a 10Gi (10240Mi) cumulative ephemeral-storage ceiling. When a non-GPU/TPU main container requests 10Gi, SkillEvaluator caps main to 10176Mi so main (10176Mi) + skillevaluator-metadata-probe (64Mi) equals 10240Mi. Standard GKE clusters, GPU/TPU pods, and explicit storage requests above 10Gi are not lowered.

Kata mode

--env-mode kata uses the same hardened Docker Compose backend as Docker mode, including its credential handoff and container containment, but Harbor adds a final Compose overlay that runs every task service under a Kata Containers runtime, so agent execution and verification happen inside KVM microVMs instead of on the host kernel. It needs a Linux host with /dev/kvm, Docker Compose v2, and a Kata runtime handler registered with Docker (for example with Kata’s kata-manager.sh). skillevaluator doctor --env-mode kata checks all three, including that Docker registers the selected handler.

Two environment kwargs are accepted:

Harbor’s Kata backend has limits that SkillEvaluator keeps:

• Image builds are not VM-isolated. Dockerfile RUN steps execute on the host’s BuildKit builder; the microVM boundary covers the agent and verifier.
• Non-public network policies are rejected before start, because Kata VMs cannot join the egress-control sidecar’s network namespace. Generated tasks use public networking.
• Compose service-name DNS does not work inside the microVM, so tasks whose Compose model declares services other than main, such as a database sidecar, are rejected before the run starts.

Local mode

--env-mode local skips containers entirely: the agent CLI already installed on your machine runs the trials, wrapped in an OS-level sandbox. On Linux that is Bubblewrap — kernel namespace isolation, a read-only view of the system, writes confined to the run directories, and a network namespace that is isolated when the run is airgapped and shared for model egress when network is allowed. SkillEvaluator resolves bwrap on PATH to an absolute path once, smoke-tests that binary, and launches the same binary for every command. On macOS it is Seatbelt, which confines reads and writes and — even when network is allowed — denies inbound connections, socket binding, and Unix sockets (except the mDNSResponder DNS socket); it cannot provide full process isolation, so it is the weaker of the two. Native Windows is hard-unsupported and fails closed before agent-runtime discovery — SKILLEVALUATOR_LOCAL_SANDBOX=prefer or off is not a bypass; the error directs you to WSL2 or --env-mode docker. Only claude-code, codex, and opencode can run in local mode.

Local mode does not install the pinned CLI releases. Each trial checks the installed version against the selected model instead and fails before the agent starts when it is too old: Claude 5.5 and later models need Claude Code 2.1.280 or newer, and GPT-6 models need Codex 0.159.1 or newer.

The sandbox policy is controlled by environment variables:

Full semantics for each variable are in Environment Variables. doctor --env-mode local verifies the sandbox backend and the installed agent CLIs before you commit to a run.

When Bubblewrap is installed but fails its startup probe, the error includes Bubblewrap’s own failure line. Inside a container, pivot_root: Operation not permitted can mean the container’s seccomp profile blocks Bubblewrap; only the options the container was launched with can change that. On such a host, use --env-mode docker where a Docker daemon is available, or set SKILLEVALUATOR_LOCAL_SANDBOX=prefer (advisory-only guardrails) or off for skills you fully trust.

Workspace modes

SkillEvaluator stages the agent’s workspace itself, in both arms, so the skill under test never leaks into the without-skill baseline — that isolation is what keeps Skill Lift trustworthy. Three flags control what else the agent can see:

Use group mode when the skill delegates to companion skills and evaluating it alone would be unfair; keep the default otherwise — a smaller workspace is a cleaner experiment.

Select a reproducible decoy set

tier3 select-decoys ranks real skills from a local description-embedding catalog and saves a selection manifest. Selection accepts skill-only catalogs with at most 1,000 entries. Build the catalog without --full-body and use the same embedding provider and model for selection:

skillevaluator similarity-check ./skills --save-catalog ./skill-catalog.json
skillevaluator tier3 select-decoys ./skills/my-skill \
  --catalog ./skill-catalog.json --catalog-root ./skills \
  --count 4 --exclude required-helper \
  --output ./skills/my-skill/evals/decoys.json

Opt in through the target skill’s run policy. Manifest paths, like included skill paths, are relative to the target skill’s parent directory:

schema_version: 1
skill_workspace:
  mode: group
  decoy_manifest: my-skill/evals/decoys.json
  include:
    - required-helper

Selection excludes the target, skills bundled inside it, and duplicate candidates, orders equal scores by catalog-relative path, and fails if too few eligible skills remain. Required helpers stay explicit in include; the selector does not infer prerequisites. A collision between a selected skill and an explicitly included skill is an error.

The manifest records catalog and model identity, scores, the target manifest hash, and selected skill runtime hashes. Evaluation verifies the hashes and uses one snapshot of the selected skills in both arms; it makes no embedding request and does not select again. The runtime hashes exclude evaluator-only directories, including each nested skill’s evals/, exactly as staging does. Editing those configs, datasets, or selection manifests does not invalidate other skills’ selections. Agent-visible references still participate in the hash, including an ordinary references/evals/ folder.

For a collection that contains the target, or lies inside the target’s parent directory, catalog_root is stored relative to that parent (. in the example above). Identical clones and relocated collections can reuse the same manifest; evaluation checks the sources in the current checkout. An unrelated external collection retains an absolute root and must be available at that location. The runtime hashes cover everything staging copies from a selected skill, including files that git ignores or does not track (such as .DS_Store or .pytest_cache/) and empty folders, so generate a manifest you plan to commit from a fresh clone. Of the permission bits, only whether a file is executable by its owner counts, so clones made under different umasks agree. Windows checkouts can differ: Git may convert line endings there, and Python reports .exe, .bat, .cmd and .com files as executable and other files as not executable. Generate a separate selection on Windows. Regenerate after editing the target’s SKILL.md, changing selected runtime content, or moving an external collection. Manifests using an earlier digest algorithm must also be regenerated. External workspace skills retain the existing publication-evidence limitations.

Use a separate manifest filename such as decoys.json. Selection output cannot replace an evaluation dataset, run policy, embedding catalog, or existing non-manifest file.

Similarity is a heuristic for plausible alternatives, not evidence of actual routing confusion. Skill Lift remains conditional on the chosen workspace; a neighboring skill may legitimately substitute for the target. Review the selection before running the evaluation, and keep the manifest with the run policy when comparing results.

Readiness checks

doctor and health-check run the same readiness checks and exit nonzero when any check fails, so either can gate a script or CI job. Warnings from doctor --verify-models are nonblocking because the catalog result is inconclusive rather than a definitive rejection. Both commands check Tier 3 by default. --tier 1 checks the Tier 1 scanners, --tier 2 the embedding and LLM providers, and --tier all every tier; see the CLI Reference.

doctor

skillevaluator doctor --agents codex --env-mode docker

doctor prints a check-by-check table and names exactly what is missing:

doctor also accepts --agent-model AGENT=MODEL (repeatable), so you can check the exact model plan a run would use. Add --verify-models to probe each agent’s resolved model live against the provider’s catalog — one extra pass, warn, or fail row per agent. A warning means the catalog result is non-authoritative, such as a public NVIDIA Build listing or an OpenAI catalog permission failure that does not decide inference access. Browse catalog membership with skillevaluator models (top 10 by default; --limit and --json adjust the result). To prove that the agent can actually invoke the model before the measured matrix, opt into the extra Tier 3 execution with --agent-runtime-preflight. A different route used only by the standard grader is not exercised by that smoke and remains unproven until grading begins.

health-check

skillevaluator health-check

health-check runs the same checks and prints the same table as a plain doctor run — the only difference is that it lacks two of doctor’s flags: --verify-models and --agent-model. Use it as an install smoke test or a CI preflight.

Debug a run

Staged-task and raw Harbor job directories are transient: they are deleted by default when a run finishes, on success and on failure alike. Retain them with --harbor-keep-jobs, then open them with Harbor’s own trajectory browser — the final Artifacts panel prints the exact harbor-view command whenever a retained jobs directory exists:

skillevaluator tier3 evaluate ./my-skill --agents codex --harbor-keep-jobs
skillevaluator harbor-view <jobs-dir>

The retained tree holds the full agent trajectory and grader output for every trial — the fastest way to see what an agent actually did. The results layout and where <jobs-dir> lands on disk are in Reports & Results; debugging custom graders this way is covered in Custom Graders & Tasks.

Next steps