Providers & Credentials

For NVIDIA Build, getting started requires just two variables: the provider and its API key. Default chat and embedding models are already configured:

export SKILL_EVAL_LLM_PROVIDER=nv_build
export NVIDIA_API_KEY='nvapi-...'

The same key covers Tier 1 LLM checks, Tier 2 embeddings and chat, and supported Tier 3 agents in Docker, Kata, or local mode. OpenAI likewise starts with SKILL_EVAL_LLM_PROVIDER=openai and OPENAI_API_KEY, using Codex for Tier 3. External scanners, agent runtimes, and sandboxes remain prerequisites; model credentials do not install them.

skillevaluator tier1 ./my-skill
skillevaluator tier2 ./my-skill
skillevaluator tier3 ./my-skill

No model override is needed. Defaults are pinned in the installed release, not automatically selected as the latest available model. Set SKILL_EVAL_LLM_MODEL only when you want a different evaluator model; the standard judge inherits it unless a judge override is set. Embeddings use their own model, and Tier 3 agent selection is separate. With NVIDIA Build, OpenCode follows SKILL_EVAL_LLM_MODEL, while Codex and Claude Code keep the Super default unless you override their models.

Anthropic and Bedrock need a separate embedding provider for Tier 2; custom OpenAI-compatible endpoints require an explicit URL and key, with optional model overrides. Those advanced setups are covered below.

For the exhaustive variable-by-variable inventory, see Environment Variables.

Tier 3 executes trials through Harbor, the open-source agent evaluation framework. This page separates the credentials SkillEvaluator and the live agents need from the backend configuration Harbor itself reads.

What needs a key

Tiers describe evaluation depth; model credentials are a separate concern. Keyless checks do not need an LLM provider, although dependency audits can contact package and vulnerability services.

validate PATH runs all three tiers for skills and needs the provider and Tier 3 runtime setup. Use --tiers 1 for keyless static checks. --no-dedup alone skips Tier 2 but still runs Tier 3.

Before Tier 3 prepares images or stages Harbor tasks, tier3 evaluate also makes a bounded credential-bearing catalog request for each distinct agent route and for the standard-grading route when that grader’s effective model is known. A definitive native-provider credential rejection or deterministic Bedrock credential/configuration error fails immediately. Catalog permission and membership are not authoritative for OpenAI Responses access, and NVIDIA Build or a compatible gateway may expose a catalog without authenticating inference, so those checks are reported as degraded and evaluation continues. When enabled with --agent-runtime-preflight or harbor.agent_runtime_preflight: true, the extra agent-only execution proves agent routes only: when a degraded credential route is used solely by the standard grader, inference on that route remains unproven until grading begins. The same applies when native Harbor must resolve at runtime whether a task or step verifier declaration supersedes the configured judge fallback. Redacted per-route outcomes are retained under run_config.credential_validation in result.json, including when an enabled agent runtime preflight fails. The extra execution is disabled by default.

Choose an LLM provider

Set SKILL_EVAL_LLM_PROVIDER to one of openai, anthropic, nv_build, bedrock, or openai-compatible. The selected provider determines which credential SkillEvaluator reads. Each ships a default evaluator model that SKILL_EVAL_LLM_MODEL overrides.

export SKILL_EVAL_LLM_PROVIDER=openai
export OPENAI_API_KEY='sk-...'

Auto-selection

When SKILL_EVAL_LLM_PROVIDER is unset and exactly one of these credentials is present, SkillEvaluator selects the matching provider:

• NVIDIA_API_KEY → nv_build
• OPENAI_API_KEY → openai
• ANTHROPIC_API_KEY → anthropic

If none of those variables is set, LLM-backed commands fail with a configuration error. If more than one is exported, resolution also fails — SKILL_EVAL_LLM_PROVIDER is required when multiple public provider credentials are configured; there is no silent preference order.

Embeddings

Tier 2 uses an OpenAI-compatible embeddings API. SKILL_EVAL_EMBEDDING_PROVIDER selects it; when unset, it falls back to your SKILL_EVAL_LLM_PROVIDER (or the same auto-selection order as above).

Use SKILL_EVAL_EMBEDDING_MODEL and SKILL_EVAL_EMBEDDING_BASE_URL to override defaults; for openai-compatible, the base URL falls back to SKILL_EVAL_LLM_BASE_URL. The nv_build endpoint is fixed to https://integrate.api.nvidia.com/v1 — only its model can be overridden.

NVIDIA Build uses the same NVIDIA_API_KEY for chat and embeddings; no extra key is needed. Deduplication sends document inputs with input_type=passage, as required by the NVIDIA embedding API.

The previous NVIDIA default, nvidia/nv-embed-v1, is retired. Remove any explicit override to that model to use the current default. Rebuild comparison catalogs and embedding caches when changing models: vectors from different models cannot be compared, and similarity scores may change. A provider failure is reported as INCOMPLETE, not a duplicate-content finding.

export SKILL_EVAL_EMBEDDING_PROVIDER=openai
export OPENAI_API_KEY='sk-...'

Fully local setup

The openai-compatible provider accepts any local OpenAI-compatible server (Ollama, vLLM, llama.cpp, NVIDIA NIM), so Tier 2 can run with no external calls at all. Example recipe using Ollama in Docker:

docker run -d --name ollama -p 11434:11434 ollama/ollama
docker exec ollama ollama pull nomic-embed-text   # embeddings
docker exec ollama ollama pull qwen2.5:0.5b       # chat LLM for dedup analysis

export SKILL_EVAL_EMBEDDING_PROVIDER=openai-compatible
export SKILL_EVAL_EMBEDDING_BASE_URL=http://localhost:11434/v1
export SKILL_EVAL_EMBEDDING_MODEL=nomic-embed-text
export SKILL_EVAL_EMBEDDING_API_KEY=local-no-key  # must be set; local servers ignore the value
# context-optimization-check and dedup-scan also use a chat LLM:
export SKILL_EVAL_LLM_PROVIDER=openai-compatible
export SKILL_EVAL_LLM_BASE_URL=http://localhost:11434/v1
export SKILL_EVAL_LLM_MODEL=qwen2.5:0.5b
export SKILL_EVAL_LLM_API_KEY=local-no-key

skillevaluator similarity-check ./skills
skillevaluator context-optimization-check ./my-skill

similarity-check compares skills directly or through a local catalog and needs only the embedding variables. context-optimization-check (also available as its dedup-scan alias) additionally uses the chat model. Analysis quality tracks that model — the tiny model above proves the plumbing; pick a stronger local model for verdicts you intend to act on.

Tier 3 credentials: two roles

Live evaluation has two credential roles. They can share a key when the selected provider supports both:

Both roles resolve from the operator’s environment; a skill’s evals/config.yml can never supply, rename, or redirect a credential. Export them in the shell or CI job that runs SkillEvaluator: Harbor starts in an empty, evaluator-owned working directory with .env loading disabled, so a .env or .env.local file near your project is not read. What each agent needs depends on the provider and --env-mode:

• NVIDIA Build, one key (--env-mode docker, kata, or local): a single NVIDIA_API_KEY powers the evaluator and all three supported agents — opencode calls NVIDIA Build directly, while claude-code and codex route through SkillEvaluator’s compatibility bridge (in-container in Docker and Kata modes; an authenticated in-process host bridge in local mode). Agent models must be full NVIDIA Build catalog IDs in publisher/model form.
• NVIDIA Build on Harbor cloud modes (daytona, e2b, …): the bridge is not available. claude-code needs its own native ANTHROPIC_API_KEY, codex needs a full OpenAI Responses API credential (OPENAI_API_KEY plus OPENAI_BASE_URL), and each needs an explicit model via --agent-model or harbor.agents.<agent>.model.
• Matching native provider: when the evaluator provider is the agent’s own (for example anthropic with claude-code, or openai with codex), the same key serves both roles automatically.
• OpenAI-compatible gateway: Codex, OpenCode, and Claude Code inherit the shared URL and key. The gateway must support the selected harness’s API and model. Explicit Claude credentials and endpoint settings take precedence as described above.

When --agents is omitted, SkillEvaluator chooses that native route: NVIDIA Build uses OpenCode, OpenAI uses Codex, and Anthropic uses Claude Code. Passing --agents always replaces the automatic selection.

Judge failures are not zero scores

Standard Tier 3 grading treats a required judge transport, authentication, timeout, HTTP, or exhausted response-parsing error as an evaluation infrastructure failure. A condition containing a failed or incomplete default trial publishes none of its scores, dimensions, pass@k, or quality findings, and no cross-condition lift is published. Redacted diagnostics and coverage information remain available. A genuine model score of 0.0 is still a valid, published score.

In default_plus_custom grading, a custom score cannot rescue failed standard judging. custom_only grading is unchanged: it does not invoke the standard judge and continues to publish a valid custom result.

The agent’s model lives in the skill’s evals/config.yml:

schema_version: 1
harbor:
  agents:
    codex:
      model: gpt-6.1-sol

You can also supply that model at invocation time with --agent-model codex=MODEL. See Tier 3: Live Evaluation for the full run workflow.

In Docker mode, credential values are never rendered on the docker compose exec command line — the bridge receives them through a mode-0600 file inside the container that is consumed and removed before the agent starts.

List the models your key can access

Before pinning --agent-model or SKILL_EVAL_LLM_MODEL, check what the selected provider actually serves:

skillevaluator models --limit 20
skillevaluator models --json

skillevaluator models prints a filtered, bounded view of the provider catalog (--limit accepts 1–100, default 10; --json emits machine-readable output). It uses the same provider and credential resolution as evaluation, so it checks catalog routing and model discovery where that surface is supported. NVIDIA Build and some compatible gateways can expose a public catalog, so a listed model is not proof that the inference key works or that a given agent harness supports it. Tier 3’s optional real per-agent runtime preflight is the stronger check for agent routes; enable it with --agent-runtime-preflight when that extra billable execution is appropriate. A separate degraded standard-grader route remains unproven until grading begins. Bedrock exposes no HTTP model catalog, so models errors on that provider; use doctor --verify-models there instead.

runtime_env is for task values, not credentials

runtime_env in evals/config.yml remains valid for non-credential task values — plain names the task environment needs. Credential names are rejected outright:

• Any operator-owned credential name (OPENAI_API_KEY, OPENAI_BASE_URL, ANTHROPIC_API_KEY, ANTHROPIC_BASE_URL, NVIDIA_API_KEY, CLAUDE_CODE_USE_BEDROCK) fails the run with harbor.runtime_env contains operator-owned credential name(s).
• A ${NVIDIA_API_KEY}-style reference to an operator-owned credential inside any value is rejected too, so a credential cannot be aliased under a different name.

Blocked runtime_env names

Names that control the launcher, credentials, or dynamic runtime are rejected:

Configure the selected Harbor backend in the host environment instead. Non-secret constructor kwargs are only for Harbor-native non-Docker backends. They are operator-only: use repeatable --environment-kwarg KEY=VALUE / --ek KEY=VALUE flags on tier3 evaluate, doctor, and health-check. Skill-owned evals/config.yml cannot set them. docker and local reject any environment kwargs, and kata accepts only kata_runtime and kata_dns. Harbor runtime-policy fields for resource overrides, mounts, networks, extra Compose definitions, SkillEvaluator’s own lifecycle fields (delete, force_build, keep_containers), Harbor’s stream and enable_environment_dir_upload controls, and Kubernetes pod security are reserved. Other backend-specific timeout and retention knobs remain operator-owned. Never pass secrets there; credentials remain host environment variables.

SkillSpector bridge

The --llm security analysis runs SkillSpector, which has its own provider environment (SKILLSPECTOR_PROVIDER plus per-provider credential variables). SkillEvaluator bridges your configured provider automatically for that invocation, so the one key above is all you need. For nv_build, the bridge selects SkillSpector’s nv_build provider, so its Build model registry sets output-token limits, and sends the key through SkillSpector’s OpenAI-compatible provider path (OPENAI_API_KEY, with OPENAI_BASE_URL pointed at the public Build endpoint). The same NVIDIA_API_KEY covers this too; it does not create a second NVIDIA credential name.

The subprocess environment is isolated. Only the selected provider settings and basic process environment variables (PATH, HOME, TLS bundles, temp dirs) pass through — other ambient credentials in your shell are never forwarded. Your SkillSpector model settings (SKILLSPECTOR_MODEL_<SLOT>, SKILLSPECTOR_MODEL_REGISTRY, SKILLSPECTOR_REASONING_EFFORT) pass through too; SKILLSPECTOR_TEMPERATURE does not.

SkillSpector sends its own structured-output requests, so it has its own default model per provider instead of following the chat default. On Anthropic and Bedrock it stays on Claude Opus 5 while the chat default is Claude Opus 5.5. An explicit SKILL_EVAL_LLM_MODEL still reaches SkillSpector, and SKILLSPECTOR_MODEL overrides both. SkillSpector v2.13.3 does not return structured output for Claude 5.5 models: if SKILL_EVAL_LLM_MODEL names one, also set SKILLSPECTOR_MODEL to a supported model such as claude-opus-5 (Bedrock: us.anthropic.claude-opus-5), or the --llm security stage is reported incomplete. See SkillSpector bridge for the defaults.

Setting SKILLSPECTOR_PROVIDER yourself overrides the bridge, but it must name a supported public provider (openai, anthropic, or bedrock — or nv_build to force the Build mapping); anything else fails closed and the subprocess receives no credential.

Troubleshooting

Next steps