Providers & Credentials

This page gets SkillEvaluator’s LLM-backed features working: choosing a provider, wiring embeddings for Tier 2, and separating evaluator credentials from agent credentials in Tier 3. Most of Tier 1 needs no credential at all — start with the map below to see whether you need a key in the first place.

For the exhaustive variable-by-variable inventory, see Environment Variables.

Tier 3 executes trials through Harbor, the open-source agent evaluation framework. This page separates the credentials SkillEvaluator and the live agents need from the backend configuration Harbor itself reads.

What needs a key

Tiers describe evaluation depth; credentials are a separate axis. This is the complete map — everything not listed under “needs a provider” runs fully offline:

validate without --no-dedup stays usable keyless: the Tier 2 dedup pass skips gracefully when no embedding provider is configured.

Before Tier 3 prepares images or stages Harbor tasks, tier3 evaluate also makes a bounded credential-bearing catalog request for each distinct agent route and for the standard-grading route when that grader’s effective model is known. A definitive native-provider credential rejection or deterministic Bedrock credential/configuration error fails immediately. Catalog permission and membership are not authoritative for OpenAI Responses access, and NVIDIA Build or a compatible gateway may expose a catalog without authenticating inference, so those checks are reported as degraded and evaluation continues to the real per-agent runtime preflight. That smoke proves agent routes only: when a degraded credential route is used solely by the standard grader, inference on that route remains unproven until grading begins. The same applies when native Harbor must resolve at runtime whether a task or step verifier declaration supersedes the configured judge fallback. Redacted per-route outcomes are retained under run_config.credential_validation in result.json, including when that later agent runtime preflight fails.

Choose an LLM provider

Set SKILL_EVAL_LLM_PROVIDER to one of openai, anthropic, nv_build, bedrock, or openai-compatible. The selected provider determines which credential SkillEvaluator reads. All but openai-compatible ship a default model that SKILL_EVAL_LLM_MODEL overrides.

export SKILL_EVAL_LLM_PROVIDER=openai
export OPENAI_API_KEY='sk-...'

Auto-selection

When SKILL_EVAL_LLM_PROVIDER is unset and exactly one of these credentials is present, SkillEvaluator selects the matching provider:

• NVIDIA_API_KEY → nv_build
• OPENAI_API_KEY → openai
• ANTHROPIC_API_KEY → anthropic

If none of those variables is set, LLM-backed commands fail with a configuration error. If more than one is exported, resolution also fails — SKILL_EVAL_LLM_PROVIDER is required when multiple public provider credentials are configured; there is no silent preference order.

Embeddings

Tier 2 uses an OpenAI-compatible embeddings API. SKILL_EVAL_EMBEDDING_PROVIDER selects it; when unset, it falls back to your SKILL_EVAL_LLM_PROVIDER (or the same auto-selection order as above).

Use SKILL_EVAL_EMBEDDING_MODEL and SKILL_EVAL_EMBEDDING_BASE_URL to override defaults; for openai-compatible, the base URL falls back to SKILL_EVAL_LLM_BASE_URL. The nv_build endpoint is fixed to https://integrate.api.nvidia.com/v1 — only its model can be overridden.

export SKILL_EVAL_EMBEDDING_PROVIDER=openai
export OPENAI_API_KEY='sk-...'

Fully local setup

The openai-compatible provider accepts any local OpenAI-compatible server (Ollama, vLLM, llama.cpp, NVIDIA NIM), so Tier 2 can run with no external calls at all. Example recipe using Ollama in Docker:

docker run -d --name ollama -p 11434:11434 ollama/ollama
docker exec ollama ollama pull nomic-embed-text   # embeddings
docker exec ollama ollama pull qwen2.5:0.5b       # chat LLM for dedup analysis

export SKILL_EVAL_EMBEDDING_PROVIDER=openai-compatible
export SKILL_EVAL_EMBEDDING_BASE_URL=http://localhost:11434/v1
export SKILL_EVAL_EMBEDDING_MODEL=nomic-embed-text
export SKILL_EVAL_EMBEDDING_API_KEY=local-no-key  # must be set; local servers ignore the value
# context-optimization-check and dedup-scan also use a chat LLM:
export SKILL_EVAL_LLM_PROVIDER=openai-compatible
export SKILL_EVAL_LLM_BASE_URL=http://localhost:11434/v1
export SKILL_EVAL_LLM_MODEL=qwen2.5:0.5b
export SKILL_EVAL_LLM_API_KEY=local-no-key

skillevaluator similarity-check ./skills
skillevaluator context-optimization-check ./my-skill

similarity-check compares skills directly or through a local catalog and needs only the embedding variables. context-optimization-check (also available as its dedup-scan alias) additionally uses the chat model. Analysis quality tracks that model — the tiny model above proves the plumbing; pick a stronger local model for verdicts you intend to act on.

Tier 3 credentials: two roles

Live evaluation involves two distinct credentials, and they are not interchangeable:

Both roles resolve from the operator’s environment; a skill’s evals/config.yml can never supply, rename, or redirect a credential. What each agent needs depends on the provider and --env-mode:

• NVIDIA Build, one key (--env-mode docker or local): a single NVIDIA_API_KEY powers the evaluator and all three supported agents — opencode calls NVIDIA Build directly, while claude-code and codex route through SkillEvaluator’s compatibility bridge (in-container in Docker mode; an authenticated in-process host bridge in local mode). Agent models must be full NVIDIA Build catalog IDs in publisher/model form.
• NVIDIA Build on Harbor cloud modes (daytona, e2b, …): the bridge is not available. claude-code needs its own native ANTHROPIC_API_KEY, codex needs a full OpenAI Responses API credential (OPENAI_API_KEY plus OPENAI_BASE_URL), and each needs an explicit model via --agent-model or harbor.agents.<agent>.model.
• Matching native provider: when the evaluator provider is the agent’s own (for example anthropic with claude-code, or openai with codex), the same key serves both roles automatically.

Judge failures are not zero scores

Standard Tier 3 grading treats a required judge transport, authentication, timeout, HTTP, or exhausted response-parsing error as an evaluation infrastructure failure. A condition containing a failed or incomplete default trial publishes none of its scores, dimensions, pass@k, or quality findings, and no cross-condition lift is published. Redacted diagnostics and coverage information remain available. A genuine model score of 0.0 is still a valid, published score.

In default_plus_custom grading, a custom score cannot rescue failed standard judging. custom_only grading is unchanged: it does not invoke the standard judge and continues to publish a valid custom result.

The agent’s model lives in the skill’s evals/config.yml:

schema_version: 1
harbor:
  agents:
    codex:
      model: gpt-5.6-sol

You can also supply that model at invocation time with --agent-model codex=MODEL. See Tier 3: Live Evaluation for the full run workflow.

In Docker mode, credential values are never rendered on the docker compose exec command line — the bridge receives them through a mode-0600 file inside the container that is consumed and removed before the agent starts.

List the models your key can access

Before pinning --agent-model or SKILL_EVAL_LLM_MODEL, check what the selected provider actually serves:

skillevaluator models --limit 20
skillevaluator models --json

skillevaluator models prints a filtered, bounded view of the provider catalog (--limit accepts 1–100, default 10; --json emits machine-readable output). It uses the same provider and credential resolution as evaluation, so it checks catalog routing and model discovery where that surface is supported. NVIDIA Build and some compatible gateways can expose a public catalog, so a listed model is not proof that the inference key works or that a given agent harness supports it. Tier 3’s real per-agent runtime preflight is the stronger check for agent routes; a separate degraded standard-grader route remains unproven until grading begins. Bedrock exposes no HTTP model catalog, so models errors on that provider; use doctor --verify-models there instead.

runtime_env is for task values, not credentials

runtime_env in evals/config.yml remains valid for non-credential task values — plain names the task environment needs. Credential names are rejected outright:

• Any operator-owned credential name (OPENAI_API_KEY, OPENAI_BASE_URL, ANTHROPIC_API_KEY, ANTHROPIC_BASE_URL, NVIDIA_API_KEY, CLAUDE_CODE_USE_BEDROCK) fails the run with harbor.runtime_env contains operator-owned credential name(s).
• A ${NVIDIA_API_KEY}-style reference to an operator-owned credential inside any value is rejected too, so a credential cannot be aliased under a different name.

Blocked runtime_env names

Names that control the launcher, credentials, or dynamic runtime are rejected:

Configure the selected Harbor backend in the host environment instead.

SkillSpector bridge

The --llm security analysis runs SkillSpector, which has its own provider environment (SKILLSPECTOR_PROVIDER plus per-provider credential variables). SkillEvaluator bridges your configured provider automatically for that invocation, so the one key above is all you need. For nv_build, the bridge uses SkillSpector’s OpenAI-compatible provider path pointed at the public Build endpoint, so the same NVIDIA_API_KEY covers this too; it does not create a second NVIDIA credential name.

The subprocess environment is isolated. Only the selected provider settings and basic process environment variables (PATH, HOME, TLS bundles, temp dirs) pass through — other ambient credentials in your shell are never forwarded.

Setting SKILLSPECTOR_PROVIDER yourself overrides the bridge, but it must name a supported public provider (openai, anthropic, or bedrock — or nv_build to force the Build mapping); anything else fails closed and the subprocess receives no credential.

Troubleshooting

Next steps