Providers & Credentials

This page gets SkillEvaluator’s LLM-backed features working: choosing a provider, wiring embeddings for Tier 2, and separating evaluator credentials from agent credentials in Tier 3. Most of Tier 1 needs no credential at all — start with the map below to see whether you need a key in the first place.

For the exhaustive variable-by-variable inventory, see Environment Variables.

Tier 3 executes trials through Harbor, the open-source agent evaluation framework. This page separates the credentials SkillEvaluator and the live agents need from the backend configuration Harbor itself reads.

What needs a key

Tiers describe evaluation depth; credentials are a separate axis. This is the complete map — everything not listed under “needs a provider” runs fully offline:

validate without --no-dedup stays usable keyless: the Tier 2 dedup pass skips gracefully when no embedding provider is configured.

Choose an LLM provider

Set SKILL_EVAL_LLM_PROVIDER to one of openai, anthropic, nv_build, bedrock, or openai-compatible. The selected provider determines which credential SkillEvaluator reads. All but openai-compatible ship a default model that SKILL_EVAL_LLM_MODEL overrides.

$
export SKILL_EVAL_LLM_PROVIDER=openai
$
export OPENAI_API_KEY='sk-...'

Auto-selection

When SKILL_EVAL_LLM_PROVIDER is unset and exactly one of these credentials is present, SkillEvaluator selects the matching provider:

• NVIDIA_API_KEY → nv_build
• OPENAI_API_KEY → openai
• ANTHROPIC_API_KEY → anthropic

If none of those variables is set, LLM-backed commands fail with a configuration error. If more than one is exported, resolution also fails — SKILL_EVAL_LLM_PROVIDER is required when multiple public provider credentials are configured; there is no silent preference order.

Embeddings

Tier 2 uses an OpenAI-compatible embeddings API. SKILL_EVAL_EMBEDDING_PROVIDER selects it; when unset, it falls back to your SKILL_EVAL_LLM_PROVIDER (or the same auto-selection order as above).

Use SKILL_EVAL_EMBEDDING_MODEL and SKILL_EVAL_EMBEDDING_BASE_URL to override defaults; for openai-compatible, the base URL falls back to SKILL_EVAL_LLM_BASE_URL. The nv_build endpoint is fixed to https://integrate.api.nvidia.com/v1 — only its model can be overridden.

$
export SKILL_EVAL_EMBEDDING_PROVIDER=openai
$
export OPENAI_API_KEY='sk-...'

Fully local setup

The openai-compatible provider accepts any local OpenAI-compatible server (Ollama, vLLM, llama.cpp, NVIDIA NIM), so Tier 2 can run with no external calls at all. Example recipe using Ollama in Docker:

$
docker run -d --name ollama -p 11434:11434 ollama/ollama
$
docker exec ollama ollama pull nomic-embed-text   # embeddings
$
docker exec ollama ollama pull qwen2.5:0.5b       # chat LLM for dedup analysis

$
export SKILL_EVAL_EMBEDDING_PROVIDER=openai-compatible
$
export SKILL_EVAL_EMBEDDING_BASE_URL=http://localhost:11434/v1
$
export SKILL_EVAL_EMBEDDING_MODEL=nomic-embed-text
$
export SKILL_EVAL_EMBEDDING_API_KEY=local-no-key  # must be set; local servers ignore the value
$
$
# context-optimization-check and dedup-scan also use a chat LLM:
$
export SKILL_EVAL_LLM_PROVIDER=openai-compatible
$
export SKILL_EVAL_LLM_BASE_URL=http://localhost:11434/v1
$
export SKILL_EVAL_LLM_MODEL=qwen2.5:0.5b
$
export SKILL_EVAL_LLM_API_KEY=local-no-key

$
skillevaluator similarity-check ./skills
$
skillevaluator context-optimization-check ./my-skill

similarity-check compares skills directly or through a local catalog and needs only the embedding variables. context-optimization-check (also available as its dedup-scan alias) additionally uses the chat model. Analysis quality tracks that model — the tiny model above proves the plumbing; pick a stronger local model for verdicts you intend to act on.

Tier 3 credentials: two roles

Live evaluation involves two distinct credentials, and they are not interchangeable:

Both roles resolve from the operator’s environment; a skill’s evals/config.yml can never supply, rename, or redirect a credential. What each agent needs depends on the provider and --env-mode:

• NVIDIA Build, one key (--env-mode docker or local): a single NVIDIA_API_KEY powers the evaluator and all three supported agents — opencode calls NVIDIA Build directly, while claude-code and codex route through SkillEvaluator’s compatibility bridge (in-container in Docker mode; an authenticated in-process host bridge in local mode). Agent models must be full NVIDIA Build catalog IDs in publisher/model form.
• NVIDIA Build on Harbor cloud modes (daytona, e2b, …): the bridge is not available. claude-code needs its own native ANTHROPIC_API_KEY, codex needs a full OpenAI Responses API credential (OPENAI_API_KEY plus OPENAI_BASE_URL), and each needs an explicit model via --agent-model or harbor.agents.<agent>.model.
• Matching native provider: when the evaluator provider is the agent’s own (for example anthropic with claude-code, or openai with codex), the same key serves both roles automatically.

The agent’s model lives in the skill’s evals/config.yml:

1
schema_version: 1
2
harbor:
3
  agents:
4
    codex:
5
      model: gpt-5.4-mini

You can also supply that model at invocation time with --agent-model codex=MODEL. See Tier 3: Live Evaluation for the full run workflow.

In Docker mode, credential values are never rendered on the docker compose exec command line — the bridge receives them through a mode-0600 file inside the container that is consumed and removed before the agent starts.

List the models your key can access

Before pinning --agent-model or SKILL_EVAL_LLM_MODEL, check what the selected provider actually serves:

$
skillevaluator models --limit 20
$
skillevaluator models --json

skillevaluator models prints a filtered, bounded view of the authenticated provider catalog (--limit accepts 1–100, default 10; --json emits machine-readable output). It uses the same provider and credential resolution as evaluation, so it doubles as a quick check that your key works. A model appearing in the catalog is not proof a given agent harness supports it — use doctor --verify-models for that. Bedrock exposes no HTTP model catalog, so models errors on that provider; use doctor --verify-models there too.

runtime_env is for task values, not credentials

runtime_env in evals/config.yml remains valid for non-credential task values — plain names the task environment needs. Credential names are rejected outright:

• Any operator-owned credential name (OPENAI_API_KEY, OPENAI_BASE_URL, ANTHROPIC_API_KEY, ANTHROPIC_BASE_URL, NVIDIA_API_KEY, CLAUDE_CODE_USE_BEDROCK) fails the run with harbor.runtime_env contains operator-owned credential name(s).
• A ${NVIDIA_API_KEY}-style reference to an operator-owned credential inside any value is rejected too, so a credential cannot be aliased under a different name.

Blocked runtime_env names

Names that control the launcher, credentials, or dynamic runtime are rejected:

Configure the selected Harbor backend in the host environment instead.

SkillSpector bridge

The --llm security analysis runs SkillSpector, which has its own provider environment (SKILLSPECTOR_PROVIDER plus per-provider credential variables). SkillEvaluator bridges your configured provider automatically for that invocation, so the one key above is all you need. For nv_build, the bridge uses SkillSpector’s OpenAI-compatible provider path pointed at the public Build endpoint, so the same NVIDIA_API_KEY covers this too; it does not create a second NVIDIA credential name.

The subprocess environment is isolated. Only the selected provider settings and basic process environment variables (PATH, HOME, TLS bundles, temp dirs) pass through — other ambient credentials in your shell are never forwarded.

Setting SKILLSPECTOR_PROVIDER yourself overrides the bridge, but it must name a supported public provider (openai, anthropic, or bedrock — or nv_build to force the Build mapping); anything else fails closed and the subprocess receives no credential.

Troubleshooting

Next steps