Environment Variables

Every environment variable SkillEvaluator reads, grouped by feature area. Each group links to the guide that explains the workflow the variables belong to — this page is the inventory, not the tutorial.

Unless a variable’s row specifies otherwise, boolean variables accept 1, true, yes, or on (case-insensitive) as true, and 0, false, no, or off as false. Any other value that is set, including an empty one, also counts as false.

Two historical SkillEvaluator prefixes coexist. SKILL_EVAL_* covers provider and model configuration plus a small set of Tier 3 engine knobs; SKILLEVALUATOR_* covers product-level validation, report paths, scanner paths, and sandbox/runtime behavior. The prefixes are part of each variable’s literal public name and are not interchangeable. The separately documented SKILLSPECTOR_* family belongs to SkillSpector, provider credential families belong to their providers, and Harbor, the open-source agent evaluation framework used for Tier 3, owns the HARBOR_* in-container grader contract. None belongs to either SkillEvaluator prefix.

LLM provider

These variables select and configure the provider used for LLM-backed checks and judging. The setup walkthrough, including per-provider export blocks and default models, is in Providers & Credentials.

LLM retries and judge time budget

SkillEvaluator retries transient LLM failures itself. The retry settings apply on the host to every chat call made through the shared LLM client, including Tier 1 --llm checks, Tier 2 LLM analysis, eval dataset generation, and Tier 3 report suggestions. They also apply in the Harbor verifier to the direct provider calls made by the accuracy, goal_accuracy, and behavior_check judges. Embedding requests and the canonical OpenAI RAGAS goal scorer use their own client and scoring path, so these settings do not apply to them.

Retries cover HTTP 408, 429, and every 5xx response except 501 and 505, including 520–524 proxy errors and 529 overload responses. Connection errors and timeouts are also retried, but TLS certificate verification failures are not: retrying cannot repair an untrusted or invalid certificate. The host LLM client inspects the underlying HTTP status in wrapped SDK errors; an adapter’s generic 500 or 503 does not hide a terminal failure. In the verifier, Bedrock throttling and service-unavailable errors are also retried. Other HTTP errors, such as 400, 401, 403, or 404, are not retried. When a retryable response carries a Retry-After header, either in seconds or as an HTTP date, the next attempt waits that long plus 0.1 to 0.5 seconds of jitter instead of the computed backoff. An unparseable value waits the base delay. If Retry-After asks for longer than SKILL_EVAL_LLM_RETRY_MAX_DELAY, the call fails immediately with the provider’s error.

In the verifier, SKILL_EVAL_LLM_JUDGE_BUDGET_SEC also bounds retries. Each request’s timeout is cut to the judge’s remaining time, and a retry whose wait would pass the deadline ends the judge with a timeout. The RAGAS goal scorer runs under the same budget. A judge that runs out of time is an infrastructure failure, not a 0.0 score (see Tier 3 judge failure semantics). The three required judges run one after another inside the generated task’s 600-second verifier timeout, so the default leaves time to write a failed-run report.

Set these variables in the host environment that runs skillevaluator. When one is set, Tier 3 stages it into the generated Harbor task’s [verifier.env], so the verifier uses the same value. They are never added to the agent’s environment, and evals/config.yml runtime_env cannot set SKILL_EVAL_* names.

Anthropic endpoint roots

When anthropic is the selected evaluator provider, endpoint precedence is SKILL_EVAL_LLM_BASE_URL, then ANTHROPIC_BASE_URL, then the Anthropic default. A root such as https://gateway.example.com and a legacy terminal /v1 root such as https://gateway.example.com/team/v1 both produce request paths with exactly one /v1/messages suffix for the Anthropic SDK, the Tier 3 verifier, and a matching Claude Code agent. An independently credentialed Claude Code route applies the same normalization and validation to ANTHROPIC_BASE_URL when the evaluator uses another provider. Routing through path prefixes, ports, IP addresses, internationalized domain names, and ordinary safe percent escapes is preserved.

Anthropic configuration fails before Harbor starts for a full /v1/messages endpoint, embedded credentials, a query or fragment, whitespace or control characters, a backslash, an invalid authority, interior empty or dot path segments, or ambiguous multi-layer endpoint encoding. This normalization applies only to Anthropic. OpenAI-compatible endpoints commonly include /v1, which remains unchanged.

Auto-selection

When SKILL_EVAL_LLM_PROVIDER is unset, auto-selection only happens when exactly one public credential is exported:

• NVIDIA_API_KEY → nv_build
• OPENAI_API_KEY → openai
• ANTHROPIC_API_KEY → anthropic

If none is set, LLM-backed commands fail with a configuration error. If more than one is set, that is also a configuration error — there is no silent preference order, so set SKILL_EVAL_LLM_PROVIDER explicitly when multiple keys are exported.

SkillSpector bridge

The --llm security analysis runs SkillSpector, which has its own provider environment. SkillEvaluator bridges your configured provider into that invocation with an isolated subprocess environment: only the selected provider’s settings, the SkillSpector model and deadline settings below, and a small set of basic process variables are forwarded, so ambient credentials for other providers never reach SkillSpector. SKILLSPECTOR_TEMPERATURE is not forwarded, because current reasoning models reject a custom temperature. The nv_build provider is bridged as SKILLSPECTOR_PROVIDER=nv_build, so SkillSpector sizes requests from its NVIDIA Build model registry, while the key travels as OPENAI_API_KEY with OPENAI_BASE_URL pointing at the public NVIDIA Build endpoint.

Embeddings

Tier 2 semantic checks use an OpenAI-compatible embeddings API. Workflow context lives in Providers & Credentials and Tier 2: Deduplication.

Validation (Tier 1)

These variables tune the static and security checks. The check-by-check guide is Tier 1: Validation.

Live evaluation (Tier 3)

These variables control where results land and how Harbor confines local mode. The run workflow is Tier 3: Live Evaluation; execution environments are covered in Agents & Sandboxes.

Harbor process environment

SkillEvaluator starts every Harbor process in an empty, evaluator-owned working directory with .env loading and Harbor telemetry disabled:

• Harbor does not read .env or .env.local files from your project or current directory. Export variables in the shell or CI job that runs skillevaluator instead.
• An evaluation run passes Harbor only an allowlisted environment: basic process, locale, and temporary-directory variables such as PATH, HOME, LANG, and TMPDIR; proxy and CA-bundle variables; the selected backend’s own variables (see Agents & Sandboxes); the evaluator provider’s settings and credential, plus the AWS_* chain for bedrock; each selected agent’s credentials; validated runtime_env values; and the Tier 3 judge, retry, and evidence-budget variables on this page. Other variables in your shell do not reach Harbor.
• Relative paths in allowlisted path variables, such as KUBECONFIG, DOCKER_CONFIG, SSL_CERT_FILE, or AWS_SHARED_CREDENTIALS_FILE, resolve against the directory where you ran skillevaluator.
• On Windows, PROGRAMFILES, PROGRAMW6432, and PROGRAMDATA reach Harbor so Docker Desktop can discover Compose. PYTHONUTF8 and PYTHONIOENCODING are also forwarded when set; use PYTHONUTF8=1 and PYTHONIOENCODING=utf-8 for a non-UTF-8 locale. See the PowerShell setup.
• Setting HARBOR_TELEMETRY or PYTHON_DOTENV_DISABLED yourself has no effect; SkillEvaluator always overrides both.
• Tier 3 needs python-dotenv 1.2.0 or newer, which the tier3 extra installs. With an older version installed, the prerequisite check fails before Harbor starts.

Judge evidence budgets

SKILL_EVAL_JUDGE_CHARACTER_FACTS=1 optionally appends Unicode character metadata to standard Tier 3 judge prompts. It is off by default; true, yes, and on also enable it (case-insensitive). This operator-owned setting is forwarded only to standard verifier jobs, never to agent environments or custom_only jobs. When configured, its value is recorded as judge.character_facts in run_config.json. See character-sensitive judging for scope, limits, and the effect on the goal-accuracy route.

Before the Harbor verifier calls the accuracy, goal_accuracy, and behavior_check judges, it condenses the agent’s trajectory into a separate evidence excerpt for each judge. These budgets cap each excerpt in characters. When a run produces more evidence than fits, lower-priority material is dropped or truncated first so the judge still sees the agent’s final response. Larger budgets give judges more context and cost more prompt tokens.

Each value must be an integer. Values below 1 are treated as 1, and non-integer values such as 8k or 1.5 use the default. Set these variables in the host environment that runs skillevaluator. As with the retry settings, Tier 3 stages any that are set into the generated task’s [verifier.env], and never into the agent’s environment.

Tier 3 judge failure semantics

In standard grading, a required judge transport, authentication, timeout, HTTP, or exhausted response-parsing error is an evaluation infrastructure failure, not a numeric 0.0. A condition containing a failed or incomplete default trial publishes none of its scores, dimensions, pass@k, or quality findings, and no cross-condition lift is published. Redacted diagnostics and coverage information remain available. A genuine model score of 0.0 remains valid and is published normally.

default_plus_custom cannot use a custom score to rescue failed standard judging. custom_only is unchanged and does not invoke the standard judge.

In-container grader contract

The HARBOR_* variables are read by your custom grader inside the task container — you never set them yourself, and each has a fixed in-container default. The full reward contract, with defaults and examples, lives in Custom Graders & Tasks.

Set at runtime, not committed

See also

• Providers & Credentials — the setup walkthrough behind the provider variables
• CLI Reference — the per-run flags that override many of these variables
• Gate Your CI — supplying these variables as masked CI secrets