Performance Dashboard
Use the dashboard to compare existing evaluation results without running new agent trials. It reads saved JSON reports and retained Tier 3 runs; missing measurements remain unavailable.
Start the dashboard
Install the optional Python dashboard and open saved JSON reports or individual retained Tier 3 run directories:
From a checkout, use uv run --extra dashboard skillevaluator dashboard ....
For live reload while developing the dashboard, run
PYTHONPATH=src uv run --extra dashboard streamlit run src/skillevaluator/reporting/dashboard_ui/skillevaluator_dashboard.py -- PATH;
PYTHONPATH=src lets Streamlit watch the package modules for changes.
The server runs at http://127.0.0.1:8501, or the next free port when 8501 is
busy; pass --port 8502 to require a specific port. Use --no-browser to
suppress opening a browser, and Ctrl+C to stop it.
The dashboard runs from its own package directory, so Streamlit settings or
secrets in your project’s .streamlit/ folder are not applied. It accepts
browser connections only for 127.0.0.1 and localhost.
Browser uploads accept up to 16 MB (16,000,000 bytes) per file. Local JSON files
accept up to 16 MiB (16,777,216 bytes); larger files produce a notice while other
valid reports remain available.
Compare saved runs
Filter the sortable tables by skill, model, harness, recorded harness features, condition, and execution status. The status and condition filters narrow the performance table only, so comparisons keep both arms; the other filters also remove arms from comparisons. Download the visible performance or comparison rows as CSV. Each performance row is one run, agent (harness and model), recorded feature configuration, and condition. Per-trial columns are means over the row’s measured trials and stay blank when any trial is unmeasured. They help compare rows with different trial counts, but runs on different datasets are still not like-for-like. The comparison table reports with skill minus without skill differences, with percentage changes for usage relative to without skill. A negative usage difference means less usage; a positive score difference means higher performance. Comparisons stay within the same report and agent and require successful conditions with matching trial coverage, the same dataset digest, and the same attempt policy. The score difference preserves the evaluator’s recorded Skill Lift, including negative, zero, or unavailable values. Cases judged by different models are excluded from lift, so it can differ from subtracting the displayed arm scores. Pass-rate and resource differences can remain available when score lift is unavailable. If every positive with-skill trial has confirmed zero activation, the comparison is inconclusive and all differences stay unavailable. See Skill Lift for these eligibility rules.
When a pair’s differences stay blank, the Comparison reason column says why. Repeated attempts pair by case, as in the HTML report: each arm must record the same number of attempts for every case, and each attempt must have its own trial ID. A trajectory above the report loader’s size bound leaves that trial’s token counts unavailable in canonical JSON but does not block score comparisons. Truncation limited to diagnostic details, such as evidence, raw reward fields, or non-best agent details, also leaves comparisons available, with a notice. Truncation that drops trials or other trial artifacts, or that is not explained, keeps comparisons unavailable.
Reload reports
The dashboard reads each input once and reuses it while you filter or switch views. Rewriting a JSON file passed on the command line, or adding an entry directly inside a directory input, is picked up on the next interaction. Edits deeper inside a directory input (for example a run that is still writing trials) or in reports a catalog references appear after Reload reports in the sidebar; a browser refresh does not re-read them. Overlapping inputs, such as a results directory and a run inside it, are loaded once. Byte-identical uploads are skipped with a notice.
Measurement limits
Missing measurements stay unavailable, and cost is shown only when recorded in
USD. Canonical Tier 3 JSON records each single-step trial’s agent execution time
(agent_seconds) when Harbor’s timestamps carry a timezone. Timestamps without a
timezone are treated as unavailable, as in the HTML report’s
measured agent resources. Agent time excludes
environment setup and verification. Canonical JSON never records USD cost; open
the retained run directory to show Harbor’s recorded agent_result.cost_usd and
multi-step totals. For retained runs whose trajectory lacks token totals or
cannot be read, the dashboard falls back to Harbor’s agent_result token
counters and keeps the trial’s agent time and cost.
Exports written before v0.5.0 could encode missing token counters as zero, so
when a canonical JSON report’s evaluator_version is earlier than 0.5.0 the
dashboard shows its zero token counters as unavailable and adds a notice; open
the retained run directory to recover recorded counters.
Cached tokens are shown separately and are not added again
to prompt plus completion tokens. No pricing estimates or new agent evaluations
are performed. Adverse conditions and harness features are displayed when
explicitly recorded; the dashboard does not run these experiments.
Since v0.5.0, canonical JSON collapses multi-step trials without per-trial token or time totals, so they show as unmeasured. Older exports that repeat a trial ID across reward rows show an ambiguity notice. Open the retained run directory to recover totals from authoritative step results.
Recorded harness features
Features are experiment metadata, not inferred from the harness name. Current
SkillEvaluator commands do not record this field; add it to the canonical JSON
or to run_config.json by hand. In a canonical Tier 3 JSON payload, an agent or
condition can include:
harness_features can be a list of strings, a non-empty string, or a JSON
object. For canonical JSON the condition’s value wins over the agent’s, which
wins over the report’s. For retained runs the order is the condition’s
summary.json, then the agent entry in run_config.json, then the root of
run_config.json. Edit the run directory’s run_config.json; the copy embedded
in result.json is used only when that file cannot be read. An empty feature
list records that no features were enabled and is shown as none (recorded);
an absent field is displayed as unknown. Differing recorded models or
feature configurations are kept separate when pairing conditions.
An explicitly imported adverse condition uses conditions.adverse,
trials_adverse, num_trials_adverse, and adverse for its score in the same
agent payload. It appears in the performance table; usage differences are
calculated for with/without pairs only. Adverse conditions are import-only
metadata: no evaluator command produces them, and the dashboard does not run them.