Reproducible Terminal Demo
Use one example from terminal replay to your own skill. This demo uses the public
text-analyzer reference skill, a two-line text fixture, and the released 0.8.0
CLI. It does not require an account, key, scanner, Docker runtime or GPU.
Replay and run the same example
Install Git and uv. These commands use Python 3.13, provisioned by uv, and avoid shell-specific environment variable syntax. The replay below is validated on Linux. Native macOS and Windows replay still need validation; cross-platform Python alone does not establish that coverage.
The demo files come from your checkout; --isolated --no-project --with uses
the public release wheel, not an editable development install. The 0.8.0 wheel
is available on GitHub Releases;
do not assume the same version is already on PyPI. Installation needs
internet access. The demo then runs locally. For standalone pip installations
and supported Python versions, see Installation.
The runner prints each command and its real terminal output. It copies the
installed skill into reports/starter/text-analyzer, stages the public input
and task, runs the analysis script, and evaluates quality. It also makes a
separate intentionally shortened documentation copy to demonstrate a regression.
Your source checkout and the installed reference skill are not edited.
The success summary starts with PASS: starter output and quality regression checked.
An expected Error: quality check failed appears earlier for the seeded
regression. The runner itself exits 0 only after the report checks succeed.
It never overwrites an existing output directory: use --output reports/starter-2
to repeat it.
This executable replay is the demo source, not a video recording. To produce the
same demonstration, run the command above; quality.txt and regressed.txt
retain the terminal output, and receipt.json records the version and time.
Expected scalar values live in
examples/starter/expected.json;
they are checked against new results, not substituted for them. The scores
in this guide are from the pinned 0.8.0 release. The current source can score
the same skill differently, so a run with your checkout’s own evaluator is
checked against the checkout scores in that file instead. Historical
benchmark scores are not evidence for this run.
Interpret the reports
In the released CLI, the passing command can print All validations passed
without mentioning publication completeness. Open the JSON to see
publication_status and its reasons; the demo runner also prints an explicit
publication INCOMPLETE reminder. Do not infer publication readiness from the
terminal PASS line alone.
The failing quality report has publication_status: fail; a known failure takes
precedence over missing evidence. Do not interpret incomplete evidence as a
zero score, a pass, or proof of a skill defect. Fix execution/setup problems and
rerun before drawing performance conclusions. For live agent results, compare
with-skill and without-skill outcomes, inspect failed attempts and coverage, and
use the verdict and completion fields together; see Reports & Results.
Replay a single command manually
From the repository root after the demo, use the same copied skill and threshold:
The runner invokes this installed Click CLI through the same Python interpreter
with python -m skillevaluator.cli; it does not mock the evaluator or render a
prebuilt report. The usual quality threshold is 70; this demo deliberately uses
80 so the seeded regression fails.
Add model feedback or a live agent
For fresh model-backed instruction feedback on this same skill, configure an account and key using NVIDIA Build or another public provider. Never put a key in a report or fixture. Then run:
Model catalogs and account access change. A visible catalog entry does not
guarantee inference access. If the default model returns HTTP 404 or 410, select
a model available to your account with SKILL_EVAL_LLM_MODEL as described in
configuration. Authentication errors, rate limits, and
unavailable models are not low rubric scores. Inspect the JSON findings for
llm_unavailable; do not present a fallback response as a completed evaluation.
For example, a fresh public Build rubric run was verified with
SKILL_EVAL_LLM_MODEL=nvidia/nemotron-3.5-bolt-e4b. Set the variable in the same
shell before the rubric command (export SKILL_EVAL_LLM_MODEL=nvidia/nemotron-3.5-bolt-e4b
on POSIX shells, or $env:SKILL_EVAL_LLM_MODEL="nvidia/nemotron-3.5-bolt-e4b" in
PowerShell). This is an instruction-feedback example, not a promised score or
an agent benchmark. If the response has llm_invalid_response, the call reached
the model but did not produce valid rubric evidence; retain it and retry with a
supported model rather than interpreting it as a completed low-scoring run.
Rubric feedback is not agent uplift. For the paired agent experiment, keep the
same copied task and input and follow the
live evaluation prerequisites.
After installing the tier3 extra, configuring evaluator/agent credentials and
starting Docker, run:
The authored task is evals/evals.json; evals/files/input.txt is staged into
the sandbox at /workspace/input/input.txt. Read both before paying for agent
runs. One tiny task is a plumbing example, not representative coverage or
independent validation of a grader. Model and sandbox usage can incur charges.
Read Eval Datasets for additional cases, grading and policy,
and retain all completed attempts and errors rather than publishing only a score.
Adapt the example
- Copy
reports/starter/text-analyzerto a new skill directory. Keep output reports outside that directory; do not start from the regressed copy. - Change
SKILL.md, the script, andevals/files/input.txtto your use case. Update the task prompt and expected outcome together. Preserve the input’s final newline if you still expect 27 characters. - Check the script independently, then run
quality-checkon your directory. Inspect individual findings rather than optimizing only the aggregate score. - For agent impact, add varied realistic cases and independently checked outcomes, then repeat the paired live evaluation. Do not carry the starter’s scores or expected values over to your own skill.
Check your understanding
Before looking at answers, open the two JSON reports and answer:
- Which command failed, and why is the demo’s final exit still 0?
- Can the passing skill claim an agent benefit or publication eligibility?
- What changed between the original and regressed copy? Which findings explain the score change?
Answers: the intentionally shortened copy fails the 80 gate; the runner expects
that failure. No agent benefit or publication readiness was established.
Only the copied documentation changed, not the script, task, or input. Inspect
quality_summary and finding_groups for the changed quality findings. These
self-checks are not a substitute for feedback from a first-time human user.
Recover and continue
The public template reference and versioned downloadable bundles were consolidated in the public template update. They require no private account. For repository automation, continue with Repository Integration; for improvements to this starter, see Contributing.