Trace Datasets
Production behavior is evidence, not an expected answer. Import creates a review bundle outside the skill; nothing is promoted until a reviewer explicitly chooses a decision. This workflow is local and does not call an LLM.
Import and review
The default output is ./my-skill-trace-review, relative to the current working
directory. Output inside the skill is refused. Existing candidates.json requires
--force to replace it. --dry-run prints classifications without writing.
Keep the original export and review bundle private; do not ship either with a skill.
The bundle contains candidates.json and redacted replay inputs under
fixtures/<candidate-id>/. Each candidate has observed evidence, provenance,
redaction counts, warnings, and a pending review. Observed read/open calls for
files outside the skill that were not exported as fixtures produce
missing replay input warnings (deduplicated, capped at 20 per candidate).
Tool-derived
suggested_assertions are hints only, never promoted automatically.
Export contract v1
Accepts JSONL (one session object per line), or a JSON wrapper:
Other wrapper formats and versions are rejected. JSONL uses the v1 session contract without a wrapper. Records are numbered from 1 (physical line numbers for JSONL).
The summary and output explicitly report candidates and skipped records:
duplicate (normalized prompt and identical fixture contents, pointing to the
earlier candidate), incomplete (missing id or prompt), unsupported (including
invalid JSON and non-object records), and other skill.
Review decisions and promotion
Edit each candidate’s review in candidates.json:
Unknown decisions and missing expectations abort the whole operation before
writing. Write expectations that define correct behavior, not merely what a
production session happened to do. Move only independently reviewed suggestions
into assertions; suggested_assertions and review notes are not eval criteria.
Accepted entries append to agentskills-shaped evals/evals.json. Legacy flat
lists or other dataset extensions require manual migration first. Fixtures copy
to evals/trace-files/<candidate-id>/; refs use
trace-files/<candidate-id>/..., outside the implicit evals/files/ pool, so
they are not staged into unrelated entries. Fixture-free entries explicitly use [].
Before accepting, the command checks which task source is selected. If native
Harbor tasks are selected, use --allow-native-harbor to explicitly accept the
source change: with task_source: auto, the new eval dataset takes precedence;
with task_source: native_harbor, the trace cases are added but are not run.
trace_source records the redacted session id, export hash, outcome, and review
decision. Re-running skips ids already present for the same session; a different
session colliding with an id is an error. Dataset validation runs after writing;
validation errors are printed and return a non-zero exit status.
Redaction guarantees and limits
Before writing, import applies the shared credential redactor and configured PII
patterns for email addresses, phone numbers, SSNs, personal macOS/Windows paths,
and Linux /home/<name> paths (replaced by /home/<user>). Placeholder
exceptions in matching patterns are respected. Prompts, observed outputs, tool
summaries, fixtures, and provenance strings are redacted; raw logs and unknown
fields are not copied. Per-candidate counts record introduced secret-redaction markers and PII
matches. The export hash is calculated from the original bytes, not redacted text.
Promotion rechecks prompts, expectations, assertions, provenance, and fixtures; if redaction would still change them, it refuses the candidate rather than silently editing the reviewed expectation. Reviewers must sanitize newly pasted material before trying again.
Pattern redaction is best effort, not a guarantee that every secret or person is detected. Reviewers must still inspect all bundle files for sensitive data, missing replay inputs, and suitability for publication. Review decisions are explicit local metadata, not authenticated signatures or proof of who reviewed.
See Eval Datasets for the promoted dataset contract.