Skill Interaction and Activation Experiments
For your first comparison, start with Run a live evaluation.
Skill interaction experiments
Use the opt-in tier3 interact command to measure whether selected skills
produce more or less improvement together than the sum of their individual
improvements. Choose one reviewed evaluation source before running:
--eval-source is a skill directory containing an existing valid evals/
dataset or native Harbor task suite. The command never generates or refines
tasks. It freezes that source and the selected skills once, then reuses the
existing Harbor evaluator and skill staging to run:
For two skills this is four configurations. For a group of N skills it is
N + 2 configurations, each with the full task suite, selected agents, and the
same --n-attempts budget. Early stopping is disabled, including when the source
config enables it. Source-config companion selections, partial task suites, and
pre_agent_setup are rejected before execution. Skill roots must have distinct
names and be self-contained: runtime snapshots include each skill’s files,
while common task inputs belong in the fixed source’s evals/files/. Results
must be outside the selected skills and evaluation source. The output root uses
--results-dir, then SKILLEVALUATOR_RESULTS_DIR, then interaction-results
beside the source. User-home shortcuts and symlinked parents are resolved before
checking that the output stays outside the inputs.
The default outcome metric is goal_accuracy; --metric accuracy is also
available. These compare task outcomes on the same cases instead of the
standard overall score, whose discovery and skill-usage components differ
between installed-skill and baseline configurations. For custom_only grading,
use --metric overall with one shared task-outcome grader.
For two skills, the interaction is:
For a selected group S:
P is the mean per-case outcome, averaging the fixed number of attempts within each case first. A positive interaction means more than additive improvement; a negative interaction means less; zero means additive. For example, baseline 0.50, individual scores 0.75 and 0.70, and combined score 0.65 yield interaction -0.30 and combined lift +0.15. Negative interaction does not imply performance below baseline. For more than two skills this is total departure from additive individual gains, rather than an isolated higher-order interaction.
These signs describe the observed estimate, without a confidence interval or statistical significance test. A small interaction can reflect trial noise; with one attempt on ten binary-outcome cases, one outcome changing in an individual or combined configuration moves the interaction by 0.10. Configurations run sequentially: baseline, each individual, then combined. Provider changes, rate limiting, or timeouts during a long experiment can therefore align with configuration order. Repeat the experiment before interpreting a small interaction as a reliable effect.
The terminal prints baseline, individual and combined scores, combined lift,
and signed interaction separately for each agent/model. Add --json for a
machine-readable report on stdout; each configuration’s trial progress goes to
stderr. Every experiment
also saves interaction.json, its fixed evaluation source, and independent
Tier 3 run directories containing reports, dataset snapshots, and trial
evidence. The interaction report records original source paths, content
identities, per-case calculations, and the reward artifacts used.
Missing trials, N/A outcomes, changed cases, unequal attempt budgets, or
different agent models, evaluator versions, or outcome scorers produce an
explicit incomplete or incomparable result with no interaction estimate and
exit code 1. Judge fallback to a different model is incomparable. Multiple
retained reward rows for one logical attempt are also incomparable rather than
being counted as extra attempts. Other completed configuration evidence remains
available for inspection.
Repeatable activation validation
Use the existing dataset generator, runner and trusted trajectory collector to
check activation separately from answer quality. Write task requests and
observable outcomes in eval-guidance.md, then run
skillevaluator create-eval-dataset ./my-skill --full --prompt eval-guidance.md, review the generated
cases, and retain a versioned dataset with these four buckets:
Template generation with --no-llm --full needs an authored negative prompt in
evals/EVAL.md to produce the negative bucket. Add the edge case yourself and
make each expected_skill explicit. An authored should_trigger takes
precedence, including false without a skill name. Unlabeled expectations and
positive routing without a target identity remain unknown. Keep task correctness
assertions outcome-oriented; a correct answer alone cannot demonstrate skill use.
Use an explicit --agent-model AGENT=MODEL when needed for your provider.
Repeat with --grading-mode default_plus_custom after configuring an authored
custom grader. --skip-baseline supports a faster activation-only iteration;
it cannot measure benefit or harm relative to the baseline. custom_only
has no trusted standard activation contract and stays unknown in this view.
Each canonical report trial now includes activation_validation:
expected_activationis true, false, or unknown (null). The without-skill arm expects non-use even for an in-scope task.observed_activationisinvoked,not_invoked, orunknown, using only the collector’s trusted trajectory evidence.expectation_statuscompares these asmatched,mismatched, orunknown. A standard baseline’s explicit null/N/A skill scores do not prevent collecting trajectory evidence; those scores stay N/A. The arm alone never proves non-use.task_outcomeretains accuracy and goal accuracy independently of routing;evidence_scopes[].workflowretains workflow order, script execution and recovery diagnostics, andresourcesretains available time/token counters.evidence_scopes[].artifactslinks to the retained reward and trajectory paths relative to the run directory. Harbor viewer links remain on the trial.
For fallback multi-step reward rows, the logical observation stays unknown and
observation_scope is reward_rows_only; individual observations remain in
evidence_scopes. This also applies when only one step row survives loading.
A collector-materialized complete root trajectory with the collector-issued
invocation_evidence_scope: logical_trial marker can support a logical
observation. Historical rows without a scope marker retain their row evidence
but cannot certify a complete logical observation. Missing or contradictory topology, unresolved subagent references,
and unsupported traces remain unknown. Copied context and unreferenced child
trajectories do not prove execution. evidence_scopes_complete describes
whether all loaded row scopes fit in the report, not whether execution was
complete. These fields do not change numeric grading or the existing lift
eligibility policy.
The following read-only recipe exports both arms from a specific retained run
without calling another model. Replace the run directory with the one printed
by evaluation. Keep this JSON beside the run’s result.json, run_config.json,
attempt_policy.json and dataset_snapshot.json; together they retain skill,
task, model, harness, environment and grading identities. Review paired task
outcomes for useful, no-benefit or harmful results, and retain the observed
result even when it contradicts the expectation.
A matched routing expectation is not a correctness pass. Unknown observations, missing rows and truncated reports are incomplete evidence, not successful non-use. Retain raw trajectories privately: they can contain task inputs and model outputs even after credential redaction.
Contributors can repeat the fixture-backed acceptance suite without provider credentials. It exercises the production collector and report projection, including correct baseline answers, missed and irrelevant activation, missing traces, copied context, referenced subagents and multi-step completeness: