Skip to navigation

Skill Interaction and Activation Experiments

View as MarkdownOpen in Claude

For your first comparison, start with Run a live evaluation.

Skill interaction experiments

Use the opt-in tier3 interact command to measure whether selected skills produce more or less improvement together than the sum of their individual improvements. Choose one reviewed evaluation source before running:

skillevaluator tier3 interact ./skills/research ./skills/summarize \
--eval-source ./skills/research --agents codex --n-attempts 3 \
--results-dir ./interaction-results

--eval-source is a skill directory containing an existing valid evals/ dataset or native Harbor task suite. The command never generates or refines tasks. It freezes that source and the selected skills once, then reuses the existing Harbor evaluator and skill staging to run:

ConfigurationInstalled skills
BaselineNone
IndividualOne run for each selected skill alone
CombinedEvery selected skill together

For two skills this is four configurations. For a group of N skills it is N + 2 configurations, each with the full task suite, selected agents, and the same --n-attempts budget. Early stopping is disabled, including when the source config enables it. Source-config companion selections, partial task suites, and pre_agent_setup are rejected before execution. Skill roots must have distinct names and be self-contained: runtime snapshots include each skill’s files, while common task inputs belong in the fixed source’s evals/files/. Results must be outside the selected skills and evaluation source. The output root uses --results-dir, then SKILLEVALUATOR_RESULTS_DIR, then interaction-results beside the source. User-home shortcuts and symlinked parents are resolved before checking that the output stays outside the inputs.

The default outcome metric is goal_accuracy; --metric accuracy is also available. These compare task outcomes on the same cases instead of the standard overall score, whose discovery and skill-usage components differ between installed-skill and baseline configurations. For custom_only grading, use --metric overall with one shared task-outcome grader.

For two skills, the interaction is:

I(A, B) = P(A, B) - P(A) - P(B) + P(empty)

For a selected group S:

I(S) = [P(S) - P(empty)] - sum(P(skill) - P(empty) for skill in S)

P is the mean per-case outcome, averaging the fixed number of attempts within each case first. A positive interaction means more than additive improvement; a negative interaction means less; zero means additive. For example, baseline 0.50, individual scores 0.75 and 0.70, and combined score 0.65 yield interaction -0.30 and combined lift +0.15. Negative interaction does not imply performance below baseline. For more than two skills this is total departure from additive individual gains, rather than an isolated higher-order interaction.

These signs describe the observed estimate, without a confidence interval or statistical significance test. A small interaction can reflect trial noise; with one attempt on ten binary-outcome cases, one outcome changing in an individual or combined configuration moves the interaction by 0.10. Configurations run sequentially: baseline, each individual, then combined. Provider changes, rate limiting, or timeouts during a long experiment can therefore align with configuration order. Repeat the experiment before interpreting a small interaction as a reliable effect.

The terminal prints baseline, individual and combined scores, combined lift, and signed interaction separately for each agent/model. Add --json for a machine-readable report on stdout; each configuration’s trial progress goes to stderr. Every experiment also saves interaction.json, its fixed evaluation source, and independent Tier 3 run directories containing reports, dataset snapshots, and trial evidence. The interaction report records original source paths, content identities, per-case calculations, and the reward artifacts used.

Missing trials, N/A outcomes, changed cases, unequal attempt budgets, or different agent models, evaluator versions, or outcome scorers produce an explicit incomplete or incomparable result with no interaction estimate and exit code 1. Judge fallback to a different model is incomparable. Multiple retained reward rows for one logical attempt are also incomparable rather than being counted as extra attempts. Other completed configuration evidence remains available for inspection.

Repeatable activation validation

Use the existing dataset generator, runner and trusted trajectory collector to check activation separately from answer quality. Write task requests and observable outcomes in eval-guidance.md, then run skillevaluator create-eval-dataset ./my-skill --full --prompt eval-guidance.md, review the generated cases, and retain a versioned dataset with these four buckets:

BucketAuthored expectationWhat to check
Explicitexpected_skill: "my-skill"A request naming the skill invokes it.
Implicitexpected_skill: "my-skill"An in-scope request without its name still invokes it.
Negativeexpected_skill: nullAn unrelated request succeeds without invoking it.
EdgeExplicit skill name or null, chosen before runningA boundary input follows the skill’s documented scope; retain the expected refusal, recovery or output.

Template generation with --no-llm --full needs an authored negative prompt in evals/EVAL.md to produce the negative bucket. Add the edge case yourself and make each expected_skill explicit. An authored should_trigger takes precedence, including false without a skill name. Unlabeled expectations and positive routing without a target identity remain unknown. Keep task correctness assertions outcome-oriented; a correct answer alone cannot demonstrate skill use.

Run the reviewed matrix through the normal harness
skillevaluator tier3 validate ./my-skill --strict
skillevaluator tier3 evaluate ./my-skill --agents codex --env-mode docker \
--n-attempts 1 --grading-mode default

Use an explicit --agent-model AGENT=MODEL when needed for your provider. Repeat with --grading-mode default_plus_custom after configuring an authored custom grader. --skip-baseline supports a faster activation-only iteration; it cannot measure benefit or harm relative to the baseline. custom_only has no trusted standard activation contract and stays unknown in this view.

Each canonical report trial now includes activation_validation:

  • expected_activation is true, false, or unknown (null). The without-skill arm expects non-use even for an in-scope task.
  • observed_activation is invoked, not_invoked, or unknown, using only the collector’s trusted trajectory evidence. expectation_status compares these as matched, mismatched, or unknown. A standard baseline’s explicit null/N/A skill scores do not prevent collecting trajectory evidence; those scores stay N/A. The arm alone never proves non-use.
  • task_outcome retains accuracy and goal accuracy independently of routing; evidence_scopes[].workflow retains workflow order, script execution and recovery diagnostics, and resources retains available time/token counters.
  • evidence_scopes[].artifacts links to the retained reward and trajectory paths relative to the run directory. Harbor viewer links remain on the trial.

For fallback multi-step reward rows, the logical observation stays unknown and observation_scope is reward_rows_only; individual observations remain in evidence_scopes. This also applies when only one step row survives loading. A collector-materialized complete root trajectory with the collector-issued invocation_evidence_scope: logical_trial marker can support a logical observation. Historical rows without a scope marker retain their row evidence but cannot certify a complete logical observation. Missing or contradictory topology, unresolved subagent references, and unsupported traces remain unknown. Copied context and unreferenced child trajectories do not prove execution. evidence_scopes_complete describes whether all loaded row scopes fit in the report, not whether execution was complete. These fields do not change numeric grading or the existing lift eligibility policy.

The following read-only recipe exports both arms from a specific retained run without calling another model. Replace the run directory with the one printed by evaluation. Keep this JSON beside the run’s result.json, run_config.json, attempt_policy.json and dataset_snapshot.json; together they retain skill, task, model, harness, environment and grading identities. Review paired task outcomes for useful, no-benefit or harmful results, and retain the observed result even when it contradicts the expectation.

Export activation acceptance evidence
python - ./my-skill ./my-skill/evals/results/RUN_ID > activation-validation.json <<'PY'
import json
import sys
from pathlib import Path
from skillevaluator.evaluation.tier3_report import agent_eval_result_from_directory
skill, run = (Path(arg).resolve() for arg in sys.argv[1:])
config = json.loads((run / "run_config.json").read_text())
result = agent_eval_result_from_directory(
skill, run,
env_mode=config.get("harbor", {}).get("environment", {}).get("value"),
use_llm_judge=False, enforce_embedded_size=False,
)
if result is None:
raise SystemExit("No retained agent results; activation is unknown")
payload = result.metadata["agent_eval"]
rows = []
for name, agent in payload["agents"].items():
for condition, key in (("with-skill", "trials"), ("without-skill", "trials_baseline")):
for trial in agent[key]:
rows.append({
"agent": name, "model": agent["model"], "condition": condition,
"entry_id": trial["entry_id"], "trial_id": trial["trial_id"],
"harbor_viewer": trial.get("harbor_viewer"),
**trial["activation_validation"],
})
print(json.dumps({
"run_dir": str(run), "run_id": payload["run_id"],
"skill_name": payload["skill_name"], "dataset_digest": payload["dataset_digest"],
"environment": payload["environment"], "grading": config.get("grading"),
"score_policy": payload["score_policy"],
"execution_status": payload["execution_status"],
"report_truncation": payload.get("report_truncation"), "rows": rows,
}, indent=2, allow_nan=False))
PY

A matched routing expectation is not a correctness pass. Unknown observations, missing rows and truncated reports are incomplete evidence, not successful non-use. Retain raw trajectories privately: they can contain task inputs and model outputs even after credential redaction.

Contributors can repeat the fixture-backed acceptance suite without provider credentials. It exercises the production collector and report projection, including correct baseline answers, missed and irrelevant activation, missing traces, copied context, referenced subagents and multi-step completeness:

Offline acceptance suite from the repository root
python -m pytest tests/reporting/test_activation_validation.py \
tests/test_harbor_negative_control_evidence.py \
tests/test_harbor_collector_multistep_trajectory.py \
tests/test_checks_routing.py tests/tier3/test_failed_skill_reads.py \
tests/tier3/test_plugin_signals.py \
--basetemp="$(mktemp -d)/activation-tests"