> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/skills/skillevaluator/llms.txt.

# Skill Interaction and Activation Experiments

> Measure skill interactions and validate activation with a reviewed evaluation matrix.

For your first comparison, start with [Run a live evaluation](/skills/skillevaluator/tier3-live-evaluation).

## Skill interaction experiments

Use the opt-in `tier3 interact` command to measure whether selected skills
produce more or less improvement together than the sum of their individual
improvements. Choose one reviewed evaluation source before running:

```bash
skillevaluator tier3 interact ./skills/research ./skills/summarize \
  --eval-source ./skills/research --agents codex --n-attempts 3 \
  --results-dir ./interaction-results
```

`--eval-source` is a skill directory containing an existing valid `evals/`
dataset or native Harbor task suite. The command never generates or refines
tasks. It freezes that source and the selected skills once, then reuses the
existing Harbor evaluator and skill staging to run:

| Configuration | Installed skills                      |
| ------------- | ------------------------------------- |
| Baseline      | None                                  |
| Individual    | One run for each selected skill alone |
| Combined      | Every selected skill together         |

For two skills this is four configurations. For a group of N skills it is
N + 2 configurations, each with the full task suite, selected agents, and the
same `--n-attempts` budget. Early stopping is disabled, including when the source
config enables it. Source-config companion selections, partial task suites, and
`pre_agent_setup` are rejected before execution. Skill roots must have distinct
names and be self-contained: runtime snapshots include each skill's files,
while common task inputs belong in the fixed source's `evals/files/`. Results
must be outside the selected skills and evaluation source. The output root uses
`--results-dir`, then `SKILLEVALUATOR_RESULTS_DIR`, then `interaction-results`
beside the source. User-home shortcuts and symlinked parents are resolved before
checking that the output stays outside the inputs.

The default outcome metric is `goal_accuracy`; `--metric accuracy` is also
available. These compare task outcomes on the same cases instead of the
standard overall score, whose discovery and skill-usage components differ
between installed-skill and baseline configurations. For `custom_only` grading,
use `--metric overall` with one shared task-outcome grader.

For two skills, the interaction is:

```text
I(A, B) = P(A, B) - P(A) - P(B) + P(empty)
```

For a selected group S:

```text
I(S) = [P(S) - P(empty)] - sum(P(skill) - P(empty) for skill in S)
```

P is the mean per-case outcome, averaging the fixed number of attempts within
each case first. A positive interaction means more than additive improvement;
a negative interaction means less; zero means additive. For example, baseline
0.50, individual scores 0.75 and 0.70, and combined score 0.65 yield interaction
-0.30 and combined lift +0.15. Negative interaction does not imply performance
below baseline. For more than two skills this is total departure from additive
individual gains, rather than an isolated higher-order interaction.

These signs describe the observed estimate, without a confidence interval or
statistical significance test. A small interaction can reflect trial noise;
with one attempt on ten binary-outcome cases, one outcome changing in an
individual or combined configuration moves the interaction by 0.10. Configurations
run sequentially: baseline, each individual, then combined. Provider changes,
rate limiting, or timeouts during a long experiment can therefore align with
configuration order. Repeat the experiment before interpreting a small
interaction as a reliable effect.

The terminal prints baseline, individual and combined scores, combined lift,
and signed interaction separately for each agent/model. Add `--json` for a
machine-readable report on stdout; each configuration's trial progress goes to
stderr. Every experiment
also saves `interaction.json`, its fixed evaluation source, and independent
Tier 3 run directories containing reports, dataset snapshots, and trial
evidence. The interaction report records original source paths, content
identities, per-case calculations, and the reward artifacts used.

Missing trials, N/A outcomes, changed cases, unequal attempt budgets, or
different agent models, evaluator versions, or outcome scorers produce an
explicit `incomplete` or `incomparable` result with no interaction estimate and
exit code 1. Judge fallback to a different model is incomparable. Multiple
retained reward rows for one logical attempt are also incomparable rather than
being counted as extra attempts. Other completed configuration evidence remains
available for inspection.

## Repeatable activation validation

Use the existing dataset generator, runner and trusted trajectory collector to
check activation separately from answer quality. Write task requests and
observable outcomes in `eval-guidance.md`, then run
`skillevaluator create-eval-dataset ./my-skill --full --prompt eval-guidance.md`, review the generated
cases, and retain a versioned dataset with these four buckets:

| Bucket   | Authored expectation                                 | What to check                                                                                           |
| -------- | ---------------------------------------------------- | ------------------------------------------------------------------------------------------------------- |
| Explicit | `expected_skill: "my-skill"`                         | A request naming the skill invokes it.                                                                  |
| Implicit | `expected_skill: "my-skill"`                         | An in-scope request without its name still invokes it.                                                  |
| Negative | `expected_skill: null`                               | An unrelated request succeeds without invoking it.                                                      |
| Edge     | Explicit skill name or `null`, chosen before running | A boundary input follows the skill's documented scope; retain the expected refusal, recovery or output. |

Template generation with `--no-llm --full` needs an authored negative prompt in
`evals/EVAL.md` to produce the negative bucket. Add the edge case yourself and
make each `expected_skill` explicit. An authored `should_trigger` takes
precedence, including `false` without a skill name. Unlabeled expectations and
positive routing without a target identity remain unknown. Keep task correctness
assertions outcome-oriented; a correct answer alone cannot demonstrate skill use.

**`Run the reviewed matrix through the normal harness`**

```bash title="Run the reviewed matrix through the normal harness"
skillevaluator tier3 validate ./my-skill --strict
skillevaluator tier3 evaluate ./my-skill --agents codex --env-mode docker \
  --n-attempts 1 --grading-mode default
```

Use an explicit `--agent-model AGENT=MODEL` when needed for your provider.
Repeat with `--grading-mode default_plus_custom` after configuring an authored
custom grader. `--skip-baseline` supports a faster activation-only iteration;
it cannot measure benefit or harm relative to the baseline. `custom_only`
has no trusted standard activation contract and stays unknown in this view.

Each canonical report trial now includes `activation_validation`:

* `expected_activation` is true, false, or unknown (`null`). The without-skill
  arm expects non-use even for an in-scope task.
* `observed_activation` is `invoked`, `not_invoked`, or `unknown`, using only
  the collector's trusted trajectory evidence. `expectation_status` compares
  these as `matched`, `mismatched`, or `unknown`.
  A standard baseline's explicit null/N/A skill scores do not prevent collecting
  trajectory evidence; those scores stay N/A. The arm alone never proves non-use.
* `task_outcome` retains accuracy and goal accuracy independently of routing;
  `evidence_scopes[].workflow` retains workflow order, script execution and
  recovery diagnostics, and `resources` retains available time/token counters.
* `evidence_scopes[].artifacts` links to the retained reward and trajectory
  paths relative to the run directory. Harbor viewer links remain on the trial.

For fallback multi-step reward rows, the logical observation stays unknown and
`observation_scope` is `reward_rows_only`; individual observations remain in
`evidence_scopes`. This also applies when only one step row survives loading.
A collector-materialized complete root trajectory with the collector-issued
`invocation_evidence_scope: logical_trial` marker can support a logical
observation. Historical rows without a scope marker retain their row evidence
but cannot certify a complete logical observation. Missing or contradictory topology, unresolved subagent references,
and unsupported traces remain unknown. Copied context and unreferenced child
trajectories do not prove execution. `evidence_scopes_complete` describes
whether all loaded row scopes fit in the report, not whether execution was
complete. These fields do not change numeric grading or the existing lift
eligibility policy.

The following read-only recipe exports both arms from a specific retained run
without calling another model. Replace the run directory with the one printed
by evaluation. Keep this JSON beside the run's `result.json`, `run_config.json`,
`attempt_policy.json` and `dataset_snapshot.json`; together they retain skill,
task, model, harness, environment and grading identities. Review paired task
outcomes for useful, no-benefit or harmful results, and retain the observed
result even when it contradicts the expectation.

**`Export activation acceptance evidence`**

```bash title="Export activation acceptance evidence"
python - ./my-skill ./my-skill/evals/results/RUN_ID > activation-validation.json <<'PY'
import json
import sys
from pathlib import Path
from skillevaluator.evaluation.tier3_report import agent_eval_result_from_directory

skill, run = (Path(arg).resolve() for arg in sys.argv[1:])
config = json.loads((run / "run_config.json").read_text())
result = agent_eval_result_from_directory(
    skill, run,
    env_mode=config.get("harbor", {}).get("environment", {}).get("value"),
    use_llm_judge=False, enforce_embedded_size=False,
)
if result is None:
    raise SystemExit("No retained agent results; activation is unknown")
payload = result.metadata["agent_eval"]
rows = []
for name, agent in payload["agents"].items():
    for condition, key in (("with-skill", "trials"), ("without-skill", "trials_baseline")):
        for trial in agent[key]:
            rows.append({
                "agent": name, "model": agent["model"], "condition": condition,
                "entry_id": trial["entry_id"], "trial_id": trial["trial_id"],
                "harbor_viewer": trial.get("harbor_viewer"),
                **trial["activation_validation"],
            })
print(json.dumps({
    "run_dir": str(run), "run_id": payload["run_id"],
    "skill_name": payload["skill_name"], "dataset_digest": payload["dataset_digest"],
    "environment": payload["environment"], "grading": config.get("grading"),
    "score_policy": payload["score_policy"],
    "execution_status": payload["execution_status"],
    "report_truncation": payload.get("report_truncation"), "rows": rows,
}, indent=2, allow_nan=False))
PY
```

A matched routing expectation is not a correctness pass. Unknown observations,
missing rows and truncated reports are incomplete evidence, not successful
non-use. Retain raw trajectories privately: they can contain task inputs and
model outputs even after credential redaction.

Contributors can repeat the fixture-backed acceptance suite without provider
credentials. It exercises the production collector and report projection,
including correct baseline answers, missed and irrelevant activation, missing
traces, copied context, referenced subagents and multi-step completeness:

**`Offline acceptance suite from the repository root`**

```bash title="Offline acceptance suite from the repository root"
python -m pytest tests/reporting/test_activation_validation.py \
  tests/test_harbor_negative_control_evidence.py \
  tests/test_harbor_collector_multistep_trajectory.py \
  tests/test_checks_routing.py tests/tier3/test_failed_skill_reads.py \
  tests/tier3/test_plugin_signals.py \
  --basetemp="$(mktemp -d)/activation-tests"
```