Skip to navigation
Closed-Loop Testing

Interpreting the Report

View as Markdown

A single Regression Testing Reporter run produces a tree of graded reports. This page explains what each file contains, what every headline metric means, and how to drill into a failing clip. Run a test first — see Quick Start Guide.

The healthy ranges below are operational guidance for reading a run, not acceptance criteria or benchmark results. Read metrics per class and per scenario; a single blended number can hide a real problem.

2D vs 3D runs. Every run scores Phase 1 (the safety decision). Phase 2 (perception quality) is added only when the 3D Sparse4D pipeline (mdx-bev + mdx-behavior) is active — a 2D-only run (mdx-events only) produces a Phase-1-only report and simply omits the Phase 2 section, which is expected, not a failure. The scenario you ran shapes what to expect from each metric; see Test Scenarios for the test cases.

Output layout

runs/multi-test-<TIMESTAMP>/
├── summary.md                      <- top-level (cross-run): open this first
├── failures.json                   <- FAIL + analyze-error clips, machine-readable
├── pss.log                         <- cross-run Safety Core gateway log
└── <scenario>/
    ├── scenes/scn_*.parquet
    ├── videos/scn_*.mp4            <- per-clip review videos
    ├── clip_logs/<scn>/            <- evidence CSVs (for the debug viewer)
    └── reports/
        ├── summary.md              <- per-scenario rollup
        ├── failures.json
        └── scn_*.md                <- per-clip detail
FileContents
Top-level summary.mdHeadline + per-run rollup table + per-clip detail with MP4 links. Open this first.
Per-scenario <scenario>/reports/summary.mdSame structure, scoped to one scenario.
Per-clip <scenario>/reports/scn_*.mdThe deepest evidence: ground-truth events vs Safety Core state, mismatch windows with timestamps, BA detection per event, MP4 links.
failures.jsonClips with verdict = FAIL, plus clips that hit an analysis error, ready for programmatic triage.

Verdict buckets

Each clip gets a verdict from its frame-by-frame match%:

  • PASS — match% ≥ 95
  • NEAR — 80 ≤ match% < 95
  • FAIL — match% < 80

A healthy run has roughly ≥ 70 % PASS clips.

Phase 1 — the safety decision

Phase 1 scores the Safety Core’s MUTE / UNMUTE decision against ground truth.

MetricWhat it measuresHealthy
match%Frame-by-frame agreement (expected mute == actual mute) at 30 Hz over the scored frames. Expected mute follows the ATL rule: forklift in the trailer, no person in the trailer, no person in the ROI. Not scored: frames whose expected state rests only on a person within --roi-edge-band-m (0.5 m) of the ROI edge, where the ground-truth point and perception’s detection can disagree, and frames within --settle-allowance-s (1.0 s) after each expected-state change. A lag beyond the allowance still counts.mean ≥ 90, median ≥ 95
raw match%The same agreement over every frame, with no frame left out.reported next to match%
missed expected-state intervalsExpected-state intervals the system never reached before the next change. A lag is measured only within its own interval, so a later transition does not answer an earlier, missed one. An interval begun by a person crossing within the edge band is neither timed nor missed; one begun by the forklift is.0; an interval shorter than the settling allowance is unreachable for any pipeline
Mute correct%Of frames GT says SHOULD mute, the fraction the system actually muted.≥ 90
Unmute correct%Of frames GT says should NOT mute, the fraction the system unmuted. This is the safety-critical direction — over-mute means the alarm was suppressed while a person was present.≥ 95
worst UNMUTE / MUTE lagMax time (ms) between a GT transition and the matching Safety Core state change, within the same expected-state interval. Long lag signals state-machine drift.UNMUTE ≤ 1 s, MUTE ≤ 5 s
over-mute / under-mute framesDirection of the mismatch. Over-mute = alarm suppressed when it shouldn’t be (safety-critical); under-mute = false alarm (noise).—

A (n=X) next to a metric is the frame count behind it. A small N (< 500) means the run barely exercised that direction — treat the percentage with caution. An n/a (unmute-test scenario) / n/a (mute-test scenario) label means the scenario’s design never exercised that direction (e.g. a person always in ROI never produces an expected-mute frame), so that column is not meaningful.

Behavior-analytics event detection

Phase 1 also checks whether perception’s behavior analytics (BA) reported the key events, matched to each ground-truth event within a ±3 s window:

EventWhat it measuresHealthy
BA ROIPer-event detection rate for “person enters the work zone”.≥ 95 % (median delay ~600–800 ms is normal)
BA TW IN / OUTPer-event detection rate for forklift trailer crossings.100 %

BA reports a direction offset that looks odd but is expected: TW IN fires ~2 s late (after the ground-truth crossing) and TW OUT ~1.5 s early (before it). BA marks the crossing from the detected bounding-box edge while ground truth marks the forklift’s centroid — those sit about half a forklift-length apart, an offset that trails the centroid on entry (BA late) and leads it on exit (BA early); a small constant BA processing delay nudges both a touch later, so the magnitudes differ. Neither the positive TW IN delay nor the negative TW OUT delay is a bug — while TW detection stays at 100 %, treat it as a geometry artifact. (BA emits no ROI exit event — it simply stops firing — so ROI exits are measured as how long BA kept firing after the person left.)

Phase 2 — perception quality

Present only when the 3D pipeline (Sparse4D → mdx-bev + mdx-behavior) was active. It scores the perception feeding the Safety Core, per class (Person / Forklift) — always read per class, since a blended number hides the forklift.

MetricQuestion it answers
detect-fail%“Was the object seen this frame?” — the % of in-coverage ground-truth frames with no matched detection. A detection question, independent of track identity. Lower is better.
tracking-loss%“Did the object keep one identity?” — the % of an actor’s matched frames that sit off its dominant track id (id-switch / fragmentation). Scored only where the object was already seen. Lower is better.
precision / recall / F1Standard detection quality, per class.
pos median vs pos p95pos median is the anchor offset (ground-truth reference point vs the 3D-box centre — geometry, not error). pos p95 is the real localization jitter — that is the spread you care about.
id-switches / split-frames%Track-id changes / frames where one physical object shows two same-class tracks (the “many ids for one object” signal).

Coverage — only score what the camera can see. A ground-truth actor outside camera view is not charged as a miss. Coverage is derived from where detections actually occurred, so a region the camera sees but where nothing was ever detected (e.g. a forklift fully lost deep inside the trailer) can fall outside coverage. Cross-check the trailer-boundary slice and the raw recall before reading a high forklift detect-fail% as a detector regression — it is usually genuine deep-trailer occlusion, not a bug.

Reading the per-run rollup

The rollup table lists each scenario. Read it in this order:

  1. Avg match% + PASS / NEAR / FAIL — at-a-glance health.
  2. Mute / Unmute correct% — which direction is failing. Low unmute = safety-critical; low mute = false-alarm noise.
  3. BA ROI / TW — should stay ≥ 95 % / 100 %. A drop means something changed in VSS / perception — escalate.
  4. n/a labels — the scenario didn’t exercise that direction; ignore that column.

Drilling into a failing clip

Open <scenario>/reports/scn_*.md when a clip is FAIL, has an unusually long reaction lag, or shows an odd Mute/Unmute mismatch. Each per-clip report has:

SectionUse
Verdict + WindowPASS/FAIL and ISO timestamps for cross-referencing.
Ground truth%char_in_roi, %char_in_trailer, %forklift_in_trailer, %expected_mute — what the test expected.
Observed Safety Core state%actual_mute + command breakdown — what the system did.
Match — scored / rawmatch% over the scored frames, raw match% over every frame, and how many frames were left out for a person at the ROI edge or for settling.
Mismatch windowsThe most useful section. Every contiguous (≥ 0.5 s) window with start / end / duration / direction. Scrub to that exact moment in the MP4.
Reaction lagUNMUTE / MUTE median + p95 + max + transition count, each within its own expected-state interval, plus the missed intervals and the intervals not timed for a person at the ROI edge.
BA detection per GT eventThe GT event → BA detected table with per-event delay. A miss here points at the perception tracker.

Drill-down workflow for a FAIL clip:

  1. Open the per-clip report.
  2. Look at over-mute% vs under-mute% — which direction failed?
  3. Read the mismatch-windows table — when (in clip-relative seconds) did it go wrong?
  4. Open the MP4, scrub to that timestamp, see what the cameras saw.
  5. Cross-check the BA detection table: did perception have valid input at that moment?
  6. If yes → Safety Core logic; if no → perception. For scrub-synced evidence, use the Debug Viewer.

Known signatures

Some patterns are known and expected — recognize them rather than filing them as new regressions.

Safety Core counter-drift

A known Safety Core issue causes the all-clear (unmute) path to drift on long runs, with a fix planned upstream. Recognize the signature so you don’t re-investigate it each time:

  • Unmute correct% < 90 %, especially with high %char_in_roi and forklift-in-trailer activity.
  • An alternating PASS/FAIL pattern aligned with the forklift-in-trailer state (FAIL while the forklift is in, PASS while out).
  • The Safety Core reporting muted while ground-truth-expected mute was 0 % — its internal state has drifted out of sync with reality.

When you see this pattern, restart the Halos compose between runs (the runner and skill already do this) and note it — don’t open a new bug for each occurrence.

Perception dead

A PERCEPTION_LIKELY_DEAD banner in summary.md means perception was dead during the run, so match% is untrustworthy. Fix perception and re-run — see Quick Start Guide Troubleshooting.

Programmatic access

For CI / scripts, failures.json gives a clean list of the FAIL (and analyze-error) clips with all fields, so you can gate a change without parsing markdown. The top-level summary.md headline lines are also stable enough to extract with simple regexes.

Next steps

  • Debug Viewer — scrub-synced root-cause analysis for a failing clip.
  • Test Scenarios — what each scenario was designed to stress.