> This page is for version 1.4.0 (Latest) (default).
> For other versions, use one of these documentation indexes:
> - 1.4.0 (Latest) (default): https://docs.nvidia.com/halos/outside-in-safety/1.4.0/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/halos/outside-in-safety/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/halos/outside-in-safety/_mcp/server.

# Overview

The **Regression Testing Reporter** is the closed-loop regression-testing layer for the Halos Outside-In Safety closed loop. It runs a scenario in Isaac Sim, records what the Safety Core *decided* alongside what the simulation *ground truth* says it *should* have decided, aligns the two frame by frame, and produces a graded, repeatable report of the end-to-end safety decision.

## What is it?

The Regression Testing Reporter builds on top of the Software-in-the-Loop (SIL) stack and turns a simulated hazard run into a *measurable* regression test. Its module, container, and directories use the short name `srr` (the source lives under `closed-loop-testing/regression-reporter/`), so the commands throughout these guides reference `srr` directly. The short name is retained from the tool's original name, Scenario Recorder + Reporter.

SIL (see [SIL Overview](/halos/outside-in-safety/testing/sil/overview)) proves the production stack — AI Perception (VSS), Safety Core, and the communication layer — runs end to end against a simulated warehouse. The Regression Testing Reporter adds the **metric layer** on top: it scores whether the system raised and cleared the safety alarm at the **right time**, and how well perception fed that decision.

![Regression Testing Reporter architecture — Isaac Sim, Halos Outside-In Safety, and the recorder in lockstep](/halos/outside-in-safety/_fern-files/_img/7591eeee6af442c0288732361af34649a95a91797bb4a8c627044d866c8abd16.webp)

## Why the Regression Testing Reporter?

The safety decision in Halos is **emergent**: no single perception result proves the end-to-end response was correct or timely. A live demo can show the safety indicator changing, but it gives no repeatable score, does not scale across many cases, and cannot safely stage real near-misses with human workers.

The Regression Testing Reporter closes that validation gap:

* **Simulation-grounded expected decisions.** Isaac Sim provides true person and
  forklift positions and safety-zone occupancy, so the *expected* MUTE / UNMUTE state is known at every frame — no manual labeling.
* **Frame-by-frame comparison.** Regression Testing Reporter records the actual Safety Core decision on
  the same simulation timeline as ground truth, so expected and observed decisions line up exactly.
* **Repeatable and unattended.** The same scenario produces the same test every
  time, so runs can gate a change the way ordinary software tests do, and can run in CI/CD without a human in the loop.
* **Aligned evidence for debugging.** Video, detections, tracking, behavior
  events, and safety logs normally live in separate stores and clocks. The Regression Testing Reporter aligns them on one 30 Hz timeline so a mismatch can be traced to its source.

## Where the Regression Testing Reporter fits

The Regression Testing Reporter is a **layer on top of an already-deployed SIL stack** — it is not a deployment path.

> **Note**
>
> Before running Regression Testing Reporter, the SIL stack (VSS Warehouse 3.3 + the Halos `sil` profile) must be deployed and healthy. Deploy it first — see [SIL — 2D Perception](/halos/outside-in-safety/testing/sil/sil-2d) (manual) or [Agentic Deployment](/halos/outside-in-safety/getting-started/agentic-deployment) (agent-driven). The Regression Testing Reporter does **not** deploy Halos or VSS.

The relationship in one line:

* **Software-in-the-Loop (SIL)** brings up the digital twin and runs the
  production safety stack against it.
* **Regression Testing Reporter** records that loop, scores the decision against ground
  truth, and reports.

## What one run produces

A single Regression Testing Reporter run splits the recording into per-clip segments at each forklift trailer crossing (where the hazard appears) and emits:

* `summary.md` — the headline report: per-clip verdicts, decision-accuracy
  metrics, behavior-analytics event detection, and — when the 3D perception pipeline is active — per-class perception quality.
* `<scenario>/reports/scn_*.md` — per-clip detail, including the exact
  mismatch windows with timestamps.
* `<scenario>/videos/scn_*.mp4` — per-clip review videos.
* `failures.json` — the machine-readable list of FAIL + analyze-error clips,
  ready for triage.
* An optional evidence bundle for the synchronized debug viewer.

See [Interpreting the Report](/halos/outside-in-safety/testing/srr/interpret-results) for how to read each of these.

## How it works (high level)

Regression Testing Reporter evaluates one simulated hazard scenario at a time. The pipeline is:

1. **Ground-truth stream.** Isaac Sim publishes true person and forklift poses
   as `/gt/*` topics (armed at launch with the `--srr-gt` flag). See [Regression Testing Reporter Ground-Truth Graph (opt-in)](/halos/outside-in-safety/testing/sil/components/action-graph#regression-testing-reporter-ground-truth-graph-opt-in) for the Action Graph that publishes them.

2. **Recording.** The Regression Testing Reporter service records both the ground-truth stream and the
   Safety Core decision (plus perception events from Kafka) into a 30 Hz `parquet` file.

3. **Clip splitting.** The recording is split into per-clip segments around each
   forklift trailer (dock) crossing, where the hazard forms.

4. **Aggregation and scoring.** Each clip is scored on two layers:

   * **Phase 1 — the safety decision.** Expected MUTE / UNMUTE (from ground
     truth) vs the actual Safety Core state, plus reaction latency and behavior-analytics event detection.
   * **Phase 2 — perception quality.** Ground-truth positions vs the 3D
     detections and tracking feeding the Safety Core (per class: detection and tracking quality). Present when the 3D pipeline is active.

5. **Review artifacts.** Regression Testing Reporter renders the graded report, the failure list, and
   per-clip review videos, and can package an evidence bundle for the synchronized [Debug Viewer](/halos/outside-in-safety/testing/srr/debug-viewer).

## Key components

| Component                   | Role                                                                                                                                     |
| --------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------- |
| **Ground-truth publishers** | Isaac Sim `--srr-gt` action graph builds the `/gt/*` topics — the reference the decision is scored against.                              |
| **Recorder**                | Records ground truth + Safety Core state + perception Kafka events into a 30 Hz `parquet` log.                                           |
| **Clip splitter**           | Splits the recording into per-clip segments at forklift trailer crossings.                                                               |
| **Aggregator**              | Scores each clip (Phase 1 decision + Phase 2 perception) and renders the reports.                                                        |
| **Debug viewer**            | Browser tool that replays the per-clip video with all evidence streams synchronized on one timeline (optional, for root-cause analysis). |

## Next steps

* [Quick Start Guide](/halos/outside-in-safety/testing/srr/quickstart) — deploy checks and run your first regression test.
* [Architecture](/halos/outside-in-safety/testing/srr/architecture) — the pipeline and data flow in detail.
* [Test Scenarios](/halos/outside-in-safety/testing/srr/scenarios) — the pre-built test cases.
* [Interpreting the Report](/halos/outside-in-safety/testing/srr/interpret-results) — how to read the report.
* [Debug Viewer](/halos/outside-in-safety/testing/srr/debug-viewer) — root-cause analysis for failing clips.