Making Halos SAFE Credible and Auditable | NVIDIA Halos Safety Assurance Framework for Evaluation (SAFE)

In order to truly support an automotive safety case, all elements of the framework – from the datasets, the simulation technologies, to the KPI analyzers – need to be credible for the specific safety claim they support. UN-R ADS clause 7.2.1 and UN-R171 DCAS Annex 5 provide a useful structure for this credibility argument.

For researchers, this credibility case maps directly to the concept of a reproducible evaluation methodology: the same principle of making evaluation pipelines transparent, auditable, and replicable that underpins benchmark design in the computer vision community.

In Halos SAFE, this credibility case is organized around five evidence areas:

Foundation and scope: define the intended use of virtual testing, the relevant safety requirement, the test objective, ODD and system boundary, assumptions, limitations, fidelity expectations, validation strategy, acceptance criteria, and the criticality of potential toolchain errors for the safety argument.
Process and governance: manage the data, people, and releases behind the evidence. This includes input and output data pedigree, data quality and coverage, traceability from KPI results back to scenarios, parameters, toolchain configuration and release, personnel competency, third-party inputs, and lifecycle support.
Technical verification: demonstrate that the toolchain is implemented correctly and behaves robustly for valid inputs. Evidence includes code verification, parameter-space exploration, sanity and consistency checks, and bounded numerical errors such as discretization, rounding, and convergence effects.
Validation and uncertainty analysis: show that simulation outputs correlate with relevant physical or real-world reference data for the intended use. This includes validation metrics, goodness-of-fit criteria, calibration, sensitivity analysis, uncertainty quantification, and safety margins where uncertainty affects interpretation of results.
Final suitability assessment: synthesize the evidence and state whether the toolchain and Halos SAFE element are suitable for the defined claim, scope, release, assumptions, and limitations.

Halos SAFE is intended to make this evidence structure explicit through an overall Halos SAFE Credibility Manual and supporting manuals for individual Halos SAFE elements. These manuals map the credibility expectations above to evidence generated through established standards and practices, including ISO 26262, ISO 21448, ISO/PAS 8800, ISO 34502, ISO 34503, and ISO/IEC 17025.