Skip to navigation
DeepStream Plugin GuidePlugins

Gst-nvinfereval

View as Markdown

The Gst-nvinfereval plugin is a pass-through element that evaluates live model accuracy against ground-truth data. It auto-detects whether it is downstream of a single inference model (Mode A) or a Gst-nvmodelmux primary/shadow pair (Mode B), and computes AP@0.5, AP@0.5:0.95, precision, recall, and F1 per model, per class, over configurable evaluation epochs. In Mode B, it can gate automatic promotion of a shadow model to primary based on configurable KPI thresholds and a minimum AP margin over the primary. The buffer is never modified; the element only reads InferenceProvenanceMeta when present and forwards it to the evaluation engine.

Inputs and Outputs

  • Inputs

    • GStreamer buffer containing NvDsBatchMeta with InferenceProvenanceMeta (from an upstream Gst-nvinfer or Gst-nvmodelmux)

  • Control parameters

    • gt-file, config-file, eval-interval, iou-threshold, cumulative
    • min-ap-margin, kpi-ap50, kpi-precision, kpi-recall, promotion-mode (A/B promotion gating, Mode B only)
    • report-file, show-gt-overlay, show-ap-table, exclude-classes, exclude-undetected

  • Output

    • The same GStreamer buffer, unmodified, passed through to the next element
    • A GstMessage posted at each epoch boundary with the evaluation report (and optionally a JSON report file)

Features

The following table summarizes the features of the plugin.

Gst-nvinfereval plugin features

FeatureDescriptionRelease
Live accuracy evaluationComputes AP@0.5, AP@0.5:0.95, precision, recall, and F1 per model and per class against ground truth over configurable evaluation epochsDS 9.2
Mode A / Mode B auto-detectionAuto-detects whether it sits downstream of a single inference model or a Gst-nvmodelmux primary/shadow pairDS 9.2
A/B promotion gatingIn Mode B, gates automatic promotion of a shadow model to primary using a minimum AP margin and configurable KPI thresholds (AP@0.5, precision, recall)DS 9.2
Ground-truth format auto-detectionAuto-detects the ground-truth format (KITTI directory | NVSchema JSONL | COCO JSON | TAO JSON)DS 9.2
Per-model / per-class config overridesSupports per-model [model-<name>] config-file sections that override GT files, KPI thresholds, and promotion marginsDS 9.2

Gst Properties

The following table describes the Gst properties of the Gst-nvinfereval plugin.

Gst-nvinfereval plugin gst properties

PropertyMeaningType and RangeExample Notes
gt-fileDefault ground-truth file or directory for all models (format auto-detected: KITTI dir, NVSchema JSONL, COCO JSON, or TAO JSON); per-model overrides go in the config-file [model-<name>] sections; leave empty to run in no-GT relative-comparison modeStringgt-file=/path/to/gt
config-filePath to a GKeyFile INI config (see configs/config_nvinfereval.txt); keys in the file override the corresponding GObject properties, and per-model [model-<name>] sections set model-specific GT files, KPI thresholds, and promotion margins; leave empty to use GObject properties onlyStringconfig-file=config_nvinfereval.txt
eval-intervalNumber of Primary frames per evaluation epoch, counted independently per model_name; shadow frames are excluded from the count so the epoch always spans exactly this many Primary frames regardless of whether a shadow model is presentInteger,
1 to 4,294,967,295		eval-interval=1000
iou-thresholdMinimum IoU for a prediction to count as a true positive in precision/recall/F1 computation; AP@0.5 and AP@0.5:0.95 always use their fixed standard thresholdsFloat,
0.0 to 1.0		iou-threshold=0.5
min-ap-marginDefault margin by which the shadow’s AP@0.5 must exceed the primary’s AP@0.5 to trigger promotion; per-model override available in config-fileFloat,
0.0 to 1.0		min-ap-margin=0.02
kpi-ap50Default minimum AP@0.5 the shadow must reach for promotion (0.0 = disabled); per-model override available in config-fileFloat,
0.0 to 1.0		kpi-ap50=0.0
kpi-precisionDefault minimum mean precision the shadow must reach (0.0 = disabled); per-model override available in config-fileFloat,
0.0 to 1.0		kpi-precision=0.0
kpi-recallDefault minimum mean recall the shadow must reach (0.0 = disabled); per-model override available in config-fileFloat,
0.0 to 1.0		kpi-recall=0.0
promotion-modeAction taken when the shadow wins an epoch (Mode B only): 0 (promote) shadow becomes primary and A/B ends; 1 (swap) shadow becomes primary, old primary becomes shadow, A/B continues; 2 (report-only) metrics reported, no REST call, pure monitoringInteger,
0 to 2		promotion-mode=0
report-fileOptional path to write JSON epoch reports (appended); leave empty to disable file output (a GstMessage is always posted)Stringreport-file=/tmp/eval_report.jsonl
show-gt-overlayDraws ground-truth bounding boxes as green rectangles via nvosd; useful for debugging GT alignmentBooleanshow-gt-overlay=0
show-ap-tablePrints the per-class AP/precision/recall table to the console at each epoch boundaryBooleanshow-ap-table=0
exclude-classesComma-separated list of class labels to exclude from mean AP/P/R; excluded classes are still shown per-class with an [EXCLUDED] noteStringexclude-classes=unknown,Box
exclude-undetectedAutomatically excludes from mean AP/P/R any class for which the model produced zero detections (TP=0, FP=0); such classes appear in the per-class table with [UNDETECTED]Booleanexclude-undetected=1
cumulativeWhen TRUE, metrics are computed over all frames since pipeline start (accumulator never resets); when FALSE, each epoch is an independent tumbling window of eval-interval Primary framesBooleancumulative=1