Gst-nvdsvisionencoder-c

The Gst-nvdsvisionencoder-c plugin is a pure C/C++ GStreamer plugin that generates ReID (Re-Identification) embeddings for tracked objects using either Triton Inference Server or a local TensorRT engine, and attaches them to DeepStream metadata as NVDS_TRACKER_OBJ_REID_META. Compared to an equivalent Python plugin, it attaches metadata directly via C pointers (avoiding wrapper/binding issues), runs without Python overhead, and produces metadata that is correctly visible to all downstream components (for example deepstream-test5-app, Gst-nvmsgconv). It also supports embedding caching with optical-flow-based staleness prediction and an optional DLA-based appearance-change validator, both of which can skip full re-inference for objects whose appearance has not meaningfully changed.

Inputs and Outputs

Inputs
- GStreamer buffer containing NvBufSurface (NVMM) frames with tracked-object metadata
Control parameters
- enable, backend, triton-url, triton-model, tensorrt-engine, onnx-model, batch-size, min-crop-size, gpu-id, device, skip-interval, embedding-classes, verbose
- smart-infer, cache-refresh-interval, cache-max-size (embedding cache)
- ofa-predict, ofa-motion-threshold, ofa-high-motion-threshold (optical-flow-based staleness prediction)
- dla-validator, dla-validator-engine, dla-core, dla-change-threshold (DLA appearance-change validator)
Output
- The same GStreamer buffer, with NVDS_TRACKER_OBJ_REID_META (ReID embeddings) attached to each tracked object’s metadata

Features

The following table summarizes the features of the plugin.

Gst-nvdsvisionencoder-c plugin features

Feature	Description	Release
ReID embedding generation	Generates ReID embeddings for tracked objects via Triton Inference Server or a local TensorRT engine	DS 9.2
NVDS_TRACKER_OBJ_REID_META attachment	Attaches generated embeddings directly to DeepStream tracked-object metadata using native C pointers	DS 9.2
Minimum crop-size filtering	Skips embedding generation for objects smaller than min-crop-size	DS 9.2
Embedding caching	Caches per-object embeddings (smart-infer) and periodically re-infers them (cache-refresh-interval), avoiding redundant inference for already-embedded tracked objects	DS 9.2
Optical-flow staleness prediction	Uses hardware optical flow motion vectors (ofa-predict) to transform cached embeddings instead of re-inferring when object motion is below a configurable threshold	DS 9.2
DLA appearance-change validator	Gates full vision-encoder inference behind a lightweight DLA (or GPU fallback) appearance-change detector	DS 9.2

Gst Properties

The following table describes the Gst properties of the Gst-nvdsvisionencoder-c plugin.

Gst-nvdsvisionencoder-c plugin gst properties

Property	Meaning	Type and Range	Example Notes
enable	Enables vision encoding	Boolean	enable=1
backend	Inference backend to use (triton or tensorrt)	String	backend=triton
triton-url	Triton Inference Server URL	String	triton-url=localhost:8001
triton-model	Triton model name	String	triton-model=radio_clip_trt
model-variant	Model variant (for compatibility)	String	model-variant=default
tensorrt-engine	Path to a TensorRT engine (.plan) file; settable while PLAYING to reload a prebuilt engine on the fly, but the new engine must match the live one’s input resolution, embedding dimension, and batch capacity	String	tensorrt-engine=/path/to/model.plan
onnx-model	Path to an ONNX model file used to build the engine if it is not found; cold-start configuration only (setting it on a running element stores it for the next start rather than loading it — use tensorrt-engine for a runtime update)	String	onnx-model=/path/to/model.onnx
batch-size	TensorRT batch size for engine building/inference	Integer, 1 to 4,294,967,295	batch-size=16
min-crop-size	Minimum crop size (width or height) below which an object is skipped	Integer, 8 to 512	min-crop-size=32
verbose	Enables verbose logging	Boolean	verbose=0
gpu-id	GPU device ID	Integer, 0 to 7	gpu-id=0
device	Device type used for inference (cuda or cpu)	String	device=cuda
skip-interval	Frame skip interval (0 = none, 1 = every other frame, 2 = every 3rd frame, etc.)	Integer, 0 to 100	skip-interval=0
embedding-classes	Semicolon-separated list of classes to generate embeddings for; empty means all classes	String	embedding-classes=Person;Vehicle
query-only	Initializes the model for REST API query handling only and skips per-frame pipeline inference	Boolean	query-only=0
smart-infer	Skips inference for tracked objects whose embedding is already cached	Boolean	smart-infer=0
cache-refresh-interval	Re-infers cached objects every N frames; 0 means never re-infer while tracked	Integer, 0 to 10,000	cache-refresh-interval=0
cache-max-size	Maximum number of object embeddings to cache; oldest entries are evicted first	Integer, 10 to 100,000	cache-max-size=10000
ofa-predict	Uses hardware optical flow motion vectors to predict embedding staleness and transform cached embeddings instead of re-inferring	Boolean	ofa-predict=0
ofa-motion-threshold	Per-object average motion magnitude, in flow-vector units (~pixels/frame), below which the cached embedding is trusted	Float, 0.0 to 100.0	ofa-motion-threshold=5.0
ofa-high-motion-threshold	Per-object motion magnitude above which full re-inference is forced because the object’s appearance likely changed significantly	Float, 0.0 to 500.0	ofa-high-motion-threshold=20.0
dla-validator	Enables a lightweight appearance-change detector on DLA (Jetson) or GPU fallback to gate full vision-encoder inference	Boolean	dla-validator=0
dla-validator-engine	Path to the TensorRT engine for the lightweight appearance-change detector	String	dla-validator-engine=/path/to/validator.plan
dla-core	DLA core to use for the validator; -1 uses GPU fallback	Integer, -1 to 7	dla-core=-1
dla-change-threshold	Probability threshold above which the validator considers the object’s appearance changed	Float, 0.0 to 1.0	dla-change-threshold=0.5