Control How NuRec Classifies Dynamic Object Tracks#

NCore object tracks describe where an object was at each point in time, but they do not record whether that object was moving. NuRec derives that itself when it ingests a dataset, and then uses the result to decide which objects to reconstruct separately from the static background.

This guide explains how that decision is made, which configuration parameters control it, and how to verify the settings a particular run actually used.

Two Separate Decisions#

The most common source of confusion is that a track passing NuRec’s motion test does not guarantee that NuRec reconstructs it as a movable object. Two independent checks apply, in order:

Dynamic classification runs at ingest and marks each track as dynamic or non-dynamic, based on its label class and how far it moved.
Model-layer routing runs when the model is built and decides which layer each track belongs to, based on the layer’s own label list and the dynamic flag from step 1.

A track must pass both to be reconstructed as a dynamic object. A track that passes the first but fails the second is not quietly folded into the background — see Why a Dynamic Track Might Not Be Reconstructed as a Dynamic Object.

What NuRec Reads from NCore#

For each cuboid track observation, NuRec uses:

The track ID, which links observations of the same object over time.
The class_id label, a free-form string that comes from whatever ontology produced the labels.
Timestamped poses and bounding boxes.
Label source metadata identifying which labeling process produced the observation.

NCore does not store an object-level dynamic classification for NuRec to consume, so NuRec computes one during ingest.

Tracks with only a single observation are discarded while tracks are assembled. NuRec cannot infer motion from one pose, so these tracks are not carried forward as non-dynamic — they are removed entirely and do not appear in the reconstruction.

Two other parameters can change how many observations a track keeps, and therefore whether it survives at all.

Parameter	Default	Description
`dataset.cuboid_tracks_params.track_min_centroid_rig_dist_m`	`3.0`	Observations whose cuboid centroid is closer than this to the ego rig are skipped, in meters.
`dataset.cuboid_tracks_params.track_extrapolate`	`true`	Adds one synthetic pose before the first and after the last observation.

The centroid check exists to discard the ego vehicle labeling itself. It runs per observation during consolidation, so it can silently thin a track that passes close to the ego vehicle. If it removes enough observations to leave fewer than two, the track is dropped entirely by the rule above.

Note

Extrapolation is applied after the single-observation check and before classification, and the synthetic poses are included in the motion metrics. A track therefore travels slightly farther than its labeled observations alone suggest, which makes it marginally easier to pass the displacement and path-length thresholds. Set track_extrapolate=false to evaluate motion strictly over labeled observations.

Classes That Are Always Dynamic#

NuRec seeds classification with a fixed list of classes that are treated as dynamic whenever the track has at least two poses, regardless of how far the object actually moved. This keeps stationary pedestrians and cyclists out of the static background.

The list contains the following eleven labels:

pedestrian, stroller, person, person_group, rider, bicycle_with_rider, bicycle, CYCLIST, motorcycle, motorcycle_with_rider, cycle

Important

Matching is exact and case-sensitive. CYCLIST is on the list but cyclist and Cyclist are not, and pedestrian is on the list but Pedestrian is not. If your ontology uses different capitalization, its tracks are classified by the motion test alone.

This list is built into NuRec and is not configurable. There is no corresponding list of classes that are always treated as non-dynamic; every other track is decided by the motion test.

The Motion Test#

Tracks that are not in the always-dynamic list become dynamic only if they pass a motion test. NuRec offers two modes, selected by use_displacement_and_distance.

Parameter	Default	Description
`dataset.cuboid_tracks_params.use_displacement_and_distance`	`true`	Selects the mode. `true` uses displacement and path length; `false` uses speed.
`dataset.cuboid_tracks_params.track_min_displacement_m`	`1.0`	Straight-line distance between the first and last pose, in meters.
`dataset.cuboid_tracks_params.track_min_distance_m`	`1.0`	Total path length travelled, in meters.
`dataset.cuboid_tracks_params.track_speed_reduction_op`	`median`	How per-pose speeds are reduced to one value. Accepts `median` or `max`.
`dataset.cuboid_tracks_params.track_min_speed_ms`	`0.1`	Speed threshold in meters per second.

In the default displacement mode, a track is dynamic if either its displacement or its total path length exceeds its threshold. The distinction matters for objects that return to where they started: a car that pulls out and parks again nearby has small displacement but large path length, and the path length check catches it.

In speed mode, NuRec reduces the pose-inferred speeds using track_speed_reduction_op and compares the result against track_min_speed_ms.

All comparisons are strictly greater-than, so a track that moves exactly 1.0 m does not pass the default displacement check.

The motion test is combined with the always-dynamic class list using a logical OR. A track whose class is on that list is dynamic even if it never moves.

Limit Motion Evaluation to Camera-Visible Intervals#

By default NuRec evaluates motion across every pose in a track. Set dataset.cuboid_tracks_params.camera_visibility to true to evaluate motion only during the intervals when the object was visible to a camera. This prevents an object that was stationary on camera but moved while out of view from being classified as dynamic.

Parameter	Default	Description
`dataset.cuboid_tracks_params.camera_visibility`	`false`	Restrict motion evaluation to camera-visible time intervals.

Note

If this option is enabled but a track has no camera-visible interval at all — for example, an object that stays behind the ego vehicle — NuRec falls back to using all of the track’s timestamps. Without this fallback, every such object would be forced static.

Split Long Pedestrian Tracks by Visible Interval#

A single pedestrian track can span a long recording, drifting in and out of camera view. Reconstructing it as one object across the whole sequence can hurt quality. Visible-track splitting divides selected tracks into shorter child tracks at gaps in camera visibility.

This feature is disabled by default.

Parameter	Default	Description
`dataset.cuboid_tracks_params.visible_track_split.enabled`	`false`	Enables splitting.
`dataset.cuboid_tracks_params.visible_track_split.target_label_classes`	`null`	Label classes eligible for splitting. Required when enabled.
`dataset.cuboid_tracks_params.visible_track_split.max_visible_duration_s`	`10.0`	Maximum camera-visible duration for each child track, in seconds.
`dataset.cuboid_tracks_params.visible_track_split.merge_gap_s`	`0.5`	Visible intervals separated by at most this gap are merged before splitting.
`dataset.cuboid_tracks_params.visible_track_split.min_child_visible_duration_s`	`3.0`	A final segment shorter than this is merged into the previous segment of the same visible interval, in seconds.
`dataset.cuboid_tracks_params.visible_track_split.min_track_distance_m`	`5.0`	Only tracks that travel more than this distance are split.
`dataset.cuboid_tracks_params.visible_track_split.camera_ids`	`null`	Cameras used for the visibility check. `null` means the training cameras.
`dataset.cuboid_tracks_params.visible_track_split.visibility_subsample`	`4`	Pixel subsampling factor for the visibility check.

Splitting runs after dynamic classification, so child tracks inherit the dynamic status of the track they came from. It is intended for pedestrian and person classes, and is not supported for generated traffic-light or static-rigid tracks.

Note

Enabling this feature without setting target_label_classes is an error, as is setting camera_ids to an empty list or naming a camera ID that the dataset does not contain.

Why a Dynamic Track Might Not Be Reconstructed as a Dynamic Object#

Dynamic classification only sets a flag on the track. Whether that track becomes a reconstructed object depends on model-layer routing, which each layer controls independently through two filters:

label_classes — the exact, case-sensitive list of class labels the layer accepts.
is_dynamic — whether the layer takes tracks that are flagged dynamic or tracks that are not.

A track must satisfy both filters to be included on a layer. The base dynamic model configuration defines two layers that include tracks, dynamic_rigids and dynamic_deformables, both requiring is_dynamic: true. Their label lists are long, so they are collapsed here for reference.

What Happens to Tracks That Aren’t Included on a Layer#

A track that isn’t included on a layer is not automatically absorbed into the background. Its fate depends on the dynamic flag, because the background layer’s point cloud is filtered by non_dynamic_points_only, which the shipped dynamic configurations enable for the background layer:

Non-dynamic tracks stay in the background point cloud. Their lidar points survive the filter, so a stationary object is reconstructed as ordinary background geometry.
Dynamic tracks are orphaned. The filter removes points inside dynamic track boxes from the background, so a dynamic track that isn’t included on a layer is excluded from the background and still not modeled as an object. It disappears rather than smearing into the scene, which typically leaves a gap along its path.

Production presets can add layers beyond the base two. Notably, a static_rigids layer reuses the rigid label list with is_dynamic: false:

model:
  layers:
    static_rigids:
      tracks:
        label_classes: ${model.layers.dynamic_rigids.tracks.label_classes}
        is_dynamic: false

With that layer present, a parked Car is included on static_rigids and reconstructed as a static track object with its own Gaussians instead of remaining plain background. Check the layers in your resolved configuration rather than assuming the base two.

Worked Examples#

These three cases show how the two decisions combine, assuming the base dynamic configuration:

Track	Flagged dynamic?	Included on a layer?	Result
Parked car labeled `Car`	No — fails motion test	No — layers require `is_dynamic: true`	Background geometry
Moving car labeled `car`	Yes — passes motion test	No — list has `Car`, not `car`	Orphaned
Pedestrian in one frame	Not evaluated — track dropped	Not reached	Absent from the scene

The parked Car is the expected outcome. Its label is in the rigid layer’s list, but the layer also requires is_dynamic: true, and a stationary vehicle never passes the motion test. Because the track is not dynamic, its points remain in the background. Under a preset that adds static_rigids, the same track becomes a static track object instead.

The moving lowercase car is the case that surprises people. The track is correctly flagged dynamic, but it isn’t included on either layer because the label comparison is case-sensitive. Being dynamic, its points are also filtered out of the background, so the vehicle is dropped from the reconstruction entirely. Adding car to the layer’s label_classes fixes it.

The single-frame pedestrian is discarded during track assembly, before classification runs. Being on the always-dynamic class list does not help, because that list is consulted only for tracks with at least two poses.

Use a Custom Ontology#

If your labels come from an ontology whose class names differ from the shipped lists, add your names to the relevant layer’s label_classes.

Warning

Overriding a list replaces it. It does not extend it. If you set label_classes to only your own labels, every default label is dropped and tracks carrying those labels stop being reconstructed as dynamic objects.

Restate the defaults you still need alongside your additions:

model:
  layers:
    dynamic_rigids:
      tracks:
        label_classes: ["automobile", "Car", "vehicle", "car", "my_ontology_truck"]

Because the always-dynamic class list is not configurable, custom pedestrian-equivalent labels are classified by the motion test alone. Consider whether your thresholds are low enough to catch slow-moving people, and add those labels to dynamic_deformables so they route correctly once flagged.

Verify the Settings a Run Actually Used#

The values in this guide are the defaults shipped with the NCore dataset configuration. Application configurations override them, so do not assume a given run used them.

Every run writes its fully resolved configuration to parsed.yaml inside the output directory:

<OUTPUT_DIR>/<RUN-ID>/config/parsed.yaml

<RUN-ID> is generated automatically for each run unless you set it explicitly, so expect a directory with a generated name between your output directory and config/. To make the path predictable, pass logger.run_id=<name> when you launch the run.

Inspect these keys to confirm the behavior described in this guide:

dataset.cuboid_tracks_params for the thresholds, mode, label-source filter, camera-visibility setting, splitting options, extrapolation, and the ego-centroid distance.
model.layers for the full list of layers. Presets can add layers that include tracks, such as static_rigids, beyond the base two.
model.layers.<layer>.tracks for each layer’s label_classes list and is_dynamic value.
model.layers.background.initialization.non_dynamic_points_only for whether dynamic track points are excluded from the background.

Inspect Which Tracks Were Classified Dynamic#

Checking parsed.yaml tells you the rules a run used, not the outcome. Training logs do not report how many tracks ended up dynamic, so when an object is missing or reconstructed in the wrong layer, inspect the classified tracks directly.

Every run exports its tracks into the USDZ artifact by default. Open artifacts/last.usdz as a zip archive and read datasource_summary.json, which holds both the complete track set and the dynamic subset, each entry carrying its track ID, label class, and flags.

To export the same data as standalone JSON, pass the export-sequence-tracks command instead of the usual training arguments:

docker run --shm-size=64g --rm --gpus all \
  -e NGC_API_KEY=${NGC_API_KEY} \
  --volume <DATASET_DIR>:/workdir/dataset \
  --volume <OUTPUT_DIR>:/workdir/output \
  nvcr.io/nvidia/nre/nre-ga:26.04 \
  export-sequence-tracks \
  --config-name=/workdir/output/<RUN-ID>/config/parsed.yaml \
  --output-dir=/workdir/output/tracks \
  --all-tracks

Use --all-tracks to export every track, or --dynamic-only to export just the tracks NuRec flagged dynamic. Comparing the two is the quickest way to confirm how a specific object was classified.

For a per-track table, the eval-ground-mesh command writes diagnostics/cuboid_stats.csv with one row per track, including track_id, category, and is_dynamic.

Note

Exported tracks can also carry a CONTROLLABLE flag, which the model adds for tracks it can pose at render time. It is separate from the dynamic classification described here, so read the is_dynamic value rather than assuming a single flag field means dynamic.

Additional Information#

These settings are optional. Most runs work with their defaults.

Select Which Label Sources to Use#

Most datasets need no change here: the default of null accepts the single label source present in the dataset. Set this parameter only when a dataset carries labels from more than one source.

NuRec applies the filter to observations before classification runs, using dataset.cuboid_tracks_params.track_label_sources.

Value	Behavior
`null` (default)	Accepts the single label source present in the dataset.
A list of sources	Accepts only the listed sources.

Valid source names are AUTOLABEL, EXTERNAL, GT_SYNTHETIC, and GT_ANNOTATION, spelled in uppercase. Each entry accepts an optional @<source-version> suffix to pin a specific version, such as AUTOLABEL@camera-DCP. Omitting the suffix accepts any version of that source, which is also what @any means. Pinning a version excludes observations that carry no version at all.

Important

The default of null works only when the dataset contains exactly one label source. If NuRec finds observations from more than one source and no explicit list is configured, ingest fails with an error naming the sources it found. Set the parameter explicitly in that case:

dataset.cuboid_tracks_params.track_label_sources=[AUTOLABEL]

Generated Static Rigid Tracks#

Non-dynamic tracks are ordinary NCore tracks that failed the motion test. Nothing generates them, and a parked car reaches a static_rigids layer through the same routing filters as any other track.

Separately, NuRec can generate cuboid tracks for static roadside objects that NCore does not label as tracks, such as traffic lights and signs. This is disabled by default.

Parameter	Default	Description
`dataset.generate_static_rigid_cuboid_tracks.enabled`	`false`	Enables generation of static rigid tracks.
`dataset.generate_static_rigid_cuboid_tracks.rigid_classes`	`["traffic light"]`	Classes to generate tracks for.
`dataset.generate_static_rigid_cuboid_tracks.max_distance_m`	`100.0`	Maximum distance at which to generate tracks, in meters.
`dataset.generate_static_rigid_cuboid_tracks.visibility_check_camera_ids`	`[camera_front_wide_120fov]`	Cameras used to check that the object is visible.
`dataset.generate_static_rigid_cuboid_tracks.point_cloud_path`	`null`	Accumulated point cloud with segment IDs. Required for lidar-free pipelines.

These generated tracks are the “generated traffic-light or static-rigid tracks” excluded from visible-track splitting.

Lidar Dynamic Point Association#

NuRec associates lidar points with dynamic objects using the classification described in this guide. The method is selected by dataset.lidar_dynamic_points.method.

Value	Description
`dynamic_tracks` (default)	Classifies points at load time using tracks that NuRec classified as dynamic.
`dynamic_flag`	Uses a precomputed per-point flag stored in NCore.

Note

Use the default. The dynamic_flag values are precomputed by a separate process and may disagree with NuRec’s own track classification, producing results inconsistent with the rest of this guide. Selecting dynamic_flag for a dataset that has no such flag data fails with an error naming the lidar and frame.

Next Steps#

Review Customize NuRec Configuration for how to pass and override configuration parameters.
See Prepare Data for Use with NuRec for converting data into the NCore format.
Consult the NCore documentation for the cuboid track schema and label conventions.

To view all available arguments for training, validation, and export utilities, run the following:

docker run --shm-size=64g --rm --gpus all \
  -e NGC_API_KEY=${NGC_API_KEY} \
  nvcr.io/nvidia/nre/nre-ga:26.04 --help

To get help for a specific export utility:

docker run --shm-size=64g --rm --gpus all \
  -e NGC_API_KEY=${NGC_API_KEY} \
  nvcr.io/nvidia/nre/nre-ga:26.04 \
  <EXPORT_UTILITY_NAME> \
  --help