Release Notes — Version 26.08#

September 2026#

This release covers everything that landed since 26.06, including the interim 26.07 development line.

Upgrade From 26.06

Work through these before upgrading. Each one is detailed under Breaking Changes.

  1. Rebuild any environment that runs against NRE outside the container against Python 3.13.

  2. If you pass car2sim.yaml, switch to car2sim_4cam.yaml or accept a six-camera rig.

  3. If you passed --renderer nrend, replace it with --renderer gsplat, and drop --no-enable-nrend.

  4. Rename difix configuration packages, option groups, and sqa_difix_* presets to their harmonizer equivalents.

  5. Replace model/tracks_calib: unicycle with elevation_aware_unicycle or direct.

  6. On Ampere GPUs, pass --tracks-timestamp-ray-stride 1 to keep 26.06 inference behavior.

Reconstructions trained on 26.06 are not reproduced exactly by any 26.08 preset, because the production presets also pick up new optimizer, tile-masking, and track-splitting defaults. Keep the 26.06 container to reproduce a 26.06 run.

Breaking Changes

  • Python 3.13: The container and the custom wheels move from CPython 3.11.14 to 3.13.15. Code that runs against NRE outside the shipped container, including gRPC clients built from the NuRec protos, must target Python 3.13. See :doc:Software Prerequisites <../basics/software>.

  • Six-Camera Car2Sim Is the Default: car2sim.yaml for Hyperion 8.1 now resolves to the six-camera preset instead of the previous four-camera rig. An unchanged command therefore trains a six-camera reconstruction, which raises VRAM use and training time and produces metrics that are not directly comparable to your 26.06 runs. Use the new car2sim_4cam.yaml to keep the four-camera rig, noting that it pins the rig only — both presets also pick up the 26.08 option-group changes, including the SelectiveAdam Gaussian optimizer, active tile masking, and five-second pedestrian track splitting. car2sim_6cam_lidar is replaced by car2sim_lidar, and car2sim_6cam_nrm is removed. See :doc:Customize NuRec Configuration <../nurec/configs>.

  • NRend Renderer Removed: NRE now has a single GSplat-based rendering path. The gRPC and render --renderer flag accepts only default and gsplat; the nrend choice and the deprecated --enable-nrend / --no-enable-nrend flags are gone. The *_nrend configuration presets (car2sim_nrend, car2sim_6cam_nrend, car2sim_lidarfree_nrend, sqa_default_nrend) and the configs/apps/prod/options/nrend option group are removed, as is the artifact.nrend export block. Replace --renderer nrend with --renderer gsplat and drop --no-enable-nrend from existing commands. See :doc:Use the NuRec gRPC API Server <../api/grpc_api_guide> and :doc:Use the NuRec Render Command <../nurec/render-command>.

  • Difix Configuration Packages Removed: The configs/difix package and the configs/apps/prod/options/difix option group are replaced by configs/harmonizer and configs/apps/prod/options/harmonizer. The sqa_difix_distill, sqa_difix_inference, and sqa_difix_distill_and_inference presets are renamed to their sqa_harmonizer_* equivalents. A configuration upgrade step (26.7.8 to 26.7.9) rewrites difix keys to harmonizer in older configs, so an older config still loads; update the keys anyway. The --enable-difix and --difix-* command-line flags are separate, are still accepted, and still emit deprecation warnings. See :doc:Use Harmonizer for Post-Processing <../nurec/harmonizer>.

  • Unicycle Track Calibration Replaced: The model/tracks_calib: unicycle preset and the unicycle-tracks-calib module are removed in favor of elevation_aware_unicycle (elevation-aware-unicycle-tracks-calib). No configuration upgrade rewrites the old name, and the replacement is not behavior-preserving — it starts calibration at step 500 instead of immediately and adds pitch clamping and short-gap fusion. Select elevation_aware_unicycle or direct explicitly.

  • Harmonizer Fails Fast Under Distributed Training: Harmonizer now raises at model load when the distributed world size is greater than 1, instead of failing partway through validation. Multi-GPU Harmonizer training runs that previously reached validation now fail at trainer setup. See the Known Issues entry for the workaround.

  • Asset Harvester Lifting Overrides Removed: The tokengs_lifting.use_ttt and tokengs_lifting.bbox_size overrides are no longer exposed. Remove them from existing Asset Harvester invocations.

  • Obfuscated Container Images Discontinued: The _obfuscated_run, _obfuscated_tools, and _obfuscated_nrm_run image variants are no longer built or published. Use the standard run, tools, and nrm_run images.

Deprecations

  • RGB Render RPCs Deprecated: render_rgb and batch_render_rgb are marked deprecated in the protobuf service in favor of render_camera and batch_render_camera. Both still work in this release. To match render_rgb exactly, call render_camera with render_products=[CAMERA_RENDER_PRODUCT_COLOR]. See :doc:Use the NuRec gRPC API Server <../api/grpc_api_guide>.

Core Improvements

  • CameraRender gRPC API: New render_camera and batch_render_camera RPCs replace the RGB-only render calls. A single request selects one or more render products through render_products — CAMERA_RENDER_PRODUCT_COLOR, CAMERA_RENDER_PRODUCT_DISTANCE_COLORIZED, and CAMERA_RENDER_PRODUCT_SEMANTICS_COLORIZED — and all requested products share one render pass. Responses carry one CameraRenderOutput per product, in request order, with per-product metadata such as the semantic class mapping and the distance colormap bound. Requests can also draw cuboid wireframes on every product through overlay_bounding_boxes. The reported grpc_api_version moves from 0.0.1 to 0.0.3; the change is additive, so clients that negotiate on major.minor are unaffected.

  • Multi-Process gRPC Serving: serve-grpc --multiproc-workers N runs the server as an owner process holding one read-only copy of the weights, shared to N workers over CUDA IPC behind a front proxy. New flags cover startup and admission control: --multiproc-bringup-concurrency, --multiproc-max-in-flight, --multiproc-admission-timeout-s, --multiproc-require-scene-cache, and --require-mps. The pool is validated for the gsplat renderer only, requires externally managed NVIDIA MPS for throughput scaling, and refuses asset editing and temporal-Harmonizer sessions. See :doc:Serve NuRec with Multiple Worker Processes <../api/grpc_multiprocess_serving>.

  • Temporal Harmonizer for gRPC Inference: serve-grpc --enable-harmonizer-temporal enables an autoregressive Harmonizer that conditions each frame on up to four previously enhanced frames. Clients open a session with the new create_harmonizer_session RPC, pass the session ID on each render, and close it with end_harmonizer_session; --max-harmonizer-sessions (default 32) bounds concurrent sessions. The bundled client exposes --temporal-harmonizer-session. The temporal model is inference-only and mutually exclusive with the stateless --enable-harmonizer. See :doc:Use Harmonizer for Post-Processing <../nurec/harmonizer>.

  • Artifact-Aware Backend Cache Keys: NRE records the artifact SHA256 in backend cache keys, so newly downloaded bytes cannot reuse a stale loaded backend when different content is published at the same artifact URL. On-demand scene registration through the x-nre-scene-url and x-nre-scene-id metadata headers, which was already available on the single-process server, is now supported by the multi-process server as well. See :doc:Use the NuRec gRPC API Server <../api/grpc_api_guide>.

  • Scene Cache Introspection and Limits: The new get_loaded_scenes RPC returns a point-in-time snapshot of cached scene instances, with per-scene loaded and reusable instance counts plus the global cache capacity. --max-backends-per-scene caps how many cached backends a single scene may hold; requests that reach the limit wait for and reuse a checked-in backend instead of building another.

  • Compact USDZ Artifacts: Optional Gaussian tensor compression writes a smaller .compact.usdz artifact. Enable it with artifact.compression.enabled (default: disabled, level 9, opacity threshold 0.005, with an exclude list). The artifact reader detects and loads compact artifacts automatically, and uncompressed artifacts are unchanged.

  • Inference Ray-Stride Tuning: The new --tracks-timestamp-ray-stride flag on render and serve-grpc subsamples the per-track rolling-shutter timestamp-estimation ray grid at inference time. Left unset it resolves to 16 on Ampere GPUs (compute capability 8.0, 8.6, and 8.7), where that kernel dominates a 4K frame, and to 1 everywhere else. Because the resolved timestamps differ slightly from stride 1, renders on an Ampere GPU are not bit-identical to 26.06 output; pass --tracks-timestamp-ray-stride 1 to restore the previous behavior and cost. See :doc:Use the NuRec Render Command <../nurec/render-command>.

  • Selective Tile Rendering: A new model.renderer.active_tile_mask option skips rendering tiles that are entirely invalid, which reduces training and inference time. It is off in the base schema, and several Hyperion 8.1 presets now select it. Masking changes what is supervised and optimized, not only what is rendered, so read the strategy table before enabling it on a custom configuration. See :doc:Customize NuRec Configuration <../nurec/configs>.

  • Parallel-Batch GSplat Rasterization: A new model.renderer.renderer_config option parallelizes both the forward and the backward rasterization pass across the training batch, reducing per-step time. It is now the default for 3DGUT configurations and requires the 3DGUT path. See :doc:Customize NuRec Configuration <../nurec/configs>.

  • Component-Only Training: A new component_only configuration overlay trains a subset of model layers and leaves the rest frozen. Six presets ship under configs/component_only, covering background-only, deformables-only, and combined variants with optional calibration and post-processing.

  • Track Calibration Options: Direct track calibration can now operate in local coordinate space through delta_frame: local, with world preserved for older checkpoints. New reusable presets include direct_lr1e-4 and elevation_aware_unicycle, exposed for Hyperion 8.1 as the dynamic_rigids_track_calibration option group. The default six-camera preset selects direct.

  • Pedestrian Visible Track Splitting: A new visible_track_split option group splits long pedestrian tracks by visible duration. The pedestrians_5s preset caps visible duration at 5 seconds for pedestrian and person classes and is selected by the production Hyperion 8.1 presets; disabled turns it off.

  • Road-Quality Metrics: eval-rendering-metrics gains four road sub-metrics selectable through --metrics: ref (road edge fidelity), rlcf (road local contrast fidelity), lm (lane-marking IoU), and rmtf (road multi-scale texture fidelity). Pass per-frame road masks with the new --road-masks-dir flag.

  • Object-Level Occlusion Estimation: Object-level rendering metrics now compute per-object occlusion ratios, reported as occlusion_gt and occlusion_ren. The new --occlusion-filter-threshold flag drops heavily occluded objects from the aggregate and reports occlusion_filtered.

  • NCore V4 Dataset Support: Object-level rendering metrics accept an NCore V4 dataset through the new --dataset-path flag as an alternative to --shard-pattern.

  • Ground-Mesh Export and Evaluation: A new export-camera-ground-mesh command exports a camera-derived ground mesh from four flags: --input, --output, --road-class-id (repeatable, default 0), and --postprocess/--no-postprocess. eval-ground-mesh adds cuboid-coverage and point-cloud hull-area KPIs behind the new --cuboid-tracks-path, --point-cloud-path, --point-cloud-margin, --point-cloud-max-range, and --hull-grid-resolution flags, and moves its diagnostic output into a diagnostics/ subdirectory so metrics files stay at the output root.

  • Point-Cloud Export Options: export-point-cloud adds --include-dynamic-points to keep points flagged dynamic, and --coordinate-system to choose between the scene-centered nre frame (default) and the world frame.

  • Heterogeneous Multi-LiDAR Reconstruction: Structured multi-LiDAR reconstruction supports per-sensor nominal model overrides keyed by lidar_id, so rigs that mix LiDAR types can be reconstructed together.

  • NRM Improvements: NRM resolves its PLY initialization path automatically from the NCore dataset layout when no explicit path is set, consumes NCore JSON paths directly, and gains a keep_movable_gaussians option (default: disabled) to retain MOVABLE Gaussians in LiDAR PLY export.

  • Per-Class Binary Mask Export: The NCore diagnostic tool adds --per-class-binary-masks to export one binary mask per requested semantic class.

  • Coverage Frame Sampler: A new dataset/samplers: coverage_image_crop option selects a coverage-based frame sampler, which can be combined with the existing hold-out split to validate on frames the model never trained on.

  • Asset Harvester Output Layout Documented: The Asset Harvester output contract is now written down rather than inferred: metadata.yaml plus one <label_class>/<track_id>/ directory per asset, each containing gaussians.ply and 3d_lifted.mp4, with assets.<track_id>.ply_file in metadata.yaml naming the PLY path. The layout itself is unchanged from 26.06. See :doc:Use Asset Harvester Output in Reconstructions <../nurec/use-ah-assets>.

  • gRPC Client Robustness: The bundled render-grpc client adds --wait-server to block until the server is reachable instead of relying on a fixed delay, --max-concurrent-renders to bound its own fan-out against an admission-limited server, and --repeat-count. It also gains the camera-product flags --output-type, --overlay-bboxes, --bbox-color, and --colorization-max-distance.

  • GSplat Kernel Migration: Fused SSIM, road Gaussian, deformation-smoothness, semantic cross-entropy, and grouped BCE losses now run through GSplat and CUDA dispatch kernels, and the MCMC strategy uses the GSplat stateless owner API. Gaussian topology mutation schedules are exposed, and the CUDA cache purge during mutation is configurable.

  • Distributed Training Scalability: Multi-GPU synchronization overhead is reduced. Gaussian count metadata is coalesced into a single all-gather instead of one per layer, backward gradient reduce-scatters are fused into one call, visibility index computation reuses the fused gather counts rather than issuing separate per-layer collectives, and pre-backward CUDA synchronization is avoided when computing training PSNR.

  • Newer Runtime Dependencies: NCore moves from 19.1.1 to 19.5.1 (19.5.4 internal), the bundled FFmpeg in the OpenCV and imageio wheels moves from 8.1.1 to 9.0.1, PyTorch Lightning moves from 2.4.0 to 2.6.5, and Transformers moves from 4.48.3 to 5.13.1. The pycuda and pytorch3d dependencies are dropped. System cuDNN is removed from the dev and run base images and is now supplied through the PyTorch wheels.

  • GB300 (Blackwell Ultra) Runtime Compatibility: NRE adds runtime support for GB300 Grace-Blackwell systems. Because CUDA 12.8 cannot emit SM 10.3 code, the custom wheels ship SM 10.0 native kernels plus embedded compute_100 PTX for forward compatibility, and an LD_PRELOAD shim remaps the reported SM 10.3 capability to SM 10.0 for PyTorch, Triton, and NVRTC. The same shim covers DGX Spark and GB10 (SM 12.1 to SM 12.0). GB300-specific runtime fixes align DAV3 LayerNorm numerics with the reference implementation. This is Grace-Blackwell enablement on aarch64, not general Linux aarch64 support — see the platform note under Known Issues.

Bug Fixes

  • Multi-GPU SelectiveAdam Correctness: Fixed a multi-GPU training bug where an owner rank skipped the optimizer update for Gaussians that were visible only on a different rank, even though the owner received a nonzero reduced gradient for them. Visibility is now unioned across ranks before the optimizer step. This affected model quality, not only training speed.

  • LiDAR RGB Projection: Fixed camera-valid LiDAR RGB projection in NCore data loading.

  • LiDAR Metrics: Corrected LiDAR metric computation, fixed sensor routing so metrics are attributed to the right LiDAR, and fixed the NRM initialization override used by LiDAR metrics.

  • LiDAR Rendering Origin: Fixed render-grpc rendering LiDAR at the rig origin instead of the sensor pose.

  • Cuboid Track Poses Under TF32: Fixed frame conversion corrupting cuboid track poses when TF32 is enabled.

  • Rigid Track Clamping: Per-track cadence metadata is now precomputed for rigid track clamping, reducing repeated recomputation.

  • Dropped Boundary Objects: Fixed obstacles being dropped in the first frame and rigid objects being dropped in the last frame, and guarded object-metric pose interpolation at clip boundaries.

  • Ground Mesh with Rig Offset: Fixed ground-mesh reconstruction when the rig has a non-zero z_offset, and included z_offset in the reconstruction settings.

  • Track Calibration Device Placement: Fixed unicycle and static track calibration mixing devices when tracks live on the GPU, and made StaticTracksCalib means deterministic.

  • Scene Cache Keying: Scene caches are keyed by both scene ID and URL, so two artifacts registered under one ID no longer collide.

  • Composite Radiance Scaling: Fixed Gaussian USD radiance scaling for composite models, and fixed GRT linear radiance when post-processing is disabled.

  • Empty Loss Handling: Fixed empty fused-loss handling, and guarded node-semantic and background-in-track reference losses against masked-out non-finite logits.

  • Validation Render Synchronization: Selected-node DDP renders are synchronized so ranks produce symmetric validation output.

  • Progressive Artifact Levels: Progressive levels are restored when loading an artifact.

  • Auxiliary Data Generation: Fixed a squeeze on single-point camera frames during auxiliary data generation, and fixed auxiliary shard discovery to use sequence_id as the file prefix.

  • Semantic Auxiliary Access: Guarded semantic auxiliary access during validation when auxiliary data is disabled.

  • Gaussian Seeding and Relocation: Gaussian initialization is seeded from every training LiDAR rather than only the first, optimizer moments are reset for relocated Gaussians, and a fused point-cloud scene extent is used as a fallback.

  • Range Request Fallback: Fixed the S3 range probe when the server does not advertise Accept-Ranges.

  • Asset Harvester Symlinks: Asset Harvester resolves Hugging Face blob symlinks correctly when locating cached weights.

  • Checkpoint Completeness: Post-batch model state is saved in final checkpoints, and NRM mounted model cache compatibility is restored.

Known Issues

  • Harmonizer does not run under distributed (multi-GPU) training: Harmonizer now fails fast instead of producing a device-mismatch error partway through. When a distributed process group is initialized with a world size greater than 1, loading the Harmonizer model raises Harmonizer does not support multi-GPU execution (world_size=N). Its TorchScript model is not multi-GPU-safe because of a shared model-cache download race and device-baked constants; 26.08 guards this condition rather than fixing the underlying model. The check keys on the distributed world size, not on how many GPUs are visible, and it runs when the model loads — at trainer setup for Harmonizer training, and at the first validation epoch for Harmonizer inference during a training run. Workaround: run Harmonizer training with a world size of 1, for example by restricting CUDA_VISIBLE_DEVICES, or apply Harmonizer at inference through the gRPC server. serve-grpc never initializes a process group, so --enable-harmonizer works on a multi-GPU host with no need to restrict visible GPUs. This replaces the 26.06 known issue describing the same limitation as a validation-time CUDA error. See :doc:Use Harmonizer for Post-Processing <../nurec/harmonizer>.

  • Temporal Harmonizer restrictions: The temporal Harmonizer is inference-only and is not available for training. It cannot be combined with the stateless --enable-harmonizer, with multi-process serving (--multiproc-workers > 0), or with the batch render RPCs — send a session’s frames one at a time, in trajectory order, awaiting each render. Component-only masks that need per-pixel semantics are also rejected for Harmonizer novel-view training.

  • Asset editing is single-process only: Under multi-process serving, edit_assets returns success=False with an UNIMPLEMENTED message and restore_model_parameters aborts with gRPC status UNIMPLEMENTED. Use a single-process serve-grpc for in-place asset edits. Per-request dynamic-actor re-posing through --enable-editing-actors works in both modes.

  • Listed scenes may not be renderable under multi-process serving: Unlike the single-process server, a multi-process get_available_scenes entry may be registered but not yet attached or validated, so a listed scene’s first render can return NOT_FOUND or UNAVAILABLE. Use get_loaded_scenes as the render-readiness check.

  • gRPC asset replacement rendering: In some gRPC asset-edit scenarios, an inserted asset may not visually replace the target scene vehicle in rendered output (for example, a grey van edit on Hyperion 8.1 scenes). Reported on RTX PRO 6000. This issue is carried over from 26.06.

  • General Linux aarch64 is not a supported platform: The supported user platform for NuRec remains Linux x86_64. The GB300 and DGX Spark enablement described under Core Improvements is targeted Grace-Blackwell support, not general aarch64 support, and it carries these caveats. It reaches you only through the -arm64 or -multiarch container tags; pulling an unsuffixed tag on an aarch64 host gives you the x86_64 image, which does not include the compatibility shim. The aarch64 and multi-architecture image builds run as non-gating nightly or on-demand CI jobs, so those tags can lag the x86_64 ones, and container security scanning covers the x86_64 image only. See :doc:Hardware Setup and Requirements <../basics/hardware>.