Release Notes — Version 26.08#
September 2026#
This release covers everything that landed since 26.06, including the interim 26.07 development line.
Upgrade From 26.06
Work through these before upgrading. Each one is detailed under Breaking Changes.
Rebuild any environment that runs against NRE outside the container against Python 3.13.
If you pass
car2sim.yaml, switch tocar2sim_4cam.yamlor accept a six-camera rig.If you passed
--renderer nrend, replace it with--renderer gsplat, and drop--no-enable-nrend.Rename
difixconfiguration packages, option groups, andsqa_difix_*presets to theirharmonizerequivalents.Replace
model/tracks_calib: unicyclewithelevation_aware_unicycleordirect.On Ampere GPUs, pass
--tracks-timestamp-ray-stride 1to keep 26.06 inference behavior.
Reconstructions trained on 26.06 are not reproduced exactly by any 26.08 preset, because the production presets also pick up new optimizer, tile-masking, and track-splitting defaults. Keep the 26.06 container to reproduce a 26.06 run.
Breaking Changes
Python 3.13: The container and the custom wheels move from CPython 3.11.14 to 3.13.15. Code that runs against NRE outside the shipped container, including gRPC clients built from the NuRec protos, must target Python 3.13. See :doc:
Software Prerequisites <../basics/software>.Six-Camera Car2Sim Is the Default:
car2sim.yamlfor Hyperion 8.1 now resolves to the six-camera preset instead of the previous four-camera rig. An unchanged command therefore trains a six-camera reconstruction, which raises VRAM use and training time and produces metrics that are not directly comparable to your 26.06 runs. Use the newcar2sim_4cam.yamlto keep the four-camera rig, noting that it pins the rig only — both presets also pick up the 26.08 option-group changes, including the SelectiveAdam Gaussian optimizer, active tile masking, and five-second pedestrian track splitting.car2sim_6cam_lidaris replaced bycar2sim_lidar, andcar2sim_6cam_nrmis removed. See :doc:Customize NuRec Configuration <../nurec/configs>.NRend Renderer Removed: NRE now has a single GSplat-based rendering path. The gRPC and
render--rendererflag accepts onlydefaultandgsplat; thenrendchoice and the deprecated--enable-nrend/--no-enable-nrendflags are gone. The*_nrendconfiguration presets (car2sim_nrend,car2sim_6cam_nrend,car2sim_lidarfree_nrend,sqa_default_nrend) and theconfigs/apps/prod/options/nrendoption group are removed, as is theartifact.nrendexport block. Replace--renderer nrendwith--renderer gsplatand drop--no-enable-nrendfrom existing commands. See :doc:Use the NuRec gRPC API Server <../api/grpc_api_guide>and :doc:Use the NuRec Render Command <../nurec/render-command>.Difix Configuration Packages Removed: The
configs/difixpackage and theconfigs/apps/prod/options/difixoption group are replaced byconfigs/harmonizerandconfigs/apps/prod/options/harmonizer. Thesqa_difix_distill,sqa_difix_inference, andsqa_difix_distill_and_inferencepresets are renamed to theirsqa_harmonizer_*equivalents. A configuration upgrade step (26.7.8to26.7.9) rewritesdifixkeys toharmonizerin older configs, so an older config still loads; update the keys anyway. The--enable-difixand--difix-*command-line flags are separate, are still accepted, and still emit deprecation warnings. See :doc:Use Harmonizer for Post-Processing <../nurec/harmonizer>.Unicycle Track Calibration Replaced: The
model/tracks_calib: unicyclepreset and theunicycle-tracks-calibmodule are removed in favor ofelevation_aware_unicycle(elevation-aware-unicycle-tracks-calib). No configuration upgrade rewrites the old name, and the replacement is not behavior-preserving — it starts calibration at step 500 instead of immediately and adds pitch clamping and short-gap fusion. Selectelevation_aware_unicycleordirectexplicitly.Harmonizer Fails Fast Under Distributed Training: Harmonizer now raises at model load when the distributed world size is greater than 1, instead of failing partway through validation. Multi-GPU Harmonizer training runs that previously reached validation now fail at trainer setup. See the Known Issues entry for the workaround.
Asset Harvester Lifting Overrides Removed: The
tokengs_lifting.use_tttandtokengs_lifting.bbox_sizeoverrides are no longer exposed. Remove them from existing Asset Harvester invocations.Obfuscated Container Images Discontinued: The
_obfuscated_run,_obfuscated_tools, and_obfuscated_nrm_runimage variants are no longer built or published. Use the standardrun,tools, andnrm_runimages.
Deprecations
RGB Render RPCs Deprecated:
render_rgbandbatch_render_rgbare marked deprecated in the protobuf service in favor ofrender_cameraandbatch_render_camera. Both still work in this release. To matchrender_rgbexactly, callrender_camerawithrender_products=[CAMERA_RENDER_PRODUCT_COLOR]. See :doc:Use the NuRec gRPC API Server <../api/grpc_api_guide>.
Core Improvements
CameraRender gRPC API: New
render_cameraandbatch_render_cameraRPCs replace the RGB-only render calls. A single request selects one or more render products throughrender_products—CAMERA_RENDER_PRODUCT_COLOR,CAMERA_RENDER_PRODUCT_DISTANCE_COLORIZED, andCAMERA_RENDER_PRODUCT_SEMANTICS_COLORIZED— and all requested products share one render pass. Responses carry oneCameraRenderOutputper product, in request order, with per-product metadata such as the semantic class mapping and the distance colormap bound. Requests can also draw cuboid wireframes on every product throughoverlay_bounding_boxes. The reportedgrpc_api_versionmoves from0.0.1to0.0.3; the change is additive, so clients that negotiate onmajor.minorare unaffected.Multi-Process gRPC Serving:
serve-grpc --multiproc-workers Nruns the server as an owner process holding one read-only copy of the weights, shared to N workers over CUDA IPC behind a front proxy. New flags cover startup and admission control:--multiproc-bringup-concurrency,--multiproc-max-in-flight,--multiproc-admission-timeout-s,--multiproc-require-scene-cache, and--require-mps. The pool is validated for thegsplatrenderer only, requires externally managed NVIDIA MPS for throughput scaling, and refuses asset editing and temporal-Harmonizer sessions. See :doc:Serve NuRec with Multiple Worker Processes <../api/grpc_multiprocess_serving>.Temporal Harmonizer for gRPC Inference:
serve-grpc --enable-harmonizer-temporalenables an autoregressive Harmonizer that conditions each frame on up to four previously enhanced frames. Clients open a session with the newcreate_harmonizer_sessionRPC, pass the session ID on each render, and close it withend_harmonizer_session;--max-harmonizer-sessions(default 32) bounds concurrent sessions. The bundled client exposes--temporal-harmonizer-session. The temporal model is inference-only and mutually exclusive with the stateless--enable-harmonizer. See :doc:Use Harmonizer for Post-Processing <../nurec/harmonizer>.Artifact-Aware Backend Cache Keys: NRE records the artifact SHA256 in backend cache keys, so newly downloaded bytes cannot reuse a stale loaded backend when different content is published at the same artifact URL. On-demand scene registration through the
x-nre-scene-urlandx-nre-scene-idmetadata headers, which was already available on the single-process server, is now supported by the multi-process server as well. See :doc:Use the NuRec gRPC API Server <../api/grpc_api_guide>.Scene Cache Introspection and Limits: The new
get_loaded_scenesRPC returns a point-in-time snapshot of cached scene instances, with per-scene loaded and reusable instance counts plus the global cache capacity.--max-backends-per-scenecaps how many cached backends a single scene may hold; requests that reach the limit wait for and reuse a checked-in backend instead of building another.Compact USDZ Artifacts: Optional Gaussian tensor compression writes a smaller
.compact.usdzartifact. Enable it withartifact.compression.enabled(default: disabled, level 9, opacity threshold 0.005, with anexcludelist). The artifact reader detects and loads compact artifacts automatically, and uncompressed artifacts are unchanged.Inference Ray-Stride Tuning: The new
--tracks-timestamp-ray-strideflag onrenderandserve-grpcsubsamples the per-track rolling-shutter timestamp-estimation ray grid at inference time. Left unset it resolves to 16 on Ampere GPUs (compute capability 8.0, 8.6, and 8.7), where that kernel dominates a 4K frame, and to 1 everywhere else. Because the resolved timestamps differ slightly from stride 1, renders on an Ampere GPU are not bit-identical to 26.06 output; pass--tracks-timestamp-ray-stride 1to restore the previous behavior and cost. See :doc:Use the NuRec Render Command <../nurec/render-command>.Selective Tile Rendering: A new
model.renderer.active_tile_maskoption skips rendering tiles that are entirely invalid, which reduces training and inference time. It is off in the base schema, and several Hyperion 8.1 presets now select it. Masking changes what is supervised and optimized, not only what is rendered, so read the strategy table before enabling it on a custom configuration. See :doc:Customize NuRec Configuration <../nurec/configs>.Parallel-Batch GSplat Rasterization: A new
model.renderer.renderer_configoption parallelizes both the forward and the backward rasterization pass across the training batch, reducing per-step time. It is now the default for 3DGUT configurations and requires the 3DGUT path. See :doc:Customize NuRec Configuration <../nurec/configs>.Component-Only Training: A new
component_onlyconfiguration overlay trains a subset of model layers and leaves the rest frozen. Six presets ship underconfigs/component_only, covering background-only, deformables-only, and combined variants with optional calibration and post-processing.Track Calibration Options: Direct track calibration can now operate in local coordinate space through
delta_frame: local, withworldpreserved for older checkpoints. New reusable presets includedirect_lr1e-4andelevation_aware_unicycle, exposed for Hyperion 8.1 as thedynamic_rigids_track_calibrationoption group. The default six-camera preset selectsdirect.Pedestrian Visible Track Splitting: A new
visible_track_splitoption group splits long pedestrian tracks by visible duration. Thepedestrians_5spreset caps visible duration at 5 seconds for pedestrian and person classes and is selected by the production Hyperion 8.1 presets;disabledturns it off.Road-Quality Metrics:
eval-rendering-metricsgains four road sub-metrics selectable through--metrics:ref(road edge fidelity),rlcf(road local contrast fidelity),lm(lane-marking IoU), andrmtf(road multi-scale texture fidelity). Pass per-frame road masks with the new--road-masks-dirflag.Object-Level Occlusion Estimation: Object-level rendering metrics now compute per-object occlusion ratios, reported as
occlusion_gtandocclusion_ren. The new--occlusion-filter-thresholdflag drops heavily occluded objects from the aggregate and reportsocclusion_filtered.NCore V4 Dataset Support: Object-level rendering metrics accept an NCore V4 dataset through the new
--dataset-pathflag as an alternative to--shard-pattern.Ground-Mesh Export and Evaluation: A new
export-camera-ground-meshcommand exports a camera-derived ground mesh from four flags:--input,--output,--road-class-id(repeatable, default 0), and--postprocess/--no-postprocess.eval-ground-meshadds cuboid-coverage and point-cloud hull-area KPIs behind the new--cuboid-tracks-path,--point-cloud-path,--point-cloud-margin,--point-cloud-max-range, and--hull-grid-resolutionflags, and moves its diagnostic output into adiagnostics/subdirectory so metrics files stay at the output root.Point-Cloud Export Options:
export-point-cloudadds--include-dynamic-pointsto keep points flagged dynamic, and--coordinate-systemto choose between the scene-centerednreframe (default) and theworldframe.Heterogeneous Multi-LiDAR Reconstruction: Structured multi-LiDAR reconstruction supports per-sensor nominal model overrides keyed by
lidar_id, so rigs that mix LiDAR types can be reconstructed together.NRM Improvements: NRM resolves its PLY initialization path automatically from the NCore dataset layout when no explicit path is set, consumes NCore JSON paths directly, and gains a
keep_movable_gaussiansoption (default: disabled) to retain MOVABLE Gaussians in LiDAR PLY export.Per-Class Binary Mask Export: The NCore diagnostic tool adds
--per-class-binary-masksto export one binary mask per requested semantic class.Coverage Frame Sampler: A new
dataset/samplers: coverage_image_cropoption selects a coverage-based frame sampler, which can be combined with the existing hold-out split to validate on frames the model never trained on.Asset Harvester Output Layout Documented: The Asset Harvester output contract is now written down rather than inferred:
metadata.yamlplus one<label_class>/<track_id>/directory per asset, each containinggaussians.plyand3d_lifted.mp4, withassets.<track_id>.ply_fileinmetadata.yamlnaming the PLY path. The layout itself is unchanged from 26.06. See :doc:Use Asset Harvester Output in Reconstructions <../nurec/use-ah-assets>.gRPC Client Robustness: The bundled
render-grpcclient adds--wait-serverto block until the server is reachable instead of relying on a fixed delay,--max-concurrent-rendersto bound its own fan-out against an admission-limited server, and--repeat-count. It also gains the camera-product flags--output-type,--overlay-bboxes,--bbox-color, and--colorization-max-distance.GSplat Kernel Migration: Fused SSIM, road Gaussian, deformation-smoothness, semantic cross-entropy, and grouped BCE losses now run through GSplat and CUDA dispatch kernels, and the MCMC strategy uses the GSplat stateless owner API. Gaussian topology mutation schedules are exposed, and the CUDA cache purge during mutation is configurable.
Distributed Training Scalability: Multi-GPU synchronization overhead is reduced. Gaussian count metadata is coalesced into a single all-gather instead of one per layer, backward gradient reduce-scatters are fused into one call, visibility index computation reuses the fused gather counts rather than issuing separate per-layer collectives, and pre-backward CUDA synchronization is avoided when computing training PSNR.
Newer Runtime Dependencies: NCore moves from 19.1.1 to 19.5.1 (19.5.4 internal), the bundled FFmpeg in the OpenCV and imageio wheels moves from 8.1.1 to 9.0.1, PyTorch Lightning moves from 2.4.0 to 2.6.5, and Transformers moves from 4.48.3 to 5.13.1. The
pycudaandpytorch3ddependencies are dropped. System cuDNN is removed from the dev and run base images and is now supplied through the PyTorch wheels.GB300 (Blackwell Ultra) Runtime Compatibility: NRE adds runtime support for GB300 Grace-Blackwell systems. Because CUDA 12.8 cannot emit SM 10.3 code, the custom wheels ship SM 10.0 native kernels plus embedded
compute_100PTX for forward compatibility, and anLD_PRELOADshim remaps the reported SM 10.3 capability to SM 10.0 for PyTorch, Triton, and NVRTC. The same shim covers DGX Spark and GB10 (SM 12.1 to SM 12.0). GB300-specific runtime fixes align DAV3 LayerNorm numerics with the reference implementation. This is Grace-Blackwell enablement on aarch64, not general Linux aarch64 support — see the platform note under Known Issues.
Bug Fixes
Multi-GPU SelectiveAdam Correctness: Fixed a multi-GPU training bug where an owner rank skipped the optimizer update for Gaussians that were visible only on a different rank, even though the owner received a nonzero reduced gradient for them. Visibility is now unioned across ranks before the optimizer step. This affected model quality, not only training speed.
LiDAR RGB Projection: Fixed camera-valid LiDAR RGB projection in NCore data loading.
LiDAR Metrics: Corrected LiDAR metric computation, fixed sensor routing so metrics are attributed to the right LiDAR, and fixed the NRM initialization override used by LiDAR metrics.
LiDAR Rendering Origin: Fixed
render-grpcrendering LiDAR at the rig origin instead of the sensor pose.Cuboid Track Poses Under TF32: Fixed frame conversion corrupting cuboid track poses when TF32 is enabled.
Rigid Track Clamping: Per-track cadence metadata is now precomputed for rigid track clamping, reducing repeated recomputation.
Dropped Boundary Objects: Fixed obstacles being dropped in the first frame and rigid objects being dropped in the last frame, and guarded object-metric pose interpolation at clip boundaries.
Ground Mesh with Rig Offset: Fixed ground-mesh reconstruction when the rig has a non-zero
z_offset, and includedz_offsetin the reconstruction settings.Track Calibration Device Placement: Fixed unicycle and static track calibration mixing devices when tracks live on the GPU, and made
StaticTracksCalibmeans deterministic.Scene Cache Keying: Scene caches are keyed by both scene ID and URL, so two artifacts registered under one ID no longer collide.
Composite Radiance Scaling: Fixed Gaussian USD radiance scaling for composite models, and fixed GRT linear radiance when post-processing is disabled.
Empty Loss Handling: Fixed empty fused-loss handling, and guarded node-semantic and background-in-track reference losses against masked-out non-finite logits.
Validation Render Synchronization: Selected-node DDP renders are synchronized so ranks produce symmetric validation output.
Progressive Artifact Levels: Progressive levels are restored when loading an artifact.
Auxiliary Data Generation: Fixed a squeeze on single-point camera frames during auxiliary data generation, and fixed auxiliary shard discovery to use
sequence_idas the file prefix.Semantic Auxiliary Access: Guarded semantic auxiliary access during validation when auxiliary data is disabled.
Gaussian Seeding and Relocation: Gaussian initialization is seeded from every training LiDAR rather than only the first, optimizer moments are reset for relocated Gaussians, and a fused point-cloud scene extent is used as a fallback.
Range Request Fallback: Fixed the S3 range probe when the server does not advertise
Accept-Ranges.Asset Harvester Symlinks: Asset Harvester resolves Hugging Face blob symlinks correctly when locating cached weights.
Checkpoint Completeness: Post-batch model state is saved in final checkpoints, and NRM mounted model cache compatibility is restored.
Known Issues
Harmonizer does not run under distributed (multi-GPU) training: Harmonizer now fails fast instead of producing a device-mismatch error partway through. When a distributed process group is initialized with a world size greater than 1, loading the Harmonizer model raises
Harmonizer does not support multi-GPU execution (world_size=N). Its TorchScript model is not multi-GPU-safe because of a shared model-cache download race and device-baked constants; 26.08 guards this condition rather than fixing the underlying model. The check keys on the distributed world size, not on how many GPUs are visible, and it runs when the model loads — at trainer setup for Harmonizer training, and at the first validation epoch for Harmonizer inference during a training run. Workaround: run Harmonizer training with a world size of 1, for example by restrictingCUDA_VISIBLE_DEVICES, or apply Harmonizer at inference through the gRPC server.serve-grpcnever initializes a process group, so--enable-harmonizerworks on a multi-GPU host with no need to restrict visible GPUs. This replaces the 26.06 known issue describing the same limitation as a validation-time CUDA error. See :doc:Use Harmonizer for Post-Processing <../nurec/harmonizer>.Temporal Harmonizer restrictions: The temporal Harmonizer is inference-only and is not available for training. It cannot be combined with the stateless
--enable-harmonizer, with multi-process serving (--multiproc-workers > 0), or with the batch render RPCs — send a session’s frames one at a time, in trajectory order, awaiting each render. Component-only masks that need per-pixel semantics are also rejected for Harmonizer novel-view training.Asset editing is single-process only: Under multi-process serving,
edit_assetsreturnssuccess=Falsewith anUNIMPLEMENTEDmessage andrestore_model_parametersaborts with gRPC statusUNIMPLEMENTED. Use a single-processserve-grpcfor in-place asset edits. Per-request dynamic-actor re-posing through--enable-editing-actorsworks in both modes.Listed scenes may not be renderable under multi-process serving: Unlike the single-process server, a multi-process
get_available_scenesentry may be registered but not yet attached or validated, so a listed scene’s first render can returnNOT_FOUNDorUNAVAILABLE. Useget_loaded_scenesas the render-readiness check.gRPC asset replacement rendering: In some gRPC asset-edit scenarios, an inserted asset may not visually replace the target scene vehicle in rendered output (for example, a grey van edit on Hyperion 8.1 scenes). Reported on RTX PRO 6000. This issue is carried over from 26.06 and a fix is still underway.
General Linux aarch64 is not a supported platform: The supported user platform for NuRec remains Linux x86_64. The GB300 and DGX Spark enablement described under Core Improvements is targeted Grace-Blackwell support, not general aarch64 support, and it carries these caveats. It reaches you only through the
-arm64or-multiarchcontainer tags; pulling an unsuffixed tag on an aarch64 host gives you the x86_64 image, which does not include the compatibility shim. The aarch64 and multi-architecture image builds run as non-gating nightly or on-demand CI jobs, so those tags can lag the x86_64 ones, and container security scanning covers the x86_64 image only. See :doc:Hardware Setup and Requirements <../basics/hardware>.