Generate Novel Views with NuRec#

NuRec supports novel view synthesis, allowing you to render from camera positions that were not present in the original dataset. You can shift the virtual camera along the X, Y, and Z axes, or apply roll-pitch-yaw rotations, to simulate a sensor rig that drove a slightly different path — for example, a lane shift, a height change, or a different vehicle orientation.

This is useful when you need to cover edge cases beyond the original trajectory: training an AV model on adjacent-lane perspectives, testing sensor configurations that weren’t physically present, or augmenting your dataset with plausible camera variations.

Note: In NuRec, novel view synthesis produces shifted camera viewpoints from a reconstructed static scene. For controllable dynamic actors (moving vehicles, pedestrians) in a live simulation, use the gRPC API with the --enable-editing-actors flag — see Use the NuRec gRPC API Server.

This guide assumes you have already trained a NuRec reconstruction model. If you haven’t, see Reconstruct a Scene with NuRec first.

How It Works#

The diagram below illustrates the X-, Y-, and Z-axis shifts available for novel view generation.

Novel view synthesis

Novel views are generated by running the NuRec validation pipeline (mode=val) with spatial translation or rotation overrides applied to the sensor position. This produces new rendered frames from the shifted viewpoints.

Run Novel View Synthesis#

This guide assumes you have already trained a NuRec reconstruction model. Modify the following command to reflect the output directory from your training run:

docker run --shm-size=64g --rm --gpus all \
  -e NGC_API_KEY=${NGC_API_KEY} \
  --volume /path/to/dataset:/workdir/dataset \
  --volume /path/to/training/output:/workdir/output \
  nvcr.io/nvidia/nre/nre-ga:26.04 \
  --config-name=/workdir/output/config/parsed.yaml \
  mode=val \
  resume=/workdir/output/checkpoints/last.ckpt \
  out_dir=/workdir/output \
  dataset.val_sensor_transl_delta_m="[0,2,0]"

Notes:

  • /path/to/training/output should point to the top-level directory containing artifacts from your training run (the directory with config/, checkpoints/, etc.).

  • This step may prompt you to configure wandb. Select option (3) to skip it.

Camera Shift Parameters#

Translation shifts — Move the virtual camera along each axis (values in meters, no spaces):

dataset.val_sensor_transl_delta_m="[x,y,z]"

For example, "[0,2,0]" shifts the camera 2 meters to the left (Y axis). Typical lane-shift values are in the 1–4 meter range. Very large shifts will move the camera outside the region covered by the training data and produce degraded results.

Rotation shifts — Apply roll-pitch-yaw Euler angles relative to the car (values in degrees, no spaces):

dataset.val_sensor_rot_delta_deg="[degree1,degree2,degree3]"

You can combine both parameters in a single command to shift and rotate simultaneously.

Use the Render Command Instead#

For quick offline novel view generation without a full validation pass, the render command is a lighter-weight option. It takes a USDZ file directly and applies fixed translation and rotation offsets via --rig-translation-offset and --rig-rotation-offset, without needing the original dataset.

Use the validation pipeline (this guide) when you need per-frame metrics alongside your renders. Use the render command when you just need images quickly.

Output Artifacts#

The novel view synthesis run produces the following files in /path/to/training/output/val/:

  • metrics.yaml — Per-frame metrics. Check the test/psnr field to evaluate reconstruction quality at the shifted viewpoint compared to training views.

  • MP4 video files showing the base reconstruction, depth map, and opacity.

  • Individual images per frame showing the base reconstruction, depth map, and opacity.

Next Steps#

Once you’ve confirmed the novel view renders look correct, use the NuRec gRPC API to serve rendered frames at arbitrary camera poses in real time, including with dynamic actor control via --enable-editing-actors.

To view all available arguments for training, validation, and export utilities, run the following:

docker run --shm-size=64g --rm --gpus all \
  -e NGC_API_KEY=${NGC_API_KEY} \
  nvcr.io/nvidia/nre/nre-ga:26.04 --help

To get help for a specific export utility:

docker run --shm-size=64g --rm --gpus all \
  -e NGC_API_KEY=${NGC_API_KEY} \
  nvcr.io/nvidia/nre/nre-ga:26.04 \
  <EXPORT_UTILITY_NAME> \
  --help
On this page