Harmonizer NIM Documentation | NVIDIA NIM Harmonizer

Harmonizer NIM serves the NVIDIA Harmonizer model (nvidia/harmonizer), an online generative enhancement model for RGB frames rendered from neural-reconstructed scenes. It is designed to reduce novel-view reconstruction artifacts and improve foreground and background appearance, lighting, and cast-shadow consistency when dynamic objects are inserted into reconstructed scenes. Temporal mode uses previously enhanced frames to improve frame-to-frame consistency. A request contains one or more rendered frames; return_policy controls which enhanced frames are returned.

The NIM exposes a custom image-to-image API over both HTTP and gRPC. The active protocol is selected at startup. The logical request and response contract is equivalent across transports, but the wire representations differ: HTTP uses base64-encoded image payloads and string parameter values, while gRPC uses binary bytes and protobuf enums.

Skip Ahead

Support Matrix

Check supported GPUs, precision, and deployment profiles.

Get Started

Choose a Docker or Kubernetes deployment and run your first request.

API Examples

Integrate HTTP or gRPC inference into your application.

Troubleshooting

Resolve startup, model-cache, and inference problems.

Use Cases

Choose the processing pattern that matches your rendered frames.

Enhance Independent Frames

Use the non-temporal lane to process each image independently.

Enhance Frame Sequences

Use temporal context to improve consistency across a sequence of frames.

Architecture

Harmonizer is a single-step, temporally conditioned image diffusion model built on the NVIDIA Cosmos backbone (nvidia/Cosmos-Predict2-0.6B-Text2Image). The public research release is open under Apache-2.0. Source code is available in the /NVIDIA/harmonizer GitHub repo, and checkpoints are available on the NVIDIA Harmonizer Hugging Face model page.

The NIM serves a single release-approved checkpoint (diffusion_harmonizer.pkl) through two processing modes over the same model:

A non-temporal image lane runs the model single-frame (V=1) for independent image-to-image enhancement. This is equivalent to the research repository’s --nontemporal mode and is the lane wired into render-loop integrations.
A temporal context lane stacks previously generated output frames as autoregressive context (V=5: the current frame plus four previous generated outputs) for higher temporal consistency. The lane falls back to V=1 for the first frames of a sequence until enough history exists.

The lane is selected per request, not at startup. Regardless of whether HTTP or gRPC was selected as the inference transport, a single running NIM serves both processing patterns without a restart. Concurrent same-shape requests are coalesced into GPU batches by an internal queue, so the same deployment dynamically serves low-latency per-frame traffic and high-throughput batch traffic. The effective batch size adapts to concurrency.

Inputs and Outputs

The NIM runs inference at 1024x576 in bfloat16 (BF16) precision. Inputs can use any positive image resolution. The runtime derives the dimensions from decoded PNG/JPEG data or from the raw tensor shape, resizes the image to the model resolution for inference, and resizes the output back to the corresponding input resolution. This resize is a direct stretch: it does not preserve aspect ratio, letterbox, or crop. Use 16:9 inputs to avoid geometric distortion during inference. Outputs are PNG, JPEG, or raw uint8 tensors, selected per request. For transport and codec performance guidance, refer to the Support Matrix.

APIs

The container exposes the following inference surface:

POST /v1/infer for the native Harmonizer request and response schema over HTTP.
harmonizer.v1.Harmonizer/Process for the same payload over binary gRPC.

For request examples and the embedded OpenAPI schema, refer to the API Reference.