Skip to navigation

Advanced Usage

View as MarkdownOpen in Claude

Model Caching

When the container launches for the first time, it downloads the required models from NGC. To avoid downloading the models on subsequent runs, you can cache them locally by using a cache directory:

# Create the cache directory on the host machine.
export LOCAL_NIM_CACHE=~/.cache/nim
mkdir -p "$LOCAL_NIM_CACHE"
chmod a+rwX "$LOCAL_NIM_CACHE" 2>/dev/null || true
# Run the container with the cache directory mounted in the appropriate location.
docker rm -f relighting-nim 2>/dev/null
docker run -d --name=relighting-nim \
--runtime=nvidia \
--gpus all \
--shm-size=8GB \
-e NGC_API_KEY=$NGC_API_KEY \
-e NV_AI4M_MAX_CONCURRENCY_PER_GPU=1 \
-e NIM_HTTP_API_PORT=8000 \
-e NIM_GRPC_API_PORT=8001 \
-p 8000:8000 \
-p 8001:8001 \
-p 9002:9002 \
-v "$LOCAL_NIM_CACHE:/opt/nim/.cache" \
nvcr.io/nim/nvidia/relighting:1.2.0
docker logs -f relighting-nim

The container runs as uid 1000, so the model folders it writes into the cache belong to uid 1000. If your user has a different uid, rm cannot delete them; remove the cached Relighting models with the container instead:

# The cache directory from the command above, set again so this also works in a new shell.
export LOCAL_NIM_CACHE=~/.cache/nim
docker run --rm -v "$LOCAL_NIM_CACHE:/opt/nim/.cache" \
nvcr.io/nim/nvidia/relighting:1.2.0 \
bash -c 'rm -rf /opt/nim/.cache/ngc/hub/models--*--relighting'

Afterwards, ls "$LOCAL_NIM_CACHE/ngc/hub" lists no models--*--relighting folder.

SSL Enablement

NVIDIA AI for Media Relighting provides an SSL mode to ensure secure communication between clients and the server by encrypting data in transit. To enable SSL, you must provide the path to the SSL certificate and key files in the container. The following example shows how to do this:

The container expects a CA certificate, a server certificate signed by it, and the matching server key. mtls mode also needs a client certificate signed by the same CA. To generate a self-signed set for local testing, in an empty directory:

mkdir -p certs && cd certs
# 1. CA
openssl req -x509 -newkey rsa:4096 -nodes -days 365 \
-keyout ssl_ca_key.pem -out ssl_ca_cert.pem \
-subj "/CN=relighting-test-ca"
# 2. Server key + CSR
openssl req -newkey rsa:4096 -nodes \
-keyout ssl_key_server.pem -out server.csr \
-subj "/CN=localhost"
# 3. Server certificate, signed by the CA. The client verifies the server's
# hostname against the certificate, so the SAN is required, not optional.
printf "subjectAltName=DNS:localhost,IP:127.0.0.1\n" > san.cnf
openssl x509 -req -in server.csr -days 365 \
-CA ssl_ca_cert.pem -CAkey ssl_ca_key.pem -CAcreateserial \
-extfile san.cnf \
-out ssl_cert_server.pem
# 4. Client key + CSR (mtls only -- the server rejects a connection with no
# client certificate)
openssl req -newkey rsa:4096 -nodes \
-keyout ssl_key_client.pem -out client.csr \
-subj "/CN=relighting-client"
# 5. Client certificate, signed by the same CA
openssl x509 -req -in client.csr -days 365 \
-CA ssl_ca_cert.pem -CAkey ssl_ca_key.pem -CAcreateserial \
-out ssl_cert_client.pem
# openssl writes ssl_key_server.pem at mode 600 (owner-only). The server runs
# inside the container as uid 1000 (user nvs), not as the uid that ran openssl,
# so grant that uid, and only that uid, read access; otherwise the server
# refuses to start:
# ERROR:inference:SSL configuration is invalid; refusing to start.
# nimlib.exceptions.SSLConfigurationError: Error verifying SSL files: [Errno 13] Permission denied
# Nothing to do when you are uid 1000. setfacl comes from the acl package.
[ "$(id -u)" = 1000 ] || setfacl -m u:1000:r ssl_key_server.pem
cd ..

This block is POSIX shell, so it runs under sh/dash as well as bash and zsh. It leaves a few scratch files next to the certificates — san.cnf, the two CSRs (server.csr, client.csr), and the CA serial (ssl_ca_cert.srl).

SSL_CERT must be an absolute path. Docker reads a relative -v source as the name of a volume rather than a directory, so a relative value silently creates an empty named volume, mounts it over /opt/nim/crt, and the server then exits reporting that it cannot find a certificate you just generated.

# Run from the directory where you ran the recipe above, so that $PWD/certs is the directory it created.
export SSL_CERT="$PWD/certs"
export LOCAL_NIM_CACHE=~/.cache/nim
mkdir -p "$LOCAL_NIM_CACHE"
chmod a+rwX "$LOCAL_NIM_CACHE" 2>/dev/null || true
docker rm -f relighting-nim 2>/dev/null
docker run -d --name=relighting-nim \
--runtime=nvidia \
--gpus all \
--shm-size=8GB \
-v "$LOCAL_NIM_CACHE:/opt/nim/.cache" \
-v "$SSL_CERT:/opt/nim/crt/:ro" \
-e NGC_API_KEY=$NGC_API_KEY \
-p 8000:8000 \
-p 8001:8001 \
-p 9002:9002 \
-e NIM_SSL_MODE="mtls" \
-e NIM_SSL_CA_CERTS_PATH="/opt/nim/crt/ssl_ca_cert.pem" \
-e NIM_SSL_CERT_PATH="/opt/nim/crt/ssl_cert_server.pem" \
-e NIM_SSL_KEY_PATH="/opt/nim/crt/ssl_key_server.pem" \
nvcr.io/nim/nvidia/relighting:1.2.0
docker logs -f relighting-nim

NIM_SSL_MODE can be set to mtls, tls, or disabled. If set to mtls, the container uses mutual TLS authentication. If set to tls, the container uses TLS authentication. For more information, refer to Environment Variables.

Verify the permissions of the SSL certificate and key files on the host machine. The container runs as uid 1000 and cannot start if the files are not readable by that user, for example a key that openssl wrote with mode 600. Grant read access to uid 1000 only, with setfacl -m u:1000:r ssl_key_server.pem, rather than making the private key world-readable. The server certificate must carry a Subject Alternative Name for the hostname that clients connect to.

NIM_SSL_MODE secures the gRPC port (8001) only. The HTTP endpoints on port 8000 stay plaintext.

To connect the sample Python client to an SSL-enabled server, set --ssl-mode to match NIM_SSL_MODE. Pass --ssl-root-cert for TLS or MTLS, and additionally --ssl-cert and --ssl-key for MTLS. Run the client from its scripts folder, with the virtual environment from Basic Inference active. The launch commands above export SSL_CERT, so the first line below keeps that value in the same shell. In a new shell, first run export SSL_CERT="{absolute path of the certs directory the recipe created}"; otherwise the first line stops with a message naming SSL_CERT:

export SSL_CERT="${SSL_CERT:?set SSL_CERT to the absolute path of the certs directory the recipe created}" && \
python relighting.py --target 127.0.0.1:8001 --video-input ../assets/sample_video.mp4 \
--ssl-mode MTLS \
--ssl-root-cert "$SSL_CERT/ssl_ca_cert.pem" \
--ssl-cert "$SSL_CERT/ssl_cert_client.pem" \
--ssl-key "$SSL_CERT/ssl_key_client.pem"

A client whose --ssl-mode does not match the server fails to connect with UNAVAILABLE, for example a plaintext client against a tls or mtls server, or --ssl-mode TLS against an mtls server.

Multiple Concurrent Inputs

By default, the NIM processes one request per GPU (NV_AI4M_MAX_CONCURRENCY_PER_GPU=1). To process several requests at once, set NV_AI4M_MAX_CONCURRENCY_PER_GPU to an integer greater than 1 on the server container. The total number of concurrent requests the server accepts is the number of GPUs multiplied by NV_AI4M_MAX_CONCURRENCY_PER_GPU.

For example, to process two requests at once on one GPU, launch the NIM with the command in Launching the NIM Container, with -e NV_AI4M_MAX_CONCURRENCY_PER_GPU=2 in place of -e NV_AI4M_MAX_CONCURRENCY_PER_GPU=1. Then send two requests at the same time from nim-clients/relighting/scripts/:

python relighting.py --target 127.0.0.1:8001 --output output_1.mp4 & p1=$!
python relighting.py --target 127.0.0.1:8001 --output output_2.mp4 & p2=$!
wait $p1; echo "request 1 exit $?"
wait $p2; echo "request 2 exit $?"

A request that arrives when every slot is busy waits up to 3 seconds for a slot. If no slot frees in that time, the request fails with RESOURCE_EXHAUSTED and the message Server is at capacity. Please retry shortly (active request limit reached). Retry the request after the running request finishes, or raise NV_AI4M_MAX_CONCURRENCY_PER_GPU and restart the container.

Each concurrent request uses its own GPU memory, and the amount grows with the input resolution. If the GPU runs out of memory while it decodes a request’s frames (for example, more concurrent 4K requests than the GPU memory can hold), that request fails with RESOURCE_EXHAUSTED. The request ends with this error, never with a shorter video reported as success. Retry with fewer concurrent requests or a lower input resolution, or lower NV_AI4M_MAX_CONCURRENCY_PER_GPU.

Requests with background_source = 2 (HDR projection) that run at the same time on one GPU must have the same video resolution. When two such requests of different resolutions overlap, the background of each comes out at the wrong size. Send them one after another, or keep NV_AI4M_MAX_CONCURRENCY_PER_GPU=1.

NIM Service Configuration Parameters via Client

Use the following options to configure each request to Relighting. Run the examples in this section from nim-clients/relighting/scripts/ with the virtual environment from Basic Inference active.

HDR/Illumination Options

  • hdri_preset_id: Specifies a built-in HDR preset to create one of several lighting moods. Mutually exclusive with hdri_image_provided.

    • 0 (Lounge): Lounge environment.
    • 1 (Cobblestone Street Night): Cobblestone street at night.
    • 2 (Glasshouse Interior): Glasshouse interior.
    • 3 (Little Paris Eiffel Tower): Little Paris Eiffel Tower.
    • 4 (Wooden Studio): Wooden studio environment.

    The following example uses the sample Python client with the Little Paris Eiffel Tower HDR preset:

    python relighting.py --target 127.0.0.1:8001 --video-input ../assets/sample_video.mp4 --hdri-id 3
  • hdri_image_provided: When set to true, indicates that a custom HDR environment map is sent inline as ImageData. Set either this field or hdri_preset_id, not both. If the configuration sets hdri_preset_id, the NIM ignores an HDR ImageData and logs a warning. The image must be an equirectangular map, twice as wide as it is tall (for example 4096 x 2048 pixels), and can be up to 150 MB and 8192 x 4096 pixels. The NIM checks the size first, so an image that breaks both rules gets the size error.

    The following example uses the sample Python client with a custom HDR file. The shipped sample_lounge.hdr is the Lounge preset, so the first command mirrors it into a different environment map:

    ffmpeg -y -i ../assets/sample_lounge.hdr -vf hflip ../assets/custom.hdr
    python relighting.py --target 127.0.0.1:8001 --video-input ../assets/sample_video.mp4 --hdr ../assets/custom.hdr
  • angle_pan_radians: Pan angle for rotating the HDR environment, in radians (range -π to π). Default is -1.5708 (-90 degrees).

    python relighting.py --target 127.0.0.1:8001 --video-input ../assets/sample_video.mp4 --pan -45 # degrees
  • angle_v_fov_radians: Vertical field of view of the HDR environment map when it is used as the background (background_source = 2), in radians (range 0.1 to π). Default is 1.0472 (60 degrees).

    python relighting.py --target 127.0.0.1:8001 --video-input ../assets/sample_video.mp4 --background-source 2 --vfov 90 # degrees

Autorotate Options

  • autorotate: Enable automatic rotation of the HDR environment during video playback for dynamic lighting effects.

  • rotation_rate: Speed of rotation in radians per second (range 0 to 2π). Default is 0.349 (20 degrees per second). Only applies when autorotate is enabled.

    The following example enables autorotate with a custom rotation speed:

    python relighting.py --target 127.0.0.1:8001 --video-input ../assets/sample_video.mp4 --autorotate --rotation-rate 45 # degrees per second

Background Source Options

  • background_source: Selects which background source to use.

    • 0 (UNSPECIFIED): Use source video frame as background (default).
    • 1 (FROM_IMAGE): Use custom background image uploaded via ImageData.
    • 2 (FROM_HDR): Use HDR environment map as background.

    The following example uses the sample Python client with the HDR environment map as the background:

    python relighting.py --target 127.0.0.1:8001 --video-input ../assets/sample_video.mp4 --background-source 2
  • Custom background image: with background_source = 1 (FROM_IMAGE), send the image inline as ImageData with IMAGE_TYPE_BACKGROUND. PNG, JPEG, and HDR are supported. The server detects the format from the image bytes, so background_image_type is not used. The NIM decodes the background image and resizes it to the video once per request, so a large image or an .hdr file adds time to each request. For the shortest requests, use a PNG or JPEG background at the output resolution. The image can be up to 150 MB and 8192 x 4096 pixels.

    ffmpeg -y -i ../assets/sample_video.mp4 -frames:v 1 ../assets/background.png
    python relighting.py --target 127.0.0.1:8001 --video-input ../assets/sample_video.mp4 --background-source 1 --background-image ../assets/background.png
  • background_color: Solid background color as an integer in 0xRRGGBB form. If the request does not set it, the NIM uses 0x808080 (gray); the sample client sets it only when you pass --background-color. Used when background_source is FROM_IMAGE and no image is uploaded. The following example composites the subject over solid blue:

    python relighting.py --target 127.0.0.1:8001 --video-input ../assets/sample_video.mp4 --background-source 1 --background-color 0x0000FF

Effect Parameters

  • foreground_gain: Controls the strength of relighting applied to the foreground subject (0.0–2.0, default: 1.0). Higher values produce more intense relighting effects.

    python relighting.py --target 127.0.0.1:8001 --video-input ../assets/sample_video.mp4 --foreground-gain 1.5
  • background_gain: Controls the strength of relighting applied to the background (0.0–2.0, default: 1.0). Applies only when background_source is 2 (FROM_HDR).

    python relighting.py --target 127.0.0.1:8001 --video-input ../assets/sample_video.mp4 --background-source 2 --background-gain 0.8

Encoding Options

To configure the encoding quality of the output video, use the output_video_encoding parameter. The encoding uses a VideoEncoding message with three mutually exclusive modes: lossy, lossless, or custom.

If the configuration message has no output_video_encoding, the NIM uses lossy mode with the auto bitrate and an IDR interval of about 2 seconds of video: the input frame rate multiplied by 2, in frames (60 frames for 30 fps input). If a lossy message does not set bitrate_mbps or idr_interval, the NIM uses the same default for that field. In lossy mode, the sample client always sends both fields.

The sample client sends one mode for each request. If you use --lossless and --custom-encoding-params together, the client stops with a configuration error. With --lossless or --custom-encoding-params, the client ignores --bitrate and --idr-interval and prints a warning.

  • Bitrate control (lossy mode): Target bitrate is specified in megabits per second (bitrate_mbps). Use 0 for auto, which selects a bitrate from the output resolution: 10 Mbps up to 720p (1280×720 pixels), 15 Mbps up to 1080p (1920×1080 pixels), and 25 Mbps above 1080p. Higher values improve quality and increase output size; lower values save bandwidth at the cost of quality.

    • In the configuration message, set output_video_encoding.lossy.bitrate_mbps to the desired Mbps (1 to 200) or 0 for auto.

    • The sample Python CLI uses --bitrate in bps (rounded to Mbps for the proto); default matches 10 Mbps (10,000,000 bps). Example explicit bitrate:

      # 20,000,000 bps = 20 Mbps
      python relighting.py --target 127.0.0.1:8001 --video-input ../assets/sample_video.mp4 --bitrate 20000000
  • IDR interval control: idr_interval is the spacing between IDR frames in frames (not seconds). The sample CLI defaults to 8 frames and always sends a value to the server. Values must be ≥ 1 (0 is invalid). Sample CLI override:

    python relighting.py --target 127.0.0.1:8001 --video-input ../assets/sample_video.mp4 --idr-interval 10
  • Lossless mode: Encodes the output video with lossless H.264 compression. Use lossless mode when the output is an intermediate file for more work, such as color grading, compositing, editing, or a quality comparison against a reference. To deliver video to viewers, use lossy mode.

    Before you use lossless mode, consider these points:

    • Output files are several times larger than lossy output, so each request transfers and stores more data.
    • The NIM converts the frames to 8-bit YUV 4:2:0 before it encodes them, as in lossy mode. The encoding adds no loss, but the color information is subsampled to 4:2:0. Lossless mode does not give 4:4:4 or RGB output.
    • The output uses the H.264 High 4:4:4 Predictive profile, which lossless H.264 requires. Some players and devices decode only the Baseline, Main, and High profiles. Make sure that your player or tool decodes this profile.
    • Bitrate and IDR interval are lossy-mode settings and do not apply in lossless mode.

    To enable lossless mode with the sample client, use the --lossless flag:

    python relighting.py --target 127.0.0.1:8001 --video-input ../assets/sample_video.mp4 --lossless
  • Custom encoding parameters: Provides fine-grained control for expert users via JSON configuration. These parameters configure properties of the DeepStream H264 encoder.

    To specify custom encoding parameters, use the --custom-encoding-params option:

    python relighting.py --target 127.0.0.1:8001 --video-input ../assets/sample_video.mp4 --custom-encoding-params '{"bitrate":5000000,"idrinterval":16,"maxbitrate":6000000}'

    The NIM accepts these encoder (nvv4l2h264enc) properties: bitrate, maxbitrate, iframeinterval, idrinterval, vbvbufsize, vbvinit, SliceIntraRefreshInterval, slice-header-spacing, preset-id, aq, cq, num-B-Frames, num-Ref-Frames, control-rate, tuning-info-id, profile, temporalaq, insert-vui, insert-sps-pps, insert-aud, disable-cabac, and extended-colorformat. Any other key, such as idr_interval or qp-range, or a value outside the property’s range makes the request fail with INVALID_ARGUMENT.

    B-frames (num-B-Frames 1 or 2) need profile 2 (Main), 4 (High), or 7 (High 4:4:4), and idrinterval and iframeinterval values that are multiples of num-B-Frames + 1. If the request does not set profile or idrinterval, the NIM uses profile 4 and the smallest such idrinterval of 8 or more. Another profile or interval makes the request fail with INVALID_ARGUMENT.

    With custom encoding parameters, the bitrate and idrinterval keys in the JSON set the bitrate and IDR interval; if the JSON has no bitrate key, the encoder uses 20 Mbps. bitrate and maxbitrate are in bits per second (5000000 is 5 Mbps), as --bitrate is; the lossy bitrate_mbps field of the configuration message is in Mbps. Use with caution because incorrect values can affect output quality or encoding stability.

The following example builds the configuration message of a request with the Python client’s generated stubs. It sends nothing and prints nothing; to send it, use generate_requests() in scripts/relighting.py:

# Run from nim-clients/relighting/scripts/ with the virtual environment active.
import sys
sys.path.append("../interfaces") # Python stubs shipped with the client
from nvidia.ai4m.relighting.v1 import relighting_pb2
from nvidia.ai4m.video.v1 import video_pb2
config = relighting_pb2.RelightConfig()
# Set HDR preset or custom HDR image
config.hdri_preset_id = 3 # Little Paris Eiffel Tower preset
# OR config.hdri_image_provided = True # Send custom HDR inline via ImageData
# Set background source (0=source video, 1=custom image, 2=HDR)
config.background_source = 2 # BACKGROUND_SOURCE_FROM_HDR (HDR projection)
# Set pan and vertical FOV angles (in radians).
# The vertical FOV applies only with background_source = 2.
config.angle_pan_radians = -1.5708 # -90 degrees
config.angle_v_fov_radians = 1.0472 # 60 degrees
# Set gain parameters. background_gain applies only with background_source = 2.
config.foreground_gain = 1.0
config.background_gain = 1.0
# Set output video encoding (lossy): 10 Mbps with 60-frame IDR interval
config.output_video_encoding.CopyFrom(
video_pb2.VideoEncoding(
lossy=video_pb2.LossyEncoding(bitrate_mbps=10, idr_interval=60)
)
)
# The configuration is the first message of the Relight request stream,
# followed by any image data and then the video chunks.
request = relighting_pb2.RelightRequest(config=config)

The sample client builds the complete request stream in generate_requests() in scripts/relighting.py.

For more information, refer to the NIM Clients GitHub repository: NVIDIA AI for Media NIM Clients.