Advanced Usage
Model Caching
When the container launches for the first time, it downloads the required models from NGC. To avoid downloading the models on subsequent runs, you can cache them locally by using a cache directory:
The container runs as uid 1000, so the model folders it writes into the cache belong to uid 1000. If your user has a different uid, rm cannot delete them; remove the cached Relighting models with the container instead:
Afterwards, ls "$LOCAL_NIM_CACHE/ngc/hub" lists no models--*--relighting folder.
SSL Enablement
NVIDIA AI for Media Relighting provides an SSL mode to ensure secure communication between clients and the server by encrypting data in transit. To enable SSL, you must provide the path to the SSL certificate and key files in the container. The following example shows how to do this:
The container expects a CA certificate, a server certificate signed by it, and the matching server key. mtls mode also needs a client certificate signed by the same CA. To generate a self-signed set for local testing, in an empty directory:
This block is POSIX shell, so it runs under sh/dash as well as bash and zsh. It leaves a few scratch files next to the certificates — san.cnf, the two CSRs (server.csr, client.csr), and the CA serial (ssl_ca_cert.srl).
SSL_CERT must be an absolute path. Docker reads a relative -v source as the name of a volume rather than a directory, so a relative value silently creates an empty named volume, mounts it over /opt/nim/crt, and the server then exits reporting that it cannot find a certificate you just generated.
NIM_SSL_MODE can be set to mtls, tls, or disabled. If set to mtls, the container uses mutual TLS authentication. If set to tls, the container uses TLS authentication.
For more information, refer to Environment Variables.
Verify the permissions of the SSL certificate and key files on the host machine. The container runs as uid 1000 and cannot start if the files are not readable by that user, for example a key that openssl wrote with mode 600. Grant read access to uid 1000 only, with setfacl -m u:1000:r ssl_key_server.pem, rather than making the private key world-readable. The server certificate must carry a Subject Alternative Name for the hostname that clients connect to.
NIM_SSL_MODE secures the gRPC port (8001) only. The HTTP endpoints on port 8000 stay plaintext.
To connect the sample Python client to an SSL-enabled server, set --ssl-mode to match NIM_SSL_MODE. Pass --ssl-root-cert for TLS or MTLS, and additionally --ssl-cert and --ssl-key for MTLS. Run the client from its scripts folder, with the virtual environment from Basic Inference active. The launch commands above export SSL_CERT, so the first line below keeps that value in the same shell. In a new shell, first run export SSL_CERT="{absolute path of the certs directory the recipe created}"; otherwise the first line stops with a message naming SSL_CERT:
A client whose --ssl-mode does not match the server fails to connect with UNAVAILABLE, for example a plaintext client against a tls or mtls server, or --ssl-mode TLS against an mtls server.
Multiple Concurrent Inputs
By default, the NIM processes one request per GPU (NV_AI4M_MAX_CONCURRENCY_PER_GPU=1). To process several requests at once, set NV_AI4M_MAX_CONCURRENCY_PER_GPU to an integer greater than 1 on the server container. The total number of concurrent requests the server accepts is the number of GPUs multiplied by NV_AI4M_MAX_CONCURRENCY_PER_GPU.
For example, to process two requests at once on one GPU, launch the NIM with the command in Launching the NIM Container, with -e NV_AI4M_MAX_CONCURRENCY_PER_GPU=2 in place of -e NV_AI4M_MAX_CONCURRENCY_PER_GPU=1. Then send two requests at the same time from nim-clients/relighting/scripts/:
A request that arrives when every slot is busy waits up to 3 seconds for a slot. If no slot frees in that time, the request fails with RESOURCE_EXHAUSTED and the message Server is at capacity. Please retry shortly (active request limit reached). Retry the request after the running request finishes, or raise NV_AI4M_MAX_CONCURRENCY_PER_GPU and restart the container.
Each concurrent request uses its own GPU memory, and the amount grows with the input resolution. If the GPU runs out of memory while it decodes a request’s frames (for example, more concurrent 4K requests than the GPU memory can hold), that request fails with RESOURCE_EXHAUSTED. The request ends with this error, never with a shorter video reported as success. Retry with fewer concurrent requests or a lower input resolution, or lower NV_AI4M_MAX_CONCURRENCY_PER_GPU.
Requests with background_source = 2 (HDR projection) that run at the same time on one GPU must have the same video resolution. When two such requests of different resolutions overlap, the background of each comes out at the wrong size. Send them one after another, or keep NV_AI4M_MAX_CONCURRENCY_PER_GPU=1.
NIM Service Configuration Parameters via Client
Use the following options to configure each request to Relighting. Run the examples in this section from nim-clients/relighting/scripts/ with the virtual environment from Basic Inference active.
HDR/Illumination Options
-
hdri_preset_id: Specifies a built-in HDR preset to create one of several lighting moods. Mutually exclusive withhdri_image_provided.0(Lounge): Lounge environment.1(Cobblestone Street Night): Cobblestone street at night.2(Glasshouse Interior): Glasshouse interior.3(Little Paris Eiffel Tower): Little Paris Eiffel Tower.4(Wooden Studio): Wooden studio environment.
The following example uses the sample Python client with the Little Paris Eiffel Tower HDR preset:
-
hdri_image_provided: When set totrue, indicates that a custom HDR environment map is sent inline asImageData. Set either this field orhdri_preset_id, not both. If the configuration setshdri_preset_id, the NIM ignores an HDRImageDataand logs a warning. The image must be an equirectangular map, twice as wide as it is tall (for example 4096 x 2048 pixels), and can be up to 150 MB and 8192 x 4096 pixels. The NIM checks the size first, so an image that breaks both rules gets the size error.The following example uses the sample Python client with a custom HDR file. The shipped
sample_lounge.hdris the Lounge preset, so the first command mirrors it into a different environment map: -
angle_pan_radians: Pan angle for rotating the HDR environment, in radians (range -π to π). Default is -1.5708 (-90 degrees). -
angle_v_fov_radians: Vertical field of view of the HDR environment map when it is used as the background (background_source = 2), in radians (range 0.1 to π). Default is 1.0472 (60 degrees).
Autorotate Options
-
autorotate: Enable automatic rotation of the HDR environment during video playback for dynamic lighting effects. -
rotation_rate: Speed of rotation in radians per second (range 0 to 2π). Default is 0.349 (20 degrees per second). Only applies when autorotate is enabled.The following example enables autorotate with a custom rotation speed:
Background Source Options
-
background_source: Selects which background source to use.0(UNSPECIFIED): Use source video frame as background (default).1(FROM_IMAGE): Use custom background image uploaded viaImageData.2(FROM_HDR): Use HDR environment map as background.
The following example uses the sample Python client with the HDR environment map as the background:
-
Custom background image: with
background_source = 1(FROM_IMAGE), send the image inline asImageDatawithIMAGE_TYPE_BACKGROUND. PNG, JPEG, and HDR are supported. The server detects the format from the image bytes, sobackground_image_typeis not used. The NIM decodes the background image and resizes it to the video once per request, so a large image or an.hdrfile adds time to each request. For the shortest requests, use a PNG or JPEG background at the output resolution. The image can be up to 150 MB and 8192 x 4096 pixels. -
background_color: Solid background color as an integer in0xRRGGBBform. If the request does not set it, the NIM uses0x808080(gray); the sample client sets it only when you pass--background-color. Used whenbackground_sourceisFROM_IMAGEand no image is uploaded. The following example composites the subject over solid blue:
Effect Parameters
-
foreground_gain: Controls the strength of relighting applied to the foreground subject (0.0–2.0, default: 1.0). Higher values produce more intense relighting effects. -
background_gain: Controls the strength of relighting applied to the background (0.0–2.0, default: 1.0). Applies only whenbackground_sourceis2(FROM_HDR).
Encoding Options
To configure the encoding quality of the output video, use the output_video_encoding parameter. The encoding uses a VideoEncoding message with three mutually exclusive modes: lossy, lossless, or custom.
If the configuration message has no output_video_encoding, the NIM uses lossy mode with the auto bitrate and an IDR interval of about 2 seconds of video: the input frame rate multiplied by 2, in frames (60 frames for 30 fps input). If a lossy message does not set bitrate_mbps or idr_interval, the NIM uses the same default for that field. In lossy mode, the sample client always sends both fields.
The sample client sends one mode for each request. If you use --lossless and --custom-encoding-params together, the client stops with a configuration error. With --lossless or --custom-encoding-params, the client ignores --bitrate and --idr-interval and prints a warning.
-
Bitrate control (lossy mode): Target bitrate is specified in megabits per second (
bitrate_mbps). Use0for auto, which selects a bitrate from the output resolution: 10 Mbps up to 720p (1280×720 pixels), 15 Mbps up to 1080p (1920×1080 pixels), and 25 Mbps above 1080p. Higher values improve quality and increase output size; lower values save bandwidth at the cost of quality.-
In the configuration message, set
output_video_encoding.lossy.bitrate_mbpsto the desired Mbps (1 to 200) or0for auto. -
The sample Python CLI uses
--bitratein bps (rounded to Mbps for the proto); default matches 10 Mbps (10,000,000 bps). Example explicit bitrate:
-
-
IDR interval control:
idr_intervalis the spacing between IDR frames in frames (not seconds). The sample CLI defaults to 8 frames and always sends a value to the server. Values must be ≥ 1 (0is invalid). Sample CLI override: -
Lossless mode: Encodes the output video with lossless H.264 compression. Use lossless mode when the output is an intermediate file for more work, such as color grading, compositing, editing, or a quality comparison against a reference. To deliver video to viewers, use lossy mode.
Before you use lossless mode, consider these points:
- Output files are several times larger than lossy output, so each request transfers and stores more data.
- The NIM converts the frames to 8-bit YUV 4:2:0 before it encodes them, as in lossy mode. The encoding adds no loss, but the color information is subsampled to 4:2:0. Lossless mode does not give 4:4:4 or RGB output.
- The output uses the H.264 High 4:4:4 Predictive profile, which lossless H.264 requires. Some players and devices decode only the Baseline, Main, and High profiles. Make sure that your player or tool decodes this profile.
- Bitrate and IDR interval are lossy-mode settings and do not apply in lossless mode.
To enable lossless mode with the sample client, use the
--losslessflag: -
Custom encoding parameters: Provides fine-grained control for expert users via JSON configuration. These parameters configure properties of the DeepStream H264 encoder.
To specify custom encoding parameters, use the
--custom-encoding-paramsoption:The NIM accepts these encoder (
nvv4l2h264enc) properties:bitrate,maxbitrate,iframeinterval,idrinterval,vbvbufsize,vbvinit,SliceIntraRefreshInterval,slice-header-spacing,preset-id,aq,cq,num-B-Frames,num-Ref-Frames,control-rate,tuning-info-id,profile,temporalaq,insert-vui,insert-sps-pps,insert-aud,disable-cabac, andextended-colorformat. Any other key, such asidr_intervalorqp-range, or a value outside the property’s range makes the request fail withINVALID_ARGUMENT.B-frames (
num-B-Frames1 or 2) needprofile2(Main),4(High), or7(High 4:4:4), andidrintervalandiframeintervalvalues that are multiples ofnum-B-Frames+ 1. If the request does not setprofileoridrinterval, the NIM usesprofile4and the smallest suchidrintervalof 8 or more. Another profile or interval makes the request fail withINVALID_ARGUMENT.With custom encoding parameters, the
bitrateandidrintervalkeys in the JSON set the bitrate and IDR interval; if the JSON has nobitratekey, the encoder uses 20 Mbps.bitrateandmaxbitrateare in bits per second (5000000is 5 Mbps), as--bitrateis; the lossybitrate_mbpsfield of the configuration message is in Mbps. Use with caution because incorrect values can affect output quality or encoding stability.
The following example builds the configuration message of a request with the Python client’s generated stubs. It sends nothing and prints nothing; to send it, use generate_requests() in scripts/relighting.py:
The sample client builds the complete request stream in generate_requests() in scripts/relighting.py.
For more information, refer to the NIM Clients GitHub repository: NVIDIA AI for Media NIM Clients.