> This page is for NVIDIA Relighting NIM, version 1.1.0.
> For other versions, use one of these documentation indexes:
> - 1.2.0 (Latest) (default): https://docs.nvidia.com/nim/maxine/relighting/1.2.0/llms.txt
> - 1.1.0: https://docs.nvidia.com/nim/maxine/relighting/1.1.0/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nim/maxine/llms.txt.

# Basic Inference

1. Perform a health check on the gRPC endpoint.
   - Install `grpcurl` from [github.com/fullstorydev/grpcurl/releases](https://github.com/fullstorydev/grpcurl/releases).

     Example commands to run on Ubuntu:

     ```bash
     wget https://github.com/fullstorydev/grpcurl/releases/download/v1.9.1/grpcurl_1.9.1_linux_amd64.deb
     sudo dpkg -i grpcurl_1.9.1_linux_amd64.deb
     ```

   - Download the health checking proto:

     ```bash
     wget https://raw.githubusercontent.com/grpc/grpc/master/src/proto/grpc/health/v1/health.proto
     ```

   - Run the health check:

     ```bash
     grpcurl --plaintext --proto health.proto localhost:8001 grpc.health.v1.Health/Check
     ```

     If the service is ready, you get a response similar to the following:

     ```bash
     { "status": "SERVING" }
     ```

   <Note>
   For using grpcurl with an SSL-enabled server, avoid using the `--plaintext` argument, and use `--cacert` with a CA certificate, `--key` with a private key, or `--cert` with a certificate file. For more details, refer to `grpcurl --help`.
   </Note>

2. Download the Relighting client code by cloning the gRPC client repository ([NVIDIA AI for Media NIM Clients](https://github.com/NVIDIA-Maxine/nim-clients)):

   ```bash
   git clone https://github.com/NVIDIA-Maxine/nim-clients.git

   # Go to the 'relighting' folder
   cd nim-clients/relighting/
   ```

3. For the Python client, install the required dependencies:

   ```bash
   # Install Python on Linux
   sudo apt-get install python3-pip
   pip install -r requirements.txt
   ```

## Compile the Protos (Optional)

If you want to use the client code provided in the NIM Clients GitHub repository ([NVIDIA AI for Media NIM Clients](https://github.com/NVIDIA-Maxine/nim-clients)), you can skip this step.

The proto files are available in the `relighting/protos` folder. You can compile them to generate client interfaces in your preferred programming language. For more details, refer to [Supported languages](https://grpc.io/docs/languages/) in the gRPC documentation.

The following example shows how to compile the protos for Python and Node.js on Linux and Windows.

### Python

The `grpcio` version needed for compilation can be referred from `requirements.txt`

To compile protos on Linux, run the following commands:

```bash
cd relighting/protos/linux/

chmod +x compile_protos.sh
./compile_protos.sh
```

To compile protos on Windows, run the following commands:

```bash
cd relighting/protos/windows/

./compile_protos.bat
```

## Running Inference via Python Script

You can use the sample client script in the [NVIDIA AI for Media NIM Clients](https://github.com/NVIDIA-Maxine/nim-clients) repo to send a gRPC request to the hosted NIM server:

1. Go to the Python relighting directory.

   ```bash
   cd relighting/scripts
   ```

2. Run the command to send a gRPC request. (All command-line parameters are optional.)

   ```bash
   python relighting.py --target <server_ip:port> --video-input <input video file path> --output <output file path and the file name> --hdri-id <hdr preset id> --ssl-mode <ssl mode value> --ssl-key <ssl key file path> --ssl-cert <ssl cert filepath> --ssl-root-cert <ssl root cert filepath>
   ```

3. The Relighting NIM supports both streaming and transactional modes. The server automatically selects the appropriate mode based on whether the input video is streamable (indicated by metadata at the start of the file). Streamable inputs are preferred for large files. For more information, see [Input Modes](#input-modes).
   ```bash
   python relighting.py --target <server_ip:port> --video-input <streamable input video file path> --output <output file path and the file name> --hdri-id <hdr preset id>
   ```

<Note>
The first inference is not indicative of the model's actual performance because it includes the time taken by the Triton Inference Server to load the models for the first time. The subsequent inference requests reflect the actual processing performance.

For Blackwell GPUs, the initial inference might time out because of the time needed to load the models. If timeout occurs, send another request; subsequent inferences reflect the actual processing performance.
</Note>

To view details of command-line arguments, run the following command:

```bash
python relighting.py -h
```

You get a response similar to the following:

```bash
usage: relighting.py [-h] [--ssl-mode {DISABLED,MTLS,TLS}] [--ssl-key SSL_KEY] [--ssl-cert SSL_CERT] [--ssl-root-cert SSL_ROOT_CERT] [--target TARGET] [--video-input VIDEO_INPUT]
                     [--output OUTPUT] [--bitrate BITRATE] [--idr-interval IDR_INTERVAL] [--lossless] [--custom-encoding-params CUSTOM_ENCODING_PARAMS] [--benchmark-grpc]
                     [--hdr HDR_FILE] [--hdri-id {0,1,2,3,4}] [--pan PAN] [--vfov VFOV] [--autorotate] [--rotation-rate ROTATION_RATE]
                     [--background-source {0,1,2}] [--background-image BACKGROUND_IMAGE] [--background-image-type {0,1,2}] [--background-color BACKGROUND_COLOR]
                     [--foreground-gain FOREGROUND_GAIN] [--background-gain BACKGROUND_GAIN] [--blur BLUR] [--specular SPECULAR]

Run Relighting with all available features

options:
  -h, --help                                           show this help message and exit

SSL:
  --ssl-mode {DISABLED,MTLS,TLS}                       Flag to set SSL mode, default is DISABLED (default: DISABLED)
  --ssl-key SSL_KEY                                    The path to ssl private key. (default: ../ssl_key/ssl_key_client.pem)
  --ssl-cert SSL_CERT                                  The path to ssl certificate chain. (default: ../ssl_key/ssl_cert_client.pem)
  --ssl-root-cert SSL_ROOT_CERT                        The path to ssl root certificate. (default: ../ssl_key/ssl_ca_cert.pem)
  --target TARGET                                      IP:port of gRPC service, when hosted locally. (default: 127.0.0.1:8001)

Input/Output:
  --video-input VIDEO_INPUT                            Path to the input video file. (default: assets/sample_video.mp4)
  --output OUTPUT                                      Path for the output video file. (default: None)
  --bitrate BITRATE                                    Output video bitrate in bps. 0 = auto. (default: 10000000)
  --idr-interval IDR_INTERVAL                          IDR interval in frames (server auto ~2 s GOP if omitted in API). (default: 8)
  --lossless                                           Enable lossless video encoding. (default: False)
  --custom-encoding-params CUSTOM_ENCODING_PARAMS      Custom encoding parameters as JSON. (default: None)
  --benchmark-grpc                                     Measure gRPC roundtrip only (exclude disk read/write). (default: False)

HDR/Illumination:
  --hdr HDR_FILE                                       Custom .hdr file. (default: None)
  --hdri-id {0,1,2,3,4}                                HDR preset: 0=Lounge, 1=Cobblestone Street Night, 2=Glasshouse Interior, 3=Little Paris Eiffel Tower, 4=Wooden Studio. (default: 0)
  --pan PAN                                            Pan angle (degrees). (default: -90.0)
  --vfov VFOV                                          Vertical FOV (degrees). (default: 60.0)

Autorotate:
  --autorotate                                         Auto-rotate the HDR environment. (default: False)
  --rotation-rate ROTATION_RATE                        Rotation speed (degrees/s). (default: 20.0)

Background:
  --background-source {0,1,2}                          0=source video, 1=custom image, 2=HDR projection. (default: 0)
  --background-image BACKGROUND_IMAGE                  Custom background image (PNG, JPG, or HDR). (default: None)
  --background-image-type {0,1,2}                      Format hint: 0=auto-detect, 1=HDRI, 2=standard (PNG/JPG). Only relevant with --background-source=1. (default: None)
  --background-color BACKGROUND_COLOR                  Solid color: 'gray' or hex '0xRRGGBB'. (default: None)

Effect Parameters:
  --foreground-gain FOREGROUND_GAIN                    Foreground relighting strength 0.0-2.0 (default: 1.0)
  --background-gain BACKGROUND_GAIN                    Background relighting strength 0.0-2.0 (default: 1.0)
  --blur BLUR                                          Background blur strength 0.0-1.0 (default: 0.0)
  --specular SPECULAR                                  Specular highlight intensity 0.0-2.0 (default: 0.0)

```

To get more information about how to run the client with these parameters, refer to the NVIDIA AI for Media NIM Clients GitHub repository: [NVIDIA AI for Media NIM Clients](https://github.com/NVIDIA-Maxine/nim-clients).

For more information about configuring advanced parameters such as bitrate, IDR interval, HDR settings, and background options, see [Advanced Usage](/nim/maxine/relighting/1.1.0/advanced-usage).

For command-line arguments that aren't specified, the script uses the following default values.

### Default Command-Line Arguments

| **Argument**             | **Default Value**                                 |
| ------------------------ | ------------------------------------------------- |
| `target`                 | `127.0.0.1:8001`                                  |
| `video-input`            | `assets/sample_video.mp4`                         |
| `output`                 | None (auto: `<input_stem>-relighting_output.mp4`) |
| `bitrate`                | `10000000`                                        |
| `idr-interval`           | `8`                                               |
| `custom-encoding-params` | `None`                                            |
| `hdri-id`                | `0` (Lounge)                                      |
| `pan`                    | `-90.0`                                           |
| `vfov`                   | `60.0`                                            |
| `autorotate`             | `False`                                           |
| `rotation-rate`          | `20.0`                                            |
| `background-source`      | `0` (source video)                                |
| `foreground-gain`        | `1.0`                                             |
| `background-gain`        | `1.0`                                             |
| `blur`                   | `0.0`                                             |
| `specular`               | `0.0`                                             |
| `ssl-mode`               | `DISABLED`                                        |
| `ssl-key`                | `../ssl_key/ssl_key_client.pem`                   |
| `ssl-cert`               | `../ssl_key/ssl_cert_client.pem`                  |
| `ssl-root-cert`          | `../ssl_key/ssl_ca_cert.pem`                      |

### Example Commands

- Basic inference with default arguments:

  ```bash
  python3 relighting.py --target 127.0.0.1:8001
  ```

- Run inference with custom input file and HDR preset:

  ```bash
  python relighting.py --target 127.0.0.1:8001 --video-input /path/to/video.mp4 --hdri-id 3
  ```

- Run inference with a streamable video (server auto-selects streaming mode):

  ```bash
  python relighting.py --target 127.0.0.1:8001 --video-input /path/to/streamable_video.mp4
  ```

- Run inference with custom HDR file:

  ```bash
  python relighting.py --target 127.0.0.1:8001 --video-input /path/to/video.mp4 --hdr /path/to/custom.hdr
  ```

- Run inference with blurred background:

  ```bash
  python relighting.py --target 127.0.0.1:8001 --video-input /path/to/video.mp4 --blur 0.5
  ```

- Run inference with custom background image:
  ```bash
  python relighting.py --target 127.0.0.1:8001 --video-input /path/to/video.mp4 --background-source 1 --background-image /path/to/bg.png
  ```

### Supported Formats

The supported format for input video is MP4 with H.264 video encoding.

### Input Modes

Relighting provides two modes for processing input files: streaming and transactional.

| Aspect                | Streaming Mode                                                       | Transactional Mode                                                                       |
| --------------------- | -------------------------------------------------------------------- | ---------------------------------------------------------------------------------------- |
| **Data Storage**      | Only frames being processed are temporarily copied in memory.        | Entire video and audio files are temporarily copied on disk.                             |
| **Processing Start**  | NIM starts processing as soon as data chunk for first frame arrives. | NIM waits to receive entire files before starting.                                       |
| **Processing Timing** | Continuous processing without waiting for complete input.            | Processing begins after all data is received.                                            |
| **Output Delivery**   | Output frames are generated and returned immediately.                | Complete output video is returned to client after inference is finished for whole video. |

#### Streaming Mode

Streaming mode is the recommended way to use Relighting. It allows inference to begin without receiving the whole video from the client. It processes video frames incrementally, and inference begins as soon as the first frame of information is available. The output frames are streamed back to the client immediately after inference.

The NIM automatically detects streamable videos and enables streaming mode. This mode delivers the lowest latency, best resource efficiency, and scales well to large files.

Use streaming mode for the following use cases:

- Best overall performance—the NIM is optimized for this path.
- Streamable video inputs.
- Applications that benefit from receiving output as it is generated, without waiting for the entire file to be uploaded.
- Large video files that benefit from incremental processing and reduced disk I/O.

Streaming mode works with streamable videos in which metadata is positioned at the beginning of the file. The NIM automatically detects streamable videos and enables streaming mode. Videos that are not streamable can be easily converted to a streamable format.

To convert your video into a streamable video, use FFmpeg with the `-movflags +faststart` option:

```bash
ffmpeg -i input.mp4 -movflags +faststart -c copy output_streamable.mp4
```

You can then specify the streamable video as input to the NIM by using the `--video-input` parameter:

```bash
python relighting.py --target 127.0.0.1:8001 --video-input ../assets/sample_video_streamable.mp4
```

#### Transactional Mode

In transactional mode, the entire video and audio files have to be received by the NIM before processing can begin. The NIM falls back to this mode when the input video is not streamable.

Transactional mode is suitable for the following use cases:

- Videos that are not optimized for streaming (such as non-streamable MP4 files in which metadata is located at the end of the file, requiring the entire file to be downloaded before playback can begin).
- Processing of small video and audio files for which streaming overhead is unnecessary.
- Applications that can wait for complete processing before receiving output.

To run Relighting in transactional mode, provide a non-streamable video as input:

```bash
python relighting.py --target 127.0.0.1:8001 --video-input ../assets/sample_video.mp4
```

<Tip>
For best performance, convert your videos to a streamable format so that the NIM can use streaming mode.
</Tip>