Configure the LLM | NVIDIA Blueprint for Video Search and Summarization

This page describes how to configure the LLM for the VSS agent workflow—both local (NIM on the same machine as the agent) and remote (LLM served elsewhere). You can run the LLM and VLM locally or remotely, in any combination that suits your setup.

For remote LLMs, as long as the API is OpenAI compatible, you can point the agent to it for reasoning and coordinating tool calls during report generation, video understanding, video summarization, and more. Remote endpoints can enable rapid startup when they are already up and running.

Agent containers default OPENAI_API_KEY to NOAPIKEYSET for local OpenAI-compatible clients, including direct NAT evaluation commands. Set a real OPENAI_API_KEY for an authenticated OpenAI endpoint; supplied Compose values and Helm extraEnv overrides take precedence.

The Base and LVS evaluation configurations give eval_llm_judge the stock reasoning model’s 32,768-token output budget so it can finish its score. When selecting a different judge model, adjust max_tokens to that model’s supported output budget. QA judges keep their analysis in the reasoning channel and return only the decimal score in their final response, as required by the score parser.

Local LLM customization

When deploying with a local LLM, the agent workflow uses a default NIM endpoint. You can customize it as follows:

Default model: nvidia/nemotron-3.5-lightning-30b-a3b for every profile (configured as a local NIM endpoint; each profile’s overrides.env carries the matching LLM_NAME and LLM_NAME_SLUG).
Custom NIM env file: Set LLM_ENV_FILE in the selected profile’s *-overrides.env file to the path (absolute or relative to deploy/docker) of an env file that overrides NIM settings. See NIM configuration settings in Prerequisites.
Different local model: Set LLM_NAME and its matching LLM_NAME_SLUG in the selected profile’s *-overrides.env file to use one of the supported local models:

LLM_NAME LLM_NAME_SLUG
nvidia/nemotron-3.5-lightning-30b-a3b (default) nemotron-3.5-lightning-30b-a3b
nvidia/NVIDIA-Nemotron-Nano-9B-v2-FP8 nvidia-nemotron-nano-9b-v2-fp8
Supported models: nvidia/nemotron-3.5-lightning-30b-a3b ships sizing for every accepted hardware profile, including the DGX-SPARK / AGX-THOR / IGX-THOR edge profiles. nvidia/NVIDIA-Nemotron-Nano-9B-v2-FP8 (11.7 GB) is the alternative for GPUs too small for the default and is only used when passed explicitly via --llm. Serve any other model through a remote endpoint; ensure GPU memory meets requirements and refer to LLM NIM documentation.
GPU assignment: Set LLM_DEVICE_ID to pin the local LLM to a specific GPU (e.g. when sharing with a local VLM). See the quickstart deploy tab-set for examples.

`LLM_NAME`	`LLM_NAME_SLUG`
`nvidia/nemotron-3.5-lightning-30b-a3b` (default)	`nemotron-3.5-lightning-30b-a3b`
`nvidia/NVIDIA-Nemotron-Nano-9B-v2-FP8`	`nvidia-nemotron-nano-9b-v2-fp8`

Setup assumptions for remote LLMs

Before you start, make sure you followed the Prerequisites section. It is also recommended to review the different parameters described in the Quickstart guide.

Register an account at NVIDIA NGC. Get an NGC API key.
Similarly login to NVIDIA NIM API catalog. Get a NIM CATALOG API key.
Ensure you are logged in to the NGC registry for docker to pull container images:
Run in the deployment host terminal
```
docker login nvcr.io
Username: $oauthtoken
Password: <PASTE_NGC_API_KEY_HERE>
```
Ensure you have NGC_CLI_API_KEY set in your environment for the agent workflow to pull NIM containers:
Run in the deployment host terminal
```
export NGC_CLI_API_KEY=<PASTE_NGC_API_KEY_HERE>
```

Copy the checked-in template to any filename matching *-overrides.env before modifying it. Files matching that pattern are ignored by Git so local settings and credentials are not committed. This guide uses user-overrides.env as a concrete example; if you choose another name, use it consistently in the deployment commands:

Run in the deployment host terminal

cp deploy/docker/developer-profiles/dev-profile-base/overrides.env \
  deploy/docker/developer-profiles/dev-profile-base/user-overrides.env

For every configuration below, save the shown values in deploy/docker/developer-profiles/dev-profile-base/user-overrides.env, then validate and start the profile with the native Docker Compose command in Quickstart.

Managed remote LLMs

Using NIM containers (hosted on build.nvidia.com)

Choose a particular NIM LLM from the NVIDIA NIM API catalog.

Deploy the agent workflow and point it to the cloud-managed remote NIM for the LLM:

   # developer-profiles/dev-profile-base/user-overrides.env
   LLM_MODE=remote
   LLM_NAME_SLUG=none
   LLM_BASE_URL=https://integrate.api.nvidia.com
   LLM_NAME=nvidia/nemotron-3.5-lightning-30b-a3b
   LLM_MODEL_TYPE=nim
   NVIDIA_API_KEY=<PASTE_NIM_CATALOG_API_KEY_HERE>
   VLM_MODE=local
   VLM_DEVICE_ID=0
   RT_VLM_DEVICE_ID=0

Examples of hosted NIM containers:

nvidia/nemotron-3.5-lightning-30b-a3b (the blueprint default)
nvidia/nemotron-3-nano-30b-a3b
openai/gpt-oss-120b

Remote VLM endpoints are supported too — see Configure the VLM — but https://integrate.api.nvidia.com does not serve the blueprint’s VLMs: nvidia/cosmos3-nano-reasoner is downloadable from the API Catalog, not hosted there. Point a remote VLM at a self-hosted or third-party endpoint, or keep the VLM local (VLM_MODE=local, the blueprint default) as shown above.

You can adapt these instructions and combinations as shown in the following sections.

Using frontier OpenAI models

Register an account at the OpenAI API platform. Get an API key.
Choose a compatible model from the list in curl https://api.openai.com/v1/models -H "Authorization: Bearer <PASTE_API_KEY_HERE>".

Deploy the agent workflow and point it to the chosen model name:

# developer-profiles/dev-profile-base/user-overrides.env
LLM_MODE=remote
LLM_NAME_SLUG=none
LLM_BASE_URL=https://api.openai.com
LLM_NAME=gpt-5.6
LLM_MODEL_TYPE=openai
OPENAI_API_KEY=<PASTE_API_KEY_HERE>
NVIDIA_API_KEY=<PASTE_NIM_CATALOG_API_KEY_HERE>
VLM_MODE=local
VLM_DEVICE_ID=0
RT_VLM_DEVICE_ID=0

Examples of compatible openAI models:

Self-hosted remote LLMs

Using downloadable NIM Containers

Check GPU requirements to ensure the machine that will host the LLM NIM container is supported.
Set NGC_API_KEY in your environment to be passed to the containers so they can pull model weights:
Run in the LLM host terminal
```
export NGC_API_KEY=<PASTE_NGC_API_KEY_HERE>
```

On the compatible remote machine within your own infrastructure, start the LLM NIM container and ensure it is reachable by the VSS agent workflow machine:

Run in the LLM host terminal

export LOCAL_NIM_CACHE=~/.cache/nim
mkdir -p "$LOCAL_NIM_CACHE"
chmod -R 777 "$LOCAL_NIM_CACHE"
docker run -it --rm \
    --gpus 'device=0' \
    --shm-size=16gb \
    -e NGC_API_KEY \
    -v "$LOCAL_NIM_CACHE:/opt/nim/.cache" \
    -p 30081:8000 \
    nvcr.io/nim/nvidia/nemotron-3-nano:latest

In case you want to run several models (LLMs, VLMs) on the same machine, make sure to assign different GPUs using —gpus ‘device=1’ flag. You may assign multiple GPUs using —gpus ‘device=1,2,3’ etc.

You may encounter errors if the remote machine does not support this model. For instance, you might see errors like torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 1.19 GiB. GPU 0 has a total capacity of 44.42 GiB of which 654.31 MiB is free. This error indicates that your GPU does not have sufficient VRAM to run the selected model. To resolve these kinds of errors, you can choose a smaller model, adjust the command with parameter arguments inspired from hardware profiles, or leverage multiple GPUs.

Run in the LLM host terminal

# Example: leveraging multiple GPUs by setting --gpus and tensor parallelism (NIM_TENSOR_PARALLEL_SIZE)
export LOCAL_NIM_CACHE=~/.cache/nim
mkdir -p "$LOCAL_NIM_CACHE"
chmod -R 777 "$LOCAL_NIM_CACHE"
docker run -it --rm \
    --gpus '"device=3,4"' \
    --shm-size=16gb \
    -e NGC_API_KEY \
    -e NIM_TENSOR_PARALLEL_SIZE=2 \
    -v "$LOCAL_NIM_CACHE:/opt/nim/.cache" \
    -p 30081:8000 \
    nvcr.io/nim/nvidia/nemotron-3-nano:latest

Start the VSS agent workflow and point it to the remote LLM NIM container:

# developer-profiles/dev-profile-base/user-overrides.env
LLM_MODE=remote
LLM_NAME_SLUG=none
LLM_BASE_URL=http://<remote-host>:30081
LLM_NAME=<llm-model-id-from-v1-models>
LLM_MODEL_TYPE=nim
NVIDIA_API_KEY=<PASTE_NIM_CATALOG_API_KEY_HERE>
VLM_MODE=local
VLM_DEVICE_ID=0
RT_VLM_DEVICE_ID=0

Examples of downloadable NIM containers:

Using vLLM Container

You can deploy an open-source LLM from Hugging Face or any other source using the NVIDIA vLLM container. Check the NVIDIA vLLM container documentation for an introduction to vLLM and the vLLM container.

Check GPU requirements to ensure the machine that will host the LLM is supported.

On the compatible remote machine within your own infrastructure, start the NVIDIA vLLM container with a Hugging Face model and ensure it is reachable by the VSS agent workflow machine:

Run in the LLM host terminal

docker run --gpus 'device=0' -it --rm -p 30081:8000 nvcr.io/nvidia/vllm:26.01-py3 \
    python3 -m vllm.entrypoints.openai.api_server \
    --model OpenAI/gpt-oss-20b \
    --trust-remote-code \
    --tensor-parallel-size 1 \
    --gpu-memory-utilization 0.85 \
    --port 8000

If you see errors such as ValueError: To serve at least one request with the models's max seq len (262144), you may need to use the --max-model-len flag to limit the maximum context window (input + output tokens per request) to allow the model to fit on smaller GPUs. While some models support up to 256K tokens, VSS agent requests typically use under 10K-20K tokens. Setting a lower value (e.g. 65536) reduces GPU memory reserved for KV cache.

Start the VSS agent workflow and point it to the remote vLLM model:

# developer-profiles/dev-profile-base/user-overrides.env
LLM_MODE=remote
LLM_NAME_SLUG=none
LLM_BASE_URL=http://<remote-host>:30081
LLM_NAME=<llm-model-id-from-v1-models>
LLM_MODEL_TYPE=openai
NVIDIA_API_KEY=<PASTE_NIM_CATALOG_API_KEY_HERE>
VLM_MODE=local
VLM_DEVICE_ID=0
RT_VLM_DEVICE_ID=0

Examples of Hugging Face LLMs to use:

See the vLLM documentation for a list of supported VLMs and LLMs.