Release Notes for NVIDIA NIM for LLMs#

This page contains the release notes for NVIDIA NIM for Large Language Models (LLMs).

Release 1.14.0#

Summary#

NVIDIA NIM for LLMs 1.14.0 introduces top-level parameter support for commonly used request parameters, improving API usability and consistency.

New Features#

The following are the new features in 1.14.0:

Added support for Oracle Cloud Infrastructure (OCI) Object Storage via the Amazon S3 Compatibility API. See Use OCI S3 Object Storage with NVIDIA NIM for LLMs for setup instructions.
Added top-level parameter support for the guided_decoding_backend, ignore_eos, and top_k fields.
- For a guided_decoding_backend migration example, see Specify a Custom Backend at Runtime.
Added TRT-LLM support for the following parameters at the top level on /v1/chat/completions. The response structure for these parameters matches the vLLM implementation:
- min_tokens
- include_stop_str_in_output
- repetition_penalty
- guided_decoding_backend
Added experimental support for the TRT-LLM PyTorch backend (with the LLM API).
- To enable, set NIM_USE_TRTLLM_LEGACY_BACKEND=0 and NIM_DISABLE_TRTLLM_PYTORCH_RT=0. See Multi-LLM NIM for more information.
- The following features are not supported:
  - LoRA
  - logprobs
  - logit_bias
  - guided_grammar
  - Air-gapped deployments
KV Cache reuse (NIM_ENABLE_KV_CACHE_REUSE) is now enabled by default for improved performance. There are security implications depending on your use case, please review Structuring Applications to Secure the KV Cache.

Deprecations#

The following fields are deprecated under nvext and should be used at the top level instead:
- ignore_eos
- top_k
- repetition_penalty
- guided_choice
- guided_json
- guided_regex
- guided_grammar
- guided_decoding_backend
- min_tokens
Note

These fields will be removed from nvext in a later release and will only be available at the top level.

The following migration example shows how to move fields from nvext to top-level parameters:

// Old (deprecated)
{
  "model": "meta/llama-3.1-8b-instruct",
  "nvext": {
    "ignore_eos": true,
    "top_k": 5
  }
}

// New (recommended)
{
  "model": "meta/llama-3.1-8b-instruct", 
  "ignore_eos": true,
  "top_k": 5
}

Known Issues Fixed in 1.14.0#

The following are the previous known issues that were fixed in 1.14.0:

No known issues were fixed in this release.

New Known Issues in 1.14.0#

The following are the new known issues discovered in 1.14.0:

Some top-level parameters can trigger log warnings.
SGLang profiles may run out of memory (OOM) under high load. Set NIM_KVCACHE_PERCENT=0.7 to help mitigate the issue.
On the TRT-LLM backend, setting temperature=0 enforces greedy decoding, making repetition_penalty ineffective.

Previous Releases#

The following are links to the previous release notes.

1.13 | 1.12 | 1.11 | 1.10 | 1.8 | 1.7 | 1.6 | 1.5 |1.4 | 1.3 | 1.2 | 1.1 | 1.0

All Current Known Issues#

The following are the current (unfixed) known issues from all previous versions:

Tip

For related information, see Troubleshoot NVIDIA NIM for LLMs.

General#

The top_logprobs parameter is not supported.
All models return a 500 when setting logprobs=2, echo=true, and stream=false; they should return a 200.
Filenames should not contain spaces if a custom fine-tuned model directory is provided.
Some stop words might not work as expected and might appear in the output.
The maximum supported context length may decrease based on memory availability.
The structured generation of regular expressions results may have unexpected responses. We recommend that you provide a strict answer format, such as \\boxed{}, to get the correct response.
The model quantization is fp8, but the logs incorrectly display it as bf16.

Deployment and Environment#

Deploying with KServe can require changing permissions for the cache directory. See the Serving models from local assets section for details.
GH200 NVIDIA driver <560.35.03 can cause a segmentation fault or hanging during deployment. Fixed in GPU driver 560.35.03
Optimized engines (TRT-LLM) aren’t supported with NVIDIA vGPU. To use optimized engines, use GPU Passthrough.
Prompts with Unicode characters in the range from 0x0e0020 to 0x0e007f can produce unpredictable responses. NVIDIA recommends that you filter these characters out of prompts before submitting the prompt to an LLM.
The container may crash when building local TensorRT LLM engines if there isn’t enough host memory. If that happens, try setting NIM_LOW_MEMORY_MODE=1.
Out-of-Bounds (OOB) sequence length with tensorrt_llm-local_build is 8K. Use the NIM_MAX_MODEL_LEN environment variable to modify the sequence length within the range of values supported by a model.
vGPU related issues:
- trtllm_buildable profiles might encounter an Out of Memory (OOM) error on vGPU systems, which can be fixed via NIM_LOW_MEMORY_MODE=1 flag.
- When using vGPU systems with trtllm_buildable profiles, you might still encounter a broken connection error. For example, client_loop: send disconnect: Broken pipe.
vLLM for A100 and H200 is not supported.
NIM with vLLM backend may intermittently enter a state where the API returns a “Service in unhealthy” message. This is a known issue with vLLM (vllm-project/vllm#5060). You must restart the NIM in this case.
You can’t deploy fp8 quantized engines on H100-NVL GPUs with deterministic generation mode on. For more information, refer to Deterministic Generation Mode in NVIDIA NIM for LLMs.
INT4/INT8 quantized profiles are not supported for Blackwell GPUs.
When using Native TLS Stack to download the model, you should set --ulimit nofile=1048576 in the docker run command. If a Helm deployment is run behind the proxy, the limit must be increased on host nodes or a custom command must be provided. See Deploying Behind a TLS Proxy for details.
Air Gap Deployments of a model like Llama 3.3 Nemotron Super 49B that uses the model directory option might not work if the model directory is in the HuggingFace format. Switch to using NIM_FT_MODEL in those cases. For more information, refer to Air Gap Deployment.
Llama-3.1-Nemotron-Ultra-253B-v1 does not work on H100s and A100s. Use H200s and B200s to deploy successfully.
Models with 8 billion parameters require NIM_KVCACHE_PERCENT=0.8 for tp=1 profiles.
NIM_ENABLE_PROMPT_LOGPROBS=1 is not supported for the TRTLLM backend.
LoRA deployments on TRT-LLM and vLLM backends can experience significant latency and throughput degradation. The performance impact can vary by model size and configuration.
PCI Access Control Services (ACS) must be disabled for NCCL to function properly in multi-GPU deployments (tp > 1). For instructions on how to disable PCI ACS, refer to the NCCL troubleshooting guide.

When deploying models, you may get Download Error - Too many open files. To resolve the error, do the following:

For Docker deployments, add the --ulimit nofile=65536:65536 option.

For Kubernetes deployments, run the following commands:

sudo mkdir -p /etc/systemd/system/containerd.service.d 
echo "[Service]" | sudo tee /etc/systemd/system/containerd.service.d/override.conf 
echo "LimitNOFILE=65536" | sudo tee -a /etc/systemd/system/containerd.service.d/override.conf 
sudo systemctl daemon-reload 
sudo systemctl restart containerd 
sudo systemctl restart kubelet

Model Support and Functionality#

Many models return a 500 error when using structured generation with context-free grammar.
Function calling and structured generation is not supported for pipeline parallelism greater than 1.
Locally-built fine tuned models are not supported with FP8 profiles.
P-Tuning isn’t supported.
No tokenizer found error when running PEFT. This warning can be safely ignored.
vllm + LoRA profiles for long context models (model_max_len > 65528) will not load resulting in ValueError: Due to limitations of the custom LoRA CUDA kernel, max_num_batched_tokens must be <= 65528 when LoRA is enabled. As a workaround you can set NIM_MAX_MODEL_LEN=65525 or lower
When repetition_penalty=2, the response time for larger models is greater. Use repetition_penalty=1 on larger models.
The following are the known issues with function calling:
- Format enforcement is not guaranteed by default. The tool_choice parameter no longer supports required as a value, despite its presence in the OpenAPI spec. This might impact the accuracy of tool calling for some models.
- Function calling no longer uses guided decoding, resulting in lower accuracy for smaller models like Llama 3.2 1B/3B Instruct.
- Smaller parameter models (<= 8 billion parameters) can have tool calling enabled but are highly inaccurate due to their limited parameter count. We don’t recommend using such models for tool calling use cases.
The following are the known issues with the custom guided decoding backend:
- The fast_outlines backend is deprecated.
- Guided decoding now defaults to xgrammar instead of outlines. For more information, refer to Structured Generation with NVIDIA NIM for LLMs.
- The guided decoding backend cannot be accessed without using constraint fields. Set guided_regex to ".*" to act as a minimal trigger for the guided decoding backend.
- The outlines guided decoding backend is not supported for sglang profiles.
- Custom guided decoding requires backends that implement the set_custom_guided_decoding_parameters method, as defined in the backend file.
- Guided decoding does not work for TP > 1 for sglang profiles.
- Deepseek R1 may produce less accurate results when using guided decoding.
DeepSeek models do not support tool calling.
LoRA does not work for mistral-nemo-12b-instruct.
The vLLM backend is not supported on Llama Nemotron models.

API and Metrics#

GET v1/metrics API is missing from the docs page (http://HOST-IP:8000/docs, where HOST-IP is the IP address of your host).
Logarithmic Probabilities (logprobs) support with echo:
- TRTLLM engine needs to be built explicitly with --gather_generation_logits
- Enabling this may impact model throughput and inter-token latency.
- NIM_MODEL_NAME must be set to the generated model repository.
logit_bias is not available for any model using the TRT-LLM backend.
logprobs=2 is only supported for TRT-LLM (optimized) configurations for Reward models; this option is supported for the vLLM (non-optimized) configurations for all models.
Empty metrics values on multi-GPU TensorRT-LLM model. Metrics items gpu_cache_usage_perc, num_request_max, num_requests_running, num_requests_waiting, and prompt_tokens_total won’t be reported for multi-GPU TensorRT-LLM model, because TensorRT-LLM currently doesn’t expose iteration statistics in orchestrator mode.
If ignore_eos=true, the model ignores EOS tokens and keeps generating until a custom stop token is encountered or the max token limit is reached (if not set, the default context window size is 128 tokens). For VLLM and simple queries, we recommend using ignore_eos=false (default).

All Current Known Issues for Specific Models#

The following are the current (unfixed) known issues from all previous versions, that are specific to a model:

Tip