NVIDIA NIM for Large Language Models

NVIDIA NIM for LLMs

• Introduction
  • High Performance Features
  • Applications
  • Architecture
    • NIM Deployment Lifecycle
  • The NVIDIA Developer Program
• Release Notes
  • All Known Issues
    • 1.8.0
    • 1.7.0
    • 1.6.0
    • 1.5.0
    • 1.4.0
    • 1.3.0
    • 1.2.3
    • 1.2.1
    • 1.2.0
    • 1.1.2
    • 1.1.1
    • 1.1.0
    • 1.0
  • Release 1.8.0
    • New Language Models
    • New Features
    • Known Issues
    • Caveats
    • Fixed Issues
  • Release 1.7.1
    • New Features
  • Release 1.7.0
    • New Language Models
    • New Features
    • Known Issues
  • Release 1.6.0
    • New Language Models
    • New Features
    • Known Issues
    • Known Issues
  • Release 1.5.0
    • New Language Models
    • New Features
    • Known Issues
  • Release 1.4.0
    • New Models
    • New Features
    • Fixed Issues
    • Known Issues
  • Release 1.3.0
    • New Language Models
    • New Features
    • Known Issues
    • Software requirements updated
  • Release 1.2.3
    • New Language Models
    • Known Issues
  • Release 1.2.1
    • New Models
    • Known Issues
  • Release 1.2.0
    • New Language Models
    • New Features
    • Fixed Issues
    • Known Issues
  • Release 1.1.2
    • New Language Models
    • New Features
    • Known Issues
  • Release 1.1.1
    • Known Issues
  • Release 1.1.0
    • New Language Models
    • New Features
    • Known Issues
  • Release 1.0
• Getting Started
  • Prerequisites
    • CPU
    • GPU
    • Software
      • Installing WSL2 for Windows
  • Launch NVIDIA NIM for LLMs
    • Option 1: From API Catalog
      • Generate an API Key
      • Login to Docker
      • Download and Launch NVIDIA NIM for LLMs
    • Option 2: From NGC
      • Generate an API key
      • Export the API key
      • Docker login to NGC
      • List available NIMs
      • Launch NIM
  • Docker Run Parameters
  • Run Inference
    • OpenAI Completion Request
    • OpenAI Chat Completion Request
    • Improving TRT-LLM Performance
    • Parameter-Efficient Fine-Tuning
  • Stopping the container
  • Kubernetes Installation
  • Serving models from local assets
• Deployment Guide
  • Deploy on your own compute
  • Additional deployment guides
  • Deploy as a managed endpoint service
• Air Gap Deployment
  • Air Gap Deployment (offline cache route)
  • Air Gap Deployment (local model directory route)
• Multi-Node Deployment
  • Multi-Node Deployment with Kubernetes
  • Preparation
  • Multi-Node Models
    • LeaderWorkerSet
    • MPI Job
  • Example: Helm chart for DeepSeek R1 using an SGLang Backend
  • Troubleshooting Multi-Node Deployments
  • Multi-Node Parameters
• Deploying with Helm
  • Prerequisites
  • Configuring Helm
    • Minimal Example
  • Storage
  • Enabling Open Telemetry Tracing and Metrics
  • Launching NIM in Kubernetes
  • Running Inference
  • Troubleshooting FAQ
  • Additional Information
  • Parameters
    • Deployment Parameters
    • Autoscaling Parameters
    • Ingress Parameters
    • Probe Parameters
    • Metrics Parameters
    • Models Parameters
    • Deprecated and Legacy Model Parameters
    • Storage Parameters
    • Service Parameters
• Tutorials
  • Playbooks
  • Platform Deployment Guides
• Configuring a NIM
  • GPU Selection
    • How many GPUs do I need?
  • Shared memory flag
  • Environment Variables
  • Volumes
• Model Profiles
  • Profile Selection
  • Automatic Profile Selection
  • Profile Details
    • Optimized Profiles vs. Local Build Profiles
    • Optimization Targets
  • Quantization
• Overview
  • Structure
    • Supported Repository Types
  • S3 Support
    • Using the Feature
      • 1. Create an S3 Bucket
      • 2. Download Files
      • 3. Upload Files to S3
        • Example
      • 4. Set the Environment Variable
      • 5. Configure AWS Credentials
      • 6. Launch NIM
        • Example Docker Command
• Benchmarking
• Models
• Supported Models
  • GPUs
  • Optimized Models
    • Bielik 11B v2.3 Instruct
      • Generic Configuration
      • Supported TRT-LLM Buildable Profiles
    • Code Llama 13B Instruct
      • Optimized Configurations
      • Generic Configuration
    • Code Llama 34B Instruct
      • Optimized Configurations
      • Generic Configuration
    • Code Llama 70B Instruct
      • Optimized Configurations
      • Generic Configuration
    • DeepSeek R1
      • Supported Configurations
    • DeepSeek R1 Distill Llama 8B
      • Optimized Configurations
      • Generic Configuration
    • DeepSeek R1 Distill Llama 70B
      • Optimized Configurations
      • Generic Configuration
    • DeepSeek R1 Distill Llama 8B RTX
      • Optimized Configurations
    • DeepSeek-R1-Distill-Qwen-32B
      • Optimized Configurations
      • Generic Configuration
    • DeepSeek-R1-Distill-Qwen-7B
      • Optimized Configurations
    • DeepSeek-R1-Distill-Qwen-14B
      • Optimized Configurations
      • Generic Configuration
      • Supported TRT-LLM Buildable Profiles
    • EuroLLM 9B Instruct
      • Generic Configuration
      • Supported TRT-LLM Buildable Profiles
    • Qwen2.5 Coder 32B Instruct
      • Generic Configuration
      • Supported TRT-LLM Buildable Profiles
    • Qwen2.5 72B Instruct
      • Optimized Configurations
      • Generic Configuration
    • Qwen2.5 7B Instruct
      • Optimized Configurations
      • Generic Configuration
      • Supported TRT-LLM Buildable Profiles
    • Gemma 2 2B
      • Generic Configuration
      • Supported TRT-LLM Buildable Profiles
    • Gemma 2 9B
      • Generic Configuration
      • Supported TRT-LLM Buildable Profiles
    • Gemma2 9B CPT Sahabat-AI v1 Instruct
      • Generic Configuration
      • Supported TRT-LLM Buildable Profiles
    • Granite 3.3 8B Instruct
      • Generic Configuration
      • Supported TRT-LLM Buildable Profiles
    • (Meta) Llama 2 7B Chat
      • Optimized Configurations
      • Generic Configuration
    • (Meta) Llama 2 13B Chat
      • Optimized Configurations
      • Generic Configuration
    • (Meta) Llama 2 70B Chat
      • Optimized Configurations
      • Generic Configuration
    • Llama 3 SQLCoder 8B
      • Optimized Configurations
      • Generic Configuration
    • Llama 3 Swallow 70B Instruct V0.1
      • Optimized Configurations
      • Generic Configuration
    • Llama 3 Taiwan 70B Instruct
      • Optimized Configurations
      • Generic Configuration
    • Llama 3.1 8B Base
      • Optimized Configurations
      • Generic Configuration
    • Llama 3.1 8B Instruct
      • Optimized Configurations
      • Generic Configuration
    • Llama 3.1 8B Instruct RTX
      • Optimized Configurations
    • Llama 3.1 Nemotron Nano 4B V1.1
      • Optimized Configurations
      • Generic Configuration
      • Supported TRT-LLM Buildable Profiles
    • Llama 3.1 Nemotron Nano 8B V1
      • Optimized Configurations
      • Generic Configuration
      • Supported TRT-LLM Buildable Profiles
    • Llama 3.1 Nemotron Ultra 253B V1
      • Optimized Configurations
      • Generic Configuration
      • Supported TRT-LLM Buildable Profiles
    • Llama 3.2 1B Instruct
      • Optimized Configurations
      • Generic Configuration
    • Llama 3.2 3B Instruct
      • Optimized Configurations
      • Generic Configuration
      • Supported TRT-LLM Buildable Profiles
    • Llama 3.1 70B Instruct
      • Optimized Configurations
      • Generic Configuration
    • Llama 3.1 405B Instruct
      • Optimized Configurations
      • Generic Configuration
    • Llama 3.1 Nemotron 70B Instruct
      • Optimized Configurations
      • Generic Configuration
    • Llama 3.1 Swallow 8B Instruct v0.1
      • Generic Configuration
      • Supported TRT-LLM Buildable Profiles
    • Llama 3.1 Swallow 70B Instruct v0.1
      • Generic Configuration
      • Supported TRT-LLM Buildable Profiles
    • Llama 3.1 Typhoon 2 8B Instruct
      • Optimized Configurations
      • Generic Configuration
    • Llama 3.3 70B Instruct
      • Optimized Configurations
      • Generic Configuration
      • Supported TRT-LLM Buildable Profiles
    • Meta Llama 3 8B Instruct
      • Optimized Configurations
      • Generic Configuration
    • Llama 3.3 Nemotron Super 49B V1
      • Optimized Configurations
      • Generic Configuration
    • Meta Llama 3 70B Instruct
      • Optimized Configurations
      • Generic Configuration
    • Mistral 7B Instruct V0.3
      • Optimized Configurations
      • Generic Configuration
    • Mistral NeMo Minitron 8B 8K Instruct
      • Generic Configuration
    • Mistral NeMo 12B Instruct RTX
      • Optimized Configurations
    • Mistral NeMo 12B Instruct
      • Optimized Configurations
      • Generic Configuration
    • Mistral Small 24b Instruct 2501
      • Generic Configuration
      • Supported TRT-LLM Buildable Profiles
    • Mixtral 8x7B Instruct V0.1
      • Optimized Configurations
      • Generic Configuration
    • Mixtral 8x22B Instruct V0.1
      • Optimized Configurations
      • Generic Configuration
    • Nemotron 4 340B Instruct
      • Optimized Configurations
      • Generic Configuration
    • Nemotron 4 340B Reward
      • Optimized Configurations
      • Generic Configuration
    • Phi 3 Mini 4K Instruct
      • Generic Configuration
    • Phind Codellama 34B V2 Instruct
      • Generic Configuration
    • Riva Translate 4B Instruct
      • Generic Configuration
    • Sarvam - M
      • Generic Configuration
    • Supported TRT-LLM buildable profiles
    • SILMA 9B Instruct v1.0
      • Generic Configuration
      • Supported TRT-LLM buildable profiles
    • StarCoder2 7B
      • Optimized Configurations
      • Generic Configuration
    • StarCoderBase 15.5B
      • Generic Configuration
      • Supported TRT-LLM Buildable Profiles
• Examples with system role
  • Message roles
    • OpenAI Chat Completion Request with Single User Question
    • OpenAI Chat Completion Request with Additional Context and Response
• API Reference
  • OpenAPI Schema
  • Experimental APIs
    • Experimental support for Llama Stack (LS) API
  • Reference
• Function Calling
  • Supported Models
  • Parameters
    • tool_choice options
  • Example Workflows
    • 1. Basic Function Calling
    • 2. Multiple Tools
    • 3. Named Tool Usage
    • 4. Parallel Tool Calling
• Using Reward Models
• Using Reasoning Models
  • Reasoning Mode
    • Detailed Thinking On
    • Detailed Thinking Off
• Llama Stack API (Experimental)
  • Installation
  • Common Components
  • Basic Usage
  • Streaming Responses
  • Tool Calling
• Utilities
  • List available model profiles
    • Example
  • Download model profiles to NIM cache
    • Example
  • Create model store
    • Example
  • Check NIM cache
    • Example
  • Set cache environment variables
    • Example
• Fine-tuned model support
  • Usage
• Observability
  • Prometheus
  • Grafana
• Structured Generation
  • JSON Schema
    • Basic Example: Movie Review
    • Advanced Example: Product Information
    • Example: Nested Structures for Event Planning
  • Regular Expressions
  • Choices
  • Context-free Grammar
• Custom Guided Decoding Backend (Experimental)
  • Loading the custom guided decoding backends
  • Specifying a custom backend at runtime
  • Custom guided decoding backend specifications
    • LogitsProcessor specifications
    • Statement level postprocessing
    • Maintaining custom state
    • Other considerations
• Parameter-Efficient Fine-Tuning
  • LoRA Setup Overview
  • LoRA Adapters
    • NeMo Format
    • Hugging Face Transformers Format
  • LoRA Model Directory Structure
  • Obtaining LoRA models
    • Downloading LoRA adapters from NGC
      • LoRA adapters for llama3-8b-instruct
      • LoRA adapters for llama3-70b-instruct
    • Downloading LoRA Adapters from Hugging Face Hub
    • Using Your Own Custom LoRA Adapters
    • Handling Mixed Batch Requests
    • PEFT Best Practices
  • PEFT Environment Variables
    • PEFT Caching and Dynamic Mixed-batch LoRA (Multi-LoRA)
      • PEFT Cache memory requirements
  • Launch NIM for LLMs with PEFT
    • Run Multi-LoRA Inference
• KV Cache Reuse (a.k.a. prefix caching)
  • How to use
  • When to use
• Acknowledgements
• Eula