NVIDIA NIM for Large Language Models

NVIDIA NIM for LLMs

• Introduction
  • High Performance Features
  • Applications
  • Architecture
    • NIM Deployment Lifecycle
  • The NVIDIA Developer Program
• Release Notes
  • All Known Issues
    • 1.8.0
    • 1.7.0
    • 1.6.0
    • 1.5.0
    • 1.4.0
    • 1.3.0
    • 1.2.3
    • 1.2.1
    • 1.2.0
    • 1.1.2
    • 1.1.1
    • 1.1.0
    • 1.0
  • Release 1.8.0
    • New Language Models
    • New Features
    • Known Issues
    • Caveats
    • Fixed Issues
  • Release 1.7.0
    • New Language Models
    • New Features
    • Known Issues
  • Release 1.6.0
    • New Language Models
    • New Features
    • Known Issues
    • Known Issues
  • Release 1.5.0
    • New Language Models
    • New Features
    • Known Issues
  • Release 1.4.0
    • New Models
    • New Features
    • Fixed Issues
    • Known Issues
  • Release 1.3.0
    • New Language Models
    • New Features
    • Known Issues
    • Software requirements updated
  • Release 1.2.3
    • New Language Models
    • Known Issues
  • Release 1.2.1
    • New Models
    • Known Issues
  • Release 1.2.0
    • New Language Models
    • New Features
    • Fixed Issues
    • Known Issues
  • Release 1.1.2
    • New Language Models
    • New Features
    • Known Issues
  • Release 1.1.1
    • Known Issues
  • Release 1.1.0
    • New Language Models
    • New Features
    • Known Issues
  • Release 1.0
• Getting Started
  • Prerequisites
    • CPU
    • GPU
    • Software
      • Installing WSL2 for Windows
  • Launch NVIDIA NIM for LLMs
    • Option 1: From API Catalog
      • Generate an API Key
      • Login to Docker
      • Download and Launch NVIDIA NIM for LLMs
    • Option 2: From NGC
      • Generate an API key
      • Export the API key
      • Docker login to NGC
      • List available NIMs
      • Launch NIM
  • Docker Run Parameters
  • Run Inference
    • OpenAI Completion Request
    • OpenAI Chat Completion Request
    • Improving TRT-LLM Performance
    • Parameter-Efficient Fine-Tuning
  • Stopping the container
  • Kubernetes Installation
  • Serving models from local assets
• Deployment Guide
  • Deploying on other platforms
• Air Gap Deployment
  • Air Gap Deployment (offline cache route)
  • Air Gap Deployment (local model directory route)
• Multi-Node Deployment
  • Multi-Node Deployment with Kubernetes
  • Preparation
  • Multi-Node Models
    • LeaderWorkerSet
    • MPI Job
  • Example: Helm chart for DeepSeek R1 using an SGLang Backend
  • Troubleshooting Multi-Node Deployments
  • Multi-Node Parameters
• Deploying with Helm
  • Prerequisites
  • Configuring Helm
    • Minimal Example
  • Storage
  • Enabling Open Telemetry Tracing and Metrics
  • Launching NIM in Kubernetes
  • Running Inference
  • Troubleshooting FAQ
  • Additional Information
  • Parameters
    • Deployment Parameters
    • Autoscaling Parameters
    • Ingress Parameters
    • Probe Parameters
    • Metrics Parameters
    • Models Parameters
    • Deprecated and Legacy Model Parameters
    • Storage Parameters
    • Service Parameters
• Tutorials
  • Playbooks
  • Platform Deployment Guides
• Configuring a NIM
  • GPU Selection
    • How many GPUs do I need?
  • Shared memory flag
  • Environment Variables
  • Volumes
• Model Profiles
  • Profile Selection
  • Automatic Profile Selection
  • Profile Details
    • Optimized Profiles vs. Local Build Profiles
    • Optimization Targets
  • Quantization
• Overview
  • Structure
    • Supported Repository Types
  • S3 Support
    • Using the Feature
      • 1. Create an S3 Bucket
      • 2. Download Files
      • 3. Upload Files to S3
        • Example
      • 4. Set the Environment Variable
      • 5. Configure AWS Credentials
      • 6. Launch NIM
        • Example Docker Command
• Benchmarking
• Models
• Supported Models
  • GPUs
  • Optimized Models
  • Code Llama 13B Instruct
    • Optimized Configurations
    • Generic Configuration
  • Code Llama 34B Instruct
    • Optimized Configurations
    • Generic Configuration
  • Code Llama 70B Instruct
    • Optimized Configurations
    • Generic Configuration
  • DeepSeek R1
    • Supported Configurations
  • DeepSeek R1 Distill Llama 8B
    • Optimized Configurations
    • Generic Configuration
  • DeepSeek R1 Distill Llama 70B
    • Optimized Configurations
    • Generic Configuration
  • DeepSeek R1 Distill Llama 8B RTX
    • Optimized Configurations
  • DeepSeek-R1-Distill-Qwen-32B
    • Optimized Configurations
    • Generic Configuration
  • DeepSeek-R1-Distill-Qwen-7B
    • Optimized Configurations
  • DeepSeek-R1-Distill-Qwen-14B
    • Optimized Configurations
    • Generic Configuration
    • Supported TRT-LLM Buildable Profiles
  • Qwen2.5 72B Instruct
    • Optimized Configurations
    • Generic Configuration
  • Qwen2.5 7B Instruct
    • Optimized Configurations
    • Generic Configuration
    • Supported TRT-LLM Buildable Profiles
  • Gemma 2 2B
    • Generic Configuration
    • Supported TRT-LLM Buildable Profiles
  • Gemma 2 9B
    • Generic Configuration
    • Supported TRT-LLM Buildable Profiles
  • (Meta) Llama 2 7B Chat
    • Optimized Configurations
    • Generic Configuration
  • (Meta) Llama 2 13B Chat
    • Optimized Configurations
    • Generic Configuration
  • (Meta) Llama 2 70B Chat
    • Optimized Configurations
    • Generic Configuration
  • Llama 3 SQLCoder 8B
    • Optimized Configurations
    • Generic Configuration
  • Llama 3 Swallow 70B Instruct V0.1
    • Optimized Configurations
    • Generic Configuration
  • Llama 3 Taiwan 70B Instruct
    • Optimized Configurations
    • Generic Configuration
  • Llama 3.1 8B Base
    • Optimized Configurations
    • Generic Configuration
  • Llama 3.1 8B Instruct
    • Optimized Configurations
    • Generic Configuration
  • Llama 3.1 8B Instruct RTX
    • Optimized Configurations
    • Generic Configuration
  • Llama 3.1 Nemotron Nano 8B V1
    • Optimized Configurations
    • Generic Configuration
    • Supported TRT-LLM Buildable Profiles
  • Llama 3.2 1B Instruct
    • Optimized Configurations
    • Generic Configuration
  • Llama 3.2 3B Instruct
    • Optimized Configurations
    • Generic Configuration
    • Supported TRT-LLM Buildable Profiles
  • Llama 3.1 70B Instruct
    • Optimized Configurations
    • Generic Configuration
  • Llama 3.1 405B Instruct
    • Optimized Configurations
    • Generic Configuration
  • Llama 3.1 Nemotron 70B Instruct
    • Optimized Configurations
    • Generic Configuration
  • Llama 3.1 Swallow 8B Instruct v0.1
    • Generic Configuration
    • Supported TRT-LLM Buildable Profiles
  • Llama 3.1 Swallow 70B Instruct v0.1
    • Generic Configuration
    • Supported TRT-LLM Buildable Profiles
  • Llama 3.3 70B Instruct
    • Optimized Configurations
    • Generic Configuration
    • Supported TRT-LLM Buildable Profiles
  • Meta Llama 3 8B Instruct
    • Optimized Configurations
    • Generic Configuration
  • Llama 3.3 Nemotron Super 49B V1
    • Optimized Configurations
    • Generic Configuration
  • Meta Llama 3 70B Instruct
    • Optimized Configurations
    • Generic Configuration
  • Mistral 7B Instruct V0.3
    • Optimized Configurations
    • Generic Configuration
  • Mistral NeMo Minitron 8B 8K Instruct
    • Generic Configuration
  • Mistral NeMo 12B Instruct RTX
    • Optimized Configurations
    • Generic Configuration
  • Mistral NeMo 12B Instruct
    • Optimized Configurations
    • Generic Configuration
  • Mixtral 8x7B Instruct V0.1
    • Optimized Configurations
    • Generic Configuration
  • Mixtral 8x22B Instruct V0.1
    • Optimized Configurations
    • Generic Configuration
  • StarCoder2 7B
    • Optimized Configurations
    • Generic Configuration
  • Nemotron 4 340B Instruct
    • Optimized Configurations
    • Generic Configuration
  • Nemotron 4 340B Instruct 128K
    • Optimized Configurations
    • Generic Configuration
  • Nemotron 4 340B Reward
    • Optimized Configurations
    • Generic Configuration
  • Phi 3 Mini 4K Instruct
    • Generic Configuration
  • Phind Codellama 34B V2 Instruct
    • Generic Configuration
  • StarCoderBase 15.5B
    • Generic Configuration
    • Supported TRT-LLM Buildable Profiles
• Examples with system role
  • Message roles
    • OpenAI Chat Completion Request with Single User Question
    • OpenAI Chat Completion Request with Additional Context and Response
• API Reference
  • OpenAPI Schema
  • Experimental APIs
    • Experimental support for Llama Stack (LS) API
  • Reference
• Function Calling
  • Supported Models
  • Parameters
    • tool_choice options
  • Example Workflows
    • 1. Basic Function Calling
    • 2. Multiple Tools
    • 3. Named Tool Usage
    • 4. Parallel Tool Calling
• Using Reward Models
• Using Reasoning Models
  • Reasoning Mode
    • Detailed Thinking On
    • Detailed Thinking Off
• Llama Stack API (Experimental)
  • Installation
  • Common Components
  • Basic Usage
  • Streaming Responses
  • Tool Calling
• Utilities
  • List available model profiles
    • Example
  • Download model profiles to NIM cache
    • Example
  • Create model store
    • Example
  • Check NIM cache
    • Example
  • Set cache environment variables
    • Example
• Fine-tuned model support
  • Usage
• Observability
  • Prometheus
  • Grafana
• Structured Generation
  • JSON Schema
    • Basic Example: Movie Review
    • Advanced Example: Product Information
    • Example: Nested Structures for Event Planning
  • Regular Expressions
  • Choices
  • Context-free Grammar
• Custom Guided Decoding Backend (Experimental)
  • Loading the custom guided decoding backends
  • Specifying a custom backend at runtime
  • Custom guided decoding backend specifications
    • LogitsProcessor specifications
    • Statement level postprocessing
    • Maintaining custom state
    • Other considerations
• Parameter-Efficient Fine-Tuning
  • LoRA Setup Overview
  • LoRA Adapters
    • NeMo Format
    • Hugging Face Transformers Format
  • LoRA Model Directory Structure
  • Obtaining LoRA models
    • Downloading LoRA adapters from NGC
      • LoRA adapters for llama3-8b-instruct
      • LoRA adapters for llama3-70b-instruct
    • Downloading LoRA Adapters from Hugging Face Hub
    • Using Your Own Custom LoRA Adapters
    • Handling Mixed Batch Requests
    • PEFT Best Practices
  • PEFT Environment Variables
    • PEFT Caching and Dynamic Mixed-batch LoRA (Multi-LoRA)
      • PEFT Cache memory requirements
  • Launch NIM for LLMs with PEFT
    • Run Multi-LoRA Inference
• KV Cache Reuse (a.k.a. prefix caching)
  • How to use
  • When to use
• Acknowledgements
• Eula