NVIDIA NIM for Large Language Models

NVIDIA NIM for LLMs

• Introduction
  • High Performance Features
  • Applications
  • Architecture
    • NIM Deployment Lifecycle
  • The NVIDIA Developer Program
• Release Notes
  • Release 1.4.0
    • Summary
    • New Models
    • New Features
    • Fixed Issues
    • Known Issues
  • Release 1.3.0
    • Summary
    • New Language Models
    • New Features
    • Known Issues
    • Software requirements updated
  • Release 1.2.3
    • Summary
    • New Language Models
    • Known Issues
  • Release 1.2.1
    • Summary
    • New Models
    • Known Issues
  • Release 1.2.0
    • Summary
    • New Language Models
    • New Features
    • Fixed Issues
    • Known Issues
  • Release 1.1.2
    • Summary
    • New Language Models
    • New Features
    • Known Issues
  • Release 1.1.1
    • Summary
    • Known Issues
  • Release 1.1.0
    • Summary
    • New Language Models
    • New Features
    • Known Issues
  • Release 1.0
• Getting Started
  • Prerequisites
    • Setup
      • Installing WSL2 for Windows
  • Launch NVIDIA NIM for LLMs
    • Option 1: From API Catalog
      • Generate an API Key
      • Login to Docker
      • Download and Launch NVIDIA NIM for LLMs
    • Option 2: From NGC
      • Generate an API key
      • Export the API key
      • Docker login to NGC
      • List available NIMs
      • Launch NIM
  • Docker Run Parameters
  • Run Inference
    • OpenAI Completion Request
    • OpenAI Chat Completion Request
    • Improving TRT-LLM Performance
    • Parameter-Efficient Fine-Tuning
  • Stopping the container
  • Kubernetes Installation
  • Serving models from local assets
• Deployment Guide
  • Deploying on other platforms
• Air Gap Deployment
  • Air Gap Deployment (offline cache route)
  • Air Gap Deployment (local model directory route)
• Multi-node Deployment
  • Multi-Node Deployment with Kubernetes
• Deploying with Helm
  • Prerequisites
  • Configuring helm
    • Minimal example
  • Storage
  • Multi-node Models
    • LeaderWorkerSet
    • MPI Job
  • Enabling Open Telemetry Tracing and Metrics
  • Launching NIM in Kubernetes
  • Running inference
  • Troubleshooting FAQ
  • Additional information
  • Parameters
    • Deployment parameters
    • Autoscaling parameters
    • Ingress parameters
    • Probe parameters
    • Metrics parameters
    • Models parameters
    • Deprecated and Legacy Model parameters
    • Storage parameters
    • Service parameters
    • Multi-node parameters
• Tutorials
  • Playbooks
  • Platform Deployment Guides
• Configuring a NIM
  • GPU Selection
    • How many GPUs do I need?
  • Shared memory flag
  • Environment Variables
  • Volumes
• Model Profiles
  • Profile Selection
  • Automatic Profile Selection
  • Profile Details
    • Optimized Profiles vs. Local Build Profiles
    • Optimization Targets
  • Quantization
• Benchmarking
• Models
• Support Matrix
  • Hardware
    • CPU
    • GPU
  • Software
  • GPUs
  • Supported Models
    • Code Llama 13B Instruct
      • Optimized configurations
      • Generic configuration
      • Supported Releases
    • Code Llama 34B Instruct
      • Optimized configurations
      • Generic configuration
      • Supported Releases
    • Code Llama 70B Instruct
      • Optimized configurations
      • Generic configuration
      • Supported Releases
    • Gemma 2 2B
      • Generic configuration
      • Supported Releases
      • Supported TRT-LLM Buildable Profiles
    • Gemma 2 9B
      • Generic configuration
      • Supported Releases
      • Supported TRT-LLM Buildable Profiles
    • (Meta) Llama 2 7B Chat
      • Optimized configurations
      • Generic configuration
      • Supported Releases
    • (Meta) Llama 2 13B Chat
      • Optimized configurations
      • Generic configuration
      • Supported Releases
    • (Meta) Llama 2 70B Chat
      • Optimized configurations
      • Generic configuration
      • Supported Releases
    • Llama 3 SQLCoder 8B
      • Optimized configurations
      • Generic configuration
      • Supported Releases
    • Llama 3 Swallow 70B Instruct V0.1
      • Optimized configurations
      • Generic configuration
      • Supported Releases
    • Llama 3 Taiwan 70B Instruct
      • Optimized configurations
      • Generic configuration
      • Supported Releases
    • Llama 3.1 8B Base
      • Optimized configurations
      • Generic configuration
      • Supported Releases
    • Llama 3.1 8B Instruct
      • Optimized configurations
      • Generic configuration
      • Supported Releases
    • Llama 3.1 70B Instruct
      • Optimized configurations
      • Generic configuration
      • Supported Releases
    • Llama 3.1 405B Instruct
      • Optimized configurations
      • Generic configuration
      • Supported Releases
    • Llama 3.1 Nemotron 70B Instruct
      • Optimized configurations
      • Generic configuration
      • Supported Releases
    • Llama 3.1 Swallow 8B Instruct v0.1
      • Generic configuration
      • Supported Releases
      • Supported TRT-LLM Buildable Profiles
    • Llama 3.1 Swallow 70B Instruct v0.1
      • Generic configuration
      • Supported Releases
      • Supported TRT-LLM Buildable Profiles
    • Meta Llama 3 8B Instruct
      • Optimized configurations
      • Generic configuration
      • Supported Releases
    • Meta Llama 3 70B Instruct
      • Optimized configurations
      • Generic configuration
      • Supported Releases
    • Mistral 7B Instruct V0.3
      • Optimized configurations
      • Generic configuration
      • Supported Releases
    • Mistral NeMo Minitron 8B 8K Instruct
      • Generic configuration
      • Supported Releases
    • Mistral NeMo 12B Instruct
      • Optimized configurations
      • Generic configuration
      • Supported Releases
    • Mixtral 8x7B Instruct V0.1
      • Optimized configurations
      • Generic configuration
      • Supported Releases
    • Mixtral 8x22B Instruct V0.1
      • Optimized configurations
      • Generic configuration
      • Supported Releases
    • Nemotron 4 340B Instruct
      • Optimized configurations
      • Generic configuration
      • Supported Releases
    • Nemotron 4 340B Instruct 128K
      • Optimized configurations
      • Generic configuration
      • Supported Releases
    • Nemotron 4 340B Reward
      • Optimized configurations
      • Generic configuration
      • Supported Releases
    • Phi 3 Mini 4K Instruct
      • Generic configuration
      • Supported Releases
    • Phind Codellama 34B V2 Instruct
      • Generic configuration
      • Supported Releases
• Examples with system role
  • Message roles
    • OpenAI Chat Completion Request with Single User Question
    • OpenAI Chat Completion Request with Additional Context and Response
• API Reference
  • OpenAPI Schema
  • Experimental APIs
    • Experimental support for Llama Stack (LS) API
  • Reference
• Function Calling
  • Supported Models
  • Parameters
    • tool_choice options
  • Example Workflows
    • 1. Basic Function Calling
    • 2. Multiple Tools
    • 3. Named Tool Usage
    • 4. Parallel Tool Calling
• Using Reward Models
• Llama Stack API (Experimental)
  • Installation
  • Common Components
  • Basic Usage
  • Streaming Responses
  • Tool Calling
• Utilities
  • List available model profiles
    • Example
  • Download model profiles to NIM cache
    • Example
  • Create model store
    • Example
  • Check NIM cache
    • Example
  • Set cache environment variables
    • Example
• Fine-tuned model support
  • Usage
• Observability
  • Prometheus
  • Grafana
• Structured Generation
  • JSON Schema
    • Basic Example: Movie Review
    • Advanced Example: Product Information
    • Example: Nested Structures for Event Planning
  • Regular Expressions
  • Choices
  • Context-free Grammar
• Parameter-Efficient Fine-Tuning
  • LoRA Setup Overview
  • LoRA Adapters
    • NeMo Format
    • Hugging Face Transformers Format
  • LoRA Model Directory Structure
  • Obtaining LoRA models
    • Downloading LoRA adapters from NGC
      • LoRA adapters for llama3-8b-instruct
      • LoRA adapters for llama3-70b-instruct
    • Downloading LoRA Adapters from Hugging Face Hub
    • Using Your Own Custom LoRA Adapters
    • Handling Mixed Batch Requests
  • PEFT Environment Variables
    • PEFT Caching and Dynamic Mixed-batch LoRA (Multi-LoRA)
      • PEFT Cache memory requirements
  • Launch NIM for LLMs with PEFT
    • Run Multi-LoRA Inference
• KV Cache Reuse (a.k.a. prefix caching)
  • How to use
  • When to use
• Acknowledgements
• Eula