NVIDIA NIM for Large Language Models (1.2.0)

NVIDIA NIM for LLMs

• Introduction
  
  • High Performance Features
  • Applications
  • Architecture
    
    • NIM Deployment Lifecycle
  • The NVIDIA Developer Program
• Release Notes
  
  • Release 1.2.3
    
    • Summary
    • New Language Models
    • Known Issues
  • Release 1.2.1
    
    • Summary
    • New Models
    • Known Issues
  • Release 1.2.0
    
    • Summary
    • New Language Models
    • New Features
    • Fixed Issues
    • Known Issues
  • Release 1.1.2
    
    • Summary
    • New Language Models
    • New Features
    • Known Issues
  • Release 1.1.1
    
    • Summary
    • Known Issues
  • Release 1.1.0
    
    • Summary
    • New Language Models
    • New Features
    • Known Issues
  • Release 1.0
• Getting Started
  
  • Prerequisites
    
    • Setup
      
      • Installing WSL2 for Windows
  • Launch NVIDIA NIM for LLMs
    
    • Option 1: From API Catalog
      
      • Generate an API Key
      • Login to Docker
      • Download and Launch NVIDIA NIM for LLMs
    • Option 2: From NGC
      
      • Generate an API key
      • Export the API key
      • Docker login to NGC
      • List available NIMs
      • Launch NIM
  • Docker Run Parameters
  • Run Inference
    
    • OpenAI Completion Request
    • OpenAI Chat Completion Request
    • Parameter-Efficient Fine-Tuning
  • Stopping the container
  • Kubernetes Installation
  • Serving models from local assets
    
    • Air Gap Deployment (offline cache route)
    • Air Gap Deployment (local model directory route)
• Deployment Guide
  
  • Deploying on other platforms
• Tutorials
  
  • Playbooks
  • Deployment Guides
• Multi-node Deployment
  
  • Multi-Node Deployment with Kubernetes
• Deploying with Helm
  
  • Prerequisites
  • Configuring helm
    
    • Minimal example
  • Storage
  • Multi-node Models
    
    • LeaderWorkerSet
    • MPI Job
  • Enabling Open Telemetry Tracing and Metrics
  • Launching NIM in Kubernetes
  • Running inference
  • Troubleshooting FAQ
  • Additional information
  • Parameters
    
    • Deployment parameters
    • Autoscaling parameters
    • Ingress parameters
    • Probe parameters
    • Metrics parameters
    • Models parameters
    • Deprecated and Legacy Model parameters
    • Storage parameters
    • Service parameters
    • Multi-node parameters
• Configuring a NIM
  
  • GPU Selection
  • Shared memory flag
  • Environment Variables
  • Volumes
• Model Profiles
  
  • Profile Selection
  • Automatic Profile Selection
  • How Profiles are Created
    
    • Optimization Targets
    • Quantization
• Benchmarking
• Models
• Support Matrix
  
  • Hardware
  • Software
  • GPUs
  • Supported Models
    
    • Code Llama 13B Instruct
      
      • Optimized configurations
      • Non-optimized configuration
      • Supported Releases
    • Code Llama 34B Instruct
      
      • Optimized configurations
      • Non-optimized configuration
      • Supported Releases
    • Code Llama 70B Instruct
      
      • Optimized configurations
      • Non-optimized configuration
      • Supported Releases
    • (Meta) Llama 2 7B Chat
      
      • Optimized configurations
      • Non-optimized configuration
      • Supported Releases
    • (Meta) Llama 2 13B Chat
      
      • Optimized configurations
      • Non-optimized configuration
      • Supported Releases
    • (Meta) Llama 2 70B Chat
      
      • Optimized configurations
      • Non-optimized configuration
      • Supported Releases
    • Llama 3 SQLCoder 8B Instruct
      
      • Optimized configurations
      • Non-optimized configuration
      • Supported Releases
    • Llama 3 Swallow 70B Instruct V0.1
      
      • Optimized configurations
      • Non-optimized configuration
      • Supported Releases
    • Llama 3 Taiwan 70B Instruct
      
      • Optimized configurations
      • Non-optimized configuration
      • Supported Releases
    • Llama 3.1 8B Base
      
      • Optimized configurations
      • Non-optimized configuration
      • Supported Releases
    • Llama 3.1 8B Instruct
      
      • Optimized configurations
      • Non-optimized configuration
      • Supported Releases
    • Llama 3.1 70B Instruct
      
      • Optimized configurations
      • Non-optimized configuration
      • Supported Releases
    • Llama 3.1 405B Instruct
      
      • Optimized configurations
      • Non-optimized configuration
      • Supported Releases
    • Llama 3.1 Nemotron 70B Instruct
      
      • Optimized configurations
      • Non-optimized configuration
      • Supported Releases
    • Meta Llama 3 8B Instruct
      
      • Optimized configurations
      • Non-optimized configuration
      • Supported Releases
    • Meta Llama 3 70B Instruct
      
      • Optimized configurations
      • Non-optimized configuration
      • Supported Releases
    • Mistral 7B Instruct V0.3
      
      • Optimized configurations
      • Non-optimized configuration
      • Supported Releases
    • Mistral NeMo Minitron 8B 8K Instruct
      
      • Non-optimized configuration
      • Supported Releases
    • Mistral NeMo 12B Instruct
      
      • Optimized configurations
      • Non-optimized configuration
      • Supported Releases
    • Mixtral 8x7B Instruct V0.1
      
      • Optimized configurations
      • Non-optimized configuration
      • Supported Releases
    • Mixtral 8x22B Instruct V0.1
      
      • Optimized configurations
      • Non-optimized configuration
      • Supported Releases
    • Nemotron 4 340B Instruct
      
      • Optimized configurations
      • Non-optimized configuration
      • Supported Releases
    • Nemotron 4 340B Instruct 128K
      
      • Optimized configurations
      • Non-optimized configuration
      • Supported Releases
    • Nemotron 4 340B Reward
      
      • Optimized configurations
      • Non-optimized configuration
      • Supported Releases
    • Phi 3 Mini 4K Instruct
      
      • Non-optimized configuration
      • Supported Releases
    • Phind Codellama 34B V2
      
      • Non-optimized configuration
      • Supported Releases
• Examples with system role
  
  • Message roles
    
    • OpenAI Chat Completion Request with Single User Question
    • OpenAI Chat Completion Request with Additional Context and Response
• API Reference
  
  • OpenAPI Schema
  • Experimental APIs
    
    • Experimental support for Llama Stack (LS) API
  • Reference
• Function Calling
  
  • Parameters
    
    • <a href="function-calling.html#tool-choice-options">tool_choice</a>
  • Example Workflows
    
    • 1. Basic Function Calling
    • 2. Multiple Tools
    • 3. Named Tool Usage
    • 4. Parallel Tool Calling
• Using Reward Models
• Llama Stack API (Experimental)
  
  • Installation
  • Common Components
  • Basic Usage
  • Streaming Responses
  • Tool Calling
• Utilities
  
  • List available model profiles
    
    • Example
  • Download model profiles to NIM cache
    
    • Example
  • Create model store
    
    • Example
  • Check NIM cache
    
    • Example
  • Set cache environment variables
    
    • Example
• Fine-tuned model support
  
  • Usage
• Observability
  
  • Prometheus
  • Grafana
• Structured Generation
  
  • JSON Schema
    
    • Basic Example: Movie Review
    • Advanced Example: Product Information
    • Example: Nested Structures for Event Planning
  • Regular Expressions
  • Choices
  • Context-free Grammar
• Parameter-Efficient Fine-Tuning
  
  • LoRA Setup Overview
  • LoRA Adapters
    
    • NeMo Format
    • Hugging Face Transformers Format
  • LoRA Model Directory Structure
  • Obtaining LoRA models
    
    • Downloading LoRA adapters from NGC
      
      • LoRA adapters for llama3-8b-instruct
      • LoRA adapters for llama3-70b-instruct
    • Downloading LoRA Adapters from Hugging Face Hub
    • Using Your Own Custom LoRA Adapters
    • Handling Mixed Batch Requests
  • PEFT Environment Variables
    
    • PEFT Caching and Dynamic Mixed-batch LoRA (Multi-LoRA)
      
      • PEFT Cache memory requirements
  • Launch NIM for LLMs with PEFT
    
    • Run Multi-LoRA Inference
• KV Cache Reuse (a.k.a. prefix caching)
  
  • How to use
  • When to use
  • Debugging
• Acknowledgements
  
  • accelerate
  • aiohttp
  • aiosignal
  • annotated-types
  • anyio
  • apscheduler
  • async-timeout
  • attrs
  • build
  • certifi
  • charset-normalizer
  • click
  • cmake
  • coloredlogs
  • datasets
  • dill
  • diskcache
  • dnspython
  • einops
  • email_validator
  • exceptiongroup
  • fastapi
  • fastapi-cli
  • filelock
  • flash-attn
  • frozenlist
  • fsspec
  • h11
  • h5py
  • httpcore
  • httptools
  • httpx
  • huggingface-hub
  • humanfriendly
  • idna
  • importlib_metadata
  • interegular
  • jinja2
  • joblib
  • jsonschema
  • jsonschema-specifications
  • lark
  • llvmlite
  • lm-format-enforcer
  • markdown-it-py
  • markupsafe
  • mdurl
  • ml-dtypes
  • mpi4py
  • mpmath
  • msgpack
  • multidict
  • multiprocess
  • nest-asyncio
  • networkx
  • ninja
  • numba
  • numpy
  • onnx
  • optimum
  • orjson
  • outlines
  • packaging
  • pandas
  • polygraphy
  • prometheus_client
  • protobuf
  • psutil
  • pulp
  • py-cpuinfo
  • pyarrow
  • pyarrow-hotfix
  • pydantic
  • pydantic_core
  • pygments
  • pynvml
  • python-dateutil
  • python-dotenv
  • python-multipart
  • pytz
  • pyyaml
  • ray
  • referencing
  • regex
  • requests
  • rich
  • rpds-py
  • scipy
  • shellingham
  • six
  • sniffio
  • starlette
  • strenum
  • sympy
  • tensorstore
  • tiktoken
  • tomli
  • torch
  • tqdm
  • transformers
  • typer
  • typing_extensions
  • tzdata
  • tzlocal
  • ujson
  • urllib3
  • uvicorn
  • uvloop
  • vllm
  • watchfiles
  • websockets
  • xformers
  • xxhash
  • yarl
  • zipp
• Eula