NVIDIA NIM for Large Language Models

NVIDIA NIM for LLMs

• Introduction
  • High Performance Features
  • Applications
  • Architecture
    • NIM Deployment Lifecycle
• Release Notes
  • Release 1.1.2
    • Summary
    • Language Models
    • New Features
    • Known Issues
  • Release 1.1.1
    • Summary
    • Known Issues
  • Release 1.1.0
    • Summary
    • Language Models
    • New Features
    • Known Issues
  • Release 1.0
• Getting Started
  • Prerequisites
    • Setup
      • Installing WSL2 for Windows
  • Launch NVIDIA NIM for LLMs
    • Option 1: From API Catalog
      • Generate an API Key
      • Login to Docker
      • Download and Launch NVIDIA NIM for LLMs
    • Option 2: From NGC
      • Generate an API key
      • Export the API key
      • Docker login to NGC
      • List available NIMs
      • Launch NIM
  • Docker Run Parameters
  • Run Inference
    • OpenAI Completion Request
    • OpenAI Chat Completion Request
    • Parameter-Efficient Fine-Tuning
  • Stopping the container
  • Kubernetes Installation
  • Serving models from local assets
• Tutorials
  • Playbooks
  • Deployment Guides
• Multi-node Deployment
  • Multi-Node Deployment with Kubernetes
• Deploying with Helm
  • Prerequisites
  • Configuring helm
    • Minimal example
  • Storage
  • Multi-node Models
    • LeaderWorkerSet
    • MPI Job
  • Launching NIM in Kubernetes
  • Running inference
  • Troubleshooting FAQ
  • Additional information
  • Parameters
    • Deployment parameters
    • Autoscaling parameters
    • Ingress parameters
    • Probe parameters
    • Metrics parameters
    • Models parameters
    • Deprecated and Legacy Model parameters
    • Storage parameters
    • Service parameters
    • Multi-node parameters
• Configuring a NIM
  • GPU Selection
  • Shared memory flag
  • Environment Variables
  • Volumes
• Model Profiles
  • Profile Selection
  • Automatic Profile Selection
  • How Profiles are Created
    • Optimization Targets
    • Quantization
• Benchmarking
• Models
• Support Matrix
  • Hardware
  • Software
  • GPUs
  • General Guidelines
    • GPUs
    • Disk Space
  • Supported Models
    • Llama 3 Swallow 70B Instruct V0.1
      • Optimized configurations
    • Llama 3 Taiwan 70B Instruct
      • Optimized configurations
    • Llama 3.1 8B Base
      • Optimized configurations
      • Non-optimized configuration
    • Llama 3.1 8B Instruct
      • Optimized configurations
      • Non-optimized configuration
    • Llama 3.1 70B Instruct
      • Optimized configurations
    • Llama 3.1 405B Instruct
    • Meta-Llama-3-8B-Instruct
      • Optimized configurations
      • Non-optimized configuration
    • Meta-Llama-3-70B-Instruct
      • Optimized configurations
      • Non-optimized configuration
    • Mistral-7B-Instruct-v0.3
      • Optimized configurations
      • Non-optimized configuration
    • Mixtral-8x7B-v0.1
      • Optimized configurations
      • Non-optimized configuration
    • Mistral-NeMo-12B-Instruct
      • Optimized configurations
      • Non-optimized configuration
    • Mixtral-8x22B-v0.1
      • Optimized configurations
    • Nemotron 4 340B Instruct
      • Optimized configurations
      • Non-optimized configuration
• API Reference
  • OpenAPI Schema
  • Experimental APIs
    • Experimental support of LS API
• Function Calling
  • Supported Models
  • Parameters
    • tool_choice options
  • Example Workflows
    • 1. Basic Function Calling
    • 2. Multiple Tools
    • 3. Named Tool Usage
    • 4. Parallel Tool Calling
• Llama Stack API (Experimental)
  • Installation
  • Basic Usage
  • Streaming Responses
  • Tool Calling
• Utilities
  • List available model profiles
    • Example
  • Download model profiles to NIM cache
    • Example
  • Create model store
    • Example
  • Check NIM cache
    • Example
  • Set cache environment variables
    • Example
• Observability
  • Prometheus
  • Grafana
• Structured Generation
  • JSON Schema
    • Basic Example: Movie Review
    • Advanced Example: Product Information
    • Example: Nested Structures for Event Planning
  • Regular Expressions
  • Choices
  • Context-free Grammar
• Parameter-Efficient Fine-Tuning
  • LoRA Setup Overview
  • LoRA Adapters
    • NeMo Format
    • Hugging Face Transformers Format
  • LoRA Model Directory Structure
  • Obtaining LoRA models
    • Downloading LoRA adapters from NGC
      • LoRA adapters for llama3-8b-instruct
      • LoRA adapters for llama3-70b-instruct
    • Downloading LoRA Adapters from Hugging Face Hub
    • Using Your Own Custom LoRA Adapters
    • Handling Mixed Batch Requests
  • PEFT Environment Variables
    • PEFT Caching and Dynamic Mixed-batch LoRA (Multi-LoRA)
      • PEFT Cache memory requirements
  • Launch NIM for LLMs with PEFT
    • Run Multi-LoRA Inference
• Acknowledgements
  • accelerate
  • aiohttp
  • aiosignal
  • annotated-types
  • anyio
  • apscheduler
  • async-timeout
  • attrs
  • build
  • certifi
  • charset-normalizer
  • click
  • cmake
  • coloredlogs
  • datasets
  • dill
  • diskcache
  • dnspython
  • einops
  • email_validator
  • exceptiongroup
  • fastapi
  • fastapi-cli
  • filelock
  • flash-attn
  • frozenlist
  • fsspec
  • h11
  • h5py
  • httpcore
  • httptools
  • httpx
  • huggingface-hub
  • humanfriendly
  • idna
  • importlib_metadata
  • interegular
  • jinja2
  • joblib
  • jsonschema
  • jsonschema-specifications
  • lark
  • llvmlite
  • lm-format-enforcer
  • markdown-it-py
  • markupsafe
  • mdurl
  • ml-dtypes
  • mpi4py
  • mpmath
  • msgpack
  • multidict
  • multiprocess
  • nest-asyncio
  • networkx
  • ninja
  • numba
  • numpy
  • onnx
  • optimum
  • orjson
  • outlines
  • packaging
  • pandas
  • polygraphy
  • prometheus_client
  • protobuf
  • psutil
  • pulp
  • py-cpuinfo
  • pyarrow
  • pyarrow-hotfix
  • pydantic
  • pydantic_core
  • pygments
  • pynvml
  • python-dateutil
  • python-dotenv
  • python-multipart
  • pytz
  • pyyaml
  • ray
  • referencing
  • regex
  • requests
  • rich
  • rpds-py
  • scipy
  • shellingham
  • six
  • sniffio
  • starlette
  • strenum
  • sympy
  • tensorstore
  • tiktoken
  • tomli
  • torch
  • tqdm
  • transformers
  • typer
  • typing_extensions
  • tzdata
  • tzlocal
  • ujson
  • urllib3
  • uvicorn
  • uvloop
  • vllm
  • watchfiles
  • websockets
  • xformers
  • xxhash
  • yarl
  • zipp
• Eula