NVIDIA Cloud Functions

Warning

This is NV-Managed documentation (NGC-managed, BYOC).

For the Self-hosted NVIDIA Cloud Functions documentation, click here.

NVIDIA Cloud Functions

• Overview
  • Function Types
  • Getting Started
• Functions Quickstart
  • Clone, Build and Push the Docker Image to NGC Private Registry
  • Create & Deploy the Function Using the Cloud Functions UI
  • Create & Deploy the Function Using the Cloud Functions API
  • Create & Deploy the Function Using the NGC CLI
  • Invoke the Function
• Tasks Quickstart
  • Clone, Build, and Push the Docker Image to NGC Private Registry
  • Create & Launch the Task Using the Cloud Functions UI
  • Create & Launch the Task Using the NGC CLI
  • Understanding the Tasks Sample
  • Verifying Task Results
  • Next Steps
• Elastic NIM
  • Benefits of Running NIM on NVCF
  • Prerequisites
• Create and Deploy the Function
  • Managing Your Function using the NGC CLI
    • Export NGC_API_KEY Environment Variable
    • List All Functions
    • Get Function Information (Specify Version)
    • Delete a Function
    • Test the NIM Function with Sample Input
  • Deployment Best Practices
  • Troubleshooting Guide
• Function Lifecycle
  • Key Concepts
  • Function States
  • Workflow
  • Function Lifecycle Endpoints
    • Function Creation, Management & Deployment
      • Function Metadata
    • Function Invocation
    • Visibility, Cluster Groups & GPUs
• Function Creation
  • Working with NGC Private Registry
    • Generate an NGC Personal API Key
    • Download & Configure the NGC CLI
    • Authenticate with NGC Docker Registry
    • (Optional) Push a Container to the NGC Private Registry
    • Best Practices with NGC Docker Registry and Cloud Functions
      • Container Versioning
      • Usage of NGC Teams
      • Security
  • Container-Based Function Creation
    • Resources
    • Container Endpoints
    • Composing a FastAPI Container
      • Create the “requirements.txt” File
      • Implement the Server
      • Create the Dockerfile
      • Build the Container & Create the Function
    • Composing a PyTriton Container
      • Create the “requirements.txt” File
      • Create the “run.py” File
      • Create the “Dockerfile”
      • Build the Docker Image
      • Push the Docker Image
      • Create the Function
      • Additional Examples
    • Creating Functions with NGC Models & Resources
    • Creating gRPC-based Functions
      • Prerequisites
      • gRPC Function Creation via UI
      • gRPC Function Creation via CLI
      • gRPC Function Creation via API
      • gRPC Function Invocation
    • Creating Low Latency Streaming (LLS AKA GameStreamSDK/WebRTC) Functions
      • Building the Streaming Server Application
      • Creating the LLS Streaming Function
      • Connecting to a streaming function with a client
        • Intermediary Proxy
        • Web Browser Client
    • Available Container Variables
    • Environment Variables
  • Helm-Based Function Creation
    • Prerequisites
    • Pull Secret Management
    • Create a Helm-based Function
    • Limitations
  • 1. Asset caching
  • 2. Inference
  • 3. Security Constraints
    • Helm Chart Overrides
• Secret Management
  • Adding Secrets During Function Creation
    • Using the UI
    • Using the API
  • Managing Secrets for Existing Functions
    • Function States and Secret Management Rules
    • Using the UI
    • Using the API
  • Listing Secrets in the API
  • Using Secrets In Inference Container
    • Secrets in Container-based Functions
    • Secrets in Helm Chart Functions
  • Considerations and Limitations
• Function Deployment
  • Deploying a Function
    • Key Concepts
    • Function Queueing
    • Autoscaling and Instance Counts
    • Deploy via the UI
    • Deploy via API
    • Deploy via CLI
    • Delete a Deployment
  • Function Deployment Status
    • Overview
    • Example Use Cases
    • Deployment Stages Details
    • Deployment Status Dashboard
      • Viewing Deployment Status
  • Deployment Failures
• Manually Deploy a NIM
  • Prerequisites
  • Create NGC Personal API Key
• Download the NIM
  • Export the NGC API Key
  • List Available NIMs
• Upload NIM to Private Registry
• Create the Function
• Validating the Function Deployment
• Troubleshooting
• Function Management
  • Function Versioning
  • Function Versioning Best Practices
    • Prerequisites
    • Function Update Flow (A/B Testing)
• Function Permissions
  • Listing Functions via API
  • Sharing a Function to Another NCA ID
• Function Invocation
  • Using the NVCF Invocation API
    • HTTP Streaming
      • Advantages of HTTP Streaming
    • gRPC
      • Connection reuse and streaming (Kit/LLS)
  • Statuses and Errors
    • Inference Container Status Codes and Responses
    • Common Function Invocation Errors
• Third-Party Registries
  • Overview
    • Docker Hub
      • Prerequisites
        • Docker Hub Account Requirements
        • Personal Access Token Setup
        • Image Architecture Requirements
      • Adding Registry Credentials
        • In the UI
        • In the CLI
        • In the NGC API
        • Base64 Encoding Credentials
      • Using Credentials
      • References
        • Docker CLI Push Example
        • Helm Charts
    • AWS ECR
      • Prerequisites
        • AWS Account Requirements
        • IAM Permissions for Private ECR
        • IAM Permissions for Public ECR
        • Creating AWS Access Keys
        • Registry Hostname Patterns
        • Image Architecture Requirements
      • Adding Registry Credentials
        • In the UI
        • In the CLI
        • In the NGC API
        • Base64 Encoding Credentials
      • Using Credentials
      • Troubleshooting
        • Common Issues and Solutions
      • References
        • AWS ECR Documentation
        • Related Documentation
  • Common Requirements
    • Image Architecture Requirements
    • Authentication Requirements
• Tasks
  • Introduction
  • Task Management and Execution
    • Lifecycle of a Task
    • Environments
    • REST API Endpoints
      • Create Task
      • Container Argument Examples
      • List Tasks
      • Retrieve Task Details
      • Delete Task
      • Cancel Task
      • List Task Events
      • List Task Results
      • Update Task Secrets
  • Environment Variables Available in Task Container
  • Models and Resources
  • Task Secrets
  • Task Results
    • Handling Task Results
      • Result Handling Strategy: UPLOAD
      • Result Handling Strategy: NONE
    • progress File Format
    • Intermediate and Final Results
  • Task Runtime Duration
• Helm Chart Based Tasks
  • Prerequisites
  • 1. Essential Helm Chart Values
  • 2. Progress Updates
  • 3. Container Image Pull Secret Management
  • 4. Sample Helm Chart values.yaml
• Create a Helm-based Task
  • Creation from UI
  • Creation from API
  • Helm Chart Values Overrides
  • Limitations
  • 1. Disk Size
  • 2. Asset caching
  • 3. Security Constraints
• NGC Account Access
  • Login as the Organization Owner
• Cloud Tasks and Cloud Functions Member Roles
• Observability Guide
  • Overview
  • NGC UI Observability
    • Overview
      • How to Access
    • Logs
    • Viewing Metrics
  • External Observability
    • Ports
    • Configuration
    • Resource Management
    • Security
  • How to Set Up External Observability on a BYOC Cluster
    • BYOC Steps
  • Error Handling
  • Appendix A: Terminology
  • Appendix B: Available Metrics
    • CPU Metrics
      • Memory Metrics
      • Filesystem Metrics
      • Network Metrics
      • Kubernetes State Metrics
      • GPU Metrics
      • NVCF Worker Service Metrics
      • NVCA Instance Type Metrics
      • OpenTelemetry Collector Metrics
      • Resource Attributes
  • Appendix C: Adding Custom Application Metrics/Logs/Traces
• Debuggability Guide
  • Overview
  • Real-Time Logs
    • Key Benefits
    • Getting Started
  • Remote Command Execution
    • Key Benefits
    • Getting Started
  • NGC CLI Requirements and Examples
    • CLI Version Requirements
    • Detailed CLI Examples
  • Supported Commands
  • Security
  • Troubleshooting
    • Common Error Codes
    • Required Permissions
  • Limitations
  • Appendix A: Terminology
• API
  • OpenAPI Specification
  • Authorization
    • Generate an NGC Personal API Key
    • API Key Scopes and Domains
    • NVCF API Status Codes
• FAQ
• Support
• Service Status

Cluster Setup & Management

• Overview
  • Management Modes
  • Authentication & Keys
  • Prerequisites
    • Supported Kubernetes Versions
    • Considerations
• NGC-Managed Clusters
  • Register the Cluster
    • Configuration
    • Cluster Maintenance Mode
      • Cordon Maintenance
      • Cordon And Drain Maintenance
    • Install the Cluster Agent
  • View & Validate Cluster Setup
    • Verify Cluster Agent Installation via UI
    • Verify Cluster Agent Installation via Terminal
  • Upgrade the Cluster Agent
    • Operator Upgrade Required
    • NVCA-Only Upgrade
    • Force NVCA Rollout
  • Cluster Key Rotation
    • Frequently Asked Questions
  • Deregister the Cluster
    • Prerequisites
    • Delete the Cluster via UI
    • Delete the Cluster Agent and the NVCA Operator
• Helm-Managed Clusters
  • Switching to Helm-Managed Mode
  • Configuration Parameters
  • Updating Configuration
  • Upgrading NVCA Version
  • Switching Back to NGC-Managed Mode
  • Limitations
  • Tips
  • Helm Values Reference
• Self-Managed Clusters
  • How Self-Managed Mode Works
  • Installing the NVCA Operator
    • Configuration
    • Install
    • Verify
  • Manual Cluster Registration
  • Uninstalling
    • Handling Stuck Resources
  • Troubleshooting
• NVCA Configuration
  • Advanced Settings
  • Cluster Features
    • Caching Support
    • Account-Isolated Clusters
    • NVLink-optimized Clusters
    • Kata Container-Isolated Workloads
  • Network Configuration
  • Key Network Requirements
  • Network Policy Customization via ConfigMap
  • Network Policy Customization via clusterNetworkCIDRs Flag
  • Advanced: Additional Configuration Options
  • CSI Volume Mount Options
  • Node Selection for Cloud Functions
  • GPU Product Name Override
  • Managing Feature Flags
    • Setting Feature Flags at Install Time
    • Verifying Feature Flags
    • Modifying Feature Flags at Runtime
  • Enable Helm Shared Storage
  • Agent Config Merging
  • Manual Instance Configuration
    • Prerequisites
    • Step 1: Create the GPU Configuration JSON
    • Step 2: Base64 Encode the Configuration
    • Step 3: Configure Helm Values
    • Step 4: Install or Upgrade the Operator
    • Configuration Fields Reference
    • Verification
    • Troubleshooting
  • Cloud Provider-Specific Notes
  • Oracle Cloud Infrastructure (OCI)
  • AWS
• Monitoring & Observability
  • Monitoring Data
    • Metrics
    • Logs
    • Tracing
• Helm Values Reference
  • How Values Are Structured
  • Shared Parameters
    • NVCA Operator
    • NGC Authentication
    • Node Selector
    • Network Policies
    • OpenTelemetry
    • Agent Configuration
  • Helm-Managed Parameters
  • Self-Managed Parameters
  • Related Documentation
• Container Cache
  • Installation
    • Prerequisites
    • Step 1. Authenticate Helm to your chart registry
    • Step 2. Create the namespace and image pull secret
    • Step 3. Create a values file
    • Step 4. Install the chart
    • Step 5. Verify the installation
  • Base Configuration
  • Configuration Sections
    • Replicas
    • Node Selection
    • Target Hosts
    • Image Configuration
    • Cache Configuration
    • Storage Configuration
    • Service Configuration
    • Metrics Configuration
    • Resource Requests and Limits
  • Architecture
    • Container Cache Architecture
    • Data Flow
  • Performance Considerations
    • Cache Size
    • Storage Performance
    • Network Configuration
  • Monitoring and Observability
    • Metrics
    • Logging
  • Troubleshooting
    • Common Issues
  • Best Practices
    • Deployment
    • Configuration
    • Maintenance
• Distributed Shader Cache (GXCache)
  • Installation
    • Prerequisites
    • Step 1. Authenticate Helm to your chart registry
    • Step 2. Create the namespace and image pull secret
    • Step 3. Create a values file
    • Step 4. Install the chart
    • Step 5. Verify the installation
  • Base Configuration
  • Configuration Sections
    • Webhook Configuration
    • Service Configuration
    • Node Selection
    • Key Management
    • Storage Configuration
    • Metrics Configuration
  • Architecture
    • GXCache Architecture
    • Data Flow
  • Performance Considerations
    • Storage Performance
    • Network Configuration
    • Cache Size Planning
  • Monitoring and Observability
    • Metrics
    • Prometheus Integration
    • Logging
  • Troubleshooting
    • Common Issues
  • Best Practices
    • Deployment
    • Configuration
    • Security
    • Maintenance
• NVIDIA KAI Scheduler Integration Guide