Operations

This page is for Self-Hosted, version v3.0 (default). For other versions, use one of these documentation indexes: - v3.0 (default): https://docs.nvidia.com/run-ai/self-hosted/v3.0/llms.txt - v2.26: https://docs.nvidia.com/run-ai/self-hosted/v2.26/llms.txt - v2.25: https://docs.nvidia.com/run-ai/self-hosted/v2.25/llms.txt - v2.24: https://docs.nvidia.com/run-ai/self-hosted/v2.24/llms.txt

Day 2 Operations covers the ongoing operational tasks that keep NVIDIA Run:ai healthy and your clusters running smoothly.

Cluster Management

• Clusters - View and manage your clusters, including status, from the NVIDIA Run:ai UI.
• Scaling - Best practices for tuning NVIDIA Run:ai system services as clusters grow in nodes or workload demand, including vertical and horizontal scaling.
• High Availability - Reduce downtime and eliminate single points of failure using dedicated system nodes and replicated core and third-party services.
• Vertical Pod Autoscaling (VPA) - Automatically right-sizes CPU and memory requests for cluster-level services based on observed usage, reducing the need for manual tuning.

Maintenance

• Node Maintenance - Manage planned and unplanned node downtime while maintaining service continuity for running workloads.
• Backup and Restore - Back up and restore an NVIDIA Run:ai deployment, covering both the cluster and control plane.

Observability

• System Monitoring - Configure health alerts for NVIDIA Run:ai clusters and connect them to your organization’s alert-management systems.
• Logs Collection - Collect NVIDIA Run:ai logs for support, including enabling verbose logging for Prometheus and the Scheduler.
• Event History - Audit log API and UI event history tracking changes to business objects like clusters, projects, and assets.