Running NVIDIA OSMO Workflows on NVIDIA Run:ai
This tutorial demonstrates how to run NVIDIA OSMO workflows on an NVIDIA Run:ai-managed Kubernetes cluster, with an OSMO backend and pool for each NVIDIA Run:ai project. OSMO runs each workflow task as a Kubernetes pod, and NVIDIA Run:ai schedules those pods under the quota of the project. You can adapt this workflow for other teams, NVIDIA Run:ai projects, and OSMO pools.
Note
This tutorial is written for OSMO 6.3.1. The links to the OSMO documentation point to the 6.3 version. A later OSMO release can use different values and steps.
In this tutorial, you will learn how to:
- Create NVIDIA Run:ai projects for OSMO workflows
- Connect OSMO to each project with a backend and a pool
- Enable preemption reporting on the NVIDIA Run:ai Scheduler
- Run an OSMO workflow and monitor it in NVIDIA Run:ai
- Check preemption reporting
How OSMO and NVIDIA Run:ai Work Together
OSMO and NVIDIA Run:ai each handle a different part of running workflows. OSMO is the workflow platform. Users submit their workflows to OSMO, and OSMO manages them. A workflow is a set of tasks that can depend on each other, and each task runs as a Kubernetes pod. The NVIDIA Run:ai Scheduler schedules these pods. It decides which pods run, which nodes they run on, and which pods are preempted when there are not enough resources for all of them.
Each part of the setup is responsible for the following:
- OSMO manages the workflows. It includes:
- The OSMO service. The central service that provides the API and UI for workflow submission, monitoring, and management. For more details, refer to Architecture in the OSMO documentation.
- OSMO users and roles. Users log in to the OSMO service from the OSMO CLI or the UI. They can log in with their organization’s credentials through an external identity provider (IdP), or with an OSMO access token. OSMO roles then control which pools they can submit workflows to. For more details, refer to Authentication Flow.
- Pools. A pool is a group of resources that users share. Users choose a pool when they submit a workflow. Each pool references the backend that runs its workflows. For more details, refer to Resource Pools.
- Backends and the backend operator. A backend is a compute backend where workflows run. The backend operator runs on the compute cluster. Its worker creates the Kubernetes pods for the tasks, and its listener reports state changes back to the OSMO service. In this tutorial, each backend uses the namespace of one NVIDIA Run:ai project. For more details, refer to Deploy Backend Operator.
- NVIDIA Run:ai schedules the pods and enforces limits. It is responsible for:
- Access to projects. NVIDIA Run:ai administrators manage access to the projects. This is separate from how OSMO manages its users.
- Projects and namespaces. Each NVIDIA Run:ai project has its own Kubernetes namespace, which OSMO uses as a backend.
- Quota and priority. NVIDIA Run:ai schedules every pod under the quota, priority, and preemption rules of the project.
- Pod placement. NVIDIA Run:ai decides which nodes the pods run on.
Each NVIDIA Run:ai project maps to one OSMO backend, one OSMO pool, and one backend-operator Helm release. The following example shows the mapping:
The following diagram shows how users reach OSMO, and how each backend operator creates pods in the namespace of its NVIDIA Run:ai project:
Prerequisites
Before you start, make sure the following requirements are met:
OSMO Prerequisites
- A running OSMO service. For the installation, refer to Deploy Service. The OSMO administrator sets up the following before you start, and this tutorial does not cover them:
- Data storage. Where OSMO stores its data. Refer to Step 1: Configure PostgreSQL and Step 3: Configure Storage Access in Deploy Service.
- External access. How users reach OSMO from outside the cluster. Refer to Networking.
- User login. How users log in to OSMO. Refer to Authentication Flow.
- The OSMO CLI, logged in to the service. For the installation, refer to Install Client.
- OSMO 6.3.1. This tutorial uses OSMO 6.3.1, with the
serviceandbackend-operatorHelm charts at version 1.3.1. The values in this tutorial match these chart versions. A later chart can use a different values structure. - Helm 3.x. Helm installs and upgrades the OSMO charts.
- Access to the OSMO images and charts. The cluster must be able to pull the OSMO images from
nvcr.io/nvidia/osmo. The commands in this tutorial install the OSMO charts from theosmoHelm repository. For how to add the repository, refer to Deploy Backend Operator in the OSMO documentation. - Kubernetes permissions. You need Kubernetes cluster-admin permissions, or delegated permissions that cover the steps in this tutorial, including cluster-scoped OSMO objects such as PriorityClasses.
NVIDIA Run:ai Prerequisites
- Administrator access. You need NVIDIA Run:ai administrator access to create projects and change the Scheduler settings. For the roles that can create projects, refer to Roles.
Step 1: Creating the NVIDIA Run:ai Projects
Create one NVIDIA Run:ai project for each OSMO pool that you want to run on NVIDIA Run:ai. This tutorial creates two, osmo-high for high-priority workflows and osmo-low for low-priority workflows. Two projects let you test preemption in Step 5.
Note
The project names are examples. If you use other names, replace osmo-high, osmo-low, and their namespaces, runai-osmo-high and runai-osmo-low, in every command and file in this tutorial.
-
Create the projects. Create
osmo-highandosmo-low, as described in Adding a New Project. NVIDIA Run:ai creates the namespace of each project,runai-osmo-highandrunai-osmo-low. -
Set the quota. Give each project the GPU and CPU quota that you want to guarantee for its pool, as described in Assigning GPU Quota and Assigning CPU Quota. How much quota each project needs depends on the priority of its OSMO workflows:
- OSMO workflows with the
NORMALorHIGHpriority are non-preemptible. They run only within the quota of the project, so giveosmo-highenough quota for them. - OSMO workflows with the
LOWpriority are preemptible. They can also run on unused capacity beyond the quota ofosmo-low, and NVIDIA Run:ai takes that capacity back when another project needs it within its quota.
- OSMO workflows with the
-
Check the namespaces. Each namespace must belong to its project and must enforce the NVIDIA Run:ai Scheduler:
Each namespace must be associated with its NVIDIA Run:ai project and queue:
runai-osmo-highwithosmo-high, andrunai-osmo-lowwithosmo-low. Pods that third-party tools create in either namespace must receive or keepschedulerName: runai-scheduler. NVIDIA Run:ai enforces the Scheduler at the namespace level for third-party Kubernetes workloads. -
Label the compute nodes (only if your pod template uses a node selector). The OSMO pod template
default_compute, which you define in Step 2, selects compute nodes with the node selectorosmo-compute: "true". If your pod template does not set a node selector, skip this item. Otherwise, label each node that can run OSMO workflows, replacing<node-name>with the name of the node:Note
Pods stay
Pendingon nodes that do not have the label that the pod template selects. When autoscaling or hibernation replaces a node, apply the label again, or add it to the node group bootstrap template.
Step 2: Connecting OSMO to Each Project
Each NVIDIA Run:ai project needs its own OSMO backend, OSMO pool, and backend-operator Helm release. The following items show only the settings that connect OSMO to NVIDIA Run:ai. For the other chart values and the service installation, refer to Deploy Service.
Note
This step connects the osmo-high project to OSMO. Repeat it to connect the osmo-low project.
-
Create the backend. In the
services.configs.backendssection of the OSMO service values, add one backend for each project. The backend uses the namespace of the project and the NVIDIA Run:ai Scheduler:scheduler_type: kai- The OSMO scheduler integration type that works with the NVIDIA Run:ai Scheduler.scheduler_name: runai-scheduler- Sets the scheduler name on every pod that OSMO creates. NVIDIA Run:ai enforces the same Scheduler at the namespace level by default, as described in Step 1. Setting it in OSMO as well means the setting still applies if enforcement is turned off.
The namespace where OSMO creates the pods of the backend comes from
backendNamespacein the backend-operator release in item 7. Do not set it in this section. -
Define the pod template (only if your nodes use a node selector). If your nodes do not use a node selector, skip this item. Otherwise, define the
default_computepod template inservices.configs.podTemplates: -
Create the pool. In
services.configs.pools, add one pool for each project. The pool points to the backend by name. Each pool uses one backend, and each backend uses one namespace, so the workflows of a pool run in one NVIDIA Run:ai project. This tutorial uses one pool and one backend for each project:common_pod_template- The pod templates that OSMO applies to the pods of the pool. A pod template is a reusable part of a pod specification, such as a node selector.default_ctrlanddefault_userare built-in templates of the OSMO service chart.default_computeis the pod template from item 2, which sets the node selector from Step 1. If your nodes do not use a node selector, leave outdefault_compute.common_resource_validations- Built-in rules of the OSMO service chart. They reject a workflow that requests more resources than the nodes have.common_default_variables- The resource values that OSMO uses when a workflow does not set them. A workflow that setsgpu: 0counts as not settinggpu. Without these values, OSMO fails the submission of such a workflow with an internal server error.default_platformandplatforms- A platform routes workflows to specific hardware in the pool. This pool has one platform,default, which workflows use when they do not choose one.
For the other pool settings, refer to Resource Pools in the OSMO deployment guide.
-
Give users access to the pools. The default
osmo-userrole allows workflow submission only to thedefaultpool. To let users submit workflows to the new pools, add a role for each pool and assign it to the users, as described in Creating a Role for a Pool. -
Apply the values. Apply the updated values to your OSMO service release with
helm upgrade, as described in Deploy Service. These values take effect only when the OSMO service reads its configuration from the Helm values, withservices.configs.enabled: true, as set in Step 4: Prepare values of Deploy Service. The OSMO service reloads the updated configuration without a restart, and the new pools can take a few minutes to appear in the pool list. -
Create the backend operator token. The backend operator authenticates to the OSMO service with a token that has the
osmo-backendrole. Create the token and store it in a Kubernetes Secret in the OSMO namespace (osmoin this tutorial), as described in Deploy Backend Operator. -
Define the backend-operator values. Define the values of one
backend-operatorrelease for each project. The release runs in the OSMO namespace (osmo). The following values are for the release ofosmo-high:name- The name of this deployment. If you leave it out, the chart uses the Helm release name.backendName- The name of the backend from item 1.backendNamespace- The namespace of the NVIDIA Run:ai project, where the workflow pods run. OSMO creates the pods of the backend in this namespace, so NVIDIA Run:ai schedules them under the quota of the project that owns it.osmoImageTag- The tag of the OSMO images that the release runs. Set it to the version of your OSMO service,6.3.1in this tutorial. The chart default islatest, which can differ from your service version.includeNamespaceUsage- The namespaces whose pods OSMO counts as workflow pods when it reports node usage. This tutorial lists the namespaces of both projects in both releases, so that each release counts the pods of the other project as workflow pods.
The release also needs the values that connect it to the OSMO service:
serviceUrl,agentNamespace, andloginMethod- SetloginMethodtotoken, because the chart default ispassword.accountTokenSecretandaccountTokenSecretKey- The name and key of the token Secret from item 6.
For these values, and for adding the OSMO Helm repository, refer to Deploy Backend Operator.
Important
Do not point two backend-operator releases at the same backend name or namespace.
-
Install the backend operator. Save the values of the release in a file, for example
osmo-backend-high-values.yaml, and install the release: -
Repeat for the other project. To connect
osmo-low, repeat this step with the namesosmo-lowandrunai-osmo-low, and use the same token Secret. Keep the sameincludeNamespaceUsagevalue, which lists both namespaces. A Kubernetes PriorityClass gives a priority level a name that pods can use. PriorityClass objects belong to the whole cluster, so only one release should create them. The release forosmo-highcreates them. In the release forosmo-low, turn off their creation:If you add more projects, turn off the creation in every release except the first.
-
Check the pools. The
osmo-highandosmo-lowpools must show asONLINEbefore you submit workflows:If a pool is not
ONLINE, check the logs of the backend listener of its release, replacing<project>with the project name,osmo-highorosmo-low:
Step 3: Enabling Preemption Reporting
When the NVIDIA Run:ai Scheduler evicts a pod of an OSMO workflow to free its resources, OSMO needs to know that the Scheduler caused the eviction. OSMO looks for a pod condition of the type DisruptionTarget with the reason PreemptionByScheduler. Without it, OSMO reports the workflow as FAILED with code 2001, treats it as a final failure, and does not reschedule it.
-
Set the condition. Add the argument
--update-pod-eviction-condition=trueto the NVIDIA Run:ai Scheduler deployment: -
Check the setting. Check that the Scheduler is using the argument:
The output must include
--update-pod-eviction-condition=true.
Important
The NVIDIA Run:ai operator can revert a direct edit of the Scheduler deployment, so check that the argument is still set before you test preemption in Step 5.
Step 4: Running and Monitoring a Workflow
Submit a small workflow to each pool, and check the pods that OSMO creates.
-
Create a workflow file. Save the following as
hello-osmo.yaml. For the workflow file format, refer to Workflow Specification. The workflow runs until you cancel it, so that you can inspect it: -
Submit the workflow to both pools:
-
Check the pods. Each pod runs in the namespace of its project, with the NVIDIA Run:ai Scheduler and the queue of its project. First, find the pod name:
The pod must be in the
Runningphase. Then check the pod, replacing<pod-name>with the name from the previous command, and do the same for the pod inrunai-osmo-low. Theschedulervalue must berunai-scheduler:Finally, check the pod group that NVIDIA Run:ai schedules the pod under. The
QUEUEvalue must be the name of the project. ThePRIORITYvalue is the OSMO priority class of the workflow:osmo-normalfor the default priority, andosmo-lowfor theLOWpriority:Note
The pod itself carries an
osmo-pool-...queue label that OSMO sets. NVIDIA Run:ai schedules the pod under the queue of the project, as the pod group shows. -
Monitor the workflows in NVIDIA Run:ai. NVIDIA Run:ai provides full monitoring for externally submitted Kubernetes workloads, including event history, metrics, and logs. Each task pod appears in the project of its namespace, named after the pod. For more details, refer to Supported Features.
-
Cancel the workflows when you finish checking. Use the workflow IDs that OSMO returned when you submitted them:
Step 5: Checking Preemption Reporting
This step checks that OSMO reports a preempted workflow correctly. Before you start, run the check in Step 3 again to confirm that the Scheduler argument is set.
-
Request GPUs in the workflow file. In the file from Step 4, set
gputo the number of free GPUs. Preemption happens only when the two workflows compete for the same GPUs. -
Submit a long-running low-priority workflow to the
osmo-lowpool. The pool must use capacity that the NVIDIA Run:ai Scheduler can reclaim, which means GPUs beyond the GPU quota ofosmo-low:Wait until its pod is in the
Runningphase before you continue. -
Submit a higher-priority workflow to the
osmo-highpool that needs the same capacity. The GPU quota ofosmo-highmust be at least the number of GPUs in the file: -
Watch the pods in the low-priority namespace. The low-priority pod is evicted to free the GPUs for the higher-priority workflow:
-
Check the state in OSMO. Use the workflow ID that OSMO returned when you submitted the low-priority workflow:
The text output of
osmo workflow querydoes not show the exit code, so use the JSON output. OSMO must report the evicted workflow asFAILED_PREEMPTEDwith exit code3006, not asFAILEDwith code2001. To have OSMO reschedule the preempted workflow, set an exit action that reschedules tasks with exit code3006, and allow retries withmax_retry_per_taskin the workflow configuration. For more details, refer to Exit Actions.