Skip to navigation
Platform ManagementPolicies

NVIDIA Run:ai Native Workload Policies

View as Markdown

This guide explains the structure and procedures for managing policies for NVIDIA Run:ai native workloads.

Note

  • Workload policies are enabled by default. If you do not see it in the menu, contact your administrator to enable it under General settings → Workloads → Policies
  • Starting from version 2.23, the creation and editing of control plane policies are no longer synchronized to the cluster. This ensures that new policy features function correctly and prevents conflicts with outdated cluster policies.
    • For cluster versions 2.21 and above, changes to control plane policies do not affect existing cluster policies, because workload policies are stored and enforced only in the control plane. Existing cluster policies impact only workloads submitted directly to the cluster, and administrators may choose to keep or manually delete these older cluster policies.
    • For cluster versions 2.18 to 2.20, policies in the cluster affect both control plane and cluster-submitted workloads. When a policy is created or edited in the control plane, the corresponding or effective policy in the cluster is automatically removed to avoid conflicts. Administrator acknowledgment is required before this change takes effect.

Workload Policies Table

The Workload policies table can be found under Policies in the NVIDIA Run:ai platform.

The Workload policies table provides a list of all the policies defined in the platform, and allows you to manage them.

The Workload policies table consists of the following columns:

ColumnDescription
PolicyThe policy name which is a combination of the policy scope and the policy type
TypeThe policy type is per NVIDIA Run:ai workload type. This allows administrators to set different policies for each workload type.
Policy architectureIndicates whether the policy targets a Standard workload (Training, Workspace, or Inference) or a Distributed workload.
StatusThe current state of the policy. Indicates whether the policy is Ready to enforce workloads.
ScopeThe scope the policy affects. Click the name of the scope to view the organizational tree diagram. You can only view the parts of the organizational tree for which you have permission to view.
Created byThe user who created the policy
Creation timeThe timestamp for when the policy was created
Last updatedThe last time the policy was updated

Customizing the Table View

  • Filter - Click ADD FILTER, select the column to filter by, and enter the filter values
  • Search - Click SEARCH and type the value to search by
  • Sort - Click each column header to sort by
  • Column selection - Click COLUMNS and select the columns to display in the table
  • Refresh - Click REFRESH to update the table with the latest data

Policy Structure

A workload policy consists of three constructs:

  • Rules - Constraints that limit or control the values of workload fields. Rules are enforced during submission and cannot be overridden by users.
  • Defaults - Suggested values for workload fields. Defaults are pre-filled during submission but can be modified by users.
  • Imposed assets - Assets (such as a PVC data source) that are automatically applied to any workload submitted under the policy’s scope.

For more information, see rules, defaults, and imposed assets.

Adding a Policy

To create a new policy:

  1. Click +NEW POLICY
  2. Select a scope
  3. Select the workload type
  4. Click +POLICY YAML
  5. In the YAML editor, type or paste a YAML policy with defaults and rules.
    You can use the following references and examples:
  6. Click SAVE POLICY

Editing a Policy

  1. Select the policy you want to edit
  2. Click EDIT
  3. Update the policy and click APPLY
  4. Click SAVE POLICY

Troubleshooting

Listed below are issues that might occur when creating or editing a policy via the YAML Editor:

IssueMessageMitigation
Cluster connectivity issuesThere’s no communication from cluster “cluster_name“. Actions may be affected, and the data may be stale.Verify that you are on a network that has been allowed access to the cluster. Reach out to your cluster administrator for instructions on verifying the issue.
Policy can’t be applied due to a rule that is occupied by a different policyField “field_name” already has rules in cluster: “cluster_id”Remove the rule from the new policy or adjust the old policy for the specific rule.
Policy is not visible in the UI-Check that the policy hasn’t been deleted.
Policy syntax is no validAdd a valid policy YAML;json: unknown field “field_name”For correct syntax check the Policy YAML reference or the Policy YAML examples.
Policy can’t be saved for some reasonThe policy couldn’t be saved due to a network or other unknown issue. Download your draft and try pasting and saving it again later.Possible cluster connectivity issues. Try updating the policy once again at a different time.

Viewing a Policy

To view a policy:

  1. Select the policy for which you want to view its policies.
  2. Click VIEW POLICY
  3. In the Policy form per workload section, view the workload rules and defaults:
    • Parameter
      The workload submission parameter that Rules and Defaults are applied to
    • Type (applicable for data sources only)
      The data source type (Git, S3, nfs, pvc etc.)
    • Default
      The default value of the Parameter
    • Rule
      Set up constraint on workload policy field
    • Source
      The origin of the applied policy (cluster, department or project)

Note

Some of the rules and defaults may be derived from policies of a parent cluster and/or department. You can see the source of each rule in the policy form. For more information, check the Scope of effectiveness documentation.

Deleting a Policy

  1. Select the policy you want to delete
  2. Click DELETE
  3. On the dialog, click DELETE to confirm the deletion

Using API

Go to the Policies API reference to view the available actions.