Node Group Configuration

DGX Cloud Lepton allows you to configure the node group, including the following configs:

• Storage
• Scheduling Policy
• Recovery Policy
• Quota
• Shapes

Storage

You can configure the storage for the node group, including Node Local Volume and Static NFS Volume.

Please refer to Storage guide for more details.

Scheduling Policy

Click on the Scheduling Policy option under the config menu, there will be a modal popup to configure the scheduling policy.

You can configure the Gang Scheduling for now, which is used to determine whether multiple replicas in the same group are shceduled together. By default the option is Enabled, which means multiple replicas in the same group are scheduled together.

Recovery Policy

Click on the Recovery Policy option under the config menu, there will be a modal popup to configure it.

Recovery policy is used to configure the following options:

Auto Pause Schedule

Whether to automatically pause schedule for unhealthy nodes.

This is useful when you want to avoid the node being scheduled to new workloads when it is unhealthy, which can cause the node to be stuck in the unhealthy state.

Auto Reboot

Sometimes there will be problems like hardware issues which will cause the node to be unhealthy. In this case, you can enable the Auto Reboot option, and a node which is unhealthy and also drained (no workload), it will be automatically rebooted to try to fix the issue.

Quota

Click on the Quota option under the config menu, you will be navigated to the Node Group Quota page.

You can see there are two sections in the page:

• Workspace Quota
• User Quota

Workspace Quota

Sometimes a node group is shared by multiple workspaces, and you want to limit the total resource usage in the node group for the current workspace.

For example, if you are in workspace "test-workspace", and you will see a quota configuration card named "test-workspace" in the workspace quota section. By default, the quota is not configured, which means the total resource usage in the node group is not limited, and the priority policy is "Best Effort".

Click on the edit button in the top right corner of the card, you can configure the following options:

• GPU: The total GPU quota for the workspace, default to unlimited.
• CPU: The total CPU quota for the workspace, default to unlimited.
• Memory: The total memory quota for the workspace, default to unlimited, unit is GB.
• Max Priority: Max priority for the workspace.
• Priority Policy: Priority policy for the workspace.
  • Best Effort: Resources are allocated according to the workload priority as much as possible.
  • Strict: Resources are allocated strictly according to the workload priority. If you selected "Strict" priority policy, you can then configure the "Strict Queueing Scope" option, determine the scope as "within the workspace" or "across workspaces".

User Quota

You can also configure the user quota for the node group, to allocate resource quota to users to run workloads.

• User: The user to allocate the quota to.
• GPU: The total GPU quota for the user, default to unlimited.
• CPU: The total CPU quota for the user, default to unlimited.
• Memory: The total memory quota for the user, default to unlimited, unit is GB.
• Max Priority: Max priority for the user.

Shapes

Shapes are used to manage the resource shapes can be used in the node group, refer to this documentation for more details.