Multinode Orchestration

Multinode deployments require either Grove + KAI Scheduler or an alternative orchestrator setup (LeaderWorkerSet + Volcano) to enable gang scheduling for workloads that span multiple nodes. See the Multinode Deployment Guide for details on orchestrator selection and configuration.

Option 1: Grove + KAI Scheduler

Grove is the default and recommended orchestrator for multinode deployments. It requires KAI Scheduler as well. There are two ways to enable Grove and KAI Scheduler, either dynamo can install it automatically (recommended for development and testing), or you can install them separately (recommended for production).

helm upgrade --install dynamo-platform dynamo-platform-$RELEASE_VERSION.tgz \
--namespace $NAMESPACE \
--create-namespace \
--set "global.grove.install=true" \
--set "global.kai-scheduler.install=true"

Option 2: LeaderWorkerSet + Volcano

If you are not using Grove for multinode, you can use LeaderWorkerSet (LWS) (>= v0.7.0) with Volcano for gang scheduling. Both must be installed before deploying multinode workloads.

1. Install Volcano:

$
helm repo add volcano-sh https://volcano-sh.github.io/helm-charts
$
helm repo update
$
helm install volcano volcano-sh/volcano -n volcano-system --create-namespace

2. Install LWS (>= v0.7.0) with Volcano gang scheduling enabled:

$
export LWS_VERSION=0.8.0
$
helm install lws oci://registry.k8s.io/lws/charts/lws \
>
  --version=$LWS_VERSION \
>
  --namespace lws-system \
>
  --create-namespace \
>
  --set gangSchedulingManagement.schedulerProvider=volcano \
>
  --wait --timeout 300s

See the LWS docs and Volcano docs for configuration options, and the Multinode Deployment Guide for orchestrator selection.