Deploy the KAI + GPU Fractions Experiment

This guide deploys 16 Qwen3-4B models, each in its own DynamoGraphDeployment (DGD), packed two per A100 40GB GPU. Unlike the KAI + HAMi experiment, each model gets a hard share of the GPU’s compute as well as its memory. The fork-built KAI-Scheduler and kai-gpu-fractioning place each worker in its own CUDA Multi-Process Service (MPS) namespace capped at 50% of the active threads, with a 19,968 MiB memory cap. The two models on a GPU run concurrently instead of taking turns.

The scripts, manifests, and results for this experiment are co-authored by @marckarp and @scheckerNV.

This is the third configuration in Interpreting GPU Sharing Results:

All commands run from examples/gpu-sharing in a checkout of the Dynamo repository. The files this guide uses are in examples/gpu-sharing/.

Requirements

You must build and install the two forks before you deploy any models. That procedure, including the full node requirements, is in Build the KAI-Scheduler and GPU Fractioning Forks and is not repeated here.

This guide assumes a multi-node Kubernetes cluster that you administer from a workstation with kubectl and helm. All 16 models run on one node with eight A100 40GB GPUs. The model cache is a hostPath directory on that node, so every frontend and worker is pinned to it. The load generator runs in its own pod on a different node and reaches each frontend through its Service DNS name.

The download Job’s init container runs as root to create the cache directory, and the DGDs mount it as hostPath. If your cluster enforces the baseline or restricted Pod Security Standard, run the experiment in a namespace labeled pod-security.kubernetes.io/enforce=privileged, and set NAMESPACE for every script and command below.

Pick the GPU node and export its name; every later step uses it:

kubectl get nodes -L nvidia.com/gpu.product,nvidia.com/gpu.count
export GPU_NODE=<node-name>
kubectl get node "$GPU_NODE" -o jsonpath='{.status.allocatable.nvidia\.com/gpu}{"\n"}'   # 8

Step 1: Install the Dynamo Platform

cd examples/gpu-sharing
common/install-dynamo-platform.sh

The script installs the dynamo-platform Helm chart, version 1.4.2, into the dynamo-system namespace and waits for its pods to be Ready. It sets global.etcd.install=true and global.nats.install=true, which is deliberate. Platform 1.4.x installs without etcd and NATS by default and configures Kubernetes-based discovery, and with that default the vllm-runtime:1.3.0 frontend never lists the model: /v1/models is empty and chat requests return 404. The generated DGDs set DYN_DISCOVERY_BACKEND=etcd, which needs the etcd and NATS the script installs.

Step 2: Download the Model to the GPU Node

NODE_NAME="$GPU_NODE" common/download-model.sh

The script runs a Kubernetes Job on $GPU_NODE that downloads Qwen/Qwen3-4B at revision 1cfa9a7208912126459214e8b04321603b3df60c into the hostPath cache HF_CACHE_DIR (default /opt/hf-cache) on that node, and prints the snapshot path. An init container running as root creates the directory and opens its permissions, because the download container, like the workers, runs as a non-root user. The script also reads NAMESPACE (default default), HF_TOKEN, and RUNTIME_IMAGE.

Step 3: Install the Forks

Follow Build the KAI-Scheduler and GPU Fractioning Forks through Step 4. Do not install the upstream KAI-Scheduler or the HAMi isolator on the same cluster: the forks are a matched pair, and the KAI + HAMi stack uses the same kai-scheduler release and namespace. If you ran the KAI + HAMi experiment on this cluster, uninstall it first.

Queues

The experiment’s workers use the default-queue queue, which the KAI-Scheduler chart creates. The KAI + HAMi install script patches both default queues to unlimited quota, but the fork chart creates them with quota 0 and limit -1, which admits over-quota work, and the published run’s records show no queue patch. If workers stay Pending with a queue-quota reason, apply the same patch that install-kai-hami.sh uses:

for q in default-parent-queue default-queue; do
  kubectl patch queue "$q" --type=merge -p '{"spec":{"resources":{
    "gpu":{"quota":-1,"limit":-1,"overQuotaWeight":1},
    "cpu":{"quota":-1,"limit":-1,"overQuotaWeight":1},
    "memory":{"quota":-1,"limit":-1,"overQuotaWeight":1}}}}'
done

Verify the Caps With the Smoke Test

smoke-test-half-gpu.yaml starts two idle pods, fraction-smoke-a and fraction-smoke-b, each requesting gpu-fraction: "0.5", the same pods as the build guide smoke test. Run it before you commit eight GPUs to the full experiment:

kubectl apply -f kai-gpu-fractions/smoke-test-half-gpu.yaml
kubectl wait --for=condition=Ready pod/fraction-smoke-a pod/fraction-smoke-b --timeout=180s
for p in fraction-smoke-a fraction-smoke-b; do
  kubectl exec "$p" -- env | grep -E 'CUDA_MPS|NVIDIA_GPU_MEMORY|NVIDIA_VISIBLE_DEVICES'
done

Both pods must report the same NVIDIA_VISIBLE_DEVICES GPU UUID, CUDA_MPS_ACTIVE_THREAD_PERCENTAGE=50, and CUDA_MPS_PINNED_DEVICE_MEM_LIMIT=0=19968M. If any of the three differs, fix the install before you continue; the build guide lists the common causes. Delete the pods before you deploy the DGDs:

kubectl delete -f kai-gpu-fractions/smoke-test-half-gpu.yaml

Step 4: Deploy the 16 DGDs

kai-gpu-fractions/gen-dgds.sh renders the 16 DGDs. With NODE_NAME set, it pins every frontend and worker to that node, where the model cache is. Render and apply in one step:

NODE_NAME="$GPU_NODE" kai-gpu-fractions/gen-dgds.sh | kubectl apply -f -
kubectl get dgd -n default      # wait until all 16 DGDs are Ready

The generator also reads NAMESPACE (default default) and HF_CACHE_DIR (default /opt/hf-cache); set them to the values you used in Step 2. It takes an optional model snapshot path as its first argument. The checked-in kai-gpu-fractions/dgds-16x.yaml is the same output without a node pin, for reference.

NUM_MODELS (default 16) sets the number of DGDs. Each worker carries the label kai.scheduler/queue: default-queue, the annotations gpu-fraction: "0.5" and nvidia.com/container.main.gpu-compute.mode: sm-sharing, schedulerName: kai-scheduler, runtimeClassName: nvidia, and a non-root security context. It has no nvidia.com/gpu resource.

Verify Placement and Caps

All 32 pods should run on $GPU_NODE:

kubectl get pods -n default -l experiment.nvidia.com/dgd -o custom-columns=NODE:.spec.nodeName --no-headers | sort | uniq -c

Expect one line, 32 <GPU_NODE>. Each GPU UUID should host exactly two workers:

for p in $(kubectl get pods -n default -l experiment.nvidia.com/role=worker -o name); do
  kubectl exec -n default "${p#pod/}" -- env | grep NVIDIA_VISIBLE
done | sort | uniq -c

Expect eight lines, each with a count of 2. Then confirm the caps on a worker, and the MPS server’s own view, which is the ground truth for enforcement:

POD=$(kubectl get pods -n default -l experiment.nvidia.com/role=worker -o name | head -1)
kubectl exec -n default "${POD#pod/}" -- env | grep -E 'CUDA_MPS_ACTIVE_THREAD_PERCENTAGE|CUDA_MPS_PINNED_DEVICE_MEM_LIMIT'
kubectl -n gpu-fractioning exec ds/gpu-fractioning-mpsd -- \
  nvidia-cuda-mps-control -p 3 namespace list --server=shared

The worker reports CUDA_MPS_ACTIVE_THREAD_PERCENTAGE=50 and CUDA_MPS_PINNED_DEVICE_MEM_LIMIT=0=19968M. The namespace list shows 16 kai_* namespaces at 50.00 active thread percentage, plus the default namespace at 10.00.

Step 5: Run the Concurrency Sweep

The frontends’ ClusterIPs are reachable only from inside the cluster, so the sweep runs in a load-generator pod, common/aiperf-client-pod.yaml. The pod requests 16 CPUs, one per concurrent AIPerf process, and its node affinity keeps it off $GPU_NODE, so load generation does not compete with the workers for host CPU.

Start the pod and copy the scripts into it:

sed "s/<gpu-node>/$GPU_NODE/" common/aiperf-client-pod.yaml | kubectl apply -f -
kubectl wait --for=condition=Ready pod/aiperf-client --timeout=300s
kubectl cp common aiperf-client:/work/common
kubectl exec aiperf-client -- bash /work/common/setup-aiperf.sh

Start the sweep as a detached process inside the pod, so that it keeps running if your kubectl exec session drops, then follow its log. Stopping tail with Ctrl+C does not stop the sweep:

kubectl exec aiperf-client -- bash -c \
  'cd /work && setsid bash -c "ENDPOINT_MODE=dns NUM_MODELS=16 RESULTS_DIR=/work/results \
     bash common/run-sweep.sh kai-fractions-16x-sweep > sweep.log 2>&1; echo \$? > sweep.exit" \
   < /dev/null > /dev/null 2>&1 &'
kubectl exec aiperf-client -- tail -f /work/sweep.log

The sweep takes several hours. It is finished when /work/sweep.exit exists; 0 means every AIPerf run succeeded. Copy the results to your workstation:

kubectl exec aiperf-client -- cat /work/sweep.exit
kubectl cp aiperf-client:/work/results ./results

The published run drove the frontends from the GPU node itself. Running the load generator on another node adds one network hop per request, which is small next to the TTFT values below.

For each concurrency in 1 2 4 8 16 32, the script sends max(40, 10 * c) requests per model after min(16, 2 * c) warmup requests. It uses streaming chat with 2048 input and 256 output tokens and seed 42. With ENDPOINT_MODE=dns, each AIPerf process targets http://<NAME_PREFIX>-<NN>-frontend.<NAMESPACE>.svc.cluster.local:8000.

Results land in results/kai-fractions-16x-sweep/c<N>/worker-NN/, with a worker-NN.log next to each directory. If any AIPerf run fails, the script prints the path of its worker-NN.log to sweep.log and exits non-zero after the last point. Re-run the placement and MPS namespace checks after the sweep; the published run still showed two workers per GPU and 16 namespaces at 50% afterward.

Expected Results

The published run (8x A100 40GB, driver 615.71.09) completed all 10,880 measured requests with zero errors and no restarts. Aggregate tok/s is the sum of the 16 per-model output tokens per second, and the latency columns are the mean of the 16 per-model p50 values.

The last column compares each row with the baseline at the same requests per GPU, as in Interpreting GPU Sharing Results, where this configuration at c is matched with the baseline at 2c. The c=32 row has no counterpart there, because the baseline would need c=64. The same table gives the HAMi comparison: at equal per-GPU load, fractions keeps 15 to 20 points more throughput than time-slicing.

Throughput grows about 7.8x from c=1 to c=32, with only a 13% gain from c=16 to c=32, so the knee is near c=16. TTFT is the main cost at saturation: the mean TTFT p90 at c=32 is about 8.5 s. Throughput varied by less than 0.6% across the 16 models at every point.

Key Configuration

Clean Up

kubectl delete pod aiperf-client -n default --ignore-not-found
NODE_NAME="$GPU_NODE" kai-gpu-fractions/gen-dgds.sh | kubectl delete -f -
kubectl delete job download-qwen3-4b -n default --ignore-not-found
kubectl delete secret download-qwen3-4b-hf-token -n default --ignore-not-found

To remove the forks, follow the uninstall steps. If MPS state is left over after repeated teardown, delete the gpu-fractioning-mpsd pods to restart the daemons. The model cache stays in /opt/hf-cache on $GPU_NODE; keep it if you plan to run the other experiments on that node.

Next Steps

• Deploy the Baseline Experiment and Deploy the KAI + HAMi Experiment produce the other two sweeps that the percentages compare against.
• Per-GPU Compute and Memory Limits with MPS explains the MPS v3 mechanism that the forks drive.
• Interpreting GPU Sharing Results explains how to read the three sweeps together.