Deploy the KAI + HAMi Experiment

This guide deploys 16 Qwen3-4B models, each in its own DynamoGraphDeployment (DGD), packed two per A100 40GB GPU. The upstream KAI-Scheduler places each worker on a half-GPU fraction, and the HAMi resource isolator enforces a per-pod memory cap. HAMi limits memory only: the two models on a GPU share all of its streaming multiprocessors (SMs), and the CUDA driver time-slices between them.

The scripts, manifests, and results for this experiment are co-authored by @marckarp and @scheckerNV.

This is the middle configuration in Interpreting GPU Sharing Results. It uses stock upstream components, so you can run it without building anything:

All commands run from examples/gpu-sharing in a checkout of the Dynamo repository. The files this guide uses are in examples/gpu-sharing/.

Requirements

This guide assumes a multi-node Kubernetes cluster that you administer from a workstation with kubectl and helm. All 16 models run on one node with eight A100 40GB GPUs. The model cache is a hostPath directory on that node, so every frontend and worker is pinned to it. The load generator runs in its own pod on a different node and reaches each frontend through its Service DNS name.

The download Job’s init container runs as root to create the cache directory, and the DGDs mount it as hostPath. If your cluster enforces the baseline or restricted Pod Security Standard, run the experiment in a namespace labeled pod-security.kubernetes.io/enforce=privileged, and set NAMESPACE for every script and command below.

Pick the GPU node and export its name; every later step uses it:

kubectl get nodes -L nvidia.com/gpu.product,nvidia.com/gpu.count
export GPU_NODE=<node-name>
kubectl get node "$GPU_NODE" -o jsonpath='{.status.allocatable.nvidia\.com/gpu}{"\n"}'   # 8

Step 1: Install the Dynamo Platform

cd examples/gpu-sharing
common/install-dynamo-platform.sh

The script installs the dynamo-platform Helm chart, version 1.4.2, into the dynamo-system namespace and waits for its pods to be Ready. It sets global.etcd.install=true and global.nats.install=true, which is deliberate. Platform 1.4.x installs without etcd and NATS by default and configures Kubernetes-based discovery, and with that default the vllm-runtime:1.3.0 frontend never lists the model: /v1/models is empty and chat requests return 404. The generated DGDs set DYN_DISCOVERY_BACKEND=etcd, which needs the etcd and NATS the script installs.

Step 2: Download the Model to the GPU Node

NODE_NAME="$GPU_NODE" common/download-model.sh

The script runs a Kubernetes Job on $GPU_NODE that downloads Qwen/Qwen3-4B at revision 1cfa9a7208912126459214e8b04321603b3df60c into the hostPath cache HF_CACHE_DIR (default /opt/hf-cache) on that node, and prints the snapshot path. An init container running as root creates the directory and opens its permissions, because the download container, like the workers, runs as a non-root user. The script also reads NAMESPACE (default default), HF_TOKEN, and RUNTIME_IMAGE.

Step 3: Install KAI-Scheduler and HAMi

kai-hami/install-kai-hami.sh

install-kai-hami.sh runs three stages and waits for the pods of each to be Ready:

The script waits up to 30 attempts at 2 seconds for each queue to exist before it patches it. It ends with KAI + HAMi stack ready.

Step 4: Deploy the 16 DGDs

kai-hami/gen-dgds.sh renders the 16 DGDs. With NODE_NAME set, it pins every frontend and worker to that node, where the model cache is. Render and apply in one step:

NODE_NAME="$GPU_NODE" kai-hami/gen-dgds.sh | kubectl apply -f -
kubectl get dgd -n default      # wait until all 16 DGDs are Ready

The generator also reads NAMESPACE (default default) and HF_CACHE_DIR (default /opt/hf-cache); set them to the values you used in Step 2. It takes an optional model snapshot path as its first argument. The checked-in kai-hami/dgds-16x.yaml is the same output without a node pin, for reference.

NUM_MODELS (default 16) sets the number of DGDs. Each worker carries the label kai.scheduler/queue: default-queue, the annotation gpu-fraction: "0.5", and schedulerName: kai-scheduler. It has no nvidia.com/gpu resource.

Verify Placement and Caps

All 32 pods should run on $GPU_NODE:

kubectl get pods -n default -l experiment.nvidia.com/dgd -o custom-columns=NODE:.spec.nodeName --no-headers | sort | uniq -c

Expect one line, 32 <GPU_NODE>. Each GPU UUID should host exactly two workers:

for p in $(kubectl get pods -n default -l experiment.nvidia.com/role=worker -o name); do
  kubectl exec -n default "${p#pod/}" -- env | grep NVIDIA_VISIBLE
done | sort | uniq -c

Expect eight lines, each with a count of 2. Then confirm that HAMi applied the memory cap to a worker:

POD=$(kubectl get pods -n default -l experiment.nvidia.com/role=worker -o name | head -1)
kubectl exec -n default "${POD#pod/}" -- nvidia-smi --query-gpu=memory.total --format=csv,noheader
kubectl exec -n default "${POD#pod/}" -- env | grep CUDA_DEVICE_MEMORY_LIMIT

nvidia-smi reports about 20,070 MiB, not 40,960, and the environment has CUDA_DEVICE_MEMORY_LIMIT=20070m. A Fail to create synthesized container dir warning from HAMi is benign.

Step 5: Run the Concurrency Sweep

The frontends’ ClusterIPs are reachable only from inside the cluster, so the sweep runs in a load-generator pod, common/aiperf-client-pod.yaml. The pod requests 16 CPUs, one per concurrent AIPerf process, and its node affinity keeps it off $GPU_NODE, so load generation does not compete with the workers for host CPU.

Start the pod and copy the scripts into it:

sed "s/<gpu-node>/$GPU_NODE/" common/aiperf-client-pod.yaml | kubectl apply -f -
kubectl wait --for=condition=Ready pod/aiperf-client --timeout=300s
kubectl cp common aiperf-client:/work/common
kubectl exec aiperf-client -- bash /work/common/setup-aiperf.sh

Start the sweep as a detached process inside the pod, so that it keeps running if your kubectl exec session drops, then follow its log. Stopping tail with Ctrl+C does not stop the sweep:

kubectl exec aiperf-client -- bash -c \
  'cd /work && setsid bash -c "ENDPOINT_MODE=dns NUM_MODELS=16 RESULTS_DIR=/work/results \
     bash common/run-sweep.sh kai-hami-16x-sweep > sweep.log 2>&1; echo \$? > sweep.exit" \
   < /dev/null > /dev/null 2>&1 &'
kubectl exec aiperf-client -- tail -f /work/sweep.log

The sweep is finished when /work/sweep.exit exists; 0 means every AIPerf run succeeded. Copy the results to your workstation:

kubectl exec aiperf-client -- cat /work/sweep.exit
kubectl cp aiperf-client:/work/results ./results

The published run drove the frontends from the GPU node itself. Running the load generator on another node adds one network hop per request, which is small next to the TTFT values below.

For each concurrency in 1 2 4 8 16 32, the script sends max(40, 10 * c) requests per model after min(16, 2 * c) warmup requests. It uses streaming chat with 2048 input and 256 output tokens and seed 42. With ENDPOINT_MODE=dns, each AIPerf process targets http://<NAME_PREFIX>-<NN>-frontend.<NAMESPACE>.svc.cluster.local:8000.

Results land in results/kai-hami-16x-sweep/c<N>/worker-NN/, with a worker-NN.log next to each directory. If any AIPerf run fails, the script prints the path of its worker-NN.log to sweep.log and exits non-zero after the last point.

Expected Results

The published run (8x A100 40GB, driver 595.91.07) used the same payload and request counts as above. Aggregate tok/s is the sum of the 16 per-model output tokens per second, and the latency columns are the mean of the 16 per-model p50 values.

The first five rows of tok/s per GPU are the HAMi column in Interpreting GPU Sharing Results, which matches rows on requests resident per GPU (this configuration at c equals the baseline at 2c). The c=32 row has no counterpart there, because the baseline would need c=64.

Throughput scales about 9.3x from c=1 to c=32, but the gain from c=16 to c=32 is only 13%: the shared GPUs saturate near c=16. At c=32 the TTFT p90 reaches about 5.5 s as prefills queue behind full batches. Per-model results were uniform to within a few percent at every point, because the two models on a GPU are identical.

Key Configuration

Clean Up

kubectl delete pod aiperf-client -n default --ignore-not-found
NODE_NAME="$GPU_NODE" kai-hami/gen-dgds.sh | kubectl delete -f -
kubectl delete job download-qwen3-4b -n default --ignore-not-found
kubectl delete secret download-qwen3-4b-hf-token -n default --ignore-not-found
helm uninstall kai-resource-isolator -n kai-resource-isolator
helm uninstall kai-scheduler -n kai-scheduler

The model cache stays in /opt/hf-cache on $GPU_NODE; keep it if you plan to run the other experiments on that node, since they use the same cache. Uninstall the KAI-Scheduler before you install the GPU fractions forks, because both use the release name kai-scheduler in the kai-scheduler namespace and the two builds do not mix.

Next Steps

• Deploy the Baseline Experiment produces the dedicated-GPU numbers that the percentages are measured against.
• Build the KAI-Scheduler and GPU Fractioning Forks, then deploy the KAI + GPU Fractions experiment, to give each model a hard 50% compute share.
• Interpreting GPU Sharing Results explains how to compare the three sweeps.