Deploy the Baseline Experiment

This guide deploys the baseline for the many-model GPU sharing benchmark: eight Qwen3-4B models, each in its own DynamoGraphDeployment (DGD), each on a dedicated A100 40GB GPU. You then sweep concurrency against all eight models at once. The numbers you collect are the denominator for every percentage in Interpreting GPU Sharing Results.

The scripts, manifests, and results for this experiment are co-authored by @marckarp and @scheckerNV.

The baseline uses the stock Kubernetes scheduler and whole-GPU requests, so it needs no scheduler or isolation add-ons. The other two experiments pack two models per GPU and differ only in how they isolate the models:

All commands run from examples/gpu-sharing in a checkout of the Dynamo repository. The files this guide uses are in examples/gpu-sharing/.

Requirements

This guide assumes a multi-node Kubernetes cluster that you administer from a workstation with kubectl and helm. All eight models run on one node with eight A100 40GB GPUs. The model cache is a hostPath directory on that node, so every frontend and worker is pinned to it. The load generator runs in its own pod on a different node and reaches each frontend through its Service DNS name.

The download Job’s init container runs as root to create the cache directory, and the DGDs mount it as hostPath. If your cluster enforces the baseline or restricted Pod Security Standard, run the experiment in a namespace labeled pod-security.kubernetes.io/enforce=privileged, and set NAMESPACE for every script and command below.

Pick the GPU node and export its name; every later step uses it:

kubectl get nodes -L nvidia.com/gpu.product,nvidia.com/gpu.count
export GPU_NODE=<node-name>
kubectl get node "$GPU_NODE" -o jsonpath='{.status.allocatable.nvidia\.com/gpu}{"\n"}'   # 8

Step 1: Install the Dynamo Platform

cd examples/gpu-sharing
common/install-dynamo-platform.sh

The script installs the dynamo-platform Helm chart, version 1.4.2, into the dynamo-system namespace and waits for its pods to be Ready. It sets global.etcd.install=true and global.nats.install=true, which is deliberate. Platform 1.4.x installs without etcd and NATS by default and configures Kubernetes-based discovery, and with that default the vllm-runtime:1.3.0 frontend never lists the model: /v1/models is empty and chat requests return 404. The generated DGDs set DYN_DISCOVERY_BACKEND=etcd, which needs the etcd and NATS the script installs.

Override PLATFORM_NAMESPACE or PLATFORM_VERSION to change the namespace or chart version.

Step 2: Download the Model to the GPU Node

NODE_NAME="$GPU_NODE" common/download-model.sh

The script runs a Kubernetes Job on $GPU_NODE that downloads Qwen/Qwen3-4B at the pinned revision 1cfa9a7208912126459214e8b04321603b3df60c into a hostPath cache on that node, and prints the snapshot path. An init container running as root creates the directory and opens its permissions, because the download container, like the workers, runs as a non-root user. The workers then run fully offline from that cache.

Step 3: Deploy the Eight DGDs

baseline/gen-dgds.sh renders the eight DGDs. With NODE_NAME set, it pins every frontend and worker to that node, where the model cache is. Render and apply in one step:

NODE_NAME="$GPU_NODE" baseline/gen-dgds.sh | kubectl apply -f -
kubectl get dgd -n default      # wait until all eight DGDs are Ready

The generator also takes an optional model snapshot path as its first argument. The checked-in baseline/dgds-8x.yaml is the same output without a node pin, for reference.

Each DGD has one frontend and one aggregated vLLM worker named qwen3-4b-NN. The worker requests and is limited to nvidia.com/gpu: "1", so the default scheduler gives each model its own GPU and the pod sees the full 40,960 MiB.

Verify Placement

All 16 pods should run on $GPU_NODE:

kubectl get pods -n default -l experiment.nvidia.com/dgd -o custom-columns=NODE:.spec.nodeName --no-headers | sort | uniq -c

Expect one line, 16 <GPU_NODE>. Each of the eight GPU UUIDs should host exactly one worker:

for p in $(kubectl get pods -n default -l experiment.nvidia.com/role=worker -o name); do
  kubectl exec -n default "${p#pod/}" -- env | grep NVIDIA_VISIBLE
done | sort | uniq -c

Expect eight lines, each with a count of 1. Then confirm that all eight frontend Services exist:

kubectl get svc -n default | grep frontend      # 8 services, qwen3-4b-NN-frontend

Step 4: Run the Concurrency Sweep

The frontends’ ClusterIPs are reachable only from inside the cluster, so the sweep runs in a load-generator pod, common/aiperf-client-pod.yaml. The pod requests 16 CPUs, and its node affinity keeps it off $GPU_NODE, so load generation does not compete with the workers for host CPU.

Start the pod and copy the scripts into it:

sed "s/<gpu-node>/$GPU_NODE/" common/aiperf-client-pod.yaml | kubectl apply -f -
kubectl wait --for=condition=Ready pod/aiperf-client --timeout=300s
kubectl cp common aiperf-client:/work/common
kubectl exec aiperf-client -- bash /work/common/setup-aiperf.sh

Start the sweep as a detached process inside the pod, so that it keeps running if your kubectl exec session drops, then follow its log. Stopping tail with Ctrl+C does not stop the sweep:

kubectl exec aiperf-client -- bash -c \
  'cd /work && setsid bash -c "ENDPOINT_MODE=dns NUM_MODELS=8 RESULTS_DIR=/work/results \
     bash common/run-sweep.sh baseline-8x-sweep > sweep.log 2>&1; echo \$? > sweep.exit" \
   < /dev/null > /dev/null 2>&1 &'
kubectl exec aiperf-client -- tail -f /work/sweep.log

NUM_MODELS defaults to 16, so you must set it to 8 here. The sweep is finished when /work/sweep.exit exists; 0 means every AIPerf run succeeded. Copy the results to your workstation:

kubectl exec aiperf-client -- cat /work/sweep.exit
kubectl cp aiperf-client:/work/results ./results

The published run drove the frontends from the GPU node itself. Running the load generator on another node adds one network hop per request, which is small next to the TTFT values below.

For each concurrency in 1 2 4 8 16 32, the script sends max(40, 10 * c) requests per model after min(16, 2 * c) warmup requests. The payload is streaming chat with 2048 input and 256 output tokens, fixed, with seed 42. With ENDPOINT_MODE=dns, each AIPerf process targets http://<NAME_PREFIX>-<NN>-frontend.<NAMESPACE>.svc.cluster.local:8000.

setup-aiperf.sh also reads AIPERF_VERSION (default 0.11.0). Results land in results/baseline-8x-sweep/c<N>/worker-NN/, with one log per worker next to each directory. If any AIPerf run fails, the script prints the path of its worker-NN.log to sweep.log and exits non-zero after the last point.

Expected Results

The published run on 8x A100 40GB, driver 615.71.09, completed all 5,440 measured requests with zero errors. Aggregate tok/s is the sum of the eight per-model output tokens per second. The latency columns combine the eight per-model p50 values, which agreed to within 3% at every point except TTFT at c=8 (297 to 333 ms across models).

The tok/s per GPU column is the baseline column in Interpreting GPU Sharing Results; each GPU hosts one model, so it equals per-model throughput. Per-worker throughput varied by less than 0.2% at every point. Your absolute numbers will differ; compare ratios against the other two experiments, not raw values.

Key Configuration

Clean Up

kubectl delete pod aiperf-client -n default --ignore-not-found
NODE_NAME="$GPU_NODE" baseline/gen-dgds.sh | kubectl delete -f -
kubectl delete job download-qwen3-4b -n default --ignore-not-found
kubectl delete secret download-qwen3-4b-hf-token -n default --ignore-not-found

The model cache stays in /opt/hf-cache on $GPU_NODE; keep it if you plan to run the other experiments on that node, since they use the same cache. To remove the platform, run helm uninstall dynamo-platform -n dynamo-system.

Next Steps

• Deploy the KAI + HAMi Experiment packs two models per GPU with memory caps and time-sliced compute.
• Build the KAI-Scheduler and GPU Fractioning Forks, then deploy the KAI + GPU Fractions experiment, which adds a hard 50% compute share per model.
• Interpreting GPU Sharing Results explains how to compare your sweep with the other two.