Load Models from MatrixHub

MatrixHub is a model registry that can make model weights available inside your network. Pointing the Hugging Face client at an in-cluster MatrixHub endpoint lets Dynamo download a pre-cached model without retrieving the weights from the public Hugging Face service.

This guide deploys an OpenAI-compatible Dynamo service for Qwen/Qwen3-0.6B on Kubernetes. It also describes an optional comparison against a public Hugging Face download.

Prerequisites

Before you begin, make sure that the following are available:

• A Kubernetes cluster with the Dynamo Operator installed.
• A GPU node with an NVIDIA GPU available to Kubernetes.
• An in-cluster MatrixHub registry reachable at http://matrixhub.example.com:30001.
• The Qwen/Qwen3-0.6B repository pre-cached in MatrixHub.
• Network access from the GPU node to nvcr.io and to the MatrixHub endpoint.
• A machine with kubectl and a kubeconfig for the cluster.

To cache the model in MatrixHub, run the following command from a host that can reach your MatrixHub deployment:

$
HF_ENDPOINT="http://matrixhub.example.com:30001" hf download Qwen/Qwen3-0.6B

Replace http://matrixhub.example.com:30001 with your MatrixHub endpoint.

Deploy Dynamo with MatrixHub

Connect to the cluster

Set the kubeconfig in each new terminal, then verify that the cluster is reachable:

$
# Replace with the path to your kubeconfig file.
$
export KUBECONFIG="/path/to/kubeconfig"
$
$
kubectl get nodes

Create the deployment manifest

Save the following manifest as dgd-vllm.yaml. Replace matrixhub.example.com:30001 and the model name as needed. If you change the model, update both --model and --served-model-name, and ensure that the repository is available in MatrixHub.

1
apiVersion: nvidia.com/v1alpha1
2
kind: DynamoGraphDeployment
3
metadata:
4
  name: vllm-qwen
5
  namespace: dynamo-system
6
spec:
7
  services:
8
    Frontend:
9
      componentType: frontend
10
      replicas: 1
11
      resources:
12
        requests:
13
          cpu: "2"
14
          memory: "4Gi"
15
        limits:
16
          cpu: "2"
17
          memory: "4Gi"
18
      extraPodSpec:
19
        mainContainer:
20
          image: nvcr.io/nvidia/ai-dynamo/vllm-runtime:1.1.1
21
          workingDir: /workspace
22
          env:
23
            - name: HF_ENDPOINT
24
              value: http://matrixhub.example.com:30001
25
          command: ["python3", "-m", "dynamo.frontend"]
26
          args: ["--http-port", "8000"]
27
28
    decode:
29
      componentType: worker
30
      subComponentType: decode
31
      replicas: 1
32
      resources:
33
        requests:
34
          cpu: "4"
35
          memory: "16Gi"
36
          custom:
37
            nvidia.com/gpu: "1"
38
        limits:
39
          cpu: "4"
40
          memory: "16Gi"
41
          custom:
42
            nvidia.com/gpu: "1"
43
      extraPodSpec:
44
        mainContainer:
45
          image: nvcr.io/nvidia/ai-dynamo/vllm-runtime:1.1.1
46
          workingDir: /workspace
47
          env:
48
            - name: HF_ENDPOINT
49
              value: http://matrixhub.example.com:30001
50
          command: ["python3", "-m", "dynamo.vllm"]
51
          args:
52
            - --model
53
            - Qwen/Qwen3-0.6B
54
            - --served-model-name
55
            - Qwen/Qwen3-0.6B
56
            - --tensor-parallel-size
57
            - "1"
58
            - --gpu-memory-utilization
59
            - "0.85"
60
            - --max-model-len
61
            - "8192"
62
            - --no-enable-log-requests

HF_ENDPOINT directs the Hugging Face client used by the frontend and worker to MatrixHub. The model repository and file layout must be compatible with Hugging Face Hub requests.

Apply and monitor the deployment

$
kubectl apply -f dgd-vllm.yaml
$
$
kubectl -n dynamo-system get pods \
>
  -l nvidia.com/dynamo-graph-deployment-name=vllm-qwen -w

Wait for both pods to report 1/1 Running. The first deployment can take several minutes while Kubernetes pulls the runtime image and starts vLLM.

To inspect the model download, first identify the decode pod, then view its logs:

$
DECODE_POD="$(kubectl -n dynamo-system get pods \
>
  -l nvidia.com/dynamo-graph-deployment-name=vllm-qwen \
>
  -o name | grep decode)"
$
$
kubectl -n dynamo-system logs "$DECODE_POD" --tail=100

In the test environment, downloading the pre-cached model from MatrixHub took approximately 10 seconds.

Verify the service

Find the frontend pod and send an OpenAI-compatible chat-completions request from inside it:

$
FRONTEND_POD="$(kubectl -n dynamo-system get pods \
>
  -l nvidia.com/dynamo-graph-deployment-name=vllm-qwen \
>
  -o name | grep frontend)"
$
$
kubectl -n dynamo-system exec "$FRONTEND_POD" -- \
>
  curl -s http://localhost:8000/v1/chat/completions \
>
    -H 'Content-Type: application/json' \
>
    -d '{"model":"Qwen/Qwen3-0.6B","messages":[{"role":"user","content":"Introduce yourself in one sentence."}],"max_tokens":64}'

Optional: compare with public Hugging Face

To measure the same deployment without MatrixHub, remove the HF_ENDPOINT environment-variable block from both the Frontend and decode containers, then redeploy:

$
kubectl apply -f dgd-vllm.yaml

Without HF_ENDPOINT, the Hugging Face client uses its normal public endpoint. In the test environment, downloading this model from public Hugging Face took approximately six minutes.

The comparison indicates that a nearby MatrixHub cache can substantially reduce the initial model-weight download time. Actual results depend on model size, cache state, network bandwidth, and the container-image cache.