FluxCD

This section describes how to use FluxCD for GitOps-based deployment of Dynamo inference graphs. GitOps enables you to manage your Dynamo deployments declaratively using Git as the source of truth. We’ll use the aggregated vLLM example to demonstrate the workflow.

Prerequisites

• A Kubernetes cluster with Dynamo Kubernetes Platform installed
• FluxCD installed in your cluster
• A Git repository to store your deployment configurations

Workflow Overview

The GitOps workflow for Dynamo deployments consists of three main steps:

1. Build and push the Dynamo Operator
2. Create and commit a DynamoGraphDeployment custom resource for initial deployment
3. Update the graph by building a new version and updating the CR for subsequent updates

Step 1: Build and Push Dynamo Operator

First, follow to See Install Dynamo Kubernetes Platform.

Step 2: Create Initial Deployment

Create a new file in your Git repository (e.g., deployments/llm-agg.yaml) with the following content:

1
apiVersion: nvidia.com/v1alpha1
2
kind: DynamoGraphDeployment
3
metadata:
4
  name: llm-agg
5
spec:
6
  pvcs:
7
    - name: vllm-model-storage
8
      size: 100Gi
9
  services:
10
    Frontend:
11
      replicas: 1
12
      envs:
13
      - name: SPECIFIC_ENV_VAR
14
        value: some_specific_value
15
    Processor:
16
      replicas: 1
17
      envs:
18
      - name: SPECIFIC_ENV_VAR
19
        value: some_specific_value
20
    VllmWorker:
21
      replicas: 1
22
      envs:
23
      - name: SPECIFIC_ENV_VAR
24
        value: some_specific_value
25
      # Add PVC for model storage
26
      volumeMounts:
27
        - name: vllm-model-storage
28
          mountPoint: /models

Commit and push this file to your Git repository. FluxCD will detect the new CR and create the initial Dynamo deployment in your cluster.

Step 3: Update Existing Deployment

To update your pipeline, just update the associated DynamoGraphDeployment CRD

The Dynamo operator will automatically reconcile it.

Monitoring the Deployment

You can monitor the deployment status using:

$
export NAMESPACE=<namespace-with-the-dynamo-operator>
$
$
# Check the DynamoGraphDeployment status
$
kubectl get dynamographdeployment llm-agg -n $NAMESPACE