Run DeepSeek-V3 Training with WorkloadRun

This guide walks through running DeepSeek-V3 BF16 training on a GPU cluster with Cluster Readiness Engine (CRE), using Megatron-Bridge and WorkloadRun. Unlike the Nemotron 5 guide, which uses raw Megatron-LM, this example uses Megatron-Bridge’s recipe system and demonstrates the developer-friendly pattern of overriding the container’s scripts/performance/ with a cloned branch checkout — so developers can iterate on their own run_script.py / utils/overrides.py without rebuilding the image.

For an introduction to WorkloadRun and when to use it instead of a full Certification, see the WorkloadRun Quick Start. For generic WorkloadRun options (targeting nodes, measurements, framework types), see Run a WorkloadRun.

Prerequisites

• A Kubernetes cluster with GPU nodes (nvidia.com/gpu.present=true) and kubectl access
• nvcrectl installed and the CRE controller running on the cluster — see Installation
• An NGC API key for pulling the NeMo container image from nvcr.io
• The DeepSeek-V3 recipe fetches the model configuration from Hugging Face Hub at startup, so worker pods need outbound access to huggingface.co (or a pre-populated Hugging Face cache mounted into the pods — see the env var comments below)

No dataset or checkpoint download is required: the recipe trains on synthetic data, and this guide runs a proxy-size model.

Megatron-LM vs Megatron-Bridge

Create the WorkloadRun manifest

Save the following as deepseek-v3-bf16.yaml:

1
apiVersion: cre.nvidia.com/v1alpha1
2
kind: WorkloadRun
3
metadata:
4
  name: deepseek-v3-bf16
5
  namespace: default
6
spec:
7
  # The NeMo 26.02.00 image has Megatron-Bridge + Megatron-Core +
8
  # DeepEP + matching transformers/flashinfer/etc. all pre-installed at
9
  # /opt/Megatron-Bridge. Nothing needs pip install at runtime.
10
  image: nvcr.io/nvidia/nemo:26.02.00
11
  framework:
12
    exec:
13
      command: ["/bin/bash", "/config/train.sh"]
14
  numNodes: 16
15
  config:
16
    inline:
17
      train.sh: |
18
        #!/bin/bash
19
        set -e
20
21
        BRIDGE_PATH=/mnt/workspace/megatron-bridge
22
        # Prepend the cloned scripts/performance/ to PYTHONPATH so a
23
        # developer's branch overrides win for runner code, while the
24
        # bridge package + megatron-core continue to come from the
25
        # container's /opt/Megatron-Bridge.
26
        export PYTHONPATH=${BRIDGE_PATH}/scripts/performance:$PYTHONPATH
27
28
        exec torchrun \
29
          --nnodes $PET_NNODES \
30
          --nproc-per-node $PET_NPROC_PER_NODE \
31
          ${BRIDGE_PATH}/scripts/performance/run_script.py \
32
          -a dummy -p dummy \
33
          -m deepseek -s v3 \
34
          -c bf16 \
35
          -g gb300 \
36
          -ng $((PET_NNODES * PET_NPROC_PER_NODE)) \
37
          -gn $PET_NPROC_PER_NODE \
38
          -ms 50 \
39
          -mb 1 -pp 1 -ep 64 \
40
          --no-detach \
41
          model.hidden_size=1024 \
42
          model.kv_channels=64 \
43
          model.pipeline_model_parallel_layout=null \
44
          model.virtual_pipeline_model_parallel_size=null \
45
          logger.log_throughput=true \
46
          train.manual_gc=true \
47
          train.manual_gc_interval=100
48
  initContainers:
49
    - name: megatron-bridge-clone
50
      image: nvcr.io/nvidia/nemo:26.02.00
51
      command: ["/bin/bash", "-c"]
52
      args:
53
        - |
54
          set -ex
55
          git clone --depth 1 -b llmb-r0.2.0 \
56
            https://github.com/NVIDIA-NeMo/Megatron-Bridge.git \
57
            /mnt/workspace/megatron-bridge
58
      volumeMounts:
59
        - name: workspace
60
          mountPath: /mnt/workspace
61
  volumes:
62
    - name: workspace
63
      emptyDir:
64
        medium: Memory
65
  volumeMounts:
66
    - name: workspace
67
      mountPath: /mnt/workspace
68
  env:
69
    # NCCL / torch.distributed tuning recommended for this model.
70
    - name: TORCH_NCCL_AVOID_RECORD_STREAMS
71
      value: "1"
72
    - name: TORCH_NCCL_HIGH_PRIORITY
73
      value: "1"
74
    - name: NCCL_NVLS_ENABLE
75
      value: "0"
76
    - name: NCCL_NET_GDR_LEVEL
77
      value: PHB
78
    - name: NCCL_NET_GDR_C2C
79
      value: "1"
80
    # MNNVL / NVLink topology hints
81
    - name: NVLINK_DOMAIN_SIZE
82
      value: "72"
83
    - name: USE_MNNVL
84
      value: "1"
85
    - name: NUM_OF_HYBRID_EP_RANKS_PER_NVLINK_DOMAIN
86
      value: "64"
87
    # CUDA + transformer-engine tuning
88
    - name: CUDA_DEVICE_MAX_CONNECTIONS
89
      value: "32"
90
    - name: NVTE_FWD_LAYERNORM_SM_MARGIN
91
      value: "20"
92
    - name: NVTE_BWD_LAYERNORM_SM_MARGIN
93
      value: "20"
94
    # HuggingFace: the DeepSeek-V3 recipe needs to fetch the model
95
    # config to introspect architecture. Leave these at "0" so HF Hub
96
    # downloads succeed (mount an HF cache instead if your cluster is
97
    # air-gapped).
98
    - name: TRANSFORMERS_OFFLINE
99
      value: "0"
100
    - name: HF_HUB_OFFLINE
101
      value: "0"
102
    - name: TOKENIZERS_PARALLELISM
103
      value: "False"
104
    # UCX env vars to avoid memory hook conflicts in the container
105
    - name: UCX_MEM_MMAP_HOOK_MODE
106
      value: none
107
    - name: UCX_MEM_CUDA_HOOK_MODE
108
      value: none
109
    - name: UCX_MEM_MALLOC_HOOKS
110
      value: "n"
111
    - name: UCX_ERROR_SIGNALS
112
      value: ""
113
  resources:
114
    limits:
115
      nvidia.com/gpu: "4"
116
      memory: 800Gi
117
      cpu: "128"
118
    requests:
119
      nvidia.com/gpu: "4"
120
      memory: 500Gi
121
      cpu: "64"
122
  goodputMeasurement:
123
    logProfileRef: megatron-bridge
124
    sampleInterval: 10s

How the branch override works

The pattern in this YAML is the developer-iteration pattern for Megatron-Bridge:

1. Container ships everything heavy. nvcr.io/nvidia/nemo:26.02.00 already has megatron-bridge, megatron-core (with FSDP), transformers, flashinfer, DeepEP, and matching CUDA/NCCL — all version-pinned to work together. No pip install at runtime.
2. Init container clones a branch of https://github.com/NVIDIA-NeMo/Megatron-Bridge into a shared emptyDir volume. Only scripts/performance/ matters here — it’s where run_script.py, utils/overrides.py, and the recipe configs live.
3. train.sh prepends the cloned scripts/performance/ to PYTHONPATH and invokes the cloned run_script.py by absolute path. Result:
   • from utils.overrides import … resolves to the developer’s branch (override wins).
   • from megatron.bridge import … and from megatron.core import … resolve to the container’s pre-installed packages (stable, version-matched).
4. No image rebuild required. A developer commits to their branch on GitHub, points the init container’s git clone -b ... at it, and re-submits the WorkloadRun. Iteration takes seconds (small clone) instead of minutes (image build/push).

Branch choice matters: pick a branch whose script + bridge API expectations match what the container ships. llmb-r0.2.0 works against nemo:26.02.00. A branch that depends on a newer bridge API will fail at import time — pick a different branch or use a newer NeMo image.

Key configuration

• numNodes: 16 — the number of nodes per job group, not a total. The orchestrator partitions all eligible GPU nodes into groups of this size and creates one job per group: numNodes: 16 on a 16-node cluster creates a single 16-node job, while numNodes: 4 on the same cluster would create four 4-node jobs that run in parallel. Set it to your full cluster size for a single full-scale run.
• -m deepseek -s v3 — selects the DeepSeek-V3 architecture.
• -c bf16 — BF16 precision (FP8 paths in this branch require a newer bridge than nemo:26.02.00 ships).
• -g gb300 — GPU-specific tuning hints. Also accepts gb200, b200, or h100 — match your cluster.
• -a dummy -p dummy --no-detach — run_script.py in this branch was written as a SLURM job submitter; --no-detach makes it run in-process under torchrun. The -a/-p flags are required by argparse but unused when not actually submitting via SLURM.
• -ng / -gn — -ng is numNodes × gpusPerNode; -gn matches gpusPerNode. The PET_NNODES and PET_NPROC_PER_NODE environment variables are injected into worker pods automatically.
• -pp 1 -ep 64 — pipeline parallelism off, 64-way expert parallelism. Works for the proxy-size model below; for the full model use -pp 4 -ep 64 -vp 4 at 256+ GPUs. The product ep × tp × pp must divide the world size; with -pp 1 and no tensor parallelism, -ep can equal the world size.
• model.hidden_size=1024 model.kv_channels=64 — proxy model dimensions so the workload fits on a 64-GPU cluster. The full DeepSeek-V3 (671B) needs 256+ GPUs; drop these overrides when you have enough GPUs.
• model.pipeline_model_parallel_layout=null model.virtual_pipeline_model_parallel_size=null — clears the recipe’s layout/VPP defaults so they don’t conflict with the -pp 1 we’re forcing.
• logger.log_throughput=true — emits TFLOPS per iteration into the training log; the built-in megatron-bridge LogProfile parses these for goodput.
• train.manual_gc=true train.manual_gc_interval=100 — explicit garbage-collection pacing recommended for this recipe.
• resources — optional. When omitted, CRE auto-sets only nvidia.com/gpu: <gpusPerNode> (both limits and requests); it does not guess memory or CPU. Set the block explicitly, as here, if you need CPU pinning or memory sizing.

CRE auto-handles NCCL environment variables, ComputeDomain and DRA setup, EFA/RoCE networking, and topology-aware orchestration based on the detected platform and GPU architecture.

Submit the WorkloadRun

$
export NGC_API_KEY=<your-ngc-api-key>
$
$
nvcrectl workloadrun run \
>
  --workload-registry nvcr.io \
>
  --workload-registry-username '$oauthtoken' \
>
  --workload-registry-password "$NGC_API_KEY" \
>
  deepseek-v3-bf16.yaml

The --workload-registry* flags create an nvcr.io image pull secret in the target namespace and inject it into the WorkloadRun automatically.

Monitor progress

Watch pods come up:

$
kubectl get pods -w

Lightweight status check:

$
nvcrectl workloadrun status deepseek-v3-bf16

Check goodput measurement in real time:

$
kubectl get goodputmeasurement -w

The training-loop log lines (in kubectl logs <pod> -c node -f) will look like:

[2026-06-11 15:19:14] iteration   2/  50 | consumed samples: 1024 |
  elapsed time per iteration (ms): 11184.3 |
  learning rate: 6.000000E-05 | global batch size: 512 |
  lm loss: 1.178900E+01 | ... | grad norm: 1.837 | ...

…with a throughput per GPU (TFLOP/s/GPU): ... field appended on each iteration thanks to logger.log_throughput=true.

Generate a report

$
nvcrectl workloadrun report deepseek-v3-bf16

Example output (16 nodes × 4 GPUs, BF16 proxy model):

╔════════════════════════════════════════════════════════════════╗
║                       WorkloadRun Report                       ║
╚════════════════════════════════════════════════════════════════╝
  Name:      deepseek-v3-bf16
  Platform:  aws
  GPU:       gb300
  Nodes:     16
┌────────────────────────────────────────────────────────────────┐
│  workloadrun/deepseek-v3-bf16                                   │
├────────────────────────────────────────────────────────────────┤
│  Status:    Succeeded                                          │
│  Runtime:   7m 52s                                             │
│  Scale:     full-scale                                         │
│  Nodes/Job: 16                                                 │
│  Jobs:      1                                                  │
│                                                                │
│  ┌ clique-0 (16 nodes) ──────────────────────────────────────┐ │
│  │  Avg Runtime Goodput:  1.00 (100%)                        │ │
│  │  Avg TFLOPs/GPU:  556.5                                   │ │
│  │  Avg Step Time:  4.78s                                    │ │
│  └───────────────────────────────────────────────────────────┘ │
└────────────────────────────────────────────────────────────────┘
┌────────────────────────────────────────────────────────────────┐
│  Summary                                                       │
├────────────────────────────────────────────────────────────────┤
│  Categories:   1/1 passed                                      │
│  Failed Nodes: none                                            │
│  Result:       PASSED                                          │
└────────────────────────────────────────────────────────────────┘

If a node fails, CRE records it in the WorkloadRun status with a reason (HardwareFailureDetected, ThresholdViolation, or WorkloadFailed); it never taints, cordons, or otherwise modifies the node.

To save as JSON:

$
nvcrectl workloadrun report deepseek-v3-bf16 --results-file report.json

Gang scheduling

If the cluster runs a gang-aware scheduler such as KAI Scheduler, add spec.gangScheduler so all pods in a job group are held until the entire gang can be placed:

1
spec:
2
  gangScheduler:
3
    schedulerName: kai-scheduler   # required; injected as schedulerName into every workload pod
4
    queue: high-priority           # optional; defaults to "default-queue"

The queue value is applied as the kai.scheduler/queue label on the pod template metadata. It must be a valid Kubernetes label value (at most 63 characters). See Run a WorkloadRun for details.

Clean up

Cancel the WorkloadRun (cascades to its Workflow, Jobs, and pods):

$
nvcrectl workloadrun cancel deepseek-v3-bf16 -n default

Changing precision

The -c flag in train.sh controls numerical precision. This example uses -c bf16. The llmb-r0.2.0 run_script.py also accepts bf16, fp8_cs, fp8_mx, fp8_sc, and nvfp4, but some FP8 variants require recipe code that’s only on newer (incompatible) branches against nemo:26.02.00 — stick with bf16 unless you’ve verified the branch supports your chosen FP8 mode. If you switch, also update metadata.name to keep the resource name aligned with what it’s actually doing.

Available models

The llmb-r0.2.0 run_script.py accepts model family + size via -m / -s:

$
# List available recipes from inside any running pod
$
kubectl exec -it <pod> -c node -- bash -c '
>
  python /opt/Megatron-Bridge/scripts/performance/run_script.py --help
>
' 2>&1 | grep -A 1 -E "(-m|-s|--task)" | head -20

Compatible models include DeepSeek-V3, GPT-OSS, LLaMA 3.1, and Qwen3. All use the same WorkloadRun pattern — swap -m deepseek -s v3 for the desired family/size, adjust parallelism (-pp, -ep, -vp) for your GPU count, and (optionally) drop the model.hidden_size=... model.kv_channels=... proxy overrides if you have enough GPUs for the full model.

Next steps

• How-to: Run a WorkloadRun — targeting nodes, measurements, framework options
• How-to: Run Nemotron 5 Training — the raw Megatron-LM equivalent of this guide
• CLI Reference: workloadrun
• API Reference: WorkloadRun