Is this page helpful?

Deploy on DGX Spark#

This guide describes how to deploy supported vision language models on two NVIDIA DGX Spark systems connected with a ConnectX-7 interconnect for distributed inference.

This guide applies to the following models:

  • GLM-5.3-Flash

Environment Configuration#

Each DGX Spark system provides the following hardware:

  • 128 GB of LPDDR5x coherent unified system memory

  • A ConnectX-7 NIC with up to 200 Gbps of network bandwidth

The two systems must be connected through ConnectX-7 and configured for RDMA over Converged Ethernet (RoCE).

Pre-Deployment Preparation#

Before starting the containers, prepare both DGX Spark systems with the required networking, memory, and optional video-runtime configuration.

  1. Stop unnecessary or memory-intensive processes on both DGX Spark systems.

  2. Connect the two systems with verified 100 Gbps QSFP28 network cables.

  3. Configure RoCE on both systems. For more information, refer to Connect Two Sparks.

  4. Prepare FFmpeg 8 on both systems if you plan to send video requests. Obtain the source from the FFmpeg 8.0 GitHub branch. The runtime directory must contain bin/ffmpeg and lib/libavcodec.so.62.

The containers use the host network stack and the following Docker options:

  • --network=host uses the host network stack.

  • --device=/dev/infiniband:/dev/infiniband:rwm maps the RoCE/InfiniBand devices into the container.

  • --ulimit memlock=-1:-1 removes the memory locking limit required for RDMA/RoCE.

Do not set hardware transport environment variables. The NIM hardware helper discovers the local RoCE devices, GID index, and network interfaces in each container.

Deploy GLM-5.3-Flash#

Start rank 0 first. The rank 0 log reports NIM_PRIMARY_NODE. Start rank 1 with that value. Both nodes use NIM_NODE_MANAGER_PORT=20000 by default.

Connected Deployment#

On both DGX Spark systems, set the image and FFmpeg paths:

export NGC_API_KEY=<your-ngc-api-key>
export NIM_IMAGE=nvcr.io/nim/zai-org/glm-5.3-flash:2.1.2-variant
export FFMPEG8_PATH="${FFMPEG8_PATH:-$HOME/nim/ffmpeg8}"

Start Rank 0#

On the first DGX Spark system, run the following command:

docker run -d --rm \
  --name nim-glm-5.3-flash-rank0 \
  --network host \
  --gpus all \
  --shm-size=16g \
  --device=/dev/infiniband:/dev/infiniband:rwm \
  --ulimit memlock=-1:-1 \
  -e NGC_API_KEY="$NGC_API_KEY" \
  -e NIM_NODE_MANAGER_PORT=20000 \
  -e NIM_SERVER_PORT=8000 \
  -v "$FFMPEG8_PATH:/opt/ffmpeg8:ro" \
  "$NIM_IMAGE"

Search the rank 0 log for NIM_PRIMARY_NODE:

docker logs nim-glm-5.3-flash-rank0 2>&1 | grep NIM_PRIMARY_NODE

The log output includes the primary-node IP address. For example:

INFO ... start worker node with NIM_PRIMARY_NODE=192.168.1.10 NIM_NODE_MANAGER_PORT=20000

Start Rank 1#

On the second DGX Spark system, set NIM_PRIMARY_NODE to the value from the rank 0 log and run the following commands:

export NIM_PRIMARY_NODE=192.168.1.10

docker run -d --rm \
  --name nim-glm-5.3-flash-rank1 \
  --network host \
  --gpus all \
  --shm-size=16g \
  --device=/dev/infiniband:/dev/infiniband:rwm \
  --ulimit memlock=-1:-1 \
  -e NGC_API_KEY="$NGC_API_KEY" \
  -e NIM_PRIMARY_NODE="$NIM_PRIMARY_NODE" \
  -e NIM_NODE_MANAGER_PORT=20000 \
  -e NIM_SERVER_PORT=8002 \
  -v "$FFMPEG8_PATH:/opt/ffmpeg8:ro" \
  "$NIM_IMAGE"

Air-Gapped Deployment#

Before starting the containers, transfer the image, the NVFP4 model files, and the FFmpeg 8 runtime to both DGX Spark systems. The examples use the same model path on both systems.

On both systems, set the image, model, and FFmpeg paths:

export NIM_IMAGE=nvcr.io/nim/zai-org/glm-5.3-flash:2.1.2-variant
export MODEL_PATH="${MODEL_PATH:-$HOME/models/GLM-5.3-Flash-NVFP4}"
export FFMPEG8_PATH="${FFMPEG8_PATH:-$HOME/nim/ffmpeg8}"

Start Rank 0#

On the first DGX Spark system, run the following command:

docker run -d --rm \
  --name nim-glm-5.3-flash-rank0 \
  --network host \
  --gpus all \
  --shm-size=16g \
  --device=/dev/infiniband:/dev/infiniband:rwm \
  --ulimit memlock=-1:-1 \
  -e NIM_NODE_MANAGER_PORT=20000 \
  -e NIM_SERVER_PORT=8000 \
  -v "$MODEL_PATH:/model:ro" \
  -e NIM_MODEL_PATH=/model \
  -v "$FFMPEG8_PATH:/opt/ffmpeg8:ro" \
  "$NIM_IMAGE"

Search the rank 0 log for NIM_PRIMARY_NODE as shown in the connected deployment, and then start rank 1 with that value.

Start Rank 1#

On the second DGX Spark system, run the following commands:

export NIM_PRIMARY_NODE=192.168.1.10

docker run -d --rm \
  --name nim-glm-5.3-flash-rank1 \
  --network host \
  --gpus all \
  --shm-size=16g \
  --device=/dev/infiniband:/dev/infiniband:rwm \
  --ulimit memlock=-1:-1 \
  -e NIM_PRIMARY_NODE="$NIM_PRIMARY_NODE" \
  -e NIM_NODE_MANAGER_PORT=20000 \
  -e NIM_SERVER_PORT=8002 \
  -v "$MODEL_PATH:/model:ro" \
  -e NIM_MODEL_PATH=/model \
  -v "$FFMPEG8_PATH:/opt/ffmpeg8:ro" \
  "$NIM_IMAGE"

Verification#

After rank 0 reports a ready state, query the API on rank 0:

curl -s http://localhost:8000/v1/models

Send inference requests to rank 0. Rank 1 connects to rank 0 automatically and does not serve client traffic.