Deploy with Dynamo
Start by serving FLUX image generation on one GPU with NVIDIA Dynamo. The walkthrough below loads a tuned checkpoint, starts the frontend and worker, and sends an image request. For video generation across multiple GPUs, continue with the WAN example in the multi-GPU section.
Your application supplies the inference callable and input conversion. AITune’s Dynamo worker registers the endpoint and converts supported outputs into responses.
If you want Dynamo to serve an exported ONNX, TensorRT, or PT2 model, follow Run Triton through Dynamo. That path uses a Triton repository and KServe gRPC.
Serve FLUX on one GPU
This walkthrough uses the existing FLUX worker and client from the AITune repository. Run the commands in the FLUX example environment, with access to FLUX.1-dev and a GPU with enough memory for the pipeline.
1. Prepare the tuned checkpoint
From the repository root, enter the example directory and install its Dynamo dependencies:
If you have not tuned the model yet:
With the default settings, this saves checkpoints/flux-dev.ait. In config.yaml, keep
Backend.tuned_model_path: "flux-dev.ait" and the generation settings used for tuning. The worker constructs the
pipeline and loads this checkpoint during startup. See Save and load checkpoints for custom paths.
2. Start the frontend
Open a terminal in examples/FLUX and start Dynamo’s HTTP frontend:
File-based discovery lets the local frontend and worker find each other without running etcd. Leave this process running.
3. Start the FLUX worker
Open a second terminal in examples/FLUX and start the worker:
The worker loads and warms up the tuned pipeline, then registers it with Dynamo. Wait for the registration message before sending a request.
4. Generate an image
From a third terminal in examples/FLUX, run the client:
The client sends a request to /v1/images/generations and saves the result as output.png. Send more requests with
different prompts using the same command. The frontend and worker keep running until you stop them. The example
processes one generation request at a time.
Multi-GPU workers
Use the WAN video generation example to serve a pipeline across multiple GPUs. It uses Diffusers context parallelism, with every rank participating in each generation request.
Prepare WAN on four GPUs
Follow the WAN environment setup and four-GPU tuning instructions first. Keep the same GPU count, context-parallel mode, and generation shapes for serving. Each rank loads its own checkpoint.
From the repository root, enter examples/WAN and install the serving dependencies:
Check config.yaml in that directory. WAN’s settings are at the top level: tuned_model_path, context_parallel,
height, width, and num_frames must match the tuning run.
Start the frontend and WAN workers
Use a separate local session for WAN; stop the FLUX worker and frontend first if you ran the single-GPU walkthrough.
From examples/WAN, start the frontend with the command above. In another terminal in
examples/WAN, launch one worker per GPU:
Wait for model registration, then send a video request from a third terminal in examples/WAN:
The client calls /v1/videos and saves output.mp4. For a launcher that starts the service, sends one request, and
shuts it down, see the WAN serving example.
Worker configuration
The callable passed to dynamo_worker() is the model implementation that AITune executes. The application creates
and loads the model before serving requests, either before starting the worker or in its setup callback.
The name model_path comes from Dynamo’s register_model() API. AITune passes this value to Dynamo as a registration
reference; AITune does not read it or use it to create the callable. model_name is an optional public alias that
clients use in requests. Keeping them separate lets a deployment use one reference for registration and expose a
stable name such as production-flux. When model_name is omitted, AITune uses model_path as the public name.
Supported modalities
Audio bytes are packed as WAV output. To serve another audio codec, return a complete Dynamo response dictionary. Plain dictionaries are forwarded to the runtime unchanged for every modality.
Customize worker startup
Use setup to initialize the model at worker startup and warmup to exercise it before endpoint registration:
The callable can also use a model loaded before worker startup. A plain callable receives the deserialized
request when mapping is omitted; an nn.Module requires a mapping to its input arguments.
For full lifecycle control, subclass DynamoWorker and implement setup() and serve(). The base class also
provides warmup() and on_ready(runtime, endpoint) hooks. With this lower-level API, your on_ready
implementation must register the model with the frontend, and serve must yield responses in Dynamo’s wire format.
The functional dynamo_worker() API handles that registration and response conversion for you.
Adapt your own multi-GPU worker
Use the same worker API for single-GPU and multi-GPU models. The application initializes the process group before
calling run() or dynamo_worker() on every rank. Construct the rank-local model before the call or in setup.
AITune starts the Dynamo endpoint only on rank 0, broadcasts each request, runs inference on every rank, and returns
only rank 0’s response.
Pass rank-local initialization that may fail through setup= and collective model warmup through warmup=. AITune
exchanges setup failures across the process group before any rank starts warmup or registers the endpoint.
AITune does not initialize or destroy the process group, add inference barriers, or synchronize CUDA. The application owns device placement, model sharding or context parallelism, and process-group cleanup. One initialized multi-rank process group represents one collective model worker. Run separate worker groups or pods for independent replicas.
Known limitations
AITune’s DynamoWorker has the following limitations:
- The
"audio"modality requires NVIDIA Dynamo 1.1 or later and serves text-to-speech requests through/v1/audio/speech. - Automatic speech recognition is not currently exposed by the worker API.