Skip to navigation

DynamoMocker

View as Markdown

DynamoMocker workload (test_template_name is DynamoMocker) runs GPU-free LLM inference simulation using dynamo.mocker and dynamo.frontend from the ai-dynamo package, then benchmarks the stack with aiperf or genai-perf.

It is a Standalone workload (no Slurm/Kubernetes/RunAI required).

Prerequisites

CloudAI automatically installs ai-dynamo, aiperf, and genai-perf into a managed Python virtual environment on first run — no manual pip install is needed.

The one prerequisite is nats-server. On many clusters nats-server is pre-installed by administrators and is already on PATH. Check if it is already available:

which nats-server

If not found, download the binary from the official releases, extract it, and add it to PATH:

# replace <version> with the latest release tag, e.g. v2.10.24
curl -L https://github.com/nats-io/nats-server/releases/download/<version>/nats-server-<version>-linux-amd64.zip \
-o /tmp/nats-server.zip
unzip /tmp/nats-server.zip -d /tmp/
mkdir -p ~/.local/bin && mv /tmp/nats-server-<version>-linux-amd64/nats-server ~/.local/bin/
export PATH="$HOME/.local/bin:$PATH" # add to ~/.bashrc to persist

An HF_TOKEN environment variable is required to download gated models from HuggingFace Hub. Set it before running:

export HF_TOKEN=<your_token>

Topologies

The workload supports two disaggregation modes, configured via cmd_args.worker.disaggregation_mode:

  • Combined (none): a single dynamo.mocker process handles both prefill and decode. Controlled by cmd_args.worker.num_workers.
  • Disaggregated (prefill_decode): separate prefill and decode mocker instances, mirroring the production ai_dynamo topology. Instance counts are set via cmd_args.worker.prefill_worker.num_nodes and cmd_args.worker.decode_worker.num_nodes.

Benchmark Tools

Select the benchmark tool with cmd_args.benchmark_tool:

  • "aiperf" (default in the provided TOML) — uses the aiperf profiler
  • "genai_perf" — uses genai-perf profile

Parameters for the active tool are configured under [cmd_args.aiperf] or [cmd_args.genai_perf].

Run Using Standalone

Set HF_TOKEN before running to allow model download from HuggingFace Hub.

uv run cloudai run \
--system-config conf/experimental/dynamo_mocker/system/standalone_system.toml \
--tests-dir conf/experimental/dynamo_mocker/test \
--test-scenario conf/experimental/dynamo_mocker/test_scenario/dynamo_mocker.toml

CloudAI will:

  1. Install ai-dynamo, aiperf, and genai-perf into a managed venv (first run only).
  2. Write a wrapper script and launch dynamo_mocker.sh.
  3. Start nats-server, dynamo.mocker (prefill and decode), and dynamo.frontend.
  4. Run the benchmark and write results to the output directory.

Review Benchmark Results

After the run completes, results are placed in results/<scenario_name>/<test_id>/:

  • benchmark_report.csv — full per-request and aggregate metrics (throughput, latency percentiles, TTFT, ITL)
  • stdout.txt / stderr.txt — orchestration log and process output
  • dynamo_prefill_0.log, dynamo_decode_0.log, dynamo_frontend.log — per-component logs
  • nats.log — NATS server log

Key summary metrics from benchmark_report.csv:

Metric,Value Output Token Throughput (tokens/sec),667.58 Request Count,50.00 Request Throughput (requests/sec),18.04

Metric,avg,p50,p99 Request Latency (ms),507.49,475.50,893.26 Time to First Token (ms),77.82,71.99,137.26 Inter Token Latency (ms),12.03,11.55,16.42

API Documentation

Command Arguments

DynamoMockerCmdArgs

Top-level command arguments for the Dynamo Mocker workload.

FieldTypeDefaultDescription
requirementsstr'ai-dynamo==1.3.0.post1 genai-perf==0.0.16 aiperf==0.11.0'Space-separated Python requirements installed into the Dynamo Mocker environment.
model_pathstr'Qwen/Qwen3-0.6B'
nats_cmdstr'nats-server -js'Command used to launch nats-server. Override with a full path if nats-server is not on PATH.
enginecloudai.workloads.dynamo_mocker.dynamo_mocker.MockerEngineArgsFactory: MockerEngineArgs
workercloudai.workloads.dynamo_mocker.dynamo_mocker.MockerWorkerConfigFactory: MockerWorkerConfig
frontendcloudai.workloads.dynamo_mocker.dynamo_mocker.MockerFrontendArgsFactory: MockerFrontendArgs
benchmark_toolLiteral['genai_perf', 'aiperf']'genai_perf'
genai_perfcloudai.workloads.dynamo_mocker.dynamo_mocker.MockerGenAIPerfArgsFactory: MockerGenAIPerfArgs
aiperfcloudai.workloads.dynamo_mocker.dynamo_mocker.MockerAIPerfArgsFactory: MockerAIPerfArgs

Test Definition

DynamoMockerTestDefinition

Test definition for the Dynamo Mocker workload.

Runs dynamo.mocker + dynamo.frontend as a lightweight GPU-free LLM inference simulator, then benchmarks it with genai-perf or aiperf via dynamo_mocker.sh. Select the benchmark tool with cmd_args.benchmark_tool (“genai_perf” or “aiperf”).

Supports two topologies mirroring ai_dynamo.sh:

  • Combined (worker.disaggregation_mode=none): single mocker handles prefill+decode
  • Disaggregated (worker.disaggregation_mode=prefill_decode): separate prefill and decode mocker instances with KV event publishing and simulated transfer

Requires: ai-dynamo and genai-perf or aiperf pip-installed in the active environment.

FieldTypeDefaultDescription
namestrRequired
descriptionstrRequired
test_template_namestrRequired
cmd_argscloudai.workloads.dynamo_mocker.dynamo_mocker.DynamoMockerCmdArgsRequired
dse_excluded_argslist[str]Factory: list
extra_env_varsdict[str, str | typing.List[str]]{}
extra_cmd_argsdict[str, str]{}
extra_container_mountslist[str][]
git_reposlist[cloudai._core.installables.git_repo.GitRepo][]
nsyscloudai.models.workload.NsysConfiguration | NoneNone
predictorcloudai.models.workload.PredictorConfig | NoneNone
training_reportcloudai.models.workload.TrainingReportConfig | NoneNone
agentstr'grid_search'
agent_stepsint1
agent_metricslist[str]['default']
agent_reward_functionstr'inverse'
agent_configdict[str, Any] | NoneNoneAgent configuration.
env_paramsdict[str, cloudai.configurator.env_params.EnvParamSpec]Factory: dictEnvironment parameters sampled by the env per trial. Sibling to cmd_args; not part of the agent’s action space. CloudAIGymEnv samples, persists them in the trajectory output, and includes them in the trajectory cache key.
success_markerstr'success-marker.txt'
failure_markerstr'failure-marker.txt'