> This page is for version dev.
> For other versions, use one of these documentation indexes:
> - 1.8.1 (default): https://docs.nvidia.com/networking/cloudai/1.8.1/llms.txt
> - 1.7.1: https://docs.nvidia.com/networking/cloudai/1.7.1/llms.txt
> - 1.6.1: https://docs.nvidia.com/networking/cloudai/1.6.1/llms.txt
> - dev: https://docs.nvidia.com/networking/cloudai/dev/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/networking/cloudai/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/networking/cloudai/_mcp/server.

DeepEP Benchmark workload (`test_template_name` is `DeepEP`) allows users to execute DeepEP (Deep Expert Parallelism) MoE (Mixture of Experts) benchmarks within the CloudAI framework.

## Overview

DeepEP is a benchmark for measuring the performance of MoE models with distributed expert parallelism. It supports:

* **Two operation modes**: Standard and Low-Latency
* **Multiple data types**: bfloat16 and FP8
* **Flexible network configurations**: With or without NVLink
* **Configurable model parameters**: Experts, tokens, hidden size, top-k
* **Performance profiling**: Kineto profiler support

## Usage Example

Test TOML example (Standard Mode):

```toml
name = "deepep_standard"
description = "DeepEP MoE Benchmark - Standard Mode"
test_template_name = "DeepEP"

[cmd_args]
docker_image_url = "<docker container url here>"
mode = "standard"
tokens = 1024
num_experts = 256
num_topk = 8
hidden_size = 7168
data_type = "bfloat16"
num_warmups = 20
num_iterations = 50
```

Test TOML example (Low-Latency Mode):

```toml
name = "deepep_low_latency"
description = "DeepEP MoE Benchmark - Low Latency Mode"
test_template_name = "DeepEP"

[cmd_args]
docker_image_url = "<docker container url here>"
mode = "low_latency"
tokens = 128
num_experts = 256
num_topk = 1
hidden_size = 7168
data_type = "bfloat16"
allow_nvlink_for_low_latency = false
allow_mnnvl = false
```

Test Scenario example:

```toml
name = "deepep-benchmark"

[[Tests]]
id = "Tests.1"
test_name = "deepep_standard"
num_nodes = 2
time_limit = "00:30:00"
```

Test-in-Scenario example:

```toml
name = "deepep-benchmark"

[[Tests]]
id = "Tests.1"
num_nodes = 2
time_limit = "00:30:00"

name = "deepep_standard"
description = "DeepEP MoE Benchmark"
test_template_name = "DeepEP"

  [Tests.cmd_args]
  docker_image_url = "<docker container url here>"
  mode = "standard"
  tokens = 1024
  num_experts = 256
  num_topk = 8
```

## API Documentation

### Command Arguments

### `DeepEPCmdArgs`

Command arguments for the official DeepEP test scripts.

| Field                         | Type                                                                         | Default               | Description |
| ----------------------------- | ---------------------------------------------------------------------------- | --------------------- | ----------- |
| `docker_image_url`            | `str`                                                                        | Required              |             |
| `subtest_name`                | `Literal['test_internode', 'test_intranode', 'test_low_latency', 'test_ep']` | `'test_internode'`    |             |
| `deep_ep_root`                | `str`                                                                        | `'/workspace/DeepEP'` |             |
| `legacy_tests_root`           | `str \| None`                                                                | `None`                |             |
| `elastic_tests_root`          | `str \| None`                                                                | `None`                |             |
| `python_executable`           | `str`                                                                        | `'python'`            |             |
| `num_processes`               | `int`                                                                        | `8`                   |             |
| `num_tokens`                  | `int`                                                                        | `4096`                |             |
| `hidden`                      | `int`                                                                        | `7168`                |             |
| `num_topk`                    | `int`                                                                        | `8`                   |             |
| `num_experts`                 | `int`                                                                        | `256`                 |             |
| `num_topk_groups`             | `int \| None`                                                                | `None`                |             |
| `allow_mnnvl`                 | `bool`                                                                       | `False`               |             |
| `test_ll_compatibility`       | `bool`                                                                       | `False`               |             |
| `pressure_test_mode`          | `int`                                                                        | `0`                   |             |
| `pressure_test`               | `bool`                                                                       | `False`               |             |
| `shrink_test`                 | `bool`                                                                       | `False`               |             |
| `disable_nvlink`              | `bool`                                                                       | `False`               |             |
| `use_logfmt`                  | `bool`                                                                       | `False`               |             |
| `num_sms`                     | `int`                                                                        | `0`                   |             |
| `num_qps`                     | `int`                                                                        | `0`                   |             |
| `num_allocated_qps`           | `int`                                                                        | `0`                   |             |
| `num_gpu_timeout_secs`        | `int`                                                                        | `100`                 |             |
| `num_cpu_timeout_secs`        | `int`                                                                        | `100`                 |             |
| `sl_idx`                      | `int`                                                                        | `0`                   |             |
| `do_cpu_sync`                 | `int`                                                                        | `1`                   |             |
| `allow_hybrid_mode`           | `int`                                                                        | `1`                   |             |
| `allow_multiple_reduction`    | `int`                                                                        | `1`                   |             |
| `prefer_overlap_with_compute` | `int`                                                                        | `0`                   |             |
| `deterministic`               | `bool`                                                                       | `False`               |             |
| `seed`                        | `int`                                                                        | `0`                   |             |
| `skip_check`                  | `bool`                                                                       | `False`               |             |
| `skip_perf_test`              | `bool`                                                                       | `False`               |             |
| `do_pressure_test`            | `bool`                                                                       | `False`               |             |
| `reuse_elastic_buffer`        | `bool`                                                                       | `False`               |             |
| `test_first_only`             | `bool`                                                                       | `False`               |             |
| `unbalanced_ratio`            | `float`                                                                      | `1.0`                 |             |
| `precise_unbalanced_ratio`    | `bool`                                                                       | `False`               |             |
| `masked_ratio`                | `float`                                                                      | `0.0`                 |             |
| `dump_profile_traces`         | `str`                                                                        | `''`                  |             |
| `ignore_local_traffic`        | `bool`                                                                       | `False`               |             |

### Test Definition

### `DeepEPTestDefinition`

Test object for official DeepEP v1/v2 test scripts.

| Field                    | Type                                                      | Default         | Description                                                                                                                                                                                                                   |
| ------------------------ | --------------------------------------------------------- | --------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `name`                   | `str`                                                     | Required        |                                                                                                                                                                                                                               |
| `description`            | `str`                                                     | Required        |                                                                                                                                                                                                                               |
| `test_template_name`     | `str`                                                     | Required        |                                                                                                                                                                                                                               |
| `cmd_args`               | `cloudai.workloads.deepep.deepep.DeepEPCmdArgs`           | Required        |                                                                                                                                                                                                                               |
| `dse_excluded_args`      | `list[str]`                                               | Factory: `list` |                                                                                                                                                                                                                               |
| `extra_env_vars`         | `dict[str, str \| typing.List[str]]`                      | `{}`            |                                                                                                                                                                                                                               |
| `extra_cmd_args`         | `dict[str, str]`                                          | `{}`            |                                                                                                                                                                                                                               |
| `extra_container_mounts` | `list[str]`                                               | `[]`            |                                                                                                                                                                                                                               |
| `git_repos`              | `list[cloudai._core.installables.git_repo.GitRepo]`       | `[]`            |                                                                                                                                                                                                                               |
| `nsys`                   | `cloudai.models.workload.NsysConfiguration \| None`       | `None`          |                                                                                                                                                                                                                               |
| `predictor`              | `cloudai.models.workload.PredictorConfig \| None`         | `None`          |                                                                                                                                                                                                                               |
| `training_report`        | `cloudai.models.workload.TrainingReportConfig \| None`    | `None`          |                                                                                                                                                                                                                               |
| `agent`                  | `str`                                                     | `'grid_search'` |                                                                                                                                                                                                                               |
| `agent_steps`            | `int`                                                     | `1`             |                                                                                                                                                                                                                               |
| `agent_metrics`          | `list[str]`                                               | `['default']`   |                                                                                                                                                                                                                               |
| `agent_reward_function`  | `str`                                                     | `'inverse'`     |                                                                                                                                                                                                                               |
| `agent_config`           | `dict[str, Any] \| None`                                  | `None`          | Agent configuration.                                                                                                                                                                                                          |
| `env_params`             | `dict[str, cloudai.configurator.env_params.EnvParamSpec]` | Factory: `dict` | Environment parameters sampled by the env per trial. Sibling to cmd_args; not part of the agent's action space. CloudAIGymEnv samples, persists them in the trajectory output, and includes them in the trajectory cache key. |