> This page is for version 1.6.1.
> For other versions, use one of these documentation indexes:
> - 1.8.1 (default): https://docs.nvidia.com/networking/cloudai/1.8.1/llms.txt
> - 1.7.1: https://docs.nvidia.com/networking/cloudai/1.7.1/llms.txt
> - 1.6.1: https://docs.nvidia.com/networking/cloudai/1.6.1/llms.txt
> - dev: https://docs.nvidia.com/networking/cloudai/dev/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/networking/cloudai/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/networking/cloudai/_mcp/server.

DeepEP Benchmark workload (`test_template_name` is `DeepEP`) allows users to execute DeepEP (Deep Expert Parallelism) MoE (Mixture of Experts) benchmarks within the CloudAI framework.

## Overview

DeepEP is a benchmark for measuring the performance of MoE models with distributed expert parallelism. It supports:

* **Two operation modes**: Standard and Low-Latency

* **Multiple data types**: bfloat16 and FP8

* **Flexible network configurations**: With or without NVLink

* **Configurable model parameters**: Experts, tokens, hidden size, top-k

* **Performance profiling**: Kineto profiler support

## Usage Example

Test TOML example (Standard Mode):

```toml
name = "deepep_standard"
description = "DeepEP MoE Benchmark - Standard Mode"
test_template_name = "DeepEP"

[cmd_args]
docker_image_url = "<docker container url here>"
mode = "standard"
tokens = 1024
num_experts = 256
num_topk = 8
hidden_size = 7168
data_type = "bfloat16"
num_warmups = 20
num_iterations = 50
```

Test TOML example (Low-Latency Mode):

```toml
name = "deepep_low_latency"
description = "DeepEP MoE Benchmark - Low Latency Mode"
test_template_name = "DeepEP"

[cmd_args]
docker_image_url = "<docker container url here>"
mode = "low_latency"
tokens = 128
num_experts = 256
num_topk = 1
hidden_size = 7168
data_type = "bfloat16"
allow_nvlink_for_low_latency = false
allow_mnnvl = false
```

Test Scenario example:

```toml
name = "deepep-benchmark"

[[Tests]]
id = "Tests.1"
test_name = "deepep_standard"
num_nodes = 2
time_limit = "00:30:00"
```

Test-in-Scenario example:

```toml
name = "deepep-benchmark"

[[Tests]]
id = "Tests.1"
num_nodes = 2
time_limit = "00:30:00"

name = "deepep_standard"
description = "DeepEP MoE Benchmark"
test_template_name = "DeepEP"

  [Tests.cmd_args]
  docker_image_url = "<docker container url here>"
  mode = "standard"
  tokens = 1024
  num_experts = 256
  num_topk = 8
```

## API Documentation

### Command Arguments

*class* cloudai.workloads.deepep.deepep.DeepEPCmdArgs(

*\**,

*docker_image_url: str*,

*mode: Literal\['standard', 'low_latency'] = 'standard'*,

*tokens: int = 1024*,

*num_experts: int = 256*,

*num_topk: int = 8*,

*hidden_size: int = 7168*,

*data_type: Literal\['bfloat16', 'fp8'] = 'bfloat16'*,

*allow_nvlink_for_low_latency: bool = False*,

*allow_mnnvl: bool = False*,

*round_scale: bool = False*,

*use_ue8m0: bool = False*,

*num_warmups: int = 20*,

*num_iterations: int = 50*,

*shuffle_columns: bool = False*,

*use_kineto_profiler: bool = False*,

*num_sms: int = 24*,

*num_qps_per_rank: int = 12*,

*config_file_path: str = '/tmp/config.yaml'*,

*results_dir: str = '/workspace/dp-benchmark/results'*,

*\*\*extra_data: Any*,

)

Bases: `CmdArgs`

DeepEP benchmark command arguments.

### Test Definition

*class* cloudai.workloads.deepep.deepep.DeepEPTestDefinition(

*\**,

*name: str*,

*description: str*,

*test_template_name: str*,

*cmd_args: [DeepEPCmdArgs](/networking/cloudai/1.6.1/workloads/deepep#cloudai.workloads.deepep.deepep.DeepEPCmdArgs "cloudai.workloads.deepep.deepep.DeepEPCmdArgs")*,

*extra_env_vars: dict\[str, str | List\[str]] = \{}*,

*extra_cmd_args: dict\[str, str] = \{}*,

*extra_container_mounts: list\[str] = \[]*,

*git_repos: list\[GitRepo] = \[]*,

*nsys: NsysConfiguration | None = None*,

*predictor: PredictorConfig | None = None*,

*agent: str = 'grid_search'*,

*agent_steps: int = 1*,

*agent_metrics: list\[str] = \['default']*,

*agent_reward_function: str = 'inverse'*,

*agent_config: dict\[str, Any] | None = None*,

)

Bases: `TestDefinition`

Test object for DeepEP MoE benchmark.

**&#x20;*property* cmd_args_dict*: dict*\*\*

Return command arguments as dict, excluding CloudAI-specific fields.