DeepEP Benchmark
DeepEP Benchmark workload (test_template_name is DeepEP) allows users to execute DeepEP (Deep Expert Parallelism) MoE (Mixture of Experts) benchmarks within the CloudAI framework.
Overview
DeepEP is a benchmark for measuring the performance of MoE models with distributed expert parallelism. It supports:
-
Two operation modes: Standard and Low-Latency
-
Multiple data types: bfloat16 and FP8
-
Flexible network configurations: With or without NVLink
-
Configurable model parameters: Experts, tokens, hidden size, top-k
-
Performance profiling: Kineto profiler support
Usage Example
Test TOML example (Standard Mode):
Test TOML example (Low-Latency Mode):
Test Scenario example:
Test-in-Scenario example:
API Documentation
Command Arguments
class cloudai.workloads.deepep.deepep.DeepEPCmdArgs(
*,
docker_image_url: str,
mode: Literal[‘standard’, ‘low_latency’] = ‘standard’,
tokens: int = 1024,
num_experts: int = 256,
num_topk: int = 8,
hidden_size: int = 7168,
data_type: Literal[‘bfloat16’, ‘fp8’] = ‘bfloat16’,
allow_nvlink_for_low_latency: bool = False,
allow_mnnvl: bool = False,
round_scale: bool = False,
use_ue8m0: bool = False,
num_warmups: int = 20,
num_iterations: int = 50,
shuffle_columns: bool = False,
use_kineto_profiler: bool = False,
num_sms: int = 24,
num_qps_per_rank: int = 12,
config_file_path: str = ‘/tmp/config.yaml’,
results_dir: str = ‘/workspace/dp-benchmark/results’,
**extra_data: Any,
)
Bases: CmdArgs
DeepEP benchmark command arguments.
Test Definition
class cloudai.workloads.deepep.deepep.DeepEPTestDefinition(
*,
name: str,
description: str,
test_template_name: str,
cmd_args: DeepEPCmdArgs,
extra_env_vars: dict[str, str | List[str]] = {},
extra_cmd_args: dict[str, str] = {},
extra_container_mounts: list[str] = [],
git_repos: list[GitRepo] = [],
nsys: NsysConfiguration | None = None,
predictor: PredictorConfig | None = None,
agent: str = ‘grid_search’,
agent_steps: int = 1,
agent_metrics: list[str] = [‘default’],
agent_reward_function: str = ‘inverse’,
agent_config: dict[str, Any] | None = None,
)
Bases: TestDefinition
Test object for DeepEP MoE benchmark.
property cmd_args_dict: dict**
Return command arguments as dict, excluding CloudAI-specific fields.