Skip to navigation

DeepEP Benchmark

View as Markdown

DeepEP Benchmark workload (test_template_name is DeepEP) allows users to execute DeepEP (Deep Expert Parallelism) MoE (Mixture of Experts) benchmarks within the CloudAI framework.

Overview

DeepEP is a benchmark for measuring the performance of MoE models with distributed expert parallelism. It supports:

  • Two operation modes: Standard and Low-Latency
  • Multiple data types: bfloat16 and FP8
  • Flexible network configurations: With or without NVLink
  • Configurable model parameters: Experts, tokens, hidden size, top-k
  • Performance profiling: Kineto profiler support

Usage Example

Test TOML example (Standard Mode):

name = "deepep_standard"
description = "DeepEP MoE Benchmark - Standard Mode"
test_template_name = "DeepEP"
[cmd_args]
docker_image_url = "<docker container url here>"
mode = "standard"
tokens = 1024
num_experts = 256
num_topk = 8
hidden_size = 7168
data_type = "bfloat16"
num_warmups = 20
num_iterations = 50

Test TOML example (Low-Latency Mode):

name = "deepep_low_latency"
description = "DeepEP MoE Benchmark - Low Latency Mode"
test_template_name = "DeepEP"
[cmd_args]
docker_image_url = "<docker container url here>"
mode = "low_latency"
tokens = 128
num_experts = 256
num_topk = 1
hidden_size = 7168
data_type = "bfloat16"
allow_nvlink_for_low_latency = false
allow_mnnvl = false

Test Scenario example:

name = "deepep-benchmark"
[[Tests]]
id = "Tests.1"
test_name = "deepep_standard"
num_nodes = 2
time_limit = "00:30:00"

Test-in-Scenario example:

name = "deepep-benchmark"
[[Tests]]
id = "Tests.1"
num_nodes = 2
time_limit = "00:30:00"
name = "deepep_standard"
description = "DeepEP MoE Benchmark"
test_template_name = "DeepEP"
[Tests.cmd_args]
docker_image_url = "<docker container url here>"
mode = "standard"
tokens = 1024
num_experts = 256
num_topk = 8

API Documentation

Command Arguments

DeepEPCmdArgs

Command arguments for the official DeepEP test scripts.

FieldTypeDefaultDescription
docker_image_urlstrRequired
subtest_nameLiteral['test_internode', 'test_intranode', 'test_low_latency', 'test_ep']'test_internode'
deep_ep_rootstr'/workspace/DeepEP'
legacy_tests_rootstr | NoneNone
elastic_tests_rootstr | NoneNone
python_executablestr'python'
num_processesint8
num_tokensint4096
hiddenint7168
num_topkint8
num_expertsint256
num_topk_groupsint | NoneNone
allow_mnnvlboolFalse
test_ll_compatibilityboolFalse
pressure_test_modeint0
pressure_testboolFalse
shrink_testboolFalse
disable_nvlinkboolFalse
use_logfmtboolFalse
num_smsint0
num_qpsint0
num_allocated_qpsint0
num_gpu_timeout_secsint100
num_cpu_timeout_secsint100
sl_idxint0
do_cpu_syncint1
allow_hybrid_modeint1
allow_multiple_reductionint1
prefer_overlap_with_computeint0
deterministicboolFalse
seedint0
skip_checkboolFalse
skip_perf_testboolFalse
do_pressure_testboolFalse
reuse_elastic_bufferboolFalse
test_first_onlyboolFalse
unbalanced_ratiofloat1.0
precise_unbalanced_ratioboolFalse
masked_ratiofloat0.0
dump_profile_tracesstr''
ignore_local_trafficboolFalse

Test Definition

DeepEPTestDefinition

Test object for official DeepEP v1/v2 test scripts.

FieldTypeDefaultDescription
namestrRequired
descriptionstrRequired
test_template_namestrRequired
cmd_argscloudai.workloads.deepep.deepep.DeepEPCmdArgsRequired
dse_excluded_argslist[str]Factory: list
extra_env_varsdict[str, str | typing.List[str]]{}
extra_cmd_argsdict[str, str]{}
extra_container_mountslist[str][]
git_reposlist[cloudai._core.installables.git_repo.GitRepo][]
nsyscloudai.models.workload.NsysConfiguration | NoneNone
predictorcloudai.models.workload.PredictorConfig | NoneNone
training_reportcloudai.models.workload.TrainingReportConfig | NoneNone
agentstr'grid_search'
agent_stepsint1
agent_metricslist[str]['default']
agent_reward_functionstr'inverse'
agent_configdict[str, Any] | NoneNoneAgent configuration.
env_paramsdict[str, cloudai.configurator.env_params.EnvParamSpec]Factory: dictEnvironment parameters sampled by the env per trial. Sibling to cmd_args; not part of the agent’s action space. CloudAIGymEnv samples, persists them in the trajectory output, and includes them in the trajectory cache key.