Skip to navigation

fio

View as Markdown

Fio runs the open-source fio tool from CloudAI. It supports standalone and Slurm systems.

Scenarios

Use conf/common/test_scenario/fio.toml for Slurm clusters. It runs the common fio test in a container and starts 4 tasks total: num_nodes = 2 and num_tasks_per_node = 2.

Use conf/common/test_scenario/fio_local.toml for local/standalone runs. It uses host-installed fio and no container image.

Fio Arguments

Put fio CLI options under cmd_args.args. CloudAI does not declare a fixed fio option list; keys pass through verbatim.

name = "fio"
description = "fio random write smoke test"
test_template_name = "Fio"
[cmd_args]
fio_binary = "fio"
[cmd_args.args]
name = "fio-smoke"
filename = "/tmp/cloudai-fio-test"
rw = "randwrite"
bs = "128k"
size = "80m"
iodepth = 1
numjobs = 1
group_reporting = true
thread = true

This emits options like --rw=randwrite and --group_reporting. Quote TOML keys when option names contain characters such as -:

[cmd_args.args]
"max-jobs" = 4

List values are CloudAI DSE sweeps:

[cmd_args.args]
bs = ["128k", "1m"]
iodepth = [1, 8, 32]

To repeat the same fio option in one run, use a nested table. Nested key names are only stable TOML item names; values become fio values:

[cmd_args.args.a]
"0" = "=foo" # --a==foo
"1" = "bar" # --a=bar
[cmd_args.args."client"]
"0" = "host1" # --client=host1
"1" = "host2" # --client=host2

For existing or complex fio configs, use job_file:

[cmd_args]
fio_binary = "/tmp/fio/fio"
job_file = "/tmp/kv_emulation.fio"

Slurm Tasks

Scenario num_nodes controls node count. cmd_args.num_tasks_per_node controls Slurm tasks per node. Total fio tasks = num_nodes * num_tasks_per_node. CloudAI passes this as --ntasks and --ntasks-per-node.

[[Tests]]
id = "Tests.fio"
test_name = "fio"
num_nodes = 2
[Tests.cmd_args]
docker_image_url = "openeuler/fio:3.42-oe2403sp3"
num_tasks_per_node = 2

Default Metric

For agent_metrics = ["default"], CloudAI aggregates parsed fio summary rows. Defaults report total bandwidth across all operations and tasks. Bandwidth metrics are normalized to MiB/s before aggregation. Latency metrics are normalized to usec before aggregation.

[cmd_args]
metric_operation = "all" # read, write, trim, all, first
metric_name = "bw" # bw, iops, latency
metric_aggregate = "sum" # sum, mean, min, max, first

Raw parsed rows and normalized metric values are written to fio_summary.csv.

API Documentation

Command Arguments

FioCmdArgs

Command line arguments for fio.

FieldTypeDefaultDescription
fio_binarystr'fio'fio executable to run. Use an absolute path for patched/custom fio builds.
job_filestr | NoneNoneOptional fio job/config file. When set, it is appended after CLI options.
argsdict[str, typing.Any]Factory: dictfio CLI options, without leading --. Keys are passed to fio verbatim.
docker_image_urlstr | NoneNoneOptional Docker image to use for Slurm container execution.
num_tasks_per_nodeint | None1Optional Slurm task count per node for multi-node fio runs.
metric_operationstr'all'Operation used for the default metric: read, write, trim, all, or first.
metric_namestr'bw'Metric used for the default metric: bw, iops, or latency.
metric_aggregatestr'sum'Aggregation used for the default metric: sum, mean, min, max, or first.

Test Definition

FioTestDefinition

Test definition for fio.

FieldTypeDefaultDescription
namestrRequired
descriptionstrRequired
test_template_namestrRequired
cmd_argscloudai.workloads.fio.fio.FioCmdArgsRequired
dse_excluded_argslist[str]Factory: list
extra_env_varsdict[str, str | typing.List[str]]{}
extra_cmd_argsdict[str, str]{}
extra_container_mountslist[str][]
git_reposlist[cloudai._core.installables.git_repo.GitRepo][]
nsyscloudai.models.workload.NsysConfiguration | NoneNone
predictorcloudai.models.workload.PredictorConfig | NoneNone
training_reportcloudai.models.workload.TrainingReportConfig | NoneNone
agentstr'grid_search'
agent_stepsint1
agent_metricslist[str]['default']
agent_reward_functionstr'inverse'
agent_configdict[str, Any] | NoneNoneAgent configuration.
env_paramsdict[str, cloudai.configurator.env_params.EnvParamSpec]Factory: dictEnvironment parameters sampled by the env per trial. Sibling to cmd_args; not part of the agent’s action space. CloudAIGymEnv samples, persists them in the trajectory output, and includes them in the trajectory cache key.