Run a Published Trace (Remote —input-file)

--input-file accepts a Hugging Face, S3, or HTTPS reference as well as a local path. AIPerf checks access, pins the exact version, downloads the data into a local cache, and then replays it exactly as it would the same file on disk.

# hf://<org>/<repo> is a placeholder; use a real published trace
aiperf profile --model my-model --url localhost:8000 --endpoint-type chat \
    --input-file hf://<org>/<repo>

To try it without publishing anything, serve a small trace over HTTP from your machine and run it by URL. Its records carry timestamps, so it replays on the recorded schedule:

mkdir -p remote-demo
cat > remote-demo/trace.jsonl << 'EOF'
{"timestamp": 0, "input_length": 1200, "output_length": 52, "hash_ids": [0, 1, 2]}
{"timestamp": 105, "input_length": 1800, "output_length": 26, "hash_ids": [0, 3, 4, 5]}
{"timestamp": 274, "input_length": 1300, "output_length": 52, "hash_ids": [1, 4, 6]}
EOF
python3 -m http.server 8099 --bind 127.0.0.1 --directory remote-demo > /dev/null 2>&1 &
trap "kill $!" EXIT
sleep 1
aiperf profile \
    --model Qwen/Qwen3-0.6B \
    --endpoint-type chat \
    --streaming \
    --url localhost:8000 \
    --input-file http://127.0.0.1:8099/trace.jsonl \
    --custom-dataset-type mooncake_trace

Remote references also work on Kubernetes, through aiperf kube or in an AIPerfJob/AIPerfSweep applied with kubectl; see Kubernetes.

To publish a trace that others can run this way, see Publishing traces.

Reference Forms

A reference to a file runs as is. A reference to a repo, folder, or prefix needs an aiperf.json manifest at its root that names the data file and its format; see Publishing traces. A single file can also have an optional <file>.aiperf.json sidecar next to it, which supplies the format and other defaults (HTTPS sidecars are only looked up for URLs without a query string).

Other schemes are rejected when the command line is parsed, and the error lists the supported ones.

# A single file in a Hugging Face repo, pinned to a commit
aiperf profile ... --input-file hf://datasets/<org>/<repo>@<commit>/data/chat.jsonl.gz \
    --custom-dataset-type mooncake_trace
# An S3 prefix that holds an aiperf.json
aiperf profile ... --input-file s3://my-traces/chat/2026-08/
# A presigned URL
aiperf profile ... --input-file "https://my-bucket.s3.amazonaws.com/chat.jsonl.gz?X-Amz-Signature=..." \
    --custom-dataset-type mooncake_trace

--custom-dataset-type is optional when a manifest or sidecar declares the format. If you pass both and they disagree, the run fails before anything is downloaded.

Parquet Datasets

Parquet files and directories of parquet shards work wherever JSONL does, for the formats that read their records through the shared reader: mooncake_trace, bailian_trace, single_turn, multi_turn, the speed_bench_* formats, and baseten_trace (a single file only; a directory of Baseten shards is rejected). Any other format fails before the run with “does not read parquet files”; convert the data to JSONL or choose a format that reads parquet.

• Rows are read as JSON records and null-valued keys are dropped at every depth (including inside nested turns or messages), which restores the record shape of the JSONL file that datasets.push_to_hub started from. Binary, timestamp, decimal, and map columns, and NaN or infinity values, are rejected with the name of the column.
• A directory reads every shard in order. When the shards are named <split>-00000-of-00002.parquet and several splits are present, AIPerf reads train.
• Files beside the shards that are not data are skipped: READMEs, aiperf.json manifests, and names starting with . or _. That covers the _SUCCESS, _committed_*, _metadata, and .crc files that Spark and Hive write, so their output folders work as-is, on S3 or locally.
• pyarrow ships with AIPerf on every platform except Windows on ARM, which has no wheel. There a remote parquet source fails before anything is downloaded.

A Hugging Face repo with no aiperf.json also runs when it uses the layout that datasets.push_to_hub writes, or the one on the refs/convert/parquet branch:

# A config folder (the only folder holding parquet, or one that has a train split)
aiperf profile ... --input-file hf://datasets/<org>/<repo>/<config> \
    --custom-dataset-type mooncake_trace
# The auto-converted parquet branch of any dataset (URL-encode the slashes in the branch)
aiperf profile ... --input-file "hf://datasets/<org>/<repo>@refs%2Fconvert%2Fparquet/<config>/train" \
    --custom-dataset-type mooncake_trace
# A single shard file
aiperf profile ... --input-file hf://datasets/<org>/<repo>/<config>/train-00000-of-00001.parquet \
    --custom-dataset-type mooncake_trace

Without a manifest, AIPerf picks the data as follows:

1. It descends into the only folder that holds parquet, or into train when several folders do.
2. In a folder of shards it reads the train split. When other splits are dropped, the run log says so at INFO level.
3. If a folder holds several datasets and none is named train, the run fails with “has several datasets under …” and the hint Use one of: ... lists a ready-to-use reference for each. If a folder holds shards of several splits and none is named train, the run fails with “holds the splits …” and the hint Use one split's shard, e.g. ... lists a reference to one shard of each split.

A manifest’s path can also name a folder of parquet shards on Hugging Face or S3. The same train split rule applies, and several splits with no train fail at the access check, before anything is downloaded. On Hugging Face only the chosen split’s shards are downloaded. On S3 the whole prefix is downloaded, and the reader then picks the same split from the local copy. For an S3 manifest folder the split error reads “its manifest folder … holds the splits … and none is named ‘train’”, and its hint asks the publisher to keep one split in the folder or to name it train.

A manifest-less remote parquet source replays on its recorded timestamp schedule only when --custom-dataset-type names a trace format. To also declare the format, add aiperf.json; see Publishing traces.

What Happens Before the Run

1. Access check. AIPerf checks that the source exists, that your credentials work, and that the manifest (if any) is valid and compatible with the run: the format must exist in this AIPerf, the requires.aiperf version specifier and requires.endpoint_types must be satisfied. It pins the exact version here (Hugging Face commit SHA, S3 object version or ETag, HTTP ETag or, without one, Last-Modified and length). Problems fail at this step, before anything is downloaded, including a parquet source whose declared format does not read parquet. The one exception is a parquet source that needs the first-record check (a trace format declared by flag or manifest, and neither --fixed-schedule nor --no-fixed-schedule): a single parquet file is downloaded here, and a parquet folder has its first shard downloaded, so that its first row can be read (step 2).
2. First-record check. For trace formats, AIPerf reads the first record. If it has timestamps, the run replays on the recorded schedule (--no-fixed-schedule opts out), just like a local trace. To decide this it reads only the leading bytes of the file, in steps of 64 KiB, 1 MiB, and 8 MiB, and downloads the whole file only when the first record is longer than that. A single columnar file (Parquet, Arrow) is read from the downloaded copy, since its schema sits at the end of the file. A parquet folder is read from its first shard only, fetched into the cache so the full download in step 3 reuses it. A weka_trace needs no check: it replays on its recorded schedule unless you pass --ignore-trace-delays or --use-think-time-only, as for a local Weka trace.
3. Download into the cache, verified against the manifest’s sha256 when present. A cache hit rehashes the cached file whenever the manifest declares a sha256. The first record is then checked against the chosen format, so a file that does not look like its declared format fails with a clear error instead of during the run.

Options that conflict with a fixed schedule (a request rate, a non-sequential --dataset-sampling-strategy, a parameter sweep) behave as they do for a local file: the trace is not promoted, and the run fails with the conflict only if the trace turns out to have timestamps.

A remote dag_jsonl file cannot be counted before it is downloaded, so it needs an explicit stop condition (--conversation-num, --request-count, or --benchmark-duration).

The run log prints the exact version used, for example:

Remote dataset hf://<org>/<repo>: using hf://datasets/<org>/<repo>@3e1f.../data/chat.jsonl.gz (version 3e1f..., 1 file(s), 48211093 bytes, sha256 9c1d4f0a7b2e)

For an hf:// source (pinned to a commit) or an S3 object with a VersionId, the using ... reference names that exact version, so copying it reproduces the run. An HTTPS URL, or an S3 object or prefix pinned by ETag or by listing, has no reference that locks a version: running it again reads whatever is current. The same information (reference, pinned version, downloaded files with their digests, and the manifest digest) is recorded under metadata.dataset in profile_export_aiperf.json, so the version that ran is always on record.

Credentials

Presigned URLs are redacted everywhere AIPerf writes them: logs, error messages, run_config.json, the command line recorded in profile_export_aiperf.json, and the provenance above. A presigned URL that has expired fails with an access-denied error that suggests generating a new one.

Pinning, Caching, and Offline Runs

• Unpinned references check for a newer version on every run and download only when it changed. Pin with @<commit> (Hugging Face) or ?versionId= (S3) for comparable results. An HTTPS URL is versioned by its ETag (or Last-Modified and length); a server that sends neither is downloaded again on every run, with a warning.
• Hugging Face files use the standard Hugging Face cache (HF_HOME). S3 and HTTPS files go to ~/.cache/aiperf/remote (AIPERF_DATASET_REMOTE_CACHE_DIR). Nothing is evicted automatically; delete the directory to reclaim disk.
• If another AIPerf process is downloading the same version, the run waits for it. If no other run is active, delete the lock file named in the error and retry.
• AIPERF_DATASET_REMOTE_OFFLINE=1 (or HF_HUB_OFFLINE=1) makes AIPerf use only what it has already downloaded and never touch the network. An unpinned reference uses the newest cached version, and a warning says so. A reference AIPerf has not downloaded before fails with a not-found error.
• An interrupted download never leaves a partial file behind, so the next run downloads again instead of serving a truncated copy. S3 and HTTPS downloads stop promptly on Ctrl-C. Ctrl-C during an hf:// download waits for the file currently downloading to finish, because the Hugging Face client is not interrupted mid-file.
• Transient failures (timeouts, HTTP 429 and 5xx) are retried automatically, three attempts in total with a growing delay. If the remote changes after AIPerf pinned it (a force-pushed repo, a replaced object), the run fails with a “changed after AIPerf pinned its version” error; run again to use the new version.

Kubernetes

aiperf kube profile, aiperf kube generate and aiperf kube sweep accept the same references, and so do AIPerfJob and AIPerfSweep resources applied with kubectl. Only the dataset-manager container downloads; workers receive the built dataset as before.

Pinning

• aiperf kube runs the access check on the machine you run it from, pins the version, and writes path: {ref: ..., resolved_version: ...} into the resource. Every pod, and every child of an aiperf kube sweep, uses that version.
• If this machine cannot read an hf:// or s3:// source but the pods can (IRSA, EKS Pod Identity, a token only the pods have), the reference is submitted unpinned with a warning, and the dataset-manager pod pins it when it starts. An https:// source that refuses this machine fails instead, because the pod would fetch the same URL.
• If this machine cannot connect to the source at all (a DNS failure, a refused connection, a timeout), for example an object store or HTTP server that only the cluster’s network reaches, the reference is submitted unpinned with the same warning, whatever its scheme. An HTTP 429 or 5xx answer is not a connection failure: the source answered, so aiperf kube retries and then fails.
• A resource that already carries a pinned version keeps it when this machine cannot read the source; the dataset-manager pod downloads that version.
• A resource written by aiperf kube generate --operator can be passed back to aiperf kube profile -f or aiperf kube sweep -f. A presigned path whose credential is <redacted> is not read on this machine at all, since only the pods hold the full URL: its pinned version is kept, or, if it has none, it is submitted unpinned with the same warning.
• aiperf kube sweep (or any run with multi-run trials) that submits a reference unpinned warns that each child pins it separately, so variations can see different versions if the source changes meanwhile. Pin it explicitly (@<commit>, ?versionId=) to avoid that.
• A trace whose fixed-schedule promotion depends on its first record cannot be submitted unpinned. Pass --fixed-schedule or --no-fixed-schedule, or run aiperf kube from a machine that can read the source.
• A resource applied with kubectl is pinned by its dataset-manager pod. An unpinned AIPerfSweep applied with kubectl resolves the reference separately in each child, so children can see different versions. Pin explicitly (@<commit>, ?versionId=) or use aiperf kube sweep.
• When the pod pins, the pinned version and any manifest defaults (format, block_size) exist only in the dataset-manager’s copy of the run, and profile_export_aiperf.json records the reference without its version. Pin on the CLI host, or in the reference, when you need full provenance.

Credentials

A presigned URL is a credential. The resource, the run ConfigMap and every spec the operator stores carry it with its signature replaced by <redacted>. Put the full URL(s) in a Secret as a JSON list and pass it to the pods; AIPerf never creates the Secret:

kubectl create secret generic aiperf-dataset-urls -n <namespace> \
    --from-literal=urls='["https://bucket.s3.amazonaws.com/trace.jsonl?X-Amz-Signature=..."]'
aiperf kube profile ... \
    --input-file 'https://bucket.s3.amazonaws.com/trace.jsonl?X-Amz-Signature=...' \
    --env-from-secrets AIPERF_INJECTED_DATASET_URLS=aiperf-dataset-urls/urls

Only the dataset-manager container, which does the download, receives the Secret; the other containers are rendered without it. The dataset-manager matches each URL to its dataset by the URL without its signing parameters, so list one URL per dataset: when you re-sign a URL, replace the old one. A resource applied with kubectl must not contain the signature: the operator rejects it, because the API server has already stored it, so rotate the URL. Write the path as aiperf kube generate --operator does, with the credential values replaced by <redacted>. aiperf kube validate catches a raw signature before you apply.

Before submission, aiperf kube warns when a presigned URL expires within an hour (queueing, for example in Kueue, can delay the download) and fails when it has already expired, before it tries to read the URL, so the error names the expiry instead of a bare access denial. The operator runs the same checks for resources applied with kubectl; see Preflight.

Storage

• Hugging Face files land in the pod’s Hugging Face home (/aiperf/hf_home, an emptyDir; AIPerf sets HF_HOME and ignores a user value). S3 and HTTPS files land in /aiperf/remote-cache, an emptyDir mounted in the dataset-manager and control-plane containers.
• Both use the node’s ephemeral storage. For multi-GB datasets, or to reuse downloads across jobs, mount a PVC and point HF_HUB_CACHE and AIPERF_DATASET_REMOTE_CACHE_DIR at paths on it; see Kubernetes configuration.

Large datasets and the startup probe

The dataset-manager downloads before its health server starts, so its startup probe fails for as long as the download runs. When the run reads a remote dataset, the dataset-manager container (and only that container) gets a startup probe that allows AIPERF_K8S_HEALTH_DATASET_DOWNLOAD_STARTUP_BUDGET_SECONDS (default 3600, one hour): its failure threshold becomes that budget divided by AIPERF_K8S_HEALTH_STARTUP_PERIOD_SECONDS, rounded up. The dataset-manager also registers with the controller only after the download, so the controller-pod containers wait for registration for that budget on top of AIPERF_SERVICE_REGISTRATION_TIMEOUT. A download that runs longer is killed by the kubelet, possibly in the middle of the transfer and before AIPerf can write its termination message; the job then fails without the [dataset-manager pod] line or the DatasetReady condition described below.

For a dataset that takes longer than the budget to download, raise AIPERF_K8S_HEALTH_DATASET_DOWNLOAD_STARTUP_BUDGET_SECONDS (at most 86400), or pre-populate a PVC-backed cache (see Storage) so the pod finds the files already downloaded. The probe variables are read by the process that renders the JobSet (the operator Deployment, or the CLI in direct mode), so set them there, not in spec.podTemplate.env; see Kubernetes configuration.

When the pod fails

A dataset that fails in the pod stops the dataset-manager before the benchmark starts. The job fails with a status.error that contains [dataset-manager pod] Cannot use <reference>: ..., the DatasetReady condition turns False with reason DatasetUnavailable, and aiperf kube debug --job-id <name> shows that reason as a DatasetReady=False finding. Gated public datasets (--public-dataset) fail the same way.

Troubleshooting

Errors read Cannot use <reference>: <what went wrong>. <how to fix it>. Find your message in the table.

If the manifest declares requires.max_context_tokens, AIPerf only warns when it is larger than --max-context-length. That comparison happens only when --max-context-length is set, and that option applies to formats that implement context filtering, which today is Weka. It never fails the run.