Use the benchmark API

This guide is for automation authors who need to run supported benchmarks from Python, consume typed results, or add a benchmark producer. For day-to-day benchmarking, start with the CLI workflows and interpretation guidance in Run benchmarks. To isolate one asset method, cached property, or sensor update, use Write micro-benchmarks.

Run one workflow

Use a typed request for a supported workflow. It applies the same launch, task, timing, schema, and output behavior as the CLI. The following runtime example measures 1000 steps after 50 warm-up steps. It disables visualizers and requests both stable schema and human-readable summary output:

from pathlib import Path
from isaaclab.benchmark import (
    BenchmarkLauncherConfig,
    BenchmarkOutputConfig,
    BenchmarkRuntimeRequest,
    run_runtime_benchmark,
)
result = run_runtime_benchmark(
    BenchmarkRuntimeRequest(
        task="Isaac-Cartpole-Direct",
        num_envs=4096,
        num_steps=1000,
        warmup_steps=50,
        seed=42,
        presets=("newton_mjwarp",),
        output=BenchmarkOutputConfig(
            path=Path("results/runtime"),
            formatters=("schema", "summary"),
        ),
        launcher=BenchmarkLauncherConfig(visualizers=()),
    )
)
print(f"Total FPS: {result.bundle.runtime.total_fps.mean:,.0f}")
for output_path in result.output_paths:
    print(output_path)

Run the file through the Isaac Lab Python wrapper:

./isaaclab.sh -p runtime_benchmark.py

The command prints the summary report. The paths in result.output_paths identify the schema and summary JSON files that were written. Use these paths in automation instead of reconstructing the timestamped names.

With a non-None task benchmark_mode and ISAACLAB_RENDER_PROFILE=1 or ISAACLAB_PHYSICS_PROFILE=1, the runtime workflow also includes scalar profiling summaries in result.bundle.extra: physics_mean_ms, physics_std_ms, physics_max_ms, physics_calls, and the corresponding render_* keys. These report mean, standard deviation, maximum time per call [ms], and call count over the captured samples. Disabled scopes contribute no keys. The summaries are included in schema and OmniPerf output without changing schema version 1.4. Raw ordered samples remain in <output_path>/profile_timings.json for local analysis.

Choose a request

Choose the request and dedicated runner that match the measurement question. ~isaaclab.benchmark.run_benchmark accepts any request type when generic dispatch is more convenient.

The examples below reuse the runtime workflow’s output and launcher objects.

Startup profiling records wall time and the most expensive functions for each phase:

from pathlib import Path
from isaaclab.benchmark import (
    BenchmarkLauncherConfig,
    BenchmarkOutputConfig,
    BenchmarkStartupRequest,
    run_startup_benchmark,
)
result = run_startup_benchmark(
    BenchmarkStartupRequest(
        task="Isaac-Cartpole-Direct",
        num_envs=4096,
        top_n=20,
        presets=("newton_mjwarp",),
        output=BenchmarkOutputConfig(
            path=Path("results/startup"),
            formatters=("schema", "summary"),
        ),
        launcher=BenchmarkLauncherConfig(visualizers=()),
    )
)
print(f"Environment creation: {result.bundle.phases['env_creation'].total_time_s:.3f} s")

Training returns the checkpoint produced by the selected RL library:

from pathlib import Path
from isaaclab.benchmark import (
    BenchmarkLauncherConfig,
    BenchmarkOutputConfig,
    BenchmarkTrainingRequest,
    run_training_benchmark,
)
result = run_training_benchmark(
    BenchmarkTrainingRequest(
        backend="rsl_rl",
        task="Isaac-Cartpole-Direct",
        num_envs=4096,
        max_iterations=500,
        warmup_steps=50,
        seed=42,
        presets=("newton_mjwarp",),
        output=BenchmarkOutputConfig(
            path=Path("results/training"),
            formatters=("schema", "summary"),
        ),
        launcher=BenchmarkLauncherConfig(visualizers=()),
    )
)
print(f"Collection FPS: {result.bundle.runtime.collection_fps.mean:,.0f}")
print(f"Checkpoint: {result.bundle.checkpoint_path}")

Pass the persisted checkpoint_path to a play request in a new process:

from pathlib import Path
from isaaclab.benchmark import (
    BenchmarkLauncherConfig,
    BenchmarkOutputConfig,
    BenchmarkPlayRequest,
    run_play_benchmark,
)
checkpoint = Path("checkpoint.txt").read_text().strip()
result = run_play_benchmark(
    BenchmarkPlayRequest(
        backend="rsl_rl",
        task="Isaac-Cartpole-Direct",
        checkpoint=checkpoint,
        num_envs=4096,
        num_steps=1000,
        warmup_steps=50,
        seed=42,
        presets=("newton_mjwarp",),
        output=BenchmarkOutputConfig(
            path=Path("results/play"),
            formatters=("schema", "summary"),
        ),
        launcher=BenchmarkLauncherConfig(visualizers=()),
    )
)
print(f"Environment + inference FPS: {result.bundle.runtime.collection_fps.mean:,.0f}")

Configure a request

All requests share the task identifier, optional environment count and seed, task configuration, output configuration, and launcher configuration.

Training exposes additional learning, video, sensor capture, and convergence options. Training and play also expose backend_args for options that truly belong to one RL library. Use typed fields, presets, and hydra_args for supported options. backend_args bypasses the common request contract. See the isaaclab.benchmark API reference for the exhaustive field list.

An empty visualizers tuple explicitly disables every visualizer. None preserves task and environment defaults. Enabling cameras, rendering, livestreaming, deterministic behavior, or animation recording changes the workload and must be kept identical across compared runs.

Choose output formats

Select one or more formatter names in ~isaaclab.benchmark.BenchmarkOutputConfig:

Use ("schema", "summary") for results intended for review or publication. The summary gives immediate feedback. The schema preserves typed data for analysis. With multiple formatters, each output filename includes its formatter name, so JSON outputs cannot overwrite each other.

Read the result

Every workflow returns ~isaaclab.benchmark.BenchmarkResult with two members:

• bundle is a ~isaaclab.benchmark.RuntimeBundle, ~isaaclab.benchmark.StartupBundle, ~isaaclab.benchmark.TrainingBundle, or ~isaaclab.benchmark.PlayBundle.
• output_paths is the exact tuple of pathlib.Path objects written by the selected formatters.

Consume typed fields rather than scraping terminal text or depending on keys in bundle.extra:

# Runtime or play
total_fps = result.bundle.runtime.total_fps.mean
step_fps = result.bundle.runtime.environment_step_timing.environment_step_fps.mean
# Startup
scene_creation_s = result.bundle.phases["env_creation"].total_time_s
# Training
collection_fps = result.bundle.runtime.collection_fps.mean
final_reward = result.bundle.learning.reward.final_raw
checkpoint = result.bundle.checkpoint_path
# Play, when at least one episode completed
mean_return = result.bundle.reward.mean if result.bundle.reward is not None else None

The common run, versions, and hardware fields carry the comparison context. runtime separates collection throughput, total throughput, startup time, and environment-step timing. For measurement definitions, see Run benchmarks. See the public API reference for every schema field.

Handle errors and process lifetime

If a workflow parser rejects a typed request, the API raises ValueError. If a workflow exits without returning a result, the API raises RuntimeError. Invalid output configuration also raises an error. Examples include an empty formatter tuple and an unknown formatter.

Run each workflow in a separate process. This gives each benchmark a clean simulator lifecycle and prevents state from carrying over between workflows. It also matches the CLI execution model. Process startup is reported separately from steady-state throughput. Save result.bundle.checkpoint_path before starting playback.

Add a custom producer

Use ~isaaclab.benchmark.BaseIsaacLabBenchmark when a producer’s lifecycle cannot be represented by a supported workflow. The framework provides phases, typed measurement and metadata records, system recorders, and formatters. You must define the timing boundary.

This executable example measures a CPU workload, records the sample distribution and configuration, updates system recorders, and always finalizes the benchmark:

import statistics
import time
from isaaclab.benchmark import (
    BaseIsaacLabBenchmark,
    IntMetadata,
    StatisticalMeasurement,
)
num_iterations = 100
benchmark = BaseIsaacLabBenchmark(
    benchmark_name="custom_cpu_workload",
    formatter_type=["json", "summary"],
    output_path="results/custom",
    output_prefix="custom_cpu_workload",
    use_recorders=True,
    workflow_metadata={
        "metadata": [{"name": "num_iterations", "data": num_iterations}]
    },
)
output_paths = ()
try:
    for _ in range(10):
        sum(value * value for value in range(10_000))
    samples_ms = []
    for _ in range(num_iterations):
        start_ns = time.perf_counter_ns()
        sum(value * value for value in range(10_000))
        samples_ms.append((time.perf_counter_ns() - start_ns) / 1.0e6)
    benchmark.add_measurement(
        "workload",
        measurement=StatisticalMeasurement(
            name="duration",
            mean=statistics.fmean(samples_ms),
            std=statistics.stdev(samples_ms),
            n=len(samples_ms),
            unit="ms",
        ),
        metadata=IntMetadata(name="warmup_iterations", data=10),
    )
    benchmark.update_manual_recorders()
finally:
    output_paths = benchmark.finalize()
for output_path in output_paths:
    print(output_path)

Choose a meaningful phase name and state units explicitly. Warm up before sampling. Report the sample count with the mean and standard deviation. For GPU work, state whether the metric is submission latency or synchronized completion latency. Place synchronization at deliberate boundaries.

When a custom producer creates one of the stable workflow bundles, call attach_bundle(bundle) before finalize() and select schema. Otherwise, use the lower-level json representation. finalize() stops optional Kit frametime recorders and gathers recorder data. It writes every selected formatter and returns the paths written.

With use_recorders=True (the default), the base class captures CPU, GPU, memory, and version metadata. Call update_manual_recorders() at least once before finalize(), and update during long workloads when current utilization samples are needed.

Set frametime_recorders=True only when a Kit application is running and the workload needs physics, render, application, or GPU frametime data. The framework enables the available Isaac Sim benchmark services and skips unavailable recorders. Frametime instrumentation is part of the workload. Keep its setting fixed across comparisons.

Choose a lower-level runner

~isaaclab.benchmark.MethodBenchmarkRunner repeats one method or property across input modes, warm-up steps, and instance counts. Use ~isaaclab.benchmark.measure_latency to define paired host-submission and device-synchronized timing boundaries. Use ~isaaclab.benchmark.LatencyBenchmarkRunner to report those structured latency samples. Neither isolated result predicts end-to-end environment or training throughput. The command matrices and extension protocol are in Write micro-benchmarks.

Troubleshooting

Run scripts through ./isaaclab.sh -p so the Isaac Lab and simulator Python environment is active. If output is missing, inspect returned output_paths and make sure custom producers call finalize(). Formatter names are lowercase and case-sensitive. Valid names are schema, summary, json, osmo, and omniperf.

Missing GPU metadata usually means nvidia-smi or CUDA device discovery is unavailable. Missing Kit frametime measurements can instead mean the relevant benchmark service is not present. It does not invalidate non-frametime phases.

For a new supported end-to-end workflow, keep typed request construction, dispatch, schema output, summary output, and CLI behavior aligned. For an isolated operation, follow Write micro-benchmarks before adding a new runner or timing convention.