Skip to navigation

aitune.torch.backend.onnx_runtime_backend

View as MarkdownOpen in Claude

ONNX Runtime backend.

Module Contents

Classes

NameDescription
ONNXExecutionProviderSupported ONNX Runtime execution providers.
ONNXRuntimeBackendBackend that runs Torch or existing ONNX models with ONNX Runtime.
ONNXRuntimeBackendConfigConfiguration for ONNXRuntimeBackend.

Functions

NameDescription
_validate_onnx_interfaceValidate the finalized ONNX interface against the recorded Torch contract.

Data

_ORT_TYPE_TO_DTYPE

logger

API

class aitune.torch.backend.ONNXExecutionProvider

Bases: enum.Enum

Supported ONNX Runtime execution providers.

Only NVIDIA GPU-backed providers are supported:

  • CUDA — CUDAExecutionProvider: standard GPU execution.
  • TENSORRT — TensorrtExecutionProvider with CUDAExecutionProvider as fallback: enables TensorRT engine compilation for maximum GPU throughput.
CUDA
= 'cuda'
TENSORRT
= 'tensorrt'
class aitune.torch.backend.ONNXRuntimeBackend(
)

Bases: Backend[ONNXRuntimeBackendConfig]

Backend that runs Torch or existing ONNX models with ONNX Runtime.

Uses the source path and shapes directly for OnnxModule. Torch modules are exported to ONNX with dynamic dimensions inferred from graph_spec. Both paths use the same ONNX Runtime executor.

Workflow:

backend = ONNXRuntimeBackend()
# build / tune as usual through ait.tune()
ait.save(model, "model.ait")
# later
ait.load(model, "model.ait")
STATE_CONFIG
= 'config'
STATE_DEVICE
= 'device'
STATE_EXTERNAL_DATA_PATHS
= 'external_data_paths'
STATE_GRAPH_SPEC
= 'graph_spec'
STATE_ONNX_DATA_PATH
= 'onnx_data_path'
STATE_ONNX_MODEL_PATH
= 'onnx_model_path'
STATE_OUTPUT_OBJECT
= 'output_object'
STATE_SAMPLES
= 'samples'
STATE_TYPE
= 'type'
_build_mode
= BuildMode.AHEAD_OF_TIME
_devices
list[str] = ['cuda']
_execution_modes
= frozenset({ExecutionMode.SINGLE_GPU})
_external_data_artifacts
list[ArtifactPath] = []
_graph_spec
GraphSpec | None = None
_input_nodes
list[NodeArg] | None = None
_onnx_data_artifact
ArtifactPath | None = None
_onnx_model_artifact
ArtifactPath | None = None
_output_nodes
list[NodeArg] | None = None
_samples
SampleStore | None = None
_session
InferenceSession | None = None
_supported_modules
= frozenset({ModuleFormat.TORCH, ModuleFormat.ONNX})
aitune.torch.backend.ONNXRuntimeBackend._activate()

Load the ONNX Runtime session from disk.

aitune.torch.backend.ONNXRuntimeBackend._artifact_additional_files(
model_path: pathlib.Path
) -> tuple[pathlib.Path, ...]

Return every tracked ONNX data file once, relative to the model.

aitune.torch.backend.ONNXRuntimeBackend._artifact_sample_inputs() -> tuple[aitune.records.TensorSample, ...]

Derive portable values from the checkpointed sample and GraphSpec.

aitune.torch.backend.ONNXRuntimeBackend._build(
module: torch.nn.Module,
cache_dir: pathlib.Path
) -> typing.Self

Use an existing ONNX file or export Torch, then load the session.

aitune.torch.backend.ONNXRuntimeBackend._create_artifact() -> aitune.records.DeploymentArtifact

Create an artifact from the saved interface and recorded shape bounds.

aitune.torch.backend.ONNXRuntimeBackend._deactivate()

Deactivate backend.

aitune.torch.backend.ONNXRuntimeBackend._deploy()

Deploy backend.

aitune.torch.backend.ONNXRuntimeBackend._export_onnx(
module: torch.nn.Module,
cache_dir: pathlib.Path
) -> None

Export a Torch module and preserve its output structure.

aitune.torch.backend.ONNXRuntimeBackend._get_execution_providers() -> list[str | tuple]

Return execution providers based on config (no automatic CPU fallback).

  • CUDA → [("CUDAExecutionProvider", {"device_id": ...})]
  • TENSORRT → ["TensorrtExecutionProvider", ("CUDAExecutionProvider", {"device_id": ...})]
aitune.torch.backend.ONNXRuntimeBackend._get_output_object(
module: torch.nn.Module,
) -> typing.Any

Get the output object from the module and sample.

Note: to avoid case where a module returns a reference to the input argument, we make a deep copy of the output object.

Parameters:

module
nn.Module

PyTorch module

sample
Sample

Sample — Sample input to use for model inference.

Returns: Any

The output object from the module.

aitune.torch.backend.ONNXRuntimeBackend._infer(
args: typing.Any = (),
kwargs: typing.Any = {}
) -> typing.Any

Run inference through the shared ONNX Runtime I/O binding executor.

aitune.torch.backend.ONNXRuntimeBackend._prepare_inputs(
args: tuple,
kwargs: dict
) -> dict[str, typing.Any]

Map args/kwargs to session input names using graph_spec locators.

Tensors are returned as-is (preserving their device); I/O binding happens in the shared run_onnx executor.

aitune.torch.backend.ONNXRuntimeBackend._prepare_outputs(
outputs: dict[str, torch.Tensor]
) -> typing.Any

Reconstruct original output structure from session output tensors.

aitune.torch.backend.ONNXRuntimeBackend._use_existing_onnx(
) -> None

Prepare artifacts and output structure from an existing ONNX model.

aitune.torch.backend.ONNXRuntimeBackend._warmup(
samples: collections.abc.Iterable[aitune.torch.module.sample_store.Sample]
) -> None

Run representative samples to initialize the execution provider.

Some execution providers defer work until inference. In particular, the TensorRT execution provider compiles the ONNX graph when it first runs. Exercising a small number of recorded samples makes those failures part of the backend build instead of deferring them until profiling or application inference.

Parameters:

samples
Iterable[Sample]

Iterable[ Sample ] — Recorded input samples for the backend build.

aitune.torch.backend.ONNXRuntimeBackend.artifact() -> aitune.records.DeploymentArtifact

Create the deployment record on request, including after deactivation.

aitune.torch.backend.ONNXRuntimeBackend.describe() -> str

Returns the description of the backend.

aitune.torch.backend.ONNXRuntimeBackend.from_dict(
module: torch.nn.Module | None,
state_dict: dict
classmethod

Creates a backend from a state_dict.

aitune.torch.backend.ONNXRuntimeBackend.key() -> str

Returns the key of the backend.

aitune.torch.backend.ONNXRuntimeBackend.to_dict() -> dict

Returns the state_dict of the backend.

class aitune.torch.backend.ONNXRuntimeBackendConfig(
use_dynamo: bool = True,
opset_version: int | None = None
)
Dataclass

Bases: BackendConfig

Configuration for ONNXRuntimeBackend.

Parameters:

use_dynamo
boolDefaults to True

If True (default), export via torch.onnx.export(dynamo=True) which calls torch.export.export internally and produces a more accurate graph (no Python-level tracing limitations). If False, use the classic trace-based exporter — faster and broader model coverage.

execution_provider
ONNXExecutionProvider | NoneDefaults to None

ONNXExecutionProvider | None — ONNX Runtime execution provider to use. When None the backend defaults to ONNXExecutionProvider.CUDA. Only ONNXExecutionProvider.CUDA and ONNXExecutionProvider.TENSORRT are supported.

opset_version
int | NoneDefaults to None

ONNX opset version passed to torch.onnx.export. None uses the torch default.

execution_provider
ONNXExecutionProvider | None = None
opset_version
int | None = None
use_dynamo
bool = True
aitune.torch.backend.ONNXRuntimeBackendConfig.__post_init__()

Post init.

aitune.torch.backend.ONNXRuntimeBackendConfig.from_dict(
data: dict
classmethod

Initialise config from a plain dict (e.g. parsed from YAML).

execution_provider may be passed as a string and will be converted to an ONNXExecutionProvider enum value automatically.

aitune.torch.backend.onnx_runtime_backend._validate_onnx_interface(
nodes: list[onnxruntime.NodeArg],
specs,
kind: str
) -> None

Validate the finalized ONNX interface against the recorded Torch contract.

aitune.torch.backend.onnx_runtime_backend._ORT_TYPE_TO_DTYPE = {'tensor(bool)': DType.BOOL, 'tensor(uint8)': DType.UINT8, 'tensor(int8)': DType...
aitune.torch.backend.onnx_runtime_backend.logger = getLogger(__name__)