Torch Inductor AOT Backend Guide
The Torch Inductor AOT backend compiles models Ahead-of-Time using PyTorch’s AOT Inductor
(torch._inductor.aoti_compile_and_package). The result is a self-contained .pt2 artifact
that can be saved, loaded, and executed without Python-interpreter overhead.
Requires PyTorch ≥ 2.6.
Overview
- AOT Compilation: Model compiled once, loaded as a native artifact at inference time
- Portable Artifact:
.pt2file contains everything needed to run inference - Dynamic Shapes: Batch and spatial dimensions automatically detected and marked dynamic
- No Python Overhead: Inference runs through a compiled runner with no Python graph tracing
- Save / Load: Artifact persists across sessions via
ait.save/ait.load
Quick Start
Loading in a later session:
Configuration Options
TorchInductorAotBackendConfig
inductor_configs
Pass any key from torch._inductor.config directly to the compiler:
See all available keys:
Dynamic Shapes
By default, dynamic shapes are inferred automatically from the data samples passed to ait.tune.
- Batch axis: detected when the same tensor dimension varies proportionally with
batch_size - Spatial / sequence axes: detected when a dimension varies independently of batch size
When input samples have different spatial sizes, use batch_sizes=[1] to prevent the
data loader from stacking tensors of mismatched shapes.
The backend uses torch.export.Dim.AUTO for spatial / sequence axes, letting PyTorch infer
valid ranges and divisibility constraints from the model automatically.
When recorded samples do not cover the full production range, provide explicit bounded dimensions on ait.Module.
See User-provided dynamic shapes and the runnable
ResNet dynamic-shapes example.
Save and Load
After ait.tune completes, the original module is offloaded to CPU to free GPU memory.
The compiled .pt2 runner is fully self-contained for inference.
Deployment Artifact
Request a deployment record from a tuned module and copy its package:
The returned DeploymentArtifact contains:
model.format="pt2", the package path, andmodel.metadata["structured_call"], which indicates whether the call uses nested inputs or container outputs.runtime.name="aotinductor"with empty runtime options. Inductor configuration controls compilation and is already reflected in the package.- Bounded input and output tensor specs named
INPUT__0,OUTPUT__0, and so on, in PyTorch’s flattened call order.
The backend creates this record only when artifact() is called. It retains the
tensor ordering captured during build, so the record can also be requested after
deactivation or directly from a backend restored from a checkpoint, without loading
the compiled runner. Calls with non-tensor leaves currently raise an error when
requesting the record; this does not prevent tuning or inference.
Comparison with Torch Inductor JIT Backend
Next Steps
- Compare with Torch Inductor JIT Backend for JIT compilation
- Compare with TensorRT Backend for maximum performance
- Learn about Tune Strategies
- Review Deployment Guide