Ahead-of-time Inspect Guide
The inspect function is a powerful tool for analyzing PyTorch models and pipelines. It helps you understand model structure, identify tuneable modules, and gather execution statistics. It can also be a first step to pick modules before ahead-of-time tuning.
Overview
Inspection provides:
- Module Discovery: Automatically finds all PyTorch modules in your model or pipeline
- Execution Tracking: Identifies which modules are executed during inference
- Performance Profiling: Measures execution time for each module
- Input and output data types: records model input and its results
Basic Usage
Inspection Parameters
Complete Signature
Parameter Details
obj (Required)
The object to inspect. Can be:
torch.nn.Module: Any PyTorch module- Callable: Any callable containing PyTorch modules (e.g., HuggingFace pipelines)
dataset (Required)
This is the source of data for your model. It can be:
torch.Tensor- sequence of strings, tensors, or dictionaries. The collate function is used to stack samples into batches.
torch.utils.data.Dataset
For customization you can use ait.DataLoaderFactory class.
inference_function (Optional)
Custom function for running inference. Useful for complex execution logic:
If the custom function uses scalar arguments that should also be tuned, use the same function later in
ait.tune() after wrapping the inspected modules. See
Inspect and tune with the same workload wrapper.
number_of_iterations (Default: 10)
Number of iterations for profiling execution time:
warmup_iterations (Default: 5)
Warmup iterations before profiling to stabilize measurements:
min_depth (Default: 0)
Minimum depth level for module discovery. Increase if root-level modules don’t work:
max_depth (Default: 5)
If a nested (child) module has a larger depth than max_depth it will be skipped from inspection.
Both min_depth and max_depth narrow the inspection search to a reasonable range.
Working with InspectedModulesInfo
The inspect function returns an InspectedModulesInfo object with several useful methods:
describe()
Display comprehensive information about discovered modules:
Output example for stable-diffusion-3-medium-diffusers:
get_modules()
This function allows getting found modules. Its basic usage returns all of them:
You can place additional criteria:
- min_execution_ratio - minimum ratio of total execution time e.g. .9
- limit - maximum number of modules to return, e.g., 5
If those criteria are not sufficient, you can manually filter modules:
Call tree
get_modules() sums the calls of each module, so it does not say where a module was called from or which other
calls ran inside it. With call_tree=True, inspect also records a call tree: every recorded module forward and
model-defined method, keyed by the chain of calls that led to it, with its calls and time.
A node’s path is its chain of callers, outermost first, joined by >. Each part is an object path from the
inspected object plus a method name: forward > dit_forward is model.dit_forward called from model.forward. The
same module called from two places gives two nodes, for example an LLM used for prefill and for generation.
tree.nodes_of(model.dit) returns the nodes of a module’s forward, one per place it was called from, and
tree.nodes_of(model, "dit_forward") those of another method.
What is recorded:
forwardof every module the object walk reaches, and the model’s own methods (dit_forward,_forward_flow), on modules and on plain pipeline objects outside any module. Methods of library classes (torch,transformers,diffusers, …) are not recorded, but methods of your own subclasses of them are;- calls of
.to(),.cuda()and similar that reach a module from code outside it, with the code that made them; - every torch op that reads a module’s parameter or buffer, or looks up its device, with the recorded call it ran in;
- times from CUDA events without synchronizing after every call, or from
time.perf_counterwithout CUDA; - tensor bytes in and out (
bytes_in,bytes_out), summed over a node’s calls: the tensors in each call’s args/kwargs and in its result, nested containers and ModelOutputs included, each tensor counted once per call. Objects thattorch.utils._pytreedoes not flatten, such as caches or your own classes, are not looked into.
call_tree=True runs one more pass of number_of_iterations, so inspection takes longer. Parts compiled with
torch.compile run compiled in the per-module pass, so calls inside them are not measured there; in the call-tree
pass they run eagerly, so their calls and weight reads are recorded too, and the call tree’s
times for them are eager times, not compiled ones. Call-tree times also include the cost of recording weight accesses
and counting tensor bytes, which run in Python. Where the GPU waits for the CPU, this inflates times, most of all for
calls with many recorded calls inside, so use them for a rough comparison, not as deployed latency. get_modules() and
describe() are unchanged: they still come from the synchronized pass. The object walk has to reach the linear
layers, so deep models need a larger max_depth (8 for BAGEL-sized pipelines).
Limits: exact nn.Sequential, nn.ModuleList and
nn.ModuleDict containers are skipped and their children attached to the caller; a call that takes a callable as an
argument (a checkpoint helper) is skipped the same way; free functions are not recorded. CUDA times come from
events on the device that was current when a call started, so work that a call runs on other GPUs is not fully
counted.
Export the call tree
tree.to_dict() returns a JSON-serializable dict. To write it to a file for tools outside the inspecting process
(no torch, no aitune), use Python’s json module:
The exported data has this structure:
iterations is the number of recorded iterations and all node values (calls, time_ms, bytes_in,
bytes_out) are totals over them — divide by iterations for per-iteration values, by calls for per-call.
nodes lists every recorded call node in the order the nodes were first called. call_path is the chain of calls
that led to the node and identifies it; the rest follows from it:
The object path is the attribute path from the inspected object to the object whose method was called, the same as
ModuleInfo.object_path: when you inspect model, model.visual is model.model.visual, language_model.layers.0
is model.language_model.layers[0], and "" is model itself (also for its own methods such as dit_forward).
Split at the last dot: method names never contain one, object paths can. The keys of modules are object paths, so
modules[object_path] gives a module node’s class name and parameter bytes. weights lists the recorded accesses
(reads, device lookups, moves) per weight. In distributed runs write the file on one rank.
Later versions may add fields without changing schema_version, so readers should ignore keys they don’t know.
schema_version changes when a field is removed, renamed or changes its meaning.
Match wrapped modules to call-tree nodes
ait.wrap names each wrapped module by its object path — model.blocks[0].attn is wrapped as blocks.0.attn —
so its name is the object_path of its call-tree nodes:
Every node below it is covered by that wrapped module. The same name is the module’s module_name in the tuning
report and is stored in its checkpoint, so the match also works after ait.save and ait.load. The root module keeps
its class name and covers the whole tree. A module reachable under two paths appears under the one the inspector
found first.
You can run inspect(..., call_tree=True) again after ait.tune or ait.load. A wrapped module’s node keeps its
call_path, so the two saved trees can be joined by call_path to compare eager and tuned time per node inside the
pipeline. Nodes inside a module tuned with a compiling backend do not appear, because the compiled module runs as
one call.
Shared weights and wrap
After tuning, AITune offloads a module’s original weights (device_after_tuning, "meta" by default). That breaks a
model when something outside the tuned module still needs one of those weights:
- a head tied to an embedding that another module uses;
- an embedding looked up outside the module, for example by the pipeline before it calls the module;
- a pipeline that reads the module’s device from its parameters, as diffusers’
_get_execution_devicedoes withnext(module.parameters()).device; - library code that moves the whole model with
.to()on every call, asSentenceTransformer.encodedoes.
The call tree finds these. shared_weights_of takes the call tree and a module, a node path or a list of them, and
returns one entry per weight — a parameter or a buffer — whose accesses list the outside uses, sorted by name:
reads ("read"), device lookups ("device") and moves ("move"). bytes is numel * element_size per weight; views of
one storage are each counted in full. Reads count per op, so a direct read such as F.linear(x, model.lm_head.weight)
in a caller is found. A module outside the unit that itself holds one of the unit’s weights counts as reading it
whenever its forward runs, because it may pass the weight to a C++ or CUDA extension torch does not see. A module
holds all its parameters and buffers, because offloading moves all of them; a module the call tree has no forward of
(an exact nn.Sequential, or a forward that takes a callable) stands for the recorded calls inside it:
Wrap modules from an inspection with call_tree=True, and tuning keeps exactly those weights on the device when it
offloads a wrapped module:
Each wrapped module logs one line for the weights it keeps (and a debug line per weight). When code outside the
module moves it with .to(), the moved buffers stay along with its parameters, because that .to() moves them
too.
Move or cast the model before wrap. Kept weights are tracked as tensor objects: a device or dtype change after
wrap, such as model.to("cuda") or model.half(), gives buffers new tensors, which are then no longer kept.
The cost is the memory of the kept weights. For weights that outside code reads, the model needs that memory
anyway. For weights that outside code only moves, it does not: E5’s SentenceTransformer.encode moves the
whole encoder with .to() on every call, so its 1.34 GB stay on the GPU next to the tuned engine. That is the same
as the device_after_tuning="cpu" workaround, whose .to() brings them back, but offloading saves nothing there.
ait.save stores the names of the kept weights with each tuned module, and ait.load keeps them on the device
when it offloads the originals, so a deployment process needs no call tree. Each wrapped module owns the weights it
keeps: wrapping it again replaces its set, and a weight stays kept while any wrapped module keeps it, also across
models in the same process.
Modules from inspect without call_tree=True, or ModuleInfo built by hand, keep nothing, so wrap and tuning
behave as before.
The call tree has these limits:
- It only knows the paths the recording ran. A weight read or a device lookup on another path (another branch, another input size) is not seen.
- It assumes the model runs on one thread, like the rest of AITune: weight reads on other threads are not seen,
and a
.to()on another thread during recording can hide reads. - It keeps following a parameter that gets new storage without
.to(), for example throughparam.data = ...or accelerate’sset_module_tensor_to_device. It loses track of a buffer that is replaced the same way.
Troubleshooting
Issue: No modules found
Models with parts compiled by torch.compile
A part compiled with torch.compile or nn.Module.compile() counts as one call: inspect times the compiled module
but not the modules it calls, so get_modules() cannot return them, and inspect logs a warning for each such part.
To inspect inside compiled parts, run inspect with compilation turned off; compiled parts then run eagerly, and their
times are eager times:
A child compiled with torch.compile(module) is an OptimizedModule that holds the original module as _orig_mod, so
its submodules are one level deeper: .encoder._orig_mod.proj instead of .encoder.proj.
Summary
Ahead-of-time inspection can be a first step in exploring your model’s structure and performance. It can also be used to select modules for tuning. For details on the tuning workflow, head to the AOT Tuning Guide.