core.extensions.te_cuda_graphs#

Transformer Engine CUDA graph execution attached to Megatron modules.

Module Contents#

Classes#

_TECudaGraphBackwardDWWrapperMixin

Skip captured eager weight gradients and replay the TE backward graph.

TECudaGraphOffloadingInterface

Coordinate TE graph execution with the current activation offload manager.

TECudaGraphBackwardReloadSyncFunction

Synchronize TE CUDA graph backward with activation reloads.

TECudaGraphMixin

Run a Megatron module through Transformer Engine CUDA graphs.

Functions#

set_te_cuda_graph_current_microbatch

Set the current microbatch on all layers that use TE CUDA graph replay.

API#

class core.extensions.te_cuda_graphs._TECudaGraphBackwardDWWrapperMixin(layer)#

Skip captured eager weight gradients and replay the TE backward graph.

Initialization

backward_dw()#

Run eager or graphed backward wgrad callables for the wrapped layer.

set_graphed_backward_dw_callable(graphed_backward_dw_callable)#

Plug the cuda-graph backward wgrad replay callable.

class core.extensions.te_cuda_graphs.TECudaGraphOffloadingInterface(manager)#

Coordinate TE graph execution with the current activation offload manager.

Initialization

_instances#

‘WeakKeyDictionary(…)’

classmethod get_instance()#

Share graph resources across layers for the lifetime of the offload manager.

cuda_graph_stream()#

Get the CUDA graph stream.

cuda_graph_event()#

Get the CUDA graph event.

forward_record() → None#

Record the forward event for cuda graph capture.

static backward_record(tensor) → torch.Tensor#

Insert TE graph synchronization with activation reloads in backward.

flush_delayed_groups()#

Flush the delayed groups.

enter_replay()#

Enter CUDA graph replay mode to enable delayed offloading.

exit_replay()#

Exit CUDA graph replay mode.

class core.extensions.te_cuda_graphs.TECudaGraphBackwardReloadSyncFunction#

Bases: torch.autograd.Function

Synchronize TE CUDA graph backward with activation reloads.

Forward is an identity. Backward records the graph synchronization event and waits for the offload manager’s H2D stream before passing gradients through.

static forward(ctx, tensor) → torch.Tensor#

Forward pass for cuda graph capture.

static backward(ctx, grad_output)#

Record the backward event and wait for the h2d stream on cuda graph stream.

class core.extensions.te_cuda_graphs.TECudaGraphMixin(*args, **kwargs)#

Run a Megatron module through Transformer Engine CUDA graphs.

When config.cuda_graph_impl == “transformer_engine”, a training-mode call to the module is redirected: while graphs are being captured it records a graph, and once te_cuda_graphs is populated it replays the graph for the current microbatch. Eval-mode calls, and calls with any other cuda_graph_impl, run forward() normally.

How to use: List this mixin before the module’s base class, so its call runs first:

    class MambaLayer(TECudaGraphMixin, GraphableMegatronModule, ...):

Every __init__ in the class chain must call super().__init__(...). This
mixin's __init__ passes its arguments through unchanged, then sets up the
graph state after the base constructor returns.

Default behavior: The whole forward() is one graph with a single tensor input, hidden_states. Modules can supply additional tensor inputs, such as masks or rotary embeddings, through their static-input and replay methods. These inputs pass through the shared methods as **kwargs so this mixin does not need each module’s argument list.

Graphing only part of the module: Subclasses override these methods (see TECudaGraphTransformerLayerMixin):

_te_cuda_graph_capture_forward: runs the selected computation during capture.
_te_cuda_graph_prepare_replay: runs any eager code that comes before the graph, then
    returns two callables: one that replays the graph, and one that takes
    the graph outputs and finishes the call.
_te_cuda_graph_finish_replay: the default for that second callable. Runs any eager
    code after the graph and returns what forward() would have returned.
is_graphable_for_scope / _get_submodules_under_cudagraphs: which
    cuda_graph_modules scopes this module covers and which submodules
    are inside the graph.

Rules for subclasses: Graph inputs must be tensors or None; drop anything else (e.g. packed_seq_params, inference_context) before replay. Bind state that the after-graph code needs (masks, context, etc.) into the callable returned by _te_cuda_graph_prepare_replay, not onto self, so one call’s state cannot leak into another.

Initialization

register_te_cuda_graph()#

Initialize TE graph state without changing this module’s parameters.

__call__(*args, **kwargs)#

Capture or replay the TE graph when this module owns the scope.

is_graphable_for_scope(modules)#

Whether this module covers one of the requested graph scopes.

set_te_cuda_graph_backward_dw_wrapper()#

Replace the backward_dw callable with dw cuda graph.

_te_cuda_graph_index(microbatch_idx)#

Select the replay slot shared by forward and deferred weight gradients.

_te_cuda_graph_backward_dw_graph(microbatch_idx)#

CUDA Graph backward weight gradient computation for current layer.

get_layer_static_inputs(seq_length, micro_batch_size)#

Get the static inputs for the layer. We assume that the module has one hidden_states input, whose shape is inferred from the seq_length, micro_batch_size, and parallel config. Override this method if the module has other inputs.

Returns:

A dictionary containing the static inputs for the layer.

Return type:

Dict[str, torch.Tensor]

setup_manual_hooks(make_hook_func)#

Set CUDA Graph manual hooks for the submodules that contain direct parameters and are covered by cudagraphs.

_get_submodules_under_cudagraphs()#

Get the submodules that are covered by cudagraphs. Return a list that only contains the module itself if the whole layer is covered by cudagraphs.

_te_cuda_graph_capture(hidden_states: torch.Tensor, *args, **kwargs)#

Capture the selected computation, including its offload dependencies.

_te_cuda_graph_capture_forward(*args, **kwargs)#

Capture the entire module by forwarding its inputs to forward().

_te_cuda_graph_replay(hidden_states: torch.Tensor, *args, **kwargs)#

Replay the graph and run its continuation with shared offload ordering.

_te_cuda_graph_prepare_replay(
hidden_states: torch.Tensor,
*args,
**kwargs,
)#

Bind tensor inputs for graph replay and its eager continuation.

_te_cuda_graph_finish_replay(outputs)#

Restore the public forward result; whole-module graphs need no adaptation.

get_offload_interface()#

Return the shared TE graph offload interface when activation offloading is enabled.

_te_cuda_graph_replay_graph(
hidden_states: torch.Tensor,
*args,
**kwargs,
)#

Prepare tensor inputs and replay this microbatch’s graph.

_get_te_cuda_graph_replay_args(
hidden_states: torch.Tensor,
*args,
**kwargs,
)#

Validate the module’s tensor inputs and add TE execution metadata.

_should_call_te_cudagraph()#

Check if we should call the TE cudagraph path.

core.extensions.te_cuda_graphs.set_te_cuda_graph_current_microbatch(model, microbatch_id)#

Set the current microbatch on all layers that use TE CUDA graph replay.

current_microbatch is read by _te_cuda_graph_replay to select the correct graph index. This helper is called from the pipeline-parallel schedule before each forward step.

Check each layer’s backend: a multimodal model can use different backends for its language decoder and vision encoder.