core.extensions.te_cuda_graphs#
Transformer Engine CUDA graph execution attached to Megatron modules.
Module Contents#
Classes#
Skip captured eager weight gradients and replay the TE backward graph. |
|
Coordinate TE graph execution with the current activation offload manager. |
|
Synchronize TE CUDA graph backward with activation reloads. |
|
Run a Megatron module through Transformer Engine CUDA graphs. |
Functions#
Set the current microbatch on all layers that use TE CUDA graph replay. |
API#
- class core.extensions.te_cuda_graphs._TECudaGraphBackwardDWWrapperMixin(layer)#
Skip captured eager weight gradients and replay the TE backward graph.
Initialization
- backward_dw()#
Run eager or graphed backward wgrad callables for the wrapped layer.
- set_graphed_backward_dw_callable(graphed_backward_dw_callable)#
Plug the cuda-graph backward wgrad replay callable.
- class core.extensions.te_cuda_graphs.TECudaGraphOffloadingInterface(manager)#
Coordinate TE graph execution with the current activation offload manager.
Initialization
- _instances#
‘WeakKeyDictionary(…)’
- classmethod get_instance()#
Share graph resources across layers for the lifetime of the offload manager.
- cuda_graph_stream()#
Get the CUDA graph stream.
- cuda_graph_event()#
Get the CUDA graph event.
- forward_record() None#
Record the forward event for cuda graph capture.
- static backward_record(tensor) torch.Tensor#
Insert TE graph synchronization with activation reloads in backward.
- flush_delayed_groups()#
Flush the delayed groups.
- enter_replay()#
Enter CUDA graph replay mode to enable delayed offloading.
- exit_replay()#
Exit CUDA graph replay mode.
- class core.extensions.te_cuda_graphs.TECudaGraphBackwardReloadSyncFunction#
Bases:
torch.autograd.FunctionSynchronize TE CUDA graph backward with activation reloads.
Forward is an identity. Backward records the graph synchronization event and waits for the offload manager’s H2D stream before passing gradients through.
- static forward(ctx, tensor) torch.Tensor#
Forward pass for cuda graph capture.
- static backward(ctx, grad_output)#
Record the backward event and wait for the h2d stream on cuda graph stream.
- class core.extensions.te_cuda_graphs.TECudaGraphMixin(*args, **kwargs)#
Run a Megatron module through Transformer Engine CUDA graphs.
When config.cuda_graph_impl == “transformer_engine”, a training-mode call to the module is redirected: while graphs are being captured it records a graph, and once te_cuda_graphs is populated it replays the graph for the current microbatch. Eval-mode calls, and calls with any other cuda_graph_impl, run forward() normally.
How to use: List this mixin before the module’s base class, so its call runs first:
class MambaLayer(TECudaGraphMixin, GraphableMegatronModule, ...): Every __init__ in the class chain must call super().__init__(...). This mixin's __init__ passes its arguments through unchanged, then sets up the graph state after the base constructor returns.
Default behavior: The whole forward() is one graph with a single tensor input, hidden_states. Modules can supply additional tensor inputs, such as masks or rotary embeddings, through their static-input and replay methods. These inputs pass through the shared methods as **kwargs so this mixin does not need each module’s argument list.
Graphing only part of the module: Subclasses override these methods (see TECudaGraphTransformerLayerMixin):
_te_cuda_graph_capture_forward: runs the selected computation during capture. _te_cuda_graph_prepare_replay: runs any eager code that comes before the graph, then returns two callables: one that replays the graph, and one that takes the graph outputs and finishes the call. _te_cuda_graph_finish_replay: the default for that second callable. Runs any eager code after the graph and returns what forward() would have returned. is_graphable_for_scope / _get_submodules_under_cudagraphs: which cuda_graph_modules scopes this module covers and which submodules are inside the graph.Rules for subclasses: Graph inputs must be tensors or None; drop anything else (e.g. packed_seq_params, inference_context) before replay. Bind state that the after-graph code needs (masks, context, etc.) into the callable returned by _te_cuda_graph_prepare_replay, not onto self, so one call’s state cannot leak into another.
Initialization
- register_te_cuda_graph()#
Initialize TE graph state without changing this module’s parameters.
- __call__(*args, **kwargs)#
Capture or replay the TE graph when this module owns the scope.
- is_graphable_for_scope(modules)#
Whether this module covers one of the requested graph scopes.
- set_te_cuda_graph_backward_dw_wrapper()#
Replace the backward_dw callable with dw cuda graph.
- _te_cuda_graph_index(microbatch_idx)#
Select the replay slot shared by forward and deferred weight gradients.
- _te_cuda_graph_backward_dw_graph(microbatch_idx)#
CUDA Graph backward weight gradient computation for current layer.
- get_layer_static_inputs(seq_length, micro_batch_size)#
Get the static inputs for the layer. We assume that the module has one hidden_states input, whose shape is inferred from the seq_length, micro_batch_size, and parallel config. Override this method if the module has other inputs.
- Returns:
A dictionary containing the static inputs for the layer.
- Return type:
Dict[str, torch.Tensor]
- setup_manual_hooks(make_hook_func)#
Set CUDA Graph manual hooks for the submodules that contain direct parameters and are covered by cudagraphs.
- _get_submodules_under_cudagraphs()#
Get the submodules that are covered by cudagraphs. Return a list that only contains the module itself if the whole layer is covered by cudagraphs.
- _te_cuda_graph_capture(hidden_states: torch.Tensor, *args, **kwargs)#
Capture the selected computation, including its offload dependencies.
- _te_cuda_graph_capture_forward(*args, **kwargs)#
Capture the entire module by forwarding its inputs to forward().
- _te_cuda_graph_replay(hidden_states: torch.Tensor, *args, **kwargs)#
Replay the graph and run its continuation with shared offload ordering.
- _te_cuda_graph_prepare_replay(
- hidden_states: torch.Tensor,
- *args,
- **kwargs,
Bind tensor inputs for graph replay and its eager continuation.
- _te_cuda_graph_finish_replay(outputs)#
Restore the public forward result; whole-module graphs need no adaptation.
- get_offload_interface()#
Return the shared TE graph offload interface when activation offloading is enabled.
- _te_cuda_graph_replay_graph(
- hidden_states: torch.Tensor,
- *args,
- **kwargs,
Prepare tensor inputs and replay this microbatch’s graph.
- _get_te_cuda_graph_replay_args(
- hidden_states: torch.Tensor,
- *args,
- **kwargs,
Validate the module’s tensor inputs and add TE execution metadata.
- _should_call_te_cudagraph()#
Check if we should call the TE cudagraph path.
- core.extensions.te_cuda_graphs.set_te_cuda_graph_current_microbatch(model, microbatch_id)#
Set the current microbatch on all layers that use TE CUDA graph replay.
current_microbatch is read by _te_cuda_graph_replay to select the correct graph index. This helper is called from the pipeline-parallel schedule before each forward step.
Check each layer’s backend: a multimodal model can use different backends for its language decoder and vision encoder.