Note
This page is a rendered version of 009_programmatic_dependent_launch.ipynb available on GitHub.
Programmatic Dependent Launch using CUTLASS Operator API#
Programmatic Dependent Launch (PDL) allows a consumer kernel to begin executing before its producer kernel has completed. See Programmatic Dependent Launch in the CUDA Programming Guide for CUDA-level details. This example demonstrates both runtime launch methods exposed by the Operator API, separately and together:
PDL stream serialization on one CUDA stream.
A programmatic event connecting two CUDA streams.
A three-kernel chain combining stream serialization and a programmatic event.
In the Operator API, PDL is controlled—and PDL-capable operators are requested—through ProgrammaticDependentLaunchControl. It can be accessed through the pdl field of GemmPerformanceControls, which is passed as the RuntimeArguments’s performance field. It can enable or disable PDL stream serialization, provide a programmatic event, and declare which operands are independent of the producer and can safely be accessed before it completes.
Set up two chained TGV GEMMs#
We will demonstrate using two sequential GEMMs that PDL allows overlapping between their executions.
We first compute intermediate = activation @ weight1 in the first GEMM (producer), and then compute output = intermediate @ weight2 in the second GEMM (consumer).
The weight operands are column-major, and the activations are row-major, as is common during inference.
[1]:
import torch
from cuda.bindings import runtime as cuda_runtime
import cutlass.operators as ops
if not (status := ops.utils.device.device_or_env_supports("100a")):
print(f"This notebook requires a Blackwell GPU (sm_100a family).\n{status.error}")
import sys
sys.exit(0)
def make_col_major_weight(
batch: int, reduction: int, output: int
) -> torch.Tensor:
return torch.randint(
-1,
2,
(batch, output, reduction),
device="cuda",
dtype=torch.float16,
).transpose(-1, -2)
torch.manual_seed(2026)
L, M, K, H, N = 1, 16, 256, 256, 128
activation = torch.randint(
-1, 2, (L, M, K), device="cuda", dtype=torch.float16
)
weight1 = make_col_major_weight(L, K, H)
weight2 = make_col_major_weight(L, H, N)
intermediate = torch.empty((L, M, H), device="cuda", dtype=torch.float16)
output = torch.empty((L, M, N), device="cuda", dtype=torch.float16)
Case 1: PDL stream serialization#
PDL stream serialization allows a later kernel on the same CUDA stream to begin executing before an earlier kernel has completed. The consumer still waits before accessing operands that depend on the producer, while declared independent operands may be accessed earlier.
ProgrammaticDependentLaunchControl has 2 key parts:
For the first operator (producer), we specify
EarlyLaunchDependentto control if and how it can signal launching its dependent (consumer) operators.For the second operator (consumer),
EarlyStartcontrols if and how a subsequent dependent operator can early start. Here, we specify thatwe use
pdl_stream_serializationto get early-launch signal from the producer kernel in the same stream, andBoperand (weights) can be consumed independent from the producer kernel (independent_operands=("B",)). This allows selecting kernels which can pre-emptively load B during their early-start, while waiting for the dependent operands to be ready. If we remove this, Operator API will conservatively select only those operators which can work correctly without this assumption.
[2]:
producer_args = ops.GemmArguments(
A=activation,
B=weight1,
out=intermediate,
accumulator_type=torch.float32,
performance=ops.arguments.GemmPerformanceControls(
pdl=ops.arguments.ProgrammaticDependentLaunchControl(
early_launch_dependent=ops.arguments.EarlyLaunchDependent(),
),
),
)
consumer_args = ops.GemmArguments(
A=intermediate,
B=weight2,
out=output,
accumulator_type=torch.float32,
performance=ops.arguments.GemmPerformanceControls(
pdl=ops.arguments.ProgrammaticDependentLaunchControl(
# Most conservative: all operands wait for the PDL dependency.
# early_start=ops.arguments.EarlyStart(pdl_stream_serialization=True),
early_start=ops.arguments.EarlyStart(
pdl_stream_serialization=True,
independent_operands=("B",),
),
),
),
)
Select operators from PDL runtime controls#
Through the RuntimeArguments above, we have expressed that we want:
A producer GEMM with
EarlyLaunchDependent, to select a kernel that signals early launch of dependent work (the consumer).A consumer GEMM with
EarlyStart, to select a kernel compiled to start early. It may independently load public GEMM operandB(its constant weight), while operandAremains dependent on theintermediatetensor from the producer. Compatible operators will internally decide where they place their device-side waits.
After this, get_operators works as usual for our RuntimeArguments: it finds all operators compatible with the operation (GEMM), operands, and PDL controls expressed above.
[3]:
producer_operators = ops.get_operators(
producer_args,
target_sm="100a",
)
consumer_operators = ops.get_operators(
consumer_args,
target_sm="100a",
)
assert producer_operators, "No PDL producer operator matched the arguments"
assert consumer_operators, "No PDL consumer operator matched the arguments"
producer = producer_operators[0]
consumer = consumer_operators[0]
print(f"Producer: {producer.metadata.operator_name}")
print(f"Consumer: {consumer.metadata.operator_name}")
Producer: cutedsl.TgvGemmOperator_sm100_tnt_AFloat16_BFloat16_outFloat16_accFloat32_tile64x8x128_pdl
Consumer: cutedsl.TgvGemmOperator_sm100_tnt_AFloat16_BFloat16_outFloat16_accFloat32_tile64x8x128_pdl_B
Compile and launch#
Compile each operator with the arguments it will use. The consumer’s runtime control affects its compiled host launch wrapper. Both kernels are then submitted to the same CUDA stream, which connects their PDL dependency through stream serialization.
[4]:
producer_artifact = producer.compile(producer_args)
consumer_artifact = consumer.compile(consumer_args)
stream = torch.cuda.Stream()
producer.run(producer_args, producer_artifact, stream=stream)
consumer.run(consumer_args, consumer_artifact, stream=stream)
stream.synchronize()
PDL changes when the consumer may begin, not the result of either operation. Verify both the producer’s intermediate tensor and the consumer’s final output.
[5]:
reference_intermediate = activation @ weight1
reference = reference_intermediate @ weight2
torch.testing.assert_close(intermediate, reference_intermediate)
torch.testing.assert_close(output, reference)
print("PDL stream-serialization chained GEMM passed")
PDL stream-serialization chained GEMM passed
Case 2: PDL programmatic event#
A programmatic event allows the producer and consumer to use arbitarary CUDA streams. The following example demonstrate the use case of different CUDA streams. The producer operator is launched on stream 1 with a runtime PDLProgrammaticEvent. With the configuration below, this event is triggered when the producer signals dependent work from its kernel code. The consumer is launched on stream 2 with runtime arguments containing EarlyStart. Stream 2 explicitly waits on the producer’s event
before launching the consumer.
We will first create arguments expressing this sequence, then launch the producer and consumer on their respective streams. Stream 2 waits on the event directly; the event does not need to be passed to the consumer.
[6]:
def check_cuda(error: cuda_runtime.cudaError_t, operation: str) -> None:
if error != cuda_runtime.cudaError_t.cudaSuccess:
raise RuntimeError(f"{operation} failed: {error}")
error, programmatic_event = cuda_runtime.cudaEventCreateWithFlags(
cuda_runtime.cudaEventDisableTiming
)
check_cuda(error, "cudaEventCreateWithFlags")
event_intermediate = torch.empty_like(intermediate)
event_output = torch.empty_like(output)
event_producer_args = ops.GemmArguments(
A=activation,
B=weight1,
out=event_intermediate,
accumulator_type=torch.float32,
performance=ops.arguments.GemmPerformanceControls(
pdl=ops.arguments.ProgrammaticDependentLaunchControl(
early_launch_dependent=ops.arguments.EarlyLaunchDependent(
pdl_programmatic_event=ops.arguments.PDLProgrammaticEvent(
programmatic_event=programmatic_event,
programmatic_event_flags=int(cuda_runtime.cudaEventRecordDefault),
programmatic_event_trigger_at_block_start=False,
),
),
),
),
)
event_consumer_args = ops.GemmArguments(
A=event_intermediate,
B=weight2,
out=event_output,
accumulator_type=torch.float32,
performance=ops.arguments.GemmPerformanceControls(
pdl=ops.arguments.ProgrammaticDependentLaunchControl(
# Most conservative: all operands wait for the PDL dependency.
# early_start=ops.arguments.EarlyStart(),
early_start=ops.arguments.EarlyStart(
independent_operands=("B",),
),
),
),
)
Note that the producer operator is passed the runtime event; the consumer operator is not.
Next, we will compile both operators and define the chain of launches and events:
Launch the producer on stream 1 with the
programmatic_event.Stream 2 asynchronously waits for
programmatic_event.Launch the consumer on stream 2.
[7]:
event_producer_artifact = producer.compile(event_producer_args)
event_consumer_artifact = consumer.compile(event_consumer_args)
event_stream1 = torch.cuda.Stream()
event_stream2 = torch.cuda.Stream()
producer.run(
event_producer_args, event_producer_artifact, stream=event_stream1
)
(error,) = cuda_runtime.cudaStreamWaitEvent(
cuda_runtime.cudaStream_t(event_stream2.cuda_stream),
programmatic_event,
0,
)
check_cuda(error, "cudaStreamWaitEvent")
consumer.run(
event_consumer_args, event_consumer_artifact, stream=event_stream2
)
event_stream1.synchronize()
event_stream2.synchronize()
[8]:
torch.testing.assert_close(event_intermediate, reference_intermediate)
torch.testing.assert_close(event_output, reference)
print("PDL programmatic-event chained GEMM passed")
(error,) = cuda_runtime.cudaEventDestroy(programmatic_event)
check_cuda(error, "cudaEventDestroy")
PDL programmatic-event chained GEMM passed
Note that, in programmatic event use case, user can specify programmatic_event_trigger_at_block_start=True in the producer’s event control to request automatic signaling at producer kernel’s block start. While the default (False) results in the producer kernel signaling at the predefined place in the kernel code.
Case 3: Combine both PDL launch methods across three kernels#
Finally, combine the two PDL methods in a three-kernel dependency chain:
Kernels 1 and 2 launch on stream 1.
Kernel 2 enables stream serialization so it can start early after kernel 1, and it also signals a programmatic event.
Stream 2 waits on that event before launching kernel 3.
Kernel 2 therefore consumes one PDL dependency and produces the next one. Its runtime arguments request an operator that supports both
EarlyStartandEarlyLaunchDependent.
[9]:
P = 64
combined_weight3 = make_col_major_weight(L, N, P)
combined_intermediate1 = torch.empty_like(intermediate)
combined_intermediate2 = torch.empty_like(output)
combined_output = torch.empty((L, M, P), device="cuda", dtype=torch.float16)
error, combined_event = cuda_runtime.cudaEventCreateWithFlags(
cuda_runtime.cudaEventDisableTiming
)
check_cuda(error, "cudaEventCreateWithFlags")
combined_kernel1_args = ops.GemmArguments(
A=activation,
B=weight1,
out=combined_intermediate1,
accumulator_type=torch.float32,
performance=ops.arguments.GemmPerformanceControls(
pdl=ops.arguments.ProgrammaticDependentLaunchControl(
early_launch_dependent=ops.arguments.EarlyLaunchDependent(),
),
),
)
combined_kernel2_args = ops.GemmArguments(
A=combined_intermediate1,
B=weight2,
out=combined_intermediate2,
accumulator_type=torch.float32,
performance=ops.arguments.GemmPerformanceControls(
pdl=ops.arguments.ProgrammaticDependentLaunchControl(
early_launch_dependent=ops.arguments.EarlyLaunchDependent(
pdl_programmatic_event=ops.arguments.PDLProgrammaticEvent(
programmatic_event=combined_event,
programmatic_event_flags=int(cuda_runtime.cudaEventRecordDefault),
programmatic_event_trigger_at_block_start=False,
),
),
early_start=ops.arguments.EarlyStart(
pdl_stream_serialization=True,
independent_operands=("B",),
),
),
),
)
combined_kernel3_args = ops.GemmArguments(
A=combined_intermediate2,
B=combined_weight3,
out=combined_output,
accumulator_type=torch.float32,
performance=ops.arguments.GemmPerformanceControls(
pdl=ops.arguments.ProgrammaticDependentLaunchControl(
early_start=ops.arguments.EarlyStart(
independent_operands=("B",),
),
),
),
)
Thus, for the 2nd operation, we we request an operator that can fulfill both early-launch-dependent and early-start in its arguments.For operation 3, its device code guards the dependent load of activation, and its runtime independent-operand declaration validates that behavior before launch.
[10]:
combined_kernel1_operators = ops.get_operators(
combined_kernel1_args,
target_sm="100a",
)
combined_kernel2_operators = ops.get_operators(
combined_kernel2_args,
target_sm="100a",
)
combined_kernel3_operators = ops.get_operators(
combined_kernel3_args,
target_sm="100a",
)
assert combined_kernel1_operators, "No PDL kernel 1 operator matched the arguments"
assert combined_kernel2_operators, "No PDL kernel 2 operator matched the arguments"
assert combined_kernel3_operators, "No PDL kernel 3 operator matched the arguments"
combined_kernel1 = combined_kernel1_operators[0]
combined_kernel2 = combined_kernel2_operators[0]
combined_kernel3 = combined_kernel3_operators[0]
combined_kernel1_artifact = combined_kernel1.compile(combined_kernel1_args)
combined_kernel2_artifact = combined_kernel2.compile(combined_kernel2_args)
combined_kernel3_artifact = combined_kernel3.compile(combined_kernel3_args)
combined_stream1 = torch.cuda.Stream()
combined_stream2 = torch.cuda.Stream()
Launch the three-kernel chain. The first dependency uses same-stream PDL serialization between kernels 1 and 2. For the second dependency, stream 2 waits on kernel 2’s programmatic event before launching kernel 3.
[11]:
combined_kernel1.run(
combined_kernel1_args,
combined_kernel1_artifact,
stream=combined_stream1,
)
combined_kernel2.run(
combined_kernel2_args,
combined_kernel2_artifact,
stream=combined_stream1,
)
(error,) = cuda_runtime.cudaStreamWaitEvent(
cuda_runtime.cudaStream_t(combined_stream2.cuda_stream),
combined_event,
0,
)
check_cuda(error, "cudaStreamWaitEvent")
combined_kernel3.run(
combined_kernel3_args,
combined_kernel3_artifact,
stream=combined_stream2,
)
combined_stream1.synchronize()
combined_stream2.synchronize()
[12]:
combined_reference1 = activation @ weight1
combined_reference2 = combined_reference1 @ weight2
combined_reference = combined_reference2 @ combined_weight3
torch.testing.assert_close(combined_intermediate1, combined_reference1)
torch.testing.assert_close(combined_intermediate2, combined_reference2)
torch.testing.assert_close(combined_output, combined_reference)
print("Combined three-kernel PDL chain passed")
(error,) = cuda_runtime.cudaEventDestroy(combined_event)
check_cuda(error, "cudaEventDestroy")
Combined three-kernel PDL chain passed