Note
This page is a rendered version of 011_data_driven_sync.ipynb available on GitHub.
Data-Driven Synchronization using CUTLASS Operator API#
This notebook shows how to utilize data-driven synchronization to improve E2E latency across back-to-back GEMM Operators on Rubin.
Data-driven synchronization is most effective when paired with Programmatic Dependent Launch or communicating across multiple streams but it is not strictly necessary to understand data-driven synchronization.
How Data-Driven Synchronization Works#
The producer and consumer operators communicate through the values stored within the output tensor of the producer. A sentinel pattern is used to mark a chunk of data as ‘invalid’ and not ready to be consumed. The producer computes its data and stores them into the output, and the consumer polls that tensor and consumes chunks as the sentinel pattern is overwritten with valid data.
The sentinel pattern is defined by the TMA report_valid load as described below:
The encoding of
-0.0in the tensor’s data type is used for all non-scale factor tensors.0xff(NaN) is used for scale factor tensors.
As such, the producer operator must ensure that the chosen sentinel pattern is never valid data. Otherwise, the consumer kernel would hang waiting on the invalid chunk with the sentinel to become ready.
[1]:
import torch
from cuda.bindings import runtime as cuda_runtime
import cutlass.operators as ops
if not (status := ops.utils.device.device_or_env_supports("107a")):
print(f"This notebook requires a Rubin GPU (sm_107a family).\n{status.error}")
import sys
sys.exit(0)
Set up two back-to-back GEMMs#
First we must initialize our tensors for our intended computation.
The producer will compute intermediate = weight1 @ activation.
The consumer will read the intermediate tensor as its B operand and compute output = weight2 @ intermediate. Within the kernel, report_valid will be used on the TMA load to stream-in data from intermediate as the producer computes them.
[2]:
def make_k_major_input(batch: int, reduction: int, cols: int) -> torch.Tensor:
return torch.randint(
-1, 2, (batch, cols, reduction), device="cuda", dtype=torch.float16
).transpose(-1, -2)
def make_k_major_buffer(
batch: int, reduction: int, cols: int, fill: float | None = None
) -> torch.Tensor:
shape = (batch, cols, reduction)
storage = (
torch.empty(shape, device="cuda", dtype=torch.float16)
if fill is None
else torch.full(shape, fill, device="cuda", dtype=torch.float16)
)
return storage.transpose(-1, -2)
torch.manual_seed(2026)
L, M, K, H, N = 1, 16, 256, 256, 128
activation = make_k_major_input(L, K, M)
weight1 = torch.randint(-1, 2, (L, H, K), device="cuda", dtype=torch.float16)
# The producer operator will handle initializing the sentinel pattern for us.
intermediate = make_k_major_buffer(L, H, M)
weight2 = torch.randint(-1, 2, (L, N, H), device="cuda", dtype=torch.float16)
output = torch.empty((L, N, M), device="cuda", dtype=torch.float16)
Case 1: Same-stream Data-Driven Synchronization without PDL#
Now, we must create the GemmArguments for a producer and a consumer operator. Once created, we pass it into get_operators to get the operators that support what we need.
We use the PerformanceControls to describe that we want a producer and a consumer that synchronize through intermediate. The producer writes it with the sentinel protocol and the consumer loads it as B with TMA report_valid.
Note that this example will not result in any kernel overlap and resultant performance improvements but is shown for clarity of Operator Api usage.
[ ]:
# Populate GemmArguments for get_operators query and tensor launch arguments
producer_args = ops.GemmArguments(
A=weight1,
B=activation,
out=intermediate,
accumulator_type=torch.float32,
performance=ops.GemmPerformanceControls(
# Specify that this operator must be a data-driven sync producer
data_driven_sync=ops.DataDrivenSyncControls(
producer=ops.DataDrivenSyncProducer()
)
),
)
consumer_args = ops.GemmArguments(
A=weight2,
B=intermediate,
out=output,
accumulator_type=torch.float32,
performance=ops.GemmPerformanceControls(
data_driven_sync=ops.DataDrivenSyncControls(
# Specify that this operator needs to be a data-driven sync consumer
# and 'B' is the operand to load with report_valid
consumer=ops.DataDrivenSyncConsumer(consumed_operands=("B",)),
)
),
)
# Query for operators and grab the first match
producer_operators = ops.get_operators(
producer_args, target_sm="107a"
)
consumer_operators = ops.get_operators(
consumer_args, target_sm="107a"
)
assert producer_operators, "No data-driven sync producer matched"
assert consumer_operators, "No data-driven sync consumer for B matched"
producer = producer_operators[0]
consumer = consumer_operators[0]
print(f"Producer: {producer.metadata.operator_name}")
print(f"Consumer: {consumer.metadata.operator_name}")
Producer: cutedsl.Sm107DenseGemmDataDrivenSyncOperator_tnn_AFloat16_BFloat16_outFloat16_accFloat32_1cta_cluster1x1x1_tile128x64x32_tma_store_data_driven_producer
Consumer: cutedsl.Sm107DenseGemmDataDrivenSyncOperator_tnt_AFloat16_BFloat16_outFloat16_accFloat32_1cta_cluster1x1x1_tile128x64x32_tma_store_data_driven_consumer_consumedB
[4]:
# Compile kernels
producer_artifact = producer.compile(producer_args, target_sm="107a")
consumer_artifact = consumer.compile(consumer_args, target_sm="107a")
# Run producer and consumer on the same stream
stream = torch.cuda.Stream()
producer.run(producer_args, producer_artifact, stream=stream)
consumer.run(consumer_args, consumer_artifact, stream=stream)
stream.synchronize()
# Compute reference result to verify against using Pytorch's built-in kernels
reference_intermediate = weight1 @ activation
reference_output = weight2 @ reference_intermediate
torch.testing.assert_close(intermediate, reference_intermediate)
torch.testing.assert_close(output, reference_output)
print("Single-stream data-driven chain passed")
Single-stream data-driven chain passed
Case 2: Single-stream Data-Driven Synchronization with 3 kernels, PDL, and user-managed sentinel initialization#
This example will show how you can combine data-driven synchronization with Programmatic Dependent Launch on a single stream to improve latency and allow the consumer to compute output tiles before the producer is fully complete.
We will query and launch 3 Operators. A producer kernel, a producer+consumer kernel, and a consumer kernel.
The dependency chain is as such:
prefill -> P -> PC -> C
Unlike case 1, this does result in a performance improvement and is more indicative of realistic workloads.
Here we will also set assert_user_will_prefill=True so the selected producer does not initialize its output. This places the responsibility of synchronization of prefill before producer+consumer launch upon us. The caller will need to fill the entire workspace with the -0.0 sentinel instead before the consumer operator launch.
This gives you control over when the output tensor should be filled with the sentinel pattern instead of relying on the Operator to do so. In this example, we will use Pytorch to prefill the sentinels but a custom kernel could be used as well.
[5]:
# User pre-fills with the sentinel pattern required by report_valid to see that each chunk is currently invalid.
# Note how previously we could use torch.empty() which would cause incorrect kernel behavior now.
# For scale factors, we use `NaN` as the sentinel pattern.
prefill_stream = torch.cuda.current_stream()
sentinel_pattern = -0.0
pdl_intermediate = make_k_major_buffer(L, H, M, fill=sentinel_pattern)
assert torch.signbit(pdl_intermediate).all() and (pdl_intermediate == 0).all()
# pdl_intermediate_2 is PC's output (weight2 @ pdl_intermediate), shape (L, N, M).
pdl_intermediate_2 = make_k_major_buffer(L, N, M, fill=sentinel_pattern)
assert torch.signbit(pdl_intermediate_2).all() and (pdl_intermediate_2 == 0).all()
weight3 = torch.randint(-1, 2, (L, N, N), device="cuda", dtype=torch.float16)
pdl_output = torch.empty_like(output)
Now, we create the GemmArguments for each of the 3 Operators. The new portion here is pdl_producer_consumer_args which queries for an Operator that can serve both the producer and consumer roles for data-driven synchronization.
[6]:
pdl_producer_args = ops.GemmArguments(
A=weight1,
B=activation,
out=pdl_intermediate,
accumulator_type=torch.float32,
performance=ops.GemmPerformanceControls(
data_driven_sync=ops.DataDrivenSyncControls(
producer=ops.DataDrivenSyncProducer(
# Because we filled the intermediate with the sentinel pattern, we can query for an Operator that skips the fill.
assert_user_will_prefill=True
),
),
# Enable PDL early launch so producer launches consumer early.
pdl=ops.ProgrammaticDependentLaunchControl(
early_launch_dependent=ops.EarlyLaunchDependent(),
),
),
)
pdl_producer_consumer_args = ops.GemmArguments(
A=weight2,
B=pdl_intermediate,
out=pdl_intermediate_2,
accumulator_type=torch.float32,
performance=ops.GemmPerformanceControls(
data_driven_sync=ops.DataDrivenSyncControls(
# Enables both producer and consumer data-driven sync roles.
producer=ops.DataDrivenSyncProducer(
assert_user_will_prefill=True
),
consumer=ops.DataDrivenSyncConsumer(consumed_operands=("B",)),
),
# Enable both early launch dependents for next operator and early start to overlap with previous.
pdl=ops.ProgrammaticDependentLaunchControl(
early_launch_dependent=ops.EarlyLaunchDependent(),
early_start=ops.EarlyStart(
pdl_stream_serialization=True,
independent_operands=("A",)
),
),
),
)
pdl_consumer_args = ops.GemmArguments(
A=weight3,
B=pdl_intermediate_2,
out=pdl_output,
accumulator_type=torch.float32,
performance=ops.GemmPerformanceControls(
data_driven_sync=ops.DataDrivenSyncControls(
consumer=ops.DataDrivenSyncConsumer(consumed_operands=("B",)),
),
pdl=ops.ProgrammaticDependentLaunchControl(
# Consumer will start early on the same stream so we enable it and set A as independent.
# See PDL notebook for more details.
early_start=ops.EarlyStart(
pdl_stream_serialization=True,
independent_operands=("A",)
),
)
),
)
The rest of the code follows case 1 with one notable exception. Because we have chosen to prefill the output tensors we need to block on the prefill stream before the producer-consumer chain can launch.
[7]:
pdl_producer_operators = ops.get_operators(pdl_producer_args, target_sm="107a")
pdl_producer_consumer_operators = ops.get_operators(pdl_producer_consumer_args, target_sm="107a")
pdl_consumer_operators = ops.get_operators(pdl_consumer_args, target_sm="107a")
assert pdl_producer_operators, "No data-driven sync producer matched"
assert pdl_producer_consumer_operators, "No data-driven sync producer+consumer matched"
assert pdl_consumer_operators, "No data-driven sync consumer for B matched"
pdl_producer = pdl_producer_operators[0]
pdl_producer_consumer = pdl_producer_consumer_operators[0]
pdl_consumer = pdl_consumer_operators[0]
pdl_producer_artifact = pdl_producer.compile(pdl_producer_args, target_sm="107a")
pdl_producer_consumer_artifact = pdl_producer_consumer.compile(pdl_producer_consumer_args, target_sm="107a")
pdl_consumer_artifact = pdl_consumer.compile(pdl_consumer_args, target_sm="107a")
pdl_stream = torch.cuda.Stream()
# PDL stream must wait for prefill stream before producer-consumer pipeline can launch.
pdl_stream.wait_stream(prefill_stream)
pdl_producer.run(pdl_producer_args, pdl_producer_artifact, stream=pdl_stream)
pdl_producer_consumer.run(pdl_producer_consumer_args, pdl_producer_consumer_artifact, stream=pdl_stream)
pdl_consumer.run(pdl_consumer_args, pdl_consumer_artifact, stream=pdl_stream)
pdl_stream.synchronize()
# Compute reference result to verify against using Pytorch's built-in kernels
reference_intermediate = weight1 @ activation
reference_intermediate_2 = weight2 @ reference_intermediate
reference_output = weight3 @ reference_intermediate_2
torch.testing.assert_close(pdl_intermediate, reference_intermediate)
torch.testing.assert_close(pdl_intermediate_2, reference_intermediate_2)
torch.testing.assert_close(pdl_output, reference_output)
print("Single-stream data-driven chain with Programmatic Dependent Launch passed")
Single-stream data-driven chain with Programmatic Dependent Launch passed
Case 3: Multi-stream Data-Driven Synchronization#
Data-driven synchronization can also be used in cases where the producer and consumer operators execute on seperate streams.
Because multiple streams have no inherent synchronization like a single-stream, you are always responsible for ensuring the consumer never sees the output tensor before it is pre-filled with sentinel patterns. get_operators cannot guarantee correct usage in this case.
The table below shows what is valid and invalid when using multi-stream data-driven synchronization. Two common ways to synchronize the producer and consumer are to use PDL or to use a CUDA event based on when the producer finishes.
In each case, you are guarding that the consumer launches after the output tensor is pre-filled with the sentinel pattern and that the producer’s computed data is never overwritten by a sentinel pattern. The former prevents incorrect computation and the latter prevents a kernel hang.
Synchronize Producer Prefill before Consumer Launch |
assert_user_will_prefill |
Result |
|---|---|---|
Yes |
True |
Valid, but unnecessary |
No |
True |
Valid |
Yes |
False |
Valid |
No |
False |
Invalid |
The below example will show the flow with assert_user_will_prefill = False and will use PDL CUDA events to synchronize the producer kernel’s sentinel prefill before the consumer kernel launch.
[8]:
multistream_intermediate = make_k_major_buffer(L, H, M)
multistream_output = torch.empty((L, N, M), device="cuda", dtype=torch.float16)
def check_cuda(error: cuda_runtime.cudaError_t, operation: str) -> None:
if error != cuda_runtime.cudaError_t.cudaSuccess:
raise RuntimeError(f"{operation} failed: {error}")
# Create the programmatic event and disable timing for better performance
error, multistream_event = cuda_runtime.cudaEventCreateWithFlags(
cuda_runtime.cudaEventDisableTiming
)
check_cuda(error, "cudaEventCreateWithFlags")
[9]:
multistream_producer_args = ops.GemmArguments(
A=weight1,
B=activation,
out=multistream_intermediate,
accumulator_type=torch.float32,
performance=ops.GemmPerformanceControls(
data_driven_sync=ops.DataDrivenSyncControls(
producer=ops.DataDrivenSyncProducer(
# Query for Operator that will prefill the sentinel pattern for us.
assert_user_will_prefill=False
),
),
pdl=ops.ProgrammaticDependentLaunchControl(
early_launch_dependent=ops.EarlyLaunchDependent(
# Setup PDL event that will trigger consumer launch at producer's chosen time.
pdl_programmatic_event=ops.PDLProgrammaticEvent(
programmatic_event=multistream_event,
programmatic_event_flags=int(cuda_runtime.cudaEventRecordDefault),
programmatic_event_trigger_at_block_start=False
)
),
),
),
)
multistream_consumer_args = ops.GemmArguments(
A=weight2,
B=multistream_intermediate,
out=multistream_output,
accumulator_type=torch.float32,
performance=ops.GemmPerformanceControls(
data_driven_sync=ops.DataDrivenSyncControls(
consumer=ops.DataDrivenSyncConsumer(consumed_operands=("B",)),
),
pdl=ops.ProgrammaticDependentLaunchControl(
early_start=ops.EarlyStart(
independent_operands=("A",)
),
),
),
)
[10]:
# Code in this block follows previous cases
multistream_producer_operators = ops.get_operators(multistream_producer_args, target_sm="107a")
multistream_consumer_operators = ops.get_operators(multistream_consumer_args, target_sm="107a")
assert multistream_producer_operators, "No data-driven sync producer matched"
assert multistream_consumer_operators, "No data-driven sync consumer for B matched"
multistream_producer = multistream_producer_operators[0]
multistream_consumer = multistream_consumer_operators[0]
multistream_producer_artifact = multistream_producer.compile(multistream_producer_args, target_sm="107a")
multistream_consumer_artifact = multistream_consumer.compile(multistream_consumer_args, target_sm="107a")
[11]:
producer_stream = torch.cuda.Stream()
consumer_stream = torch.cuda.Stream()
# Producer will fill the intermediate with the sentinel pattern
multistream_producer.run(
multistream_producer_args, multistream_producer_artifact, stream=producer_stream
)
# Consumer waits for PDL event on producer stream
(error,) = cuda_runtime.cudaStreamWaitEvent(
cuda_runtime.cudaStream_t(consumer_stream.cuda_stream),
multistream_event,
0,
)
check_cuda(error, "cudaStreamWaitEvent")
multistream_consumer.run(
multistream_consumer_args, multistream_consumer_artifact, stream=consumer_stream
)
producer_stream.synchronize()
consumer_stream.synchronize()
# Compute reference result to verify against using Pytorch's built-in kernels
reference_multistream_intermediate = weight1 @ activation
reference_multistream_output = weight2 @ reference_multistream_intermediate
torch.testing.assert_close(multistream_intermediate, reference_multistream_intermediate)
torch.testing.assert_close(multistream_output, reference_multistream_output)
print("PDL CUDA event data-driven synchronization passed")
(error,) = cuda_runtime.cudaEventDestroy(multistream_event)
check_cuda(error, "cudaEventDestroy")
PDL CUDA event data-driven synchronization passed
Recap#
Use get_operators with producer and consumer controls to find the required data-driven sync roles. Prefer same-stream PDL for the usual chained-launch case, use a programmatic event when the launches belong on different streams, and use the no-PDL form when independent concurrent launching is more useful than PDL scheduling. In every reuse case, either select a producer that initializes the sentinel or restore the -0.0 workspace yourself before launching it.