Note
This page is a rendered version of 008_performance_controls.ipynb available on GitHub.
Requesting performance controls#
Performance controls are optional, and usually runtime-set, parameters that steer how an operator does its work.
Many Operators offer such controls that may include, for example, how tiles are scheduled or whether the kernel uses a persistent grid.
So far, we have used RuntimeArguments to express what computation we wish to perform, and get_operators() to find operators that fulfill that. The same RuntimeArguments can also express information about what performance controls we want to exercise.
This example shows how to:
create a set of performance controls,
use them to discover compatible operators with
get_operators,run an operator with the requested controls applied
[1]:
import torch
import cutlass.operators as ops
if not (status := ops.utils.device.device_or_env_supports("100f")):
print(f"This notebook requires sm100.\n{status.error}")
import sys
sys.exit(0)
Setup#
We will run an operator with different performance controls while computing the same fixed reference computation.
[2]:
L, M, N, K = 1, 1024, 1024, 8192
A = torch.randint(-1, 2, (L, M, K), device="cuda", dtype=torch.float16)
B = torch.randint(-1, 2, (L, K, N), device="cuda", dtype=torch.float16)
out = torch.empty((L, M, N), device="cuda", dtype=torch.float16)
reference = (A.cpu() @ B.cpu()).to(out.dtype)
target_sm = ops.utils.device.device_or_env_target_sm()
Creating performance controls#
RuntimeArguments carry both the logical computation we wish an operator to perform, but also any performance controls we want it to exercise. These can be attached to the arguments via args=RuntimeArguments(performance=...) and passed to compatible operators. Just as before, get_operators(args) finds us compatible operators that fulfill all our requested functionality.
The controls are organized by the kernel component they affect. For example, settings that influence the tile scheduler – the part of a GEMM kernel that decides the order in which output tiles are computed and how the K dimension is reduced – are grouped under ops.TileSchedulerControls.
See the docstrings of ops.GemmPerformanceControls for the current list of available performance control options.
Below, we set the performance controls to ask the scheduler to walk tiles along N and to swizzle them in bands of 4 for better L2 cache locality.
[3]:
performance = ops.GemmPerformanceControls(
tile_scheduler_controls=ops.TileSchedulerControls(
raster_order=ops.RasterOrder.ALONG_N,
swizzle_size=4,
)
)
perf_args = ops.GemmArguments(
A=A, B=B, out=out, accumulator_type=torch.float32, performance=performance
)
operators = ops.get_operators(perf_args, target_sm=target_sm)
assert operators, "No operators found for the given arguments"
print(f"{len(operators)} operators honor the raster-order + swizzle request")
2184 operators honor the raster-order + swizzle request
Running the operator#
A discovered operator is compiled and run exactly as in the other notebooks. Run it with the same arguments used to discover it, so the requested controls are the ones applied.
[4]:
operators[0].run(perf_args)
torch.testing.assert_close(out.cpu(), reference)
# clear out the result
_ = out.zero_()
Some performance controls exist as runtime-set options on some operators but compile-time fixed options on others. Querying for such controls through get_operators(args) will return all such compatible kernels that use the performance control.
If a control is baked into the kernel, then it is a property of the design. Therefore, if the operator fixes the raster order and swizzle size at compile time, it must be run with the same arguments that were used to find them through get_operators(args). Asking for a different value picks a different operator.
If a control is not baked into the kernel, we can re-run this kernel with different options without recompiling as they are dynamic controls. To find such a kernel, one can narrow the search with a metadata_filter. In the following example, we query for the universal tile scheduler, which takes two performance controls as launch arguments.
[5]:
def offers_several(domain):
"""Whether a design accepts more than one value, i.e. sets it at launch."""
return isinstance(domain, tuple) and len(domain) > 1
def is_universal_tile_scheduler(metadata):
"""Match designs that take the rasterization settings as launch arguments."""
scheduler = getattr(metadata.design, "tile_scheduler", None)
if scheduler is None:
return False
rasterization = scheduler.rasterization
return (
isinstance(rasterization, ops.metadata.TileSwizzleRasterizationMetadata)
and offers_several(rasterization.raster_order)
and offers_several(rasterization.swizzle_size)
)
universal_operators = ops.get_operators(
perf_args, metadata_filter=is_universal_tile_scheduler, target_sm=target_sm
)
assert universal_operators, "No operator sets the rasterization settings at launch"
print(f"{len(universal_operators)} operators set raster order and swizzle size at launch")
operator = universal_operators[0]
artifact = operator.compile(perf_args)
operator.run(perf_args, compiled_artifact=artifact)
torch.testing.assert_close(out.cpu(), reference)
# clear out the result
_ = out.zero_()
2184 operators set raster order and swizzle size at launch
[6]:
tweaked_performance = ops.GemmPerformanceControls(
tile_scheduler_controls=ops.TileSchedulerControls(
raster_order=ops.RasterOrder.ALONG_M,
swizzle_size=2,
)
)
tweaked_perf_args = ops.GemmArguments(
A=A, B=B, out=out, accumulator_type=torch.float32, performance=tweaked_performance
)
# The artifact compiled above serves this request too: nothing is recompiled here.
operator.run(tweaked_perf_args, compiled_artifact=artifact)
torch.testing.assert_close(out.cpu(), reference)
# clear out the result
_ = out.zero_()
Note: some operators must run with a workspace. To do so, query the size from the operator, allocate and initialize a buffer, then pass it to run.
[7]:
split_k = ops.GemmPerformanceControls(
tile_scheduler_controls=ops.TileSchedulerControls(
reduction=ops.ReductionStrategy.SPLIT_K,
split_count=2,
)
)
split_k_args = ops.GemmArguments(
A=A, B=B, out=out, accumulator_type=torch.float32, performance=split_k
)
split_k_operators = ops.get_operators(split_k_args, target_sm=target_sm)
assert split_k_operators, "No operators found for the split-K request"
print(f"{len(split_k_operators)} operators honor the split-K request")
operator = split_k_operators[0]
requirement = operator.get_workspace_size(split_k_args)
workspace = ops.workspace.Workspace(
data=torch.empty(requirement.size_bytes, device="cuda", dtype=torch.int8)
)
operator.initialize_workspace(split_k_args, workspace)
operator.run(split_k_args, workspace=workspace)
torch.testing.assert_close(out.cpu(), reference)
# clear out the result
_ = out.zero_()
1000 operators honor the split-K request
Wildcards and Exclusions#
Every setting of ops.TileSchedulerControls defaults to ..., the wildcard. That is why the requests above could name raster_order and swizzle_size while saying nothing about other parameters like split-K: the settings we do not mention stay wildcards. The operator applies its own default for that setting at launch.
[8]:
def scheduler_controls(**settings):
"""Wrap tile scheduler setting requests in GemmPerformanceControls."""
return ops.GemmPerformanceControls(
tile_scheduler_controls=ops.TileSchedulerControls(**settings)
)
def candidate_count(performance):
"""How many operators can compute our GEMM with these performance controls."""
args = ops.GemmArguments(
A=A, B=B, out=out, accumulator_type=torch.float32, performance=performance
)
return len(ops.get_operators(args, target_sm=target_sm))
operators = ops.get_operators(ops.GemmArguments(A=A, B=B, out=out, accumulator_type=torch.float32), target_sm=target_sm)
all_operators_satisfying_nonperformance_args = candidate_count(...)
any_scheduler = candidate_count(scheduler_controls())
swizzled = candidate_count(scheduler_controls(swizzle_size=4))
swizzled_along_n = candidate_count(
scheduler_controls(swizzle_size=4, raster_order=ops.RasterOrder.ALONG_N)
)
print(f"performance=... { all_operators_satisfying_nonperformance_args }")
print(f"TileSchedulerControls() {any_scheduler}")
print(f"swizzle_size=4 {swizzled}")
print(f"swizzle_size=4, raster_order=ALONG_N {swizzled_along_n}")
assert len(operators) == all_operators_satisfying_nonperformance_args
assert all_operators_satisfying_nonperformance_args >= any_scheduler >= swizzled >= swizzled_along_n > 0
performance=... 4136
TileSchedulerControls() 3928
swizzle_size=4 2184
swizzle_size=4, raster_order=ALONG_N 2184
To exclude a performance control, set it explicitly to None.
[9]:
declined_reduction = candidate_count(scheduler_controls(reduction=None))
no_reduction = candidate_count(
scheduler_controls(reduction=ops.ReductionStrategy.NO_REDUCTION)
)
assert declined_reduction == no_reduction
print(f"reduction=... {any_scheduler} operators")
print(f"reduction=None {declined_reduction} operators, the NO_REDUCTION ones")
reduction=... 3928 operators
reduction=None 2288 operators, the NO_REDUCTION ones
Incompatible Requests#
Asking for something no operator provides returns an empty list rather than raising an error. Here we ask for a swizzle width that no operator currently offers.
[10]:
unsupported = ops.GemmPerformanceControls(
tile_scheduler_controls=ops.TileSchedulerControls(swizzle_size=257)
)
unsupported_args = ops.GemmArguments(
A=A, B=B, out=out, accumulator_type=torch.float32, performance=unsupported
)
empty_operators = ops.get_operators(unsupported_args, target_sm=target_sm)
print(f"{len(empty_operators)} operators offer a swizzle width of 257")
assert len(empty_operators) == 0
0 operators offer a swizzle width of 257