Custom Reduction
For a conceptual walkthrough, see How custom reductions work (docs, GitHub).
A custom reduction reduces one or more input tensors along the same dimensions. Each input is a stream. A reducer callback that you write sets how two values of a stream combine. The callback can read the values of all streams, so a reduction such as argmax (a value stream and an index stream) is one operation.
The frontend calls the reducer when it builds the operation graph. After an expansion that succeeds, the frontend keeps the child graph and does not call the reducer again. If the reducer throws an exception, the build returns INVALID_VALUE with the exception message (Python: a RuntimeError that contains the type and the message of the Python exception). If the reducer throws an exception or the frontend refuses the child graph, the frontend does not keep the child graph, and the next build calls the reducer again. The reducer adds pointwise operations to a child graph. cuDNN runs that child graph for each combination step.
Requirements
The operation needs cuDNN 9.29.0 or later. The API is always present. With an older cuDNN, building the operation graph returns GRAPH_NOT_SUPPORTED with the message “Custom reduction requires cuDNN v9.29.0” (Python: cudnn.cudnnGraphNotSupportedError). The frontend reads the version of the loaded cuDNN library before it calls the reducer, so with an older library the reducer does not run. This is also true for a program that you compile with cuDNN 9.29 headers and run with an older library.
Semantics
- Stream
ihas the inputX[i]and the outputY[i]. All inputs have the same shape. Y[i]has the shape ofX[i], with extent 1 in each reduced dimension.Y[i]has the data type ofX[i].- The reducer gets the child graph and two lists,
lhsandrhs, with one scalar tensor for each stream. It returns one scalar tensor for each stream. - The reducer body contains only pointwise operations. A build of the body gives
GRAPH_NOT_SUPPORTEDfor any other operation before an operation in the body expands, also if the reducer builds the child graph itself, on any thread. Thus a custom reduction or an SDPAscore_modin the body never runs. A reducer can build a different graph that contains a custom reduction. Do not assign to or copy the child graph object. An operand islhs,rhs, the result of a body operation, or a compile-time constant. - The combination must be associative and commutative.
- The identity of stream
imust be a two-sided identity of the combination. For example, use 0 for a sum and -inf for a maximum. - The identity has exactly the data type of
Y[i]. The frontend does not convert identities. - The compute type of a stream is the data type of the body values of that stream. If you do not set it, the stream computes in its own data type. A compute type must be an integer type for an integer or boolean stream, and a floating-point type for a floating-point stream.
- The semantic key identifies the reducer in the graph key. Change the key when the behavior of the reducer changes.
- The graph JSON shows a custom reduction for inspection only. A graph with a custom reduction cannot be deserialized; deserialize the execution plan instead.
C++ API
The function returns one output tensor for each input, in input order.
Custom_reduction_attributes setters:
set_compute_data_types and set_compute_data_type replace each other. The last call sets the compute types.
Custom_reduction_identity holds one identity value and its exact data type. Make it from a value of the stream type:
cudnn_frontend::graph::reduction_modifiers::sum, max and min take a Custom_reduction_attributes and return it with a single-stream reducer and its semantic key. They do not set an identity or a compute type.
reduction_modifiers also has argmax and argmin. These set the reducer, the semantic key and the identities. They do not set a compute type.
argmaxandargminreduce two streams: stream 0 holds the values and stream 1 holds the index of each element along the reduced dimensions (for example 0 to N−1), in a tensor of the full input shape. Among equal values, the lowest index value wins.value_typeandindex_typeare the data types of the two streams; the identities have exactly these types. A value stream that is notFLOAT,HALF,BFLOAT16orDOUBLE, or an index stream that is notINT32orINT64, givesINVALID_VALUEfromvalidate.- The identities are
(-inf, largest index)for argmax and(+inf, largest index)for argmin, and(NaN, largest index)forIGNORE_NAN, where the largest index is the maximum of the index type. Each is a two-sided identity of its body for any index values smaller than the largest index. - The frontend builds the graph for each stream type; the cuDNN library decides which stream types and body forms it runs, and this can change with the library version. If the library does not support the graph, the build returns its status unchanged (
GRAPH_NOT_SUPPORTEDfor a not-supported status).
The rule sets the result for NaN values. Each rule gives the lowest index among equal values, and -0 equals +0.
The backend contract requires a commutative combination. The XLA body is not commutative for two NaN values, so for those rows its result depends on the reduction order.
Example, an argmax over dimension 1 of BF16 values:
Example, a sum over dimension 1:
Example, a reducer that writes the body:
Python API
pygraph.custom_reduction(inputs, reducer, reducer_semantic_key, reduction_dims, identity_values, *, compute_data_types=None, compute_data_type=None, name="")- inputs: list of input tensors, one for each stream.
- reducer: callable
reducer(graph, lhs, rhs)that returns a list with one tensor for each stream. The reducer can capture parent tensors, use them as operands and return them. It can read captured parent graph state; add reducer operations to the provided child graph. - reducer_semantic_key: string that identifies the reducer in the graph key.
- reduction_dims: list of reduced dimensions, strictly ascending.
- identity_values: list of
cudnn.custom_reduction_identity, one for each stream. - compute_data_types: optional list with one
cudnn.data_typefor each stream. - compute_data_type: optional
cudnn.data_typefor each stream. Givecompute_data_typesorcompute_data_type, not both. - Returns the list of output tensors.
cudnn.custom_reduction_identity(dtype, value)makes an identity from a Python value. Use a real number (numbers.Real, for example afloator a NumPy floating-point scalar) forFLOATandDOUBLE, and an integer (numbers.Integral) for the integer types. Aboolis not a number for these types. The value must have an exact representation indtype. Use aboolforBYTE_BOOLEAN. Usefloat("inf")andfloat("-inf")for infinite identities.cudnn.custom_reduction_identity.from_bits(dtype, bits)makes an identity from its exact storage bits. Use it forHALF,BFLOAT16, FP8 and NaN encodings.cudnn.reduction_modifiers.sum(),max()andmin()return a dictionary with thereducerandreducer_semantic_keyarguments. These are the C++reduction_modifiersreducers and keys.cudnn.reduction_modifiers.argmax(value_type, index_type=cudnn.data_type.INT32, rule=cudnn.reduction_modifiers.ArgNanRule.TORCH)andargmin(...)return a dictionary with thereducer,reducer_semantic_keyandidentity_valuesarguments.ruleis anArgNanRule(TORCH,XLA,IGNORE_NAN) or its value ("torch","xla","ignore_nan"). These are the C++ helpers: the same bodies, keys and identities, and the same rules and limits (see the C++ API).
Example, an argmax over dimension 1. x holds the values and idx holds the index of each element along dimension 1, from 0 to N−1 for a dimension of extent N.
A pointwise operation of the graph can produce an input stream. Examples, with B rows, a graph with the intermediate data type FLOAT, and idx the index tensor:
Example, a sum with a modifier:
XLA HLO examples
samples/cpp/membound/custom_reduction_xla.cpp runs the variadic argmax that XLA emits (value and s32 index, only the index read) at every distinct argmax operand shape of a corpus of XLA HLO dumps of LLM and vision models: 31 shapes. Each shape runs with Arg_nan_rule::XLA (the XLA comparator) and with Arg_nan_rule::TORCH, on rows with ties, one NaN, two NaN, signed zeros and only -inf, against a host reference. The sample also runs the second stage of an XLA split argmax.
Samples
- C++:
samples/cpp/membound/custom_reduction.cpp,custom_reduction_argmax.cpp(the argmax and argmin rules),custom_reduction_sampling.cpp(greedy decoding, speculative-decoding verify, masked argmax, Gumbel-max) andcustom_reduction_xla.cpp(XLA argmax shapes). - Python:
samples/python/71_custom_reduction_argmax.ipynb.