LayerNorm and RMSNorm in JAX
cudnn.ops.layer_norm_forward_jax and cudnn.ops.rms_norm_forward_jax
return (y,), a one-element tuple containing the normalized JAX array. They
use the existing SM100/SM103 CuTeDSL normalization kernels and support eager calls,
jax.jit, and first-order reverse-mode
differentiation through jax.grad and jax.vjp. PyTorch is not required.
These APIs are experimental.
Signatures and semantics
Both functions normalize over the trailing dimensions specified by
normalized_shape. The default is (x.shape[-1],). Explicit dimensions must
be positive, static, and match the corresponding dimensions of x exactly.
Leading dimensions identify independent normalization rows.
LayerNorm computes
y = (x - mean(x)) / sqrt(mean((x - mean(x))**2) + eps), then applies
y * weight + bias when those parameters are present. RMSNorm computes
y = x / sqrt(mean(x**2) + eps) and optionally multiplies by weight.
The means in these formulas are over the normalized dimensions.
x has dtype bfloat16, float16, or float32. Each supplied affine
parameter must have shape normalized_shape and the same dtype as x.
LayerNorm accepts weight and bias independently; omitted weight means one and
omitted bias means zero. RMSNorm has no bias argument.
The returned array y has the same shape and dtype as x. Statistics and
scratch remain internal; neither function returns or mutates them. The native backward
accumulates parameter gradients in FP32; the VJP casts each gradient to its
primal parameter’s dtype. Input gradients also match the input dtype.
The backward implementation is private and is invoked by JAX’s reverse-mode
transformations; these APIs do not expose a separate backward function.
Weight and bias gradients are not guaranteed to be bitwise repeatable, even with identical inputs and random seeds. Their FP32 atomic additions combine contributions across rows or reduction chunks in an order that can vary between launches.
normalized_shape and eps are static configuration: close over them in a
jitted function, or mark them as static arguments. Arrays remain runtime
inputs. Use a finite positive eps within the FP32 representable range.
Usage
Install the frontend and a CUDA-enabled JAX build (>=0.9.1). The frontend already depends
on CuTeDSL; these kernels require a build at least version 4.7.1 that provides
cutlass.experimental.primitives. No additional frontend dependency is introduced.
Place inputs on one SM100 or SM103 GPU per invocation. Independent local calls are allowed
in multi-process JAX programs; process count and other visible devices do not
restrict eligibility. Compilation targets sm_100a or sm_103a according to the input
device or, during tracing, JAX’s default device; launch tuning uses the same device.
Use jax.default_device to select the intended GPU for tracing and new arrays
when the system default is another architecture. Existing committed arrays retain their placement and must already
be on the intended GPU; the context does not move them. Automatic partitioning
of globally sharded arrays remains outside the qualified scope.
For multidimensional normalization, pass the complete trailing shape:
Support matrix
GroupNorm, InstanceNorm, BatchNorm, fused normalization operations, JVP,
higher derivatives, vmap, and distributed sharding are outside this API’s
qualified scope. The functions do not expose a cuDNN graph-planning interface.
Validation
The focused suite is
test/python/norm/jax/test_norm_jax.py.
It exercises the public entry points against numerical references and checks
the documented input contract. Passing collection or an environment skip does
not establish GPU support; qualify the suite on the target architecture with CUDA-enabled JAX
and the required CuTeDSL runtime.
The earlier single-GPU implementation was qualified on B200 with Python 3.12,
JAX/jaxlib 0.9.1, CUDA 13, and the internal CuTeDSL build
4.9.0a0.dev0+internal.master.20261002010000.3617316. Multiple-GPU execution
requires a separate run of the placement and compilation-reuse tests with at
least two GPUs of the target architecture visible. Mixed-architecture visibility also requires
validation with the installed CuTeDSL JAX runtime; mocked device properties
exercise frontend selection but do not qualify the runtime’s binary loading.
Run the focused suite without the parent Torch
test fixtures: