NVIDIA Optimized Frameworks

JAX Release 26.11

The NVIDIA JAX Release 26.11 is made up of two container images available on NGC: JAX and MaxText.

Contents of the JAX container

This container image contains the complete source for the following software:

  • JAX: /opt/jax
  • XLA: /opt/xla
  • Flax:/opt/flax
  • TransformerEngine:/opt/transformer-engine

The MaxText container image is based on the JAX container. Additionally, it includes:

  • MaxText: /opt/maxtext

The JAX runtime package jaxlib is prebuilt and installed in the default Python environment (/usr/local/lib/python3.10/dist-packages/jaxlib) in the container image.

Versions of packages included in both of these containers:

  • CUDA 13.4.1
    • Please refer to the CUDA DL 26.11 release notes section for the list of libraries inherited from the CUDA container.

Driver Requirements

Release 26.11 is based on CUDA 13.5.0. For comprehensive and up-to-date driver compatibility information, please refer to the following documentation:

Key Features and Enhancements

  • Starting in 26.11, the DLFW containers - JAX & MaxText, introduce general availability support for the NVIDIA Rubin GPU architecture.

XLA compiler support for Vera Rubin

  • XLA compute cost model supports Rubin GPU specs, such as its number of compute units, memory size and speed, and cache sizes. XLA can also now target newer GPU instruction versions (PTX 9.4) even if it was built with an older CUDA toolkit; the installed NVIDIA assembler (ptxas) still sets the upper limit. Two related fixes make cuDNN operations compile correctly on CUDA 13.4 for multiple GPU models, including B200.

Horizontal Scaling

  • Faster GPU-to-GPU communication at a larger scale. The kernels can also run inside CUDA graphs, which cuts per-iteration graph update time from about 0.25 ms to about 0.05 ms. This is on by default and can be turned off with --xla_gpu_enable_command_buffer=-collectives_kernel. A separate fix stops the multi-GPU barrier used by ragged-all-to-all from crashing above 32 GPUs; it now supports up to 128.
  • More collective operations and smarter scheduling. XLA adds support for collective-reduce, which combines results from all GPUs onto one chosen GPU. This is motivated by the layerwise FSDP used to optimize the Muon optimizer in Deepseek-v4. It is off by default; turn it on with --xla_gpu_emit_collective_reduce. JAX's broadcast can now choose its source GPU at run time. A scheduler bug is also fixed. Collective calls assigned to a stream were counted twice against a resource limit, which could make compilation fail when overlap limits were tight.
  • Faster transfers between CPU and GPU memory. Copies between CPU memory and GPU memory now run on two dedicated streams instead of sharing streams with compute work. This lets transfers, such as host-memory offloading, overlap with computation and communication. The setting is on by default and affects every GPU program. The changes report no speed measurements.
  • More reliable runs with many GPUs in one process. Several fixes make it safer to drive many GPUs from a single process. A timing problem in loop unrolling could make compilation inconsistent and hang multi-GPU runs; the decision is now made once, up front. Execution timing data was shared by all GPUs, which caused a data race; each GPU now keeps its own record. Each cuDNN graph is now built once per GPU, which prevents CUDNN_STATUS_EXECUTION_FAILED errors, and a new per-GPU state container supports this. Operations that cannot be captured in a CUDA graph now keep their order relative to captured ones, which prevents races. TransformerEngine also sets up its softmax custom calls before first use to avoid certain deadlock scenarios.

XLA Compiler

  • Experimental new GPU code generator (TensorIR). XLA adds an experimental code generator, called TensorIR, that writes GPU kernels using CUDA Tile IR. After a follow-up change it handles more kinds of fused operations, including FP8 and operations with several outputs. It is off by default; turn it on with --xla_gpu_experimental_enable_tensor_ir. It needs an sm_90 or newer GPU and CUDA 13.3 or newer. Because XLA's autotuner can choose it, turning it on can change which kernels run.
  • Using PGLE with NCCL symmetric memory requires NCCL 2.32.2 or newer. Training jobs that enable both profile-guided latency estimation (PGLE) and NCCL symmetric buffers (--xla_gpu_experimental_enable_nccl_symmetric_buffers=true) could fail right after PGLE recompiles the program, with an NCCL “Allocation failed” error. The cause is that XLA registers symmetric memory windows each time a GPU program is loaded, and PGLE loads a second program in the same process. Older NCCL versions did not reuse those windows, so the repeated registrations exhausted NCCL’s symmetric allocation space. NCCL 2.32 adds window reuse, which fixes the failure. Use NCCL 2.32.2 or newer whenever PGLE and symmetric buffers are both enabled. Running either feature alone is not affected.

MoE

  • DeepSeek V3 on Blackwell and Vera Rubin GPUs. DeepSeek V3 performance on MXFP8 has been optimized for both Blackwell and Vera Rubin GPUs in this release. To reproduce these results, please refer to the configuration settings that will be published in JAX-Toolbox.
  • Mixture-of-experts training with TransformerEngine on GPU. These changes help train mixture-of-experts (MoE) models, such as DeepSeek-style models, on GPU with TransformerEngine. TransformerEngine can now spread experts across several GPUs within one process, reusing XLA's NCCL communicator. It also names MoE checkpoints so they can be saved and recomputed selectively. MaxText builds on this to cut communication during training and to fix a sizing bug that could set expert capacity too low when gradients are accumulated.
  • Faster mixture-of-experts communication and math. MoE models send tokens between GPUs, and these changes make that exchange faster. The ragged all-to-all kernel now spreads network transfers over all available connections, and it sends data between machines before copying within a machine, so the two overlap. cuDNN can also compute the weight-gradient step of MoE training (a grouped matrix multiply). That needs cuDNN 9.24 or newer and an experimental flag.

Attention

  • Attention enchancements. JAX SDPA now supports flex attention, which lets users customize how attention scores are computed. TransformerEngine's fused attention now handles gradient sharding consistently in the backward pass, with more tests. In MaxText, cuDNN flash attention works during autoregressive inference when the key-value cache and the query have different batch sizes.

Profiling and Observability

  • More accurate profiling. Profiling results are more trustworthy. For operations made of several GPU kernels, cost is now measured from the first kernel's start to the last kernel's end, instead of an average that under-reported it. XLA's automatic profile-guided scheduling (AutoPGLE) now works while running under Nsight Systems, and fixed offsets mean XLA metadata such as module name and source locations shows correctly there. Mosaic GPU supports CUPTI V2, so several profiling sessions can run at once. A tool that builds matmul performance tables now handles matmuls that launch several kernels; it does not change compiled programs.

JAX Toolbox

The JAX Toolboxprojects focuses on achieving the best performance and convergence on NVIDIA Ampere, Hopper, and Blackwell, and Rubin architectures families and provides the latest deep learning models and scripts and container images for training and fine-tuning the latest open source models. These examples are tested against a nightly CI as well as each NGC container release to ensure consistent accuracy and performance over time.

Nightly Containers

In addition to projects, JAX Toolbox includes nightly containers for libraries across the JAX ecosystem.

ContainerTypeImage URI
jax-ghcr.io/nvidia/jax:jax-YYYY-MM-DD
maxtextLLM frameworkghcr.io/nvidia/jax:maxtext-YYYY-MM-DD
axlearnLLM frameworkghcr.io/nvidia/jax:axlearn-YYYY-MM-DD

Known Issues

  • None.

© Copyright 2026, NVIDIA. Last updated on Nov 17, 2026