Known Issues#

This page lists known issues and limitations in the current release.

26.08#

  • SPCX payload corruption at high expert parallelism: In the affected NCCL 2.30.5/SPCX stack, stale or uninitialized ncclIbRequest.send.onlyWriteImm state can misclassify grouped variable-split BF16 all-to-all sends, losing or corrupting payloads. MCore’s MoE alltoall dispatcher can trigger this transport issue at high EP. Disable the external network plugin and select NCCL’s internal IB transport with NCCL_NET_PLUGIN=none NCCL_NET=IB, or use HybridEP. See issue #5462.

  • Step-3.7-Flash checkpoint parity remains unverified.

  • Large-checkpoint round-trip validation may time out: All ranks export, but rank 0 then lazily loads and exhaustively compares the original Hugging Face tensors. Peers can reach synchronization early enough to exceed the default 600-second NCCL process-group timeout while rank 0 is still progressing. Increase --distributed-timeout-minutes beyond the worst-case verification duration. If exhaustive validation is impractical, run separate import and export --distributed-save workflows instead of roundtrip. See PR #5491.

26.06#

  • Moonlight and Nemotron v3 Nano model training recipe shows performance degradation with TP > 1. As a workaround, set TP=1 and use HybridEP. We have root caused this regression to the base PyTorch image upgrade from 26.02 to 26.04. We are actively investigating and looking to fix this regression soon.

  • Step-3.7-Flash forward-pass outputs have not been fully verified.

  • Some examples/scripts have known minor issues: MiniMax M2 (conversion/export saving), GLM-4.5V (exported tokenizer artifacts), FLUX (tokenizer setup), and WAN (inference setup/dependencies).

26.04#

  • The following video / image decoding packages are no longer installed by default in the NeMo Framework 26.04 container (nvcr.io/nvidia/nemo:26.04) to mitigate CVEs in their vendored native binaries:

    Workflows that depend on any of these (for example, multimodal video pipelines, qwen-vl-utils video paths, or decord[av-decode]) must reinstall them at runtime — see docker/common/README.md for instructions.

26.02#

  • AWS EKS only: Due to AWS-OFI-NCCL v1.17.0 long-running jobs suffer a memory leak that causes performance regression over time. This can be mitigated by upgrading to v1.17.3.

  • Context parallelism with sequence packing are not yet supported for Qwen 3 VL in the r0.3.0 release. Fixed in 26.02.01 (r0.3.1).

  • DeepEP is not supported in the current NeMo framework 26.02 container (nvcr.io/nvidia/nemo:26.02), which results in reduced DSv3 performance compared to the NeMo framework 25.09 container (nvcr.io/nvidia/nemo:25.09) on H100 machines. For optimal H100 performance, we recommend using the NeMo framework 25.09 container.

25.11#

  • Deepseek V3 on H100 has an issue when using DeepEP and fails with RuntimeError: DeepEP error: timeout (dispatch CPU).

  • MODEL_TFLOP/s/GPU is printed as 0 to stdout for all Hybrid models, such as Nemotron-H 56B.

25.09#

  • Pretraining DeepSeek in subchannel FP8 precision is not working. Pretraining DeepSeek with current scaling FP8 is a workaround, but MTP loss does not converge.

On this page