Llama Nemotron#

Deprecation notice: Llama Nemotron support (Super 49B, Ultra 253B, 70B, Nano 8B, and Nano 4B) is no longer actively maintained or tested against current upstream checkpoints and will be removed in Megatron Bridge 0.7.0.

Llama Nemotron is NVIDIA’s family of large language models derived from Meta’s Llama architecture, post-trained for enhanced reasoning, human chat preferences, and agentic tasks such as RAG and tool calling. The models feature neural architecture search (NAS) optimizations for improved efficiency and accuracy trade-offs.

Llama Nemotron models are supported via the Bridge system with auto-detected configuration and weight mapping.

Available Models#

Megatron Bridge supports the following Llama Nemotron model variants:

  • Llama-3.3-Nemotron-Super-49B: 49B parameters (NAS-optimized from 70B)

  • Llama-3.1-Nemotron-Ultra-253B: 253B parameters (large-scale reasoning model)

  • Llama-3.1-Nemotron-70B: 70B parameters (standard size)

  • Llama-3.1-Nemotron-Nano-8B: 8B parameters (efficient variant)

  • Llama-3.1-Nemotron-Nano-4B: 4B parameters (ultra-compact variant)

All models are ready for commercial use and support context lengths up to 128K tokens.

Model Architecture Features#

  • Neural Architecture Search (NAS): Novel approach to reduce memory footprint while maintaining accuracy

  • Heterogeneous Blocks: Non-standard and non-repetitive layer configurations for efficiency

    • Skip attention in some blocks

    • Variable FFN expansion/compression ratios between blocks

  • Multi-Phase Post-Training:

    • Supervised fine-tuning for Math, Code, Science, and Tool Calling

    • Reward-aware Preference Optimization (RPO) for chat

    • Reinforcement Learning with Verifiable Rewards (RLVR) for reasoning

    • Iterative Direct Preference Optimization (DPO) for tool calling

  • Extended Context: Native support for sequences up to 128K tokens

  • Commercial Ready: Fully licensed for commercial deployment

Conversion with 🤗 Hugging Face#

Load HF → Megatron#

from megatron.bridge import AutoBridge

# Example: Llama-3.3-Nemotron-Super-49B
bridge = AutoBridge.from_hf_pretrained(
    "nvidia/Llama-3_3-Nemotron-Super-49B-v1_5",
    trust_remote_code=True
)
provider = bridge.to_megatron_provider()

# Optionally configure parallelism before instantiating the model
provider.tensor_model_parallel_size = 2
provider.pipeline_model_parallel_size = 1

model = provider.provide_distributed_model(wrap_with_ddp=False)

Note: Heterogeneous Llama-Nemotron models (Super/Ultra) require trust_remote_code=True as they use custom DeciLMForCausalLM architecture. Homogeneous models (Nano/70B) use standard Llama architecture and don’t require this flag.

Import Checkpoint from HF#

./scripts/conversion/convert.sh import \
  --hf-model nvidia/Llama-3_3-Nemotron-Super-49B-v1_5 \
  --megatron-path /checkpoints/llama_nemotron_super_49b_megatron \
  --trust-remote-code

Export Megatron → HF#

from megatron.bridge import AutoBridge

# Load the bridge from HF model ID
bridge = AutoBridge.from_hf_pretrained(
    "nvidia/Llama-3_3-Nemotron-Super-49B-v1_5",
    trust_remote_code=True
)

# Export a trained Megatron checkpoint to HF format
bridge.export_ckpt(
    megatron_path="/results/llama_nemotron_super_49b/checkpoints/iter_0000500",
    hf_path="/exports/llama_nemotron_super_49b_hf",
)

Run Inference on Converted Checkpoint#

uv run python examples/conversion/hf_to_megatron_generate_text.py \
  --hf_model_path nvidia/Llama-3_3-Nemotron-Super-49B-v1_5 \
  --megatron_model_path /checkpoints/llama_nemotron_super_49b_megatron \
  --prompt "What is artificial intelligence?" \
  --max_new_tokens 100 \
  --tp 2 \
  --trust-remote-code

For more details, see examples/conversion/hf_to_megatron_generate_text.py

Recipes#

Training recipes for Llama Nemotron models are not currently available.

Hugging Face Model Cards & References#

Hugging Face Model Cards#

  • Llama Nemotron Collection: https://huggingface.co/collections/nvidia/llama-nemotron

  • Llama-3.3-Nemotron-Super-49B-v1.5: https://huggingface.co/nvidia/Llama-3_3-Nemotron-Super-49B-v1_5

  • Llama-3.1-Nemotron-Ultra-253B-v1: https://huggingface.co/nvidia/Llama-3_1-Nemotron-Ultra-253B-v1

  • Llama-3.1-Nemotron-Nano-8B-v1: https://huggingface.co/nvidia/Llama-3.1-Nemotron-Nano-8B-v1

  • Llama-3.1-Nemotron-Nano-4B-v1.1: https://huggingface.co/nvidia/Llama-3.1-Nemotron-Nano-4B-v1.1

Technical Papers#

Additional Resources#

  • NVIDIA Build Platform: https://build.nvidia.com/

  • Llama Nemotron Post-Training Dataset: https://huggingface.co/nvidia/Llama-Nemotron-Post-Training-Dataset