Inkling 2#
Inkling 2 Small is Thinking Machines’ image-and-text
mixture-of-experts model with 594B total and about 17B active parameters. Its 48 decoder layers replace the residual
stream with Block Attention Residuals, interleave 1,024-token sliding-window attention with full attention on every
sixth layer, and route each token to 8 of 384 experts. The release stores the routed experts in NVFP4 and MXFP8.
Megatron Bridge builds it as a native Megatron-Core HybridModel through the builder-backed Inkling2ModelConfig,
converts the complete checkpoint including the image encoder, and keeps the speculative-decoding draft network outside
the Megatron model.
Examples:
examples/models/inkling(conversion, GB200 recipes, image-text SFT)Implementation:
src/megatron/bridge/models/inkling(architecture, quantized-expert conversion, parallelism and limits)Verification: model verification card
Verified configurations#
Choose an exact recorded configuration to see its command and expected result. These selectors are generated from the authoritative verification cards and never synthesize combinations.
Run a configuration#
Choose a workflow, precision, and exact recorded combination. The command and expected result update below.
Import · CPU
â—‹ UnverifiedExact command
No runnable command is recorded for this status.
Expected result
CPU import of the complete 594B checkpoint has not been attempted. GPU import is recorded separately.
Import · GPU
✓ VerifiedExact command
./scripts/conversion/convert.sh import --executor slurm --device gpu --nodes 4 --gpus-per-node 4 --hf-model thinkingmachines/Inkling-2-Small --hf-revision 78cb6c0ef68e1f089b77e78cfd94783d05b5c41b --megatron-path work/model-verification/inkling-2-small/gpu-megatron --torch-dtype bfloat16 --tp 1 --pp 1 --ep 16 --etp 1
Expected result
Observed with 16 GB200 GPUs at TP1/PP1/EP16/ETP1: the release loads by its Hub ID, builds the complete HybridModel with the image encoder, and dequantizes the NVFP4 and MXFP8 experts to BF16 one layer at a time, in 200.6 s. The export audit under megatron_to_hf_gpu establishes its tensor content. Caveats: this ran through a private srun wrapper that imports and exports in memory rather than through this launcher, so it persisted no iter_0000000 checkpoint. The training items start from a distributed checkpoint of the same weights that a pre-release tree of this support imported and saved.
Export · CPU
â—‹ UnverifiedExact command
No runnable command is recorded for this status.
Expected result
CPU export of the complete 594B checkpoint has not been attempted. GPU export is recorded separately.
Export · GPU
✓ VerifiedExact command
./scripts/conversion/convert.sh export --executor slurm --device gpu --nodes 4 --gpus-per-node 4 --hf-model thinkingmachines/Inkling-2-Small --hf-revision 78cb6c0ef68e1f089b77e78cfd94783d05b5c41b --megatron-path work/model-verification/inkling-2-small/gpu-megatron/iter_0000000 --hf-path work/model-verification/inkling-2-small/gpu-hf-export --torch-dtype bfloat16 --tp 1 --pp 1 --ep 16 --etp 1 --distributed-save
Expected result
Observed with 16 GB200 GPUs at TP1/PP1/EP16/ETP1: the export re-quantizes the routed experts against the source scales, and an exhaustive comparison against the release finds all 1,770 tensors byte-identical, including the NVFP4 and MXFP8 experts and their scales, with no missing or extra keys. The tokenizer files, chat template, processor and quantization configurations, and the MTP draft are byte-identical to the release; config.json keeps every release key and adds dtype, image_token_id, tie_word_embeddings, and transformers_version. Caveats: the export ran in memory from the model imported under hf_to_megatron_gpu, through a private srun wrapper, rather than from a saved iter_0000000 checkpoint through this launcher. Export from a saved checkpoint was exercised by the plain BF16 export of an SFT checkpoint described under sft_export_inference.
Pretrain · GB200
✓ VerifiedRecorded metrics
- Initial loss
- 1.827291
- Final loss
- 1.745315
- Step time · last 10 avg
- 16,240.320 ms
- Model throughput · last 10 avg
- 225.530 TFLOP/s/GPU
- Token throughput · last 10 avg
- 2,017.694 tokens/s/GPU
Exact command
./scripts/training/train.sh --nodes 32 --gpus-per-node 4 --recipe inkling2_small_pretrain_128gpu_gb200_bf16_config --mode pretrain --pretrained_checkpoint work/model-verification/inkling-2-small/gpu-megatron --max_steps 50 'dataset.blend=[["work/data/rp2/head_01_inkling2_text_document"],null]' checkpoint.save=null checkpoint.load=null checkpoint.async_save=false ddp.check_for_large_grads=true rerun_state_machine.check_for_nan_in_loss=true logger.log_interval=1 logger.log_throughput=true logger.log_device_memory_used=true logger.tensorboard_dir=null logger.save_config_filepath=work/model-verification/inkling-2-small/pretrain-resolved.yaml
Expected result
Observed with 128 GB200 GPUs at TP1/PP2/EP32/ETP1 (pipeline stages of 26 and 22 decoder blocks, each on 16 consecutive nodes of one NVL72 rack holding two EP32 groups; HybridEP dispatch; full activation recompute; compiled Inkling2 operators; Inkling2 attention's FlexAttention backward tiles for Blackwell; expandable segments set in the environment as the example README instructs): the command warm-starts from the imported checkpoint with natural expert routing, keeps the image encoder frozen, and runs the first 50 of the recipe's 100 steps on the RedPajama2 head_01 prefix at GBS 1024 and sequence length 4096, at the recipe's peak LR of 3e-5 after 40 warmup steps (2.82e-5 at step 50). All 50 steps finish with finite losses and 0 skipped or NaN iterations. Caveats: run on 2026-11-16 through a private srun wrapper around scripts/training/run_recipe.py rather than train.sh, with Transformers 5.23.0 installed over the container's copy. Checkpoint saving was off, so the run saved no step-50 or step-100 checkpoint and checkpoint_resume stays unverified. An earlier 100-step run of the same data and schedule on older code and four pipeline stages reached a loss of 1.7301 at step 100.
SFT · GB200
✓ VerifiedRecorded metrics
- Initial loss
- 1.323619
- Final loss
- 0.6976618
- Step time · last 10 avg
- 1,481.150 ms
- Model throughput · last 10 avg
- 37.740 TFLOP/s/GPU
- Token throughput · last 10 avg
- 345.677 tokens/s/GPU
Exact command
./scripts/training/train.sh --nodes 32 --gpus-per-node 4 --recipe inkling2_small_sft_128gpu_gb200_bf16_config --mode sft --pretrained_checkpoint work/model-verification/inkling-2-small/gpu-megatron --max_steps 30 checkpoint.save=null checkpoint.load=null checkpoint.async_save=false ddp.check_for_large_grads=true rerun_state_machine.check_for_nan_in_loss=true logger.log_interval=1 logger.log_throughput=true logger.log_device_memory_used=true logger.tensorboard_dir=null logger.save_config_filepath=work/model-verification/inkling-2-small/sft-resolved.yaml
Expected result
Observed with 128 GB200 GPUs at TP4/PP4/EP32/ETP1 with sequence parallelism, no activation recompute, HybridEP dispatch, and compiled Inkling2 operators: the command warm-starts from the imported checkpoint and runs the first 30 of the recipe's 100 steps of full SFT on the first 10,000 Tulu 3 conversations, rendered with the release's chat template and packed to 8,192 tokens, at GBS 8 (eight data-parallel replicas with one pack each) and peak LR 5e-6 after 10 warmup steps. All 30 steps finish with finite losses and 0 skipped or NaN iterations. Caveats: run on 2026-11-11 through a private srun wrapper around scripts/training/run_recipe.py rather than train.sh, with checkpoint saving off, on a pre-release tree whose model code matches the recorded commit; its recipe reached the recorded recipe's settings through recompute overrides and rendered the data with a copy of the release's chat template that gives the same tokens and loss masks. A 100-step run of the same data and schedule on older code reached a loss of 0.5446 at step 100; sft_export_inference describes the export of its checkpoint.
Long Context · GB200
✓ VerifiedRecorded metrics
- Initial loss
- 1.153563
- Final loss
- 0.8579385
- Step time · last 10 avg
- 4,711.280 ms
- Model throughput · last 10 avg
- 47.460 TFLOP/s/GPU
- Token throughput · last 10 avg
- 434.701 tokens/s/GPU
Exact command
./scripts/training/train.sh --nodes 32 --gpus-per-node 4 --recipe inkling2_small_sft_128gpu_gb200_bf16_long_context_config --mode sft --pretrained_checkpoint work/model-verification/inkling-2-small/gpu-megatron --max_steps 20 checkpoint.save=null checkpoint.load=null checkpoint.async_save=false ddp.check_for_large_grads=true rerun_state_machine.check_for_nan_in_loss=true logger.log_interval=1 logger.log_throughput=true logger.log_device_memory_used=true logger.tensorboard_dir=null logger.save_config_filepath=work/model-verification/inkling-2-small/sft-long-context-resolved.yaml
Expected result
Observed with 128 GB200 GPUs at TP4/PP4/EP32/CP4/ETP1 with sequence parallelism, Inkling2's own context parallelism, recompute of the first 20 of 24 layers per stage, HybridEP dispatch, and compiled Inkling2 operators: the command warm-starts from the imported checkpoint and runs all 20 recipe steps of full SFT on Tulu 3 packed to 32,768 tokens at GBS 8 (two data-parallel replicas running four packs each), at peak LR 1e-6 after 2 warmup steps with decay to 0 at step 20. All 20 steps finish with finite losses and 0 skipped or NaN iterations. Caveats: run on 2026-11-11 through a private srun wrapper around scripts/training/run_recipe.py rather than train.sh, on a pre-release tree whose model code matches the recorded commit; its recipe reached the recorded recipe's settings through recompute overrides and rendered the data with a copy of the release's chat template that gives the same tokens and loss masks. The pre-release checkpoint's configuration starts full-attention length scaling at 8,192 tokens instead of this revision's 128,000, so tokens past position 8,192 of a document were scaled where this revision would not scale them.
LoRA · GB200
✓ VerifiedRecorded metrics
- Initial loss
- 1.324133
- Final loss
- 0.7994129
- Step time · last 10 avg
- 1,427.080 ms
- Model throughput · last 10 avg
- 142.310 TFLOP/s/GPU
- Token throughput · last 10 avg
- 1,435.098 tokens/s/GPU
Exact command
./scripts/training/train.sh --nodes 8 --gpus-per-node 4 --recipe inkling2_small_peft_32gpu_gb200_bf16_config --mode lora --pretrained_checkpoint work/model-verification/inkling-2-small/gpu-megatron --max_steps 30 checkpoint.save=null checkpoint.load=null checkpoint.async_save=false ddp.check_for_large_grads=true rerun_state_machine.check_for_nan_in_loss=true logger.log_interval=1 logger.log_throughput=true logger.log_device_memory_used=true logger.tensorboard_dir=null logger.save_config_filepath=work/model-verification/inkling-2-small/peft-resolved.yaml
Expected result
Observed with 32 GB200 GPUs at TP4/PP1/EP32/ETP1 with sequence parallelism, no activation recompute, HybridEP dispatch, and compiled Inkling2 operators: the command loads the imported checkpoint, freezes the base model, and trains rank-8, alpha-16 LoRA adapters on linear_q, linear_k, linear_v, and linear_proj for the first 30 of the recipe's 100 steps on the packed 8K Tulu 3 data of the sft item at GBS 8 and peak LR 1e-4 after 10 warmup steps. All 30 steps finish with finite losses and 0 skipped or NaN iterations. The TFLOP/s value is Bridge's generic estimate, which counts a full backward pass. Caveats: run on 2026-11-11 through a private srun wrapper around scripts/training/run_recipe.py rather than train.sh, with checkpoint saving off, on a pre-release tree whose model code matches the recorded commit; its recipe reached the recorded recipe's settings through recompute overrides and rendered the data with a copy of the release's chat template that gives the same tokens and loss masks. A 100-step run of the same data and schedule on older code reached a loss of 0.7560 at step 100.