Inkling-2-Mini
Inkling-2-Mini is a vision-language
Mixture-of-Experts (MoE) model from Thinking Machines Lab with 26B total and 5B active parameters. It
accepts text and image inputs, generates text, and supports a context window of up to 1M tokens. It is
a looped Inkling 2 model: twelve of its 24 decoder layers run three times for every token, and the
active-parameter count includes every pass.
NeMo AutoModel provides two image-text recipes (Inkling2ForConditionalGeneration) and a text recipe
(Inkling2ForCausalLM) for the released checkpoint. Each recipe runs on one node with four GPUs.
The training loader dequantizes the NVFP4 and MXFP8 expert weights into floating-point storage, and the recipes train the experts in BF16. The checkpoint’s auxiliary multi-token prediction module is outside the training scope of these recipes.
Fine-Tune Inkling-2-Mini
Follow the installation instructions, then run a recipe from the repository root on a node with four GB200 GPUs. For example, launch the LoRA recipe:
Choose a Workflow
Configuration
All three recipes load thinkingmachines/Inkling-2-Mini and dequantize its NVFP4 and MXFP8 expert
weights at load time (checkpoint.dequantize_base_checkpoint: true). They use flex attention,
torch_mm experts with the HybridEP dispatcher and FP32 router gates, and full activation
checkpointing. The output loss, Inkling2FusedLinearCrossEntropy, applies the logit soft cap of 30
over the 200,058 usable vocabulary rows.
The text recipe builds Inkling2ForCausalLM from the image-text checkpoint through
model.config.architectures, without the vision tower. Both MedPix recipes freeze the vision tower.
W&B is opt-in: set wandb.enable: true and configure your entity and project when launching.
The following table shows measurements of the three recipes over 100 steps on one node with four GB200 GPUs:
The checkpoint loads in under one minute. The first step takes 11 to 14 minutes while FlexAttention compiles.
Packed Sequences and Context Parallelism
The text recipe trains on Tulu 3 rows of 131,072 tokens with cp_size: 4:
The recipe reads allenai/tulu-3-sft-mixture from the Hugging Face Hub and packs it when training
starts. Conversations are rendered with the Inkling chat template in
examples/llm_finetune/inkling2/inkling2_chat_template.jinja, shuffled with seed 1234, truncated to
131,072 tokens, and packed into THD rows of 131,072 tokens with 64-token document alignment. Labels
cover the assistant turns only.
Each row holds many documents. Positions restart at every document, and local attention, full attention, and the short convolutions all stay inside a document. With context parallelism, each GPU holds one contiguous shard of the row, 32,768 tokens here.
Model Reference
Model Architecture
Looped Layers
Inkling-2-Mini is an Inkling 2 model with one change to how the layer stack runs. Layers 6-17 run three times for every token with the same weights, so 24 layers of weights give 48 layer visits. Layers 0-5 run once before the loop and layers 18-23 run once after it.
Each later pass starts again from the state that entered layer 6 and reads the previous pass through messages. Every looped layer mixes a message into its attention query, key, and value inputs and into its expert input. A low-rank gate of rank 64 decides, per channel and per token, how much of the input to keep and how much of the message to add. The first pass uses no messages.
Attention, short convolutions, experts, the output vocabulary, and the vision tower work as in Inkling-2-Small.
Distributed and Kernel Support
The recipes use the following distributed and kernel features:
- FSDP2, expert parallelism, and context parallelism with document-aware packed text.
- PyTorch grouped experts (
experts: torch_mm) and the shared HybridEP dispatcher (dispatcher: hybridep). - FlexAttention for learned relative-position and sink gradients.
- Activation checkpointing of whole decoder layers.
- Shared fused linear cross entropy over usable vocabulary rows, with BF16 projection inputs, FP32 accumulation, and the soft cap of 30.
Pipeline parallelism and tensor parallelism greater than one are not available for this model. Packed context-parallel training covers text. Image-text training uses the non-packed path.
See the LLM fine-tuning guide for shared training options.
Hugging Face Model Card
View the model card for the checkpoint:
Related Resources
Refer to the following resources for related information: