Skip to navigation

Inkling-2-Mini is a vision-language Mixture-of-Experts (MoE) model from Thinking Machines Lab with 26B total and 5B active parameters. It accepts text and image inputs, generates text, and supports a context window of up to 1M tokens. It is a looped Inkling 2 model: twelve of its 24 decoder layers run three times for every token, and the active-parameter count includes every pass. NeMo AutoModel provides two image-text recipes (Inkling2ForConditionalGeneration) and a text recipe (Inkling2ForCausalLM) for the released checkpoint. Each recipe runs on one node with four GPUs.

The training loader dequantizes the NVFP4 and MXFP8 expert weights into floating-point storage, and the recipes train the experts in BF16. The checkpoint’s auxiliary multi-token prediction module is outside the training scope of these recipes.

Fine-Tune Inkling-2-Mini

Follow the installation instructions, then run a recipe from the repository root on a node with four GB200 GPUs. For example, launch the LoRA recipe:

automodel examples/vlm_finetune/inkling2/inkling2_mini_medpix_lora_ep4.yaml --nproc-per-node 4

Choose a Workflow

GoalStart Here
Full image-text SFT on MedPixinkling2_mini_medpix_sft_ep4.yaml
Image-text LoRA on MedPixinkling2_mini_medpix_lora_ep4.yaml
128K packed text SFT on Tulu 3 with context parallelisminkling2_mini_tulu3_packed128k_ep4_cp4.yaml
Prepare your environmentFollow the installation instructions.

Configuration

SettingMedPix SFTMedPix LoRATulu 3 128K SFT
ModelInkling2ForConditionalGenerationInkling2ForConditionalGenerationInkling2ForCausalLM, text only
GPUs4 GB200, one node4 GB200, one node4 GB200, one node
ParallelismEP4, TP1; FSDP2EP4, TP1; FSDP2EP4, CP4, TP1; FSDP2
SequenceNonpacked, up to 4,097 tokensNonpacked, up to 4,097 tokensPacked THD, 131,072 tokens
BatchGlobal 16, local 4Global 16, local 4Global 1, local 1
Trained weightsLanguage modelLoRA rank 8, alpha 32, on attention projections and routed expertsLanguage model
Weights and optimizerBF16 weights; FusedAdam with FP32 master weights and BF16 momentsBF16 weights; FusedAdam with FP32 master weights and BF16 momentsBF16 weights; FusedAdam with FP32 master weights and BF16 moments
Learning rate5e-6, constant1e-4, constant5e-6, constant
Steps100100100

All three recipes load thinkingmachines/Inkling-2-Mini and dequantize its NVFP4 and MXFP8 expert weights at load time (checkpoint.dequantize_base_checkpoint: true). They use flex attention, torch_mm experts with the HybridEP dispatcher and FP32 router gates, and full activation checkpointing. The output loss, Inkling2FusedLinearCrossEntropy, applies the logit soft cap of 30 over the 200,058 usable vocabulary rows.

The text recipe builds Inkling2ForCausalLM from the image-text checkpoint through model.config.architectures, without the vision tower. Both MedPix recipes freeze the vision tower.

W&B is opt-in: set wandb.enable: true and configure your entity and project when launching.

The following table shows measurements of the three recipes over 100 steps on one node with four GB200 GPUs:

RecipeFirst LossMean Loss of the Last 10 StepsTime per StepPeak Memory per GPU
MedPix SFT1.931.406.3 s94 GiB
MedPix LoRA1.931.434.5 s44 GiB
Tulu 3 128K SFT0.670.6218.9 s116 GiB

The checkpoint loads in under one minute. The first step takes 11 to 14 minutes while FlexAttention compiles.

Packed Sequences and Context Parallelism

The text recipe trains on Tulu 3 rows of 131,072 tokens with cp_size: 4:

automodel examples/llm_finetune/inkling2/inkling2_mini_tulu3_packed128k_ep4_cp4.yaml --nproc-per-node 4

The recipe reads allenai/tulu-3-sft-mixture from the Hugging Face Hub and packs it when training starts. Conversations are rendered with the Inkling chat template in examples/llm_finetune/inkling2/inkling2_chat_template.jinja, shuffled with seed 1234, truncated to 131,072 tokens, and packed into THD rows of 131,072 tokens with 64-token document alignment. Labels cover the assistant turns only.

Each row holds many documents. Positions restart at every document, and local attention, full attention, and the short convolutions all stay inside a document. With context parallelism, each GPU holds one contiguous shard of the row, 32,768 tokens here.

Model Reference

Model Architecture

PropertyValue
Checkpointthinkingmachines/Inkling-2-Mini
ArchitectureInkling2ForConditionalGeneration with model_type: inkling2_loop, a vision-language model with a routed Mixture-of-Experts decoder whose middle layers run three times
Parameters26B total; 5B active per token, counting every pass through the looped layers
InputsText and images
Context lengthUp to 1M tokens
Language backbone24 layers, hidden size 2,304; 48 layer visits per token
LoopLayers 6-17 run three times; layers 0-5 and 18-23 run once
Attention24 query heads, 8 KV heads, head dimension 128
Attention patternFive local layers followed by one full layer, repeated four times
Local attention window1,024 tokens, causal
Experts128 routed experts per layer, top-8 sigmoid routing, intermediate size 1,152
Routing scale8
Vocabulary201,024 embedding rows; 200,058 usable token IDs
Output logitsSoft cap of 30; padded vocabulary rows excluded
Vision tower23 layers, hidden size 1,024, 16 attention heads
Vision projectionNative 20-by-20 patches; neighboring 2-by-2 features merged before projection
Expert weightsNVFP4 in layers 0-19, MXFP8 in layers 20-23

Looped Layers

Inkling-2-Mini is an Inkling 2 model with one change to how the layer stack runs. Layers 6-17 run three times for every token with the same weights, so 24 layers of weights give 48 layer visits. Layers 0-5 run once before the loop and layers 18-23 run once after it.

Each later pass starts again from the state that entered layer 6 and reads the previous pass through messages. Every looped layer mixes a message into its attention query, key, and value inputs and into its expert input. A low-rank gate of rank 64 decides, per channel and per token, how much of the input to keep and how much of the message to add. The first pass uses no messages.

Attention, short convolutions, experts, the output vocabulary, and the vision tower work as in Inkling-2-Small.

Distributed and Kernel Support

The recipes use the following distributed and kernel features:

  • FSDP2, expert parallelism, and context parallelism with document-aware packed text.
  • PyTorch grouped experts (experts: torch_mm) and the shared HybridEP dispatcher (dispatcher: hybridep).
  • FlexAttention for learned relative-position and sink gradients.
  • Activation checkpointing of whole decoder layers.
  • Shared fused linear cross entropy over usable vocabulary rows, with BF16 projection inputs, FP32 accumulation, and the soft cap of 30.

Pipeline parallelism and tensor parallelism greater than one are not available for this model. Packed context-parallel training covers text. Image-text training uses the non-packed path.

See the LLM fine-tuning guide for shared training options.

Hugging Face Model Card

View the model card for the checkpoint:

Refer to the following resources for related information: