Skip to navigation

Inkling-2-Small is a vision-language Mixture-of-Experts (MoE) model from Thinking Machines Lab with 594B total and 17B active parameters. It accepts text and image inputs, generates text, and supports a context window of up to 1M tokens. NeMo AutoModel provides a text recipe (Inkling2ForCausalLM) and two image-text recipes (Inkling2ForConditionalGeneration) for the released checkpoint.

The training loader dequantizes expert weights into floating-point storage. Numerical equivalence to the original quantized inference runtime is not established. The checkpoint’s auxiliary multi-token prediction module is outside the training scope of these recipes.

Fine-Tune Inkling-2-Small

Follow the installation instructions. The recipes run on multiple GB200 nodes with four GPUs each. Use the Slurm launcher guide for multi-node runs. For example, launch the LoRA recipe on 8 nodes:

uv run torchrun --nnodes=8 --nproc-per-node=4 \
--node-rank="$NODE_RANK" --master-addr="$MASTER_ADDR" --master-port="$MASTER_PORT" \
examples/vlm_finetune/finetune.py \
--config examples/vlm_finetune/inkling2/inkling2_medpix_lora_ep32.yaml

Run the text recipe with examples/llm_finetune/finetune.py.

Choose a Workflow

GoalStart Here
128K packed text SFT on Tulu 3inkling2_tulu3_packed128k_ep32_pp2_cp32.yaml
Image-text LoRA on MedPixinkling2_medpix_lora_ep32.yaml
Full image-text SFT on MedPixinkling2_medpix_sft_ep64_pp4.yaml
Prepare your environmentFollow the installation instructions.

Configuration

SettingTulu 3 128K SFTMedPix LoRAMedPix SFT
ModelInkling2ForCausalLM, text onlyInkling2ForConditionalGenerationInkling2ForConditionalGeneration
GPUs64 GB200, 16 nodes32 GB200, 8 nodes256 GB200, 64 nodes
ParallelismEP32, PP2, CP32, TP1; FSDP2EP32, four replicas, TP1; FSDP2EP64, PP4, TP1; FSDP2
Pipeline scheduleInterleaved 1F1B, 12 layers per stageNoneInterleaved 1F1B, 6 layers per stage
SequencePacked THD, 131,072 tokensNonpacked, up to 4,097 tokensNonpacked, up to 4,097 tokens
BatchGlobal 8, local 8Global 32, local 1Global 256, local 4
Trained weightsLanguage modelLoRA rank 8, alpha 32, on attention projections and routed expertsLanguage model
Weights and optimizerBF16 weights; FusedAdam with FP32 master weights and BF16 momentsBF16 weights; FusedAdam with FP32 master weights and BF16 momentsFP32 weights; AdamW
Learning rate5e-6, constant5e-6, constant5e-6, 10 warmup steps
Steps50100100

All three recipes load thinkingmachines/Inkling-2-Small and dequantize its NVFP4 and MXFP8 expert weights at load time (checkpoint.dequantize_base_checkpoint: true). They use flex attention, torch_mm experts with FP32 router gates, and full activation checkpointing. The output loss, Inkling2FusedLinearCrossEntropy, applies the logit soft cap of 30 over the 200,058 usable vocabulary rows.

The text recipe builds Inkling2ForCausalLM from the image-text checkpoint through model.config.architectures, without the vision tower. It reads Tulu 3 from the Hugging Face Hub and packs it at startup. Both MedPix recipes freeze the vision tower.

W&B is opt-in: set wandb.enable: true and configure your entity and project when launching.

Model Reference

Model Architecture

PropertyValue
Checkpointsthinkingmachines/Inkling-2-Small (NVFP4), thinkingmachines/Inkling-2-Small-MXFP8
ArchitectureInkling2ForConditionalGeneration, a vision-language model with a routed Mixture-of-Experts decoder
Parameters594B total, 17B active
InputsText and images
Context lengthUp to 1M tokens
Language backbone48 layers, hidden size 4,608
Attention48 query heads, 8 KV heads, head dimension 128
Attention patternFive local layers followed by one full layer, repeated eight times
Local attention window1,024 tokens, causal
Experts384 routed experts per layer, top-8 sigmoid routing, intermediate size 2,304
Routing scale8
Vocabulary201,024 embedding rows; 200,058 usable token IDs
Output logitsSoft cap of 30; padded vocabulary rows excluded
Vision tower23 layers, hidden size 1,024, 16 attention heads
Vision projectionNative 20-by-20 patches; neighboring 2-by-2 features merged before projection
Expert weightsInkling-2-Small: NVFP4 in layers 0-39, MXFP8 in layers 40-47; Inkling-2-Small-MXFP8: MXFP8 in all layers

Attention and Residuals

Local layers use causal sliding-window attention. Every sixth layer uses full causal attention, allowing information to travel beyond the local window. Attention includes learned relative-position terms and attention sinks. The text decoder does not use rotary position embeddings.

The decoder also applies causal short convolutions with four taps. Each output mixes the current position with up to three previous positions. Attention-based residual aggregation uses blocks of six decoder layers, with normalization of the residual values.

Experts and Output Vocabulary

Experts use a SwiGLU variant with separate gate and up soft caps of 4 and 25. The checkpoint stores packed gate/up weights in an interleaved layout. Each token selects eight of the 384 experts using sigmoid router scores. A routing scale of 8 controls the contribution of the selected experts.

The output projection has 201,024 rows, of which 200,058 correspond to usable token IDs. The extra rows pad the matrix dimensions for computation. They do not represent sequence padding tokens. A logit soft cap of 30 bounds the output scores before the probability distribution is computed.

Vision Features and Checkpoint Format

The vision encoder processes native 20-by-20 image patches. The projector concatenates each neighboring 2-by-2 group of vision features, then maps that vector into the language hidden space.

Thinking Machines publishes two checkpoints. Inkling-2-Small is the native NVFP4 release: its experts are NVFP4 in layers 0-39 and MXFP8 in layers 40-47. Inkling-2-Small-MXFP8 stores every expert in MXFP8. Both checkpoints keep the remaining parameters in floating point, record the layout in hf_quant_config.json, and include an auxiliary multi-token prediction module in mtp.safetensors.

Prepare the Checkpoint

Follow the installation instructions and Slurm cluster guide. Use a CUDA environment with PyTorch FlexAttention, grouped matrix multiplication, Triton gradient clipping, and Transformer Engine’s FusedAdam for the LoRA and Tulu 3 recipes.

Each released checkpoint records quantization in hf_quant_config.json. The native model reads that sidecar directly and stages the packed expert tensors through the shared checkpoint loader, decoding them into floating-point storage. Set the path to a complete downloaded checkpoint directory, accessible on every training node:

export INKLING2_MODEL=/path/to/Inkling-2-Small

Both the image-text and text recipes use this directory. The Tulu 3 recipe selects Inkling2ForCausalLM through model.config.architectures. It constructs the text backbone without a vision tower. No separate metadata-preparation helper or checkpoint copy is required.

Run MedPix LoRA

Set shared output paths and a W&B run identity. Authenticate W&B in the training environment through your usual account setup:

export INKLING2_RUN_DIR=/path/to/shared/inkling2-medpix-lora
export WANDB_ENTITY=your-wandb-entity
export WANDB_PROJECT=inkling2
export WANDB_RUN_ID=inkling2-medpix-lora-001
mkdir -p "${INKLING2_RUN_DIR}/wandb"

Within an eight-node allocation with four GB200 GPUs per node, run one launcher per node. Set NODE_RANK from 0 to 7 and MASTER_ADDR to the first node’s reachable hostname:

uv run torchrun --nnodes=8 --nproc-per-node=4 \
--node-rank="${NODE_RANK}" \
--rdzv-id="${WANDB_RUN_ID}" \
--rdzv-backend=c10d --rdzv-endpoint="${MASTER_ADDR}:29500" \
-m nemo_automodel.cli.app \
examples/vlm_finetune/inkling2/inkling2_medpix_lora_ep32.yaml \
--model.pretrained_model_name_or_path "${INKLING2_MODEL}" \
--processor.pretrained_model_name_or_path "${INKLING2_MODEL}" \
--checkpoint.enabled=true \
--checkpoint.checkpoint_dir="${INKLING2_RUN_DIR}/checkpoints" \
--wandb.enable=true --wandb.mode=online \
--wandb.entity="${WANDB_ENTITY}" --wandb.project="${WANDB_PROJECT}" \
--wandb.id="${WANDB_RUN_ID}" --wandb.name="${WANDB_RUN_ID}" \
--wandb.dir="${INKLING2_RUN_DIR}/wandb"

The LoRA recipe uses local batch 1 and global batch 32. Base weights and LoRA parameters use BF16, with FP32 router computation. TE FusedAdam maintains FP32 master weights and BF16 optimizer moments. W&B and checkpoint writing are disabled in the portable YAML and enabled explicitly by this command.

Run MedPix SFT

Use the MedPix SFT recipe in a 64-node allocation with four GB200 GPUs per node:

uv run torchrun --nnodes=64 --nproc-per-node=4 \
--node-rank="${NODE_RANK}" \
--rdzv-id="${WANDB_RUN_ID}" \
--rdzv-backend=c10d --rdzv-endpoint="${MASTER_ADDR}:29500" \
-m nemo_automodel.cli.app \
examples/vlm_finetune/inkling2/inkling2_medpix_sft_ep64_pp4.yaml \
--model.pretrained_model_name_or_path "${INKLING2_MODEL}" \
--processor.pretrained_model_name_or_path "${INKLING2_MODEL}" \
--checkpoint.enabled=true \
--checkpoint.checkpoint_dir="${INKLING2_RUN_DIR}/checkpoints" \
--wandb.enable=true --wandb.mode=online \
--wandb.entity="${WANDB_ENTITY}" --wandb.project="${WANDB_PROJECT}" \
--wandb.id="${WANDB_RUN_ID}" --wandb.name="${WANDB_RUN_ID}" \
--wandb.dir="${INKLING2_RUN_DIR}/wandb"

Before this launch, set a new INKLING2_RUN_DIR and WANDB_RUN_ID for SFT, create its W&B directory, and set NODE_RANK from 0 to 63. This recipe uses FP32 model storage with BF16 FSDP compute and PyTorch AdamW. It trains the language model while freezing the vision tower.

PP4 uses two virtual chunks per stage, six decoder layers per chunk, Interleaved1F1B, and pipeline microbatch size 1. Local batch 4 supplies four microbatches per pipeline iteration; global batch 256 preserves that relationship across the 64 data replicas. EP shards expert ownership across those replicas. It is not an extra divisor for the data batch. Stage 0 owns the vision encoder, and each microbatch receives its matching image patches.

Checkpoint and Resume

The portable recipes disable checkpoint writing by default. With checkpoints enabled, MedPix LoRA saves every 25 steps and MedPix SFT saves every 50. A resumable save includes model or adapter weights, optimizer shards, scheduler, and data-loader state. Full SFT optimizer checkpoints require substantially more storage than adapter-only weights. Wait for checkpoint completion before stopping a job.

To resume MedPix SFT, use the same recipe, base checkpoint, optimizer, and distributed topology. Keep the W&B run ID and append these arguments to its launch command:

--checkpoint.restore_from="${INKLING2_RUN_DIR}/checkpoints/epoch_0_step_49" \
--step_scheduler.max_steps=100 \
--wandb.resume=allow

Use the actual completed checkpoint directory. The example name uses the zero-based step index after 50 updates. W&B resume reconnects metrics, and checkpoint.restore_from restores training state.

Packed Context Parallelism

The recipe reads allenai/tulu-3-sft-mixture from the Hugging Face Hub and packs it when training starts. Conversations are rendered with the Inkling chat template in examples/llm_finetune/inkling2/inkling2_chat_template.jinja, shuffled with seed 1234, truncated to 131,072 tokens, and packed into 416 THD rows of 131,072 tokens with 64-token document alignment. Labels cover the assistant turns only. THD cumulative lengths and position IDs retain document boundaries. Local attention and short convolutions cannot cross those boundaries.

Use a 16-node allocation with four GB200 GPUs per node:

uv run torchrun --nnodes=16 --nproc-per-node=4 \
--node-rank="${NODE_RANK}" \
--rdzv-id="${WANDB_RUN_ID}" \
--rdzv-backend=c10d --rdzv-endpoint="${MASTER_ADDR}:29500" \
-m nemo_automodel.cli.app \
examples/llm_finetune/inkling2/inkling2_tulu3_packed128k_ep32_pp2_cp32.yaml \
--model.pretrained_model_name_or_path "${INKLING2_MODEL}" \
--dataset.tokenizer.pretrained_model_name_or_path "${INKLING2_MODEL}" \
--wandb.enable=true --wandb.mode=online \
--wandb.entity="${WANDB_ENTITY}" --wandb.project="${WANDB_PROJECT}" \
--wandb.id="${WANDB_RUN_ID}" --wandb.name="${WANDB_RUN_ID}" \
--wandb.dir="${INKLING2_RUN_DIR}/wandb"

Choose a new run ID and output directory and set NODE_RANK from 0 to 15. PP2/CP32 leaves one data replica. Local and global batches are both 8. Interleaved1F1B uses twelve layers per virtual chunk and microbatch size 1. The recipe uses FlexAttention with the Inkling CP implementation, BF16 model storage, and TE FusedAdam with FP32 master weights and BF16 moments.

Distributed and Kernel Support

The recipes use the following distributed and kernel features:

  • FSDP2, expert parallelism, pipeline parallelism with Interleaved1F1B, and context parallelism with document-aware packed text.
  • PyTorch grouped experts (experts: torch_mm) and the shared HybridEP dispatcher (dispatcher: hybridep). The torch dispatcher remains available for reference checks.
  • FlexAttention for learned relative-position and sink gradients, and a dense PyTorch reference path for small numerical checks.
  • Activation checkpointing and separate dense and MoE reshard controls.
  • Shared fused linear cross entropy over usable vocabulary rows, with BF16 projection inputs, FP32 accumulation, and the soft cap of 30.

The recipes now select HybridEP; full-model training with this dispatcher remains to be qualified. Tensor parallelism greater than one, MTP training, and quantized activation training are not qualified by these recipes. Packed CP training covers text. Image-text pipeline training uses the non-packed path.

See the LLM fine-tuning guide for shared training options.

Hugging Face Model Card

View the model cards for both checkpoints:

Refer to the following resources for related information: