LFM3-1.7B
LFM3-1.7B has a checked-in NeMo AutoModel recipe for text generation.
The Hugging Face configuration declares the Lfm3ForCausalLM architecture: gated short convolutions interleaved with full and sliding-window grouped-query attention.
Fine-Tune LFM3-1.7B
Follow the installation instructions, then run the recipe from the repository root:
Choose a Workflow
Configuration
NeMo AutoModel runs its own Lfm3ForCausalLM, so the checkpoint loads without remote code on the pinned Transformers release. Full-attention layers rotate half of each attention head, as the released weights were trained. Transformers 5.23.0 reads the released config.json as a full rotation, which changes its outputs on long inputs. Checkpoints exported by NeMo AutoModel record the rotation as proportional and load correctly in both.
Packed Sequences and Context Parallelism
The packed recipe trains on Tulu-3 rows of 131,072 tokens with cp_size: 8 on one node with 8 GPUs:
Each row holds many documents. Positions restart at every document, and the short convolutions, full attention, and sliding-window attention all stay inside a document, so a document gives the same outputs packed as it does alone.
With context parallelism, each GPU holds one contiguous shard of the row, 16,384 tokens here. A layer that needs tokens before the shard fetches only the part of the shard’s first document that earlier GPUs hold: 6 tokens for a convolution, up to 255 keys and values for sliding-window attention, and the whole part for full attention.
Packed THD rows work at any cp_size, with one row per GPU when cp_size > 1. Unpacked batches also run with cp_size > 1. Their padding counts as a document boundary, so real tokens never attend to or convolve over padding. Batched (non-THD) packing needs a FlashAttention backend, and its document indices are honored under context parallelism too.
With distributed.activation_checkpointing: true, whole decoder layers are recomputed in the backward pass, the convolution layers included. A row of 131,072 tokens then trains at every cp_size, down to a whole row on one GPU. The following table shows measurements with the packed recipe, activation checkpointing, and 8 rows per step on 8x H100 80 GB:
The four layouts give the same training loss on the same rows: over 100 steps it differs by 2.5e-5 on average between cp_size values, as between two runs of one layout.
Model Reference
Model Architecture
Available Models
The following Hugging Face checkpoint is available:
Related Resources
Refer to the following guide for related workflows: