Skip to navigation

LFM3-1.7B has a checked-in NeMo AutoModel recipe for text generation. The Hugging Face configuration declares the Lfm3ForCausalLM architecture: gated short convolutions interleaved with full and sliding-window grouped-query attention.

Fine-Tune LFM3-1.7B

Follow the installation instructions, then run the recipe from the repository root:

automodel examples/llm_finetune/lfm3/lfm3_1p7b_squad.yaml --nproc-per-node 8

Choose a Workflow

GoalStart Here
Run the primary recipe for this modelUse the recipe configuration.
Train on long packed sequences with context parallelismUse the 131,072-token Tulu-3 recipe.

Configuration

SettingConfiguration
Hardware-
StrategyFSDP2; tp_size=1, cp_size=1
Nodes-
Features-
Advanced-

NeMo AutoModel runs its own Lfm3ForCausalLM, so the checkpoint loads without remote code on the pinned Transformers release. Full-attention layers rotate half of each attention head, as the released weights were trained. Transformers 5.23.0 reads the released config.json as a full rotation, which changes its outputs on long inputs. Checkpoints exported by NeMo AutoModel record the rotation as proportional and load correctly in both.

Packed Sequences and Context Parallelism

The packed recipe trains on Tulu-3 rows of 131,072 tokens with cp_size: 8 on one node with 8 GPUs:

automodel examples/llm_finetune/lfm3/lfm3_1p7b_tulu3_packed128k_cp8.yaml --nproc-per-node 8
SettingConfiguration
Hardware8x H100 80 GB, one node
StrategyFSDP2; tp_size=1, cp_size=8
DataTulu-3 shuffled with a fixed seed, then THD-packed to 131,072 tokens; 4 rows per step
AttentionFlashAttention 4 (attn_implementation: flash_attention_4)
Throughput167K tokens/s, 3.1 s per step
Peak memory66 GiB per GPU

Each row holds many documents. Positions restart at every document, and the short convolutions, full attention, and sliding-window attention all stay inside a document, so a document gives the same outputs packed as it does alone.

With context parallelism, each GPU holds one contiguous shard of the row, 16,384 tokens here. A layer that needs tokens before the shard fetches only the part of the shard’s first document that earlier GPUs hold: 6 tokens for a convolution, up to 255 keys and values for sliding-window attention, and the whole part for full attention.

Packed THD rows work at any cp_size, with one row per GPU when cp_size > 1. Unpacked batches also run with cp_size > 1. Their padding counts as a document boundary, so real tokens never attend to or convolve over padding. Batched (non-THD) packing needs a FlashAttention backend, and its document indices are honored under context parallelism too.

With distributed.activation_checkpointing: true, whole decoder layers are recomputed in the backward pass, the convolution layers included. A row of 131,072 tokens then trains at every cp_size, down to a whole row on one GPU. The following table shows measurements with the packed recipe, activation checkpointing, and 8 rows per step on 8x H100 80 GB:

cp_sizeTokens per GPUPeak Memory per GPUThroughput
1131,07239 GiB155K tokens/s
265,53628 GiB152K tokens/s
432,76819 GiB142K tokens/s
816,38415 GiB128K tokens/s

The four layouts give the same training loss on the same rows: over 100 steps it differs by 2.5e-5 on average between cp_size values, as between two runs of one layout.

Model Reference

Model Architecture

PropertyValue
TaskText generation
ArchitectureLfm3ForCausalLM

Available Models

The following Hugging Face checkpoint is available:

Refer to the following guide for related workflows: