> This page is for version Nightly (default).
> For other versions, use one of these documentation indexes:
> - Nightly (default): https://docs.nvidia.com/nemo/automodel/nightly/llms.txt
> - Latest: https://docs.nvidia.com/nemo/automodel/latest/llms.txt
> - 0.7.0 · 26.10: https://docs.nvidia.com/nemo/automodel/v0.7/llms.txt
> - 0.5.0 · 26.06: https://docs.nvidia.com/nemo/automodel/v0.5/llms.txt
> - 0.4.0 · 26.04: https://docs.nvidia.com/nemo/automodel/v0.4/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/automodel/llms.txt.

# LFM3-1.7B

> Use NeMo AutoModel to fine-tune LFM3-1.7B for LLM workflows with checked-in text generation recipes, including Tulu-3 packed to 131,072 tokens with context parallelism.

[LFM3-1.7B](https://huggingface.co/LiquidAI/LFM3-1.7B) has a checked-in NeMo AutoModel recipe for text generation.
The Hugging Face configuration declares the `Lfm3ForCausalLM` architecture: gated short convolutions interleaved with full and sliding-window grouped-query attention.

## Fine-Tune LFM3-1.7B

Follow the [installation instructions](/nemo/automodel/nightly/get-started/installation), then run the recipe from the repository root:

```bash
automodel examples/llm_finetune/lfm3/lfm3_1p7b_squad.yaml --nproc-per-node 8
```

## Choose a Workflow

| Goal                                                    | Start Here                                                                                                                                                |
| ------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Run the primary recipe for this model                   | Use the [recipe configuration](https://github.com/NVIDIA-NeMo/Automodel/blob/main/examples/llm_finetune/lfm3/lfm3_1p7b_squad.yaml).                       |
| Train on long packed sequences with context parallelism | Use the [131,072-token Tulu-3 recipe](https://github.com/NVIDIA-NeMo/Automodel/blob/main/examples/llm_finetune/lfm3/lfm3_1p7b_tulu3_packed128k_cp8.yaml). |

### Configuration

| Setting  | Configuration                   |
| -------- | ------------------------------- |
| Hardware | -                               |
| Strategy | FSDP2; `tp_size=1`, `cp_size=1` |
| Nodes    | -                               |
| Features | -                               |
| Advanced | -                               |

> **Note**
>
> NeMo AutoModel runs its own `Lfm3ForCausalLM`, so the checkpoint loads without remote code on the pinned Transformers release. Full-attention layers rotate half of each attention head, as the released weights were trained. Transformers 5.23.0 reads the released `config.json` as a full rotation, which changes its outputs on long inputs. Checkpoints exported by NeMo AutoModel record the rotation as `proportional` and load correctly in both.

## Packed Sequences and Context Parallelism

The packed recipe trains on Tulu-3 rows of 131,072 tokens with `cp_size: 8` on one node with 8 GPUs:

```bash
automodel examples/llm_finetune/lfm3/lfm3_1p7b_tulu3_packed128k_cp8.yaml --nproc-per-node 8
```

| Setting     | Configuration                                                                         |
| ----------- | ------------------------------------------------------------------------------------- |
| Hardware    | 8x H100 80 GB, one node                                                               |
| Strategy    | FSDP2; `tp_size=1`, `cp_size=8`                                                       |
| Data        | Tulu-3 shuffled with a fixed seed, then THD-packed to 131,072 tokens; 4 rows per step |
| Attention   | FlashAttention 4 (`attn_implementation: flash_attention_4`)                           |
| Throughput  | 167K tokens/s, 3.1 s per step                                                         |
| Peak memory | 66 GiB per GPU                                                                        |

Each row holds many documents. Positions restart at every document, and the short convolutions, full attention, and sliding-window attention all stay inside a document, so a document gives the same outputs packed as it does alone.

With context parallelism, each GPU holds one contiguous shard of the row, 16,384 tokens here. A layer that needs tokens before the shard fetches only the part of the shard's first document that earlier GPUs hold: 6 tokens for a convolution, up to 255 keys and values for sliding-window attention, and the whole part for full attention.

Packed THD rows work at any `cp_size`, with one row per GPU when `cp_size > 1`. Unpacked batches also run with `cp_size > 1`. Their padding counts as a document boundary, so real tokens never attend to or convolve over padding. Batched (non-THD) packing needs a FlashAttention backend, and its document indices are honored under context parallelism too.

With `distributed.activation_checkpointing: true`, whole decoder layers are recomputed in the backward pass, the convolution layers included. A row of 131,072 tokens then trains at every `cp_size`, down to a whole row on one GPU. The following table shows measurements with the packed recipe, activation checkpointing, and 8 rows per step on 8x H100 80 GB:

| `cp_size` | Tokens per GPU | Peak Memory per GPU | Throughput    |
| --------- | -------------- | ------------------- | ------------- |
| 1         | 131,072        | 39 GiB              | 155K tokens/s |
| 2         | 65,536         | 28 GiB              | 152K tokens/s |
| 4         | 32,768         | 19 GiB              | 142K tokens/s |
| 8         | 16,384         | 15 GiB              | 128K tokens/s |

The four layouts give the same training loss on the same rows: over 100 steps it differs by 2.5e-5 on average between `cp_size` values, as between two runs of one layout.

## Model Reference

### Model Architecture

| Property     | Value             |
| ------------ | ----------------- |
| Task         | Text generation   |
| Architecture | `Lfm3ForCausalLM` |

### Available Models

The following Hugging Face checkpoint is available:

* [`LiquidAI/LFM3-1.7B`](https://huggingface.co/LiquidAI/LFM3-1.7B)

## Related Resources

Refer to the following guide for related workflows:

* [LLM Fine-Tuning Guide](/nemo/automodel/nightly/recipes-e2e-examples/sft-peft)