> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/automodel/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/automodel/_mcp/server.

# nemo_automodel.components.models.qwen2.model

Custom Qwen2 model implementation for NeMo Automodel.

This module provides a self-contained Qwen2 implementation with separate
HuggingFace-style q/k/v and gate/up projections.

Example (YAML):

```python
model:
  _target_: nemo_automodel.NeMoAutoModelForCausalLM.from_pretrained
  pretrained_model_name_or_path: Qwen/Qwen2.5-7B
```

## Module Contents

### Classes

| Name                                                                                         | Description                                                                           |
| -------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------- |
| [`Qwen2Attention`](#nemo_automodel-components-models-qwen2-model-Qwen2Attention)             | Multi-headed attention with separate QKV projections — HuggingFace default layout.    |
| [`Qwen2DecoderLayer`](#nemo_automodel-components-models-qwen2-model-Qwen2DecoderLayer)       | Single Qwen2 decoder layer with RMSNorm, attention, and MLP.                          |
| [`Qwen2ForCausalLM`](#nemo_automodel-components-models-qwen2-model-Qwen2ForCausalLM)         | Qwen2 model with causal language modeling head.                                       |
| [`Qwen2Model`](#nemo_automodel-components-models-qwen2-model-Qwen2Model)                     | Qwen2 transformer model (embeddings + decoder layers + norm).                         |
| [`Qwen2PreTrainedModel`](#nemo_automodel-components-models-qwen2-model-Qwen2PreTrainedModel) | Abstract class for Qwen2 pretrained models.                                           |
| [`Qwen2SeparateMLP`](#nemo_automodel-components-models-qwen2-model-Qwen2SeparateMLP)         | SwiGLU MLP with separate gate\_proj and up\_proj -- identical to HuggingFace default. |

### Data

[`ModelClass`](#nemo_automodel-components-models-qwen2-model-ModelClass)

[`__all__`](#nemo_automodel-components-models-qwen2-model-__all__)

[`check_model_inputs`](#nemo_automodel-components-models-qwen2-model-check_model_inputs)

### API

```python
class nemo_automodel.components.models.qwen2.model.Qwen2Attention(
    config: transformers.Qwen2Config,
    layer_idx: int,
    backend: 'BackendConfig' | None = None
)
```

**Bases:** `Module`

Multi-headed attention with separate QKV projections — HuggingFace default layout.

**`attention_dropout`** `= config.attention_dropout`

---

**`backend`** `= backend or BackendConfig()`

---

**`head_dim`**

---

**`k_proj`**

---

**`num_key_value_groups`**

---

**`o_proj`**

---

**`q_proj`**

---

**`rope_backend`** `= self.backend.rope`

---

**`rope_fusion`** `= self.backend.rope_fusion`

---

**`scaling`** `= self.head_dim ** -0.5`

---

**`sliding_window`**

---

**`v_proj`**

---

```python
nemo_automodel.components.models.qwen2.model.Qwen2Attention.forward(
    hidden_states: torch.Tensor,
    position_embeddings: tuple[torch.Tensor, torch.Tensor],
    attention_mask: torch.Tensor | None,
    past_key_values: transformers.cache_utils.Cache | None = None,
    cache_position: torch.LongTensor | None = None,
    kwargs: transformers.processing_utils.Unpack[transformers.utils.TransformersKwargs] = {}
) -> tuple[torch.Tensor, torch.Tensor]
```

Run Qwen2 attention over padded BSHD or packed THD states.

**Parameters:**

**`hidden_states`** `torch.Tensor`

Hidden states `[B, S, H]` or packed local states
`[T, H]`. `B` is batch, `S` is sequence, `T` is local
total tokens, and `H` is hidden size.

---

**`position_embeddings`** `tuple[torch.Tensor, torch.Tensor]`

RoPE tensors `(cos, sin)` or fused
`(cos, sin, freqs_cis)`. Local tensors use `[B, S, D]` or
`[T, D]` and the fused raw table uses `[S, 1, 1, D]`.

---

**`attention_mask`** `torch.Tensor | None`

Padded attention mask for BSHD; THD uses cumulative
document lengths from `kwargs`.

---

**`past_key_values`** `Cache | None` — default: None

Optional BSHD KV cache; unsupported for THD.

---

**`cache_position`** `torch.LongTensor | None` — default: None

Optional BSHD cache positions `[S]`.

---

**`**kwargs`** `Unpack[TransformersKwargs]` — default: \{}

THD requires `qkv_format='thd'` and `cu_seqlens`
`[N + 1]`; CP additionally supplies `cp_size` and `cp_rank`.

---

**Returns:** `torch.Tensor`

Attention output shaped like `hidden_states` and optional BSHD

```python
class nemo_automodel.components.models.qwen2.model.Qwen2DecoderLayer(
    config: transformers.Qwen2Config,
    layer_idx: int,
    backend: nemo_automodel.components.models.common.BackendConfig
)
```

**Bases:** `GradientCheckpointingLayer`

Single Qwen2 decoder layer with RMSNorm, attention, and MLP.

**`attention_type`** `= config.layer_types[layer_idx]`

---

**`hidden_size`** `= config.hidden_size`

---

**`input_layernorm`**

---

**`mlp`** `= Qwen2SeparateMLP(config=config)`

---

**`post_attention_layernorm`**

---

**`self_attn`**

---

```python
nemo_automodel.components.models.qwen2.model.Qwen2DecoderLayer.forward(
    hidden_states: torch.Tensor,
    attention_mask: torch.Tensor | None = None,
    position_ids: torch.LongTensor | None = None,
    past_key_values: transformers.cache_utils.Cache | None = None,
    use_cache: bool | None = False,
    cache_position: torch.LongTensor | None = None,
    position_embeddings: tuple[torch.Tensor, torch.Tensor] | None = None,
    kwargs: transformers.processing_utils.Unpack[transformers.utils.TransformersKwargs] = {}
) -> torch.Tensor
```

```python
class nemo_automodel.components.models.qwen2.model.Qwen2ForCausalLM(
    config: transformers.Qwen2Config,
    backend: nemo_automodel.components.models.common.BackendConfig | None = None
)
```

**Bases:** [HFCheckpointingMixin](/nemo-automodel/nemo_automodel/components/models/common/hf_checkpointing_mixin#nemo_automodel-components-models-common-hf_checkpointing_mixin-HFCheckpointingMixin), [Qwen2PreTrainedModel](#nemo_automodel-components-models-qwen2-model-Qwen2PreTrainedModel)

Qwen2 model with causal language modeling head.

Uses separate q/k/v and gate/up projections -- HuggingFace layout.

**`_pp_plan`** `= {'lm_head': (['hidden_states'], ['logits'])}`

---

**`_tied_weights_keys`** `= {'lm_head.weight': 'model.embed_tokens.weight'}`

---

**`_tp_plan`** `= {'lm_head': 'colwise_rep'}`

---

**`backend`** `= backend or BackendConfig()`

---

**`lm_head`**

---

**`model`** `= Qwen2Model(config=config, backend=(self.backend))`

---

**`state_dict_adapter`** `= Qwen2StateDictAdapter(config=(self.config))`

---

**`tie_word_embeddings_support`** `TieSupport = TieSupport.BOTH`

---

**`vocab_size`** `= config.vocab_size`

---

```python
nemo_automodel.components.models.qwen2.model.Qwen2ForCausalLM.forward(
    input_ids: torch.LongTensor | None = None,
    attention_mask: torch.Tensor | None = None,
    position_ids: torch.LongTensor | None = None,
    past_key_values: transformers.cache_utils.Cache | None = None,
    inputs_embeds: torch.FloatTensor | None = None,
    labels: torch.LongTensor | None = None,
    use_cache: bool | None = None,
    output_attentions: bool | None = None,
    output_hidden_states: bool | None = None,
    return_dict: bool | None = None,
    cache_position: torch.LongTensor | None = None,
    logits_to_keep: typing.Union[int, torch.Tensor] = 0,
    kwargs: transformers.processing_utils.Unpack[transformers.utils.TransformersKwargs] = {}
) -> transformers.modeling_outputs.CausalLMOutputWithPast
```

Run causal LM projection for BSHD or packed THD inputs.

**Parameters:**

**`input_ids`** `torch.LongTensor | None` — default: None

Token IDs `[B, S]` or packed local IDs `[T]`.

---

**`attention_mask`** `torch.Tensor | None` — default: None

Optional padded mask. THD uses `cu_seqlens`.

---

**`position_ids`** `torch.LongTensor | None` — default: None

Position IDs `[B, S]` or packed local IDs `[T]`.

---

**`past_key_values`** `Cache | None` — default: None

Optional BSHD KV cache; unsupported for THD.

---

**`inputs_embeds`** `torch.FloatTensor | None` — default: None

Optional hidden inputs `[B, S, H]` or `[T, H]`.

---

**`labels`** `torch.LongTensor | None` — default: None

Optional labels `[B, S]` or packed `[T]`.

---

**`use_cache`** `bool | None` — default: None

Whether to update the BSHD KV cache.

---

**`output_attentions`** `bool | None` — default: None

Whether to request attention outputs.

---

**`output_hidden_states`** `bool | None` — default: None

Whether to return per-layer hidden states.

---

**`return_dict`** `bool | None` — default: None

Whether to return `CausalLMOutputWithPast`.

---

**`cache_position`** `torch.LongTensor | None` — default: None

Optional BSHD cache positions `[S]`.

---

**`logits_to_keep`** `Union[int, torch.Tensor]` — default: 0

Positions to project from hidden size `H` to
vocabulary size `V`.

---

**`**kwargs`** `Unpack[TransformersKwargs]` — default: \{}

THD metadata. `cu_seqlens` is `[N + 1]` and CP adds
`cp_size` and `cp_rank`.

---

**Returns:** `CausalLMOutputWithPast`

Causal LM output with BSHD logits `[B, S, V]`. Packed local

```python
nemo_automodel.components.models.qwen2.model.Qwen2ForCausalLM.get_input_embeddings()
```

```python
nemo_automodel.components.models.qwen2.model.Qwen2ForCausalLM.get_output_embeddings()
```

```python
nemo_automodel.components.models.qwen2.model.Qwen2ForCausalLM.set_input_embeddings(
    value
)
```

```python
nemo_automodel.components.models.qwen2.model.Qwen2ForCausalLM.set_output_embeddings(
    new_embeddings
)
```

```python
nemo_automodel.components.models.qwen2.model.Qwen2ForCausalLM.tie_weights(
    _args: object = (),
    _kwargs: object = {}
) -> None
```

```python
class nemo_automodel.components.models.qwen2.model.Qwen2Model(
    config: transformers.Qwen2Config,
    backend: nemo_automodel.components.models.common.BackendConfig
)
```

**Bases:** [Qwen2PreTrainedModel](#nemo_automodel-components-models-qwen2-model-Qwen2PreTrainedModel)

Qwen2 transformer model (embeddings + decoder layers + norm).

**`embed_tokens`**

---

**`has_sliding_layers`** `= 'sliding_attention' in self.config.layer_types`

---

**`layers`**

---

**`norm`**

---

**`padding_idx`** `= config.pad_token_id`

---

**`rotary_emb`**

---

**`vocab_size`** `= config.vocab_size`

---

```python
nemo_automodel.components.models.qwen2.model.Qwen2Model.forward(
    input_ids: torch.LongTensor | None = None,
    attention_mask: torch.Tensor | None = None,
    position_ids: torch.LongTensor | None = None,
    past_key_values: transformers.cache_utils.Cache | None = None,
    inputs_embeds: torch.FloatTensor | None = None,
    use_cache: bool | None = None,
    output_attentions: bool | None = None,
    output_hidden_states: bool | None = None,
    return_dict: bool | None = None,
    cache_position: torch.LongTensor | None = None,
    kwargs: transformers.processing_utils.Unpack[transformers.utils.TransformersKwargs] = {}
) -> transformers.modeling_outputs.BaseModelOutputWithPast
```

Run the Qwen2 decoder in padded BSHD or packed THD layout.

**Parameters:**

**`input_ids`** `torch.LongTensor | None` — default: None

Token IDs `[B, S]` or packed local IDs `[T]`.

---

**`attention_mask`** `torch.Tensor | None` — default: None

Optional padded mask `[B, S]`. THD uses packed
document boundaries instead.

---

**`position_ids`** `torch.LongTensor | None` — default: None

Position IDs `[B, S]` or packed local IDs `[T]`.

---

**`past_key_values`** `Cache | None` — default: None

Optional BSHD generation cache; unsupported for THD.

---

**`inputs_embeds`** `torch.FloatTensor | None` — default: None

Alternative hidden inputs `[B, S, H]` or `[T, H]`.

---

**`use_cache`** `bool | None` — default: None

Whether to update the BSHD KV cache.

---

**`output_attentions`** `bool | None` — default: None

Whether to request attention outputs.

---

**`output_hidden_states`** `bool | None` — default: None

Whether to retain per-layer hidden states.

---

**`return_dict`** `bool | None` — default: None

Whether to return `BaseModelOutputWithPast`.

---

**`cache_position`** `torch.LongTensor | None` — default: None

Optional BSHD cache positions `[S]`.

---

**`**kwargs`** `Unpack[TransformersKwargs]` — default: \{}

THD metadata including `qkv_format`, `cu_seqlens`,
`max_seqlen`, `cp_size`, and `cp_rank`.

---

**Returns:** `BaseModelOutputWithPast`

Decoder output with final states `[B, S, H]` or packed local

```python
class nemo_automodel.components.models.qwen2.model.Qwen2PreTrainedModel()
```

**Bases:** `PreTrainedModel`

Abstract class for Qwen2 pretrained models.

**`_can_record_outputs`**

---

**`_no_split_modules`** `= ['Qwen2DecoderLayer']`

---

**`_skip_keys_device_placement`** `= ['past_key_values']`

---

**`base_model_prefix`** `= 'model'`

---

```python
class nemo_automodel.components.models.qwen2.model.Qwen2SeparateMLP(
    config: transformers.Qwen2Config
)
```

**Bases:** `Module`

SwiGLU MLP with separate gate\_proj and up\_proj -- identical to HuggingFace default.

**`act_fn`** `= ACT2FN[config.hidden_act]`

---

**`down_proj`**

---

**`gate_proj`**

---

**`hidden_size`** `= config.hidden_size`

---

**`intermediate_size`** `= config.intermediate_size`

---

**`up_proj`**

---

```python
nemo_automodel.components.models.qwen2.model.Qwen2SeparateMLP.forward(
    x: torch.Tensor
) -> torch.Tensor
```

```python
nemo_automodel.components.models.qwen2.model.ModelClass = Qwen2ForCausalLM
```

```python
nemo_automodel.components.models.qwen2.model.__all__ = ['Qwen2ForCausalLM']
```

```python
nemo_automodel.components.models.qwen2.model.check_model_inputs = get_check_model_inputs_decorator()
```