> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/automodel/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/automodel/_mcp/server.

# nemo_automodel.components.models.qwen3.model

Dense Qwen3 implementation with packed THD context parallelism.

## Module Contents

### Classes

| Name                                                                                         | Description                                                          |
| -------------------------------------------------------------------------------------------- | -------------------------------------------------------------------- |
| [`Qwen3Attention`](#nemo_automodel-components-models-qwen3-model-Qwen3Attention)             | HuggingFace Qwen3 attention extended with a packed THD TE path.      |
| [`Qwen3DecoderLayer`](#nemo_automodel-components-models-qwen3-model-Qwen3DecoderLayer)       | Qwen3 decoder layer using the packed-aware attention implementation. |
| [`Qwen3ForCausalLM`](#nemo_automodel-components-models-qwen3-model-Qwen3ForCausalLM)         | Dense Qwen3 causal LM with packed THD context parallelism.           |
| [`Qwen3Model`](#nemo_automodel-components-models-qwen3-model-Qwen3Model)                     | Dense Qwen3 decoder supporting padded BSHD and packed THD layouts.   |
| [`Qwen3PreTrainedModel`](#nemo_automodel-components-models-qwen3-model-Qwen3PreTrainedModel) | Base class for the dense Qwen3 implementation.                       |

### Data

[`ModelClass`](#nemo_automodel-components-models-qwen3-model-ModelClass)

[`__all__`](#nemo_automodel-components-models-qwen3-model-__all__)

[`check_model_inputs`](#nemo_automodel-components-models-qwen3-model-check_model_inputs)

### API

```python
class nemo_automodel.components.models.qwen3.model.Qwen3Attention(
    config: transformers.Qwen3Config,
    layer_idx: int,
    backend: nemo_automodel.components.models.common.BackendConfig
)
```

**Bases:** `HFQwen3Attention`

HuggingFace Qwen3 attention extended with a packed THD TE path.

```python
nemo_automodel.components.models.qwen3.model.Qwen3Attention.forward(
    hidden_states: torch.Tensor,
    position_embeddings: tuple[torch.Tensor, torch.Tensor] | tuple[torch.Tensor, torch.Tensor, torch.Tensor],
    attention_mask: torch.Tensor | None,
    past_key_values: transformers.cache_utils.Cache | None = None,
    kwargs: transformers.processing_utils.Unpack[transformers.utils.TransformersKwargs] = {}
) -> tuple[torch.Tensor, torch.Tensor | None]
```

Run Qwen3 attention over padded BSHD or packed THD states.

**Parameters:**

Hidden states `[B, S, H]` or packed local states
`[T, H]`. `B` is batch, `S` is sequence, `T` is local
total tokens, and `H` is hidden size.

RoPE tensors `(cos, sin)` or fused
`(cos, sin, freqs_cis)`. Local tensors use `[B, S, D]` or
`[T, D]` and the fused raw table uses `[S, 1, 1, D]`.

Padded attention mask for BSHD; THD uses cumulative
document lengths from `kwargs`.

Optional BSHD KV cache; unsupported for THD.

THD requires `qkv_format='thd'` and `cu_seqlens`
`[N + 1]`; CP additionally supplies `cp_size` and `cp_rank`.

**Returns:** `torch.Tensor`

Attention output shaped like `hidden_states` and optional BSHD

```python
class nemo_automodel.components.models.qwen3.model.Qwen3DecoderLayer(
    config: transformers.Qwen3Config,
    layer_idx: int,
    backend: nemo_automodel.components.models.common.BackendConfig
)
```

**Bases:** `HFQwen3DecoderLayer`

Qwen3 decoder layer using the packed-aware attention implementation.

```python
class nemo_automodel.components.models.qwen3.model.Qwen3ForCausalLM(
    config: transformers.Qwen3Config,
    backend: nemo_automodel.components.models.common.BackendConfig | None = None
)
```

**Bases:** [HFCheckpointingMixin](/nemo-automodel/nemo_automodel/components/models/common/hf_checkpointing_mixin#nemo_automodel-components-models-common-hf_checkpointing_mixin-HFCheckpointingMixin), [Qwen3PreTrainedModel](#nemo_automodel-components-models-qwen3-model-Qwen3PreTrainedModel), `GenerationMixin`

Dense Qwen3 causal LM with packed THD context parallelism.

```python
nemo_automodel.components.models.qwen3.model.Qwen3ForCausalLM.forward(
    input_ids: torch.LongTensor | None = None,
    attention_mask: torch.Tensor | None = None,
    position_ids: torch.LongTensor | None = None,
    past_key_values: transformers.cache_utils.Cache | None = None,
    inputs_embeds: torch.FloatTensor | None = None,
    labels: torch.LongTensor | None = None,
    use_cache: bool | None = None,
    output_attentions: bool | None = None,
    output_hidden_states: bool | None = None,
    return_dict: bool | None = None,
    cache_position: torch.LongTensor | None = None,
    logits_to_keep: int | torch.Tensor = 0,
    kwargs: transformers.processing_utils.Unpack[transformers.utils.TransformersKwargs] = {}
) -> transformers.modeling_outputs.CausalLMOutputWithPast
```

Run causal LM projection for BSHD or packed THD inputs.

**Parameters:**

Token IDs `[B, S]` or packed local IDs `[T]`.

Optional padded mask. THD uses `cu_seqlens`.

Position IDs `[B, S]` or packed local IDs `[T]`.

Optional BSHD KV cache; unsupported for THD.

Optional hidden inputs `[B, S, H]` or `[T, H]`.

Optional labels `[B, S]` or packed `[T]`.

Whether to update the BSHD KV cache.

Whether to request attention outputs.

Whether to return per-layer hidden states.

Whether to return `CausalLMOutputWithPast`.

Optional BSHD cache positions `[S]`.

Positions to project from hidden size `H` to
vocabulary size `V`.

THD metadata. `cu_seqlens` is `[N + 1]` and CP adds
`cp_size` and `cp_rank`.

**Returns:** `CausalLMOutputWithPast`

Causal LM output with BSHD logits `[B, S, V]`. Packed local

```python
nemo_automodel.components.models.qwen3.model.Qwen3ForCausalLM.get_input_embeddings() -> torch.nn.Module
```

```python
nemo_automodel.components.models.qwen3.model.Qwen3ForCausalLM.get_output_embeddings() -> torch.nn.Module
```

```python
nemo_automodel.components.models.qwen3.model.Qwen3ForCausalLM.set_input_embeddings(
    value: torch.nn.Module
) -> None
```

```python
nemo_automodel.components.models.qwen3.model.Qwen3ForCausalLM.set_output_embeddings(
    new_embeddings: torch.nn.Module
) -> None
```

```python
nemo_automodel.components.models.qwen3.model.Qwen3ForCausalLM.tie_weights(
    _args: object = (),
    _kwargs: object = {}
) -> None
```

```python
class nemo_automodel.components.models.qwen3.model.Qwen3Model(
    config: transformers.Qwen3Config,
    backend: nemo_automodel.components.models.common.BackendConfig
)
```

**Bases:** [Qwen3PreTrainedModel](#nemo_automodel-components-models-qwen3-model-Qwen3PreTrainedModel)

Dense Qwen3 decoder supporting padded BSHD and packed THD layouts.

```python
nemo_automodel.components.models.qwen3.model.Qwen3Model.forward(
    input_ids: torch.LongTensor | None = None,
    attention_mask: torch.Tensor | None = None,
    position_ids: torch.LongTensor | None = None,
    past_key_values: transformers.cache_utils.Cache | None = None,
    inputs_embeds: torch.FloatTensor | None = None,
    use_cache: bool | None = None,
    output_attentions: bool | None = None,
    output_hidden_states: bool | None = None,
    return_dict: bool | None = None,
    cache_position: torch.LongTensor | None = None,
    kwargs: transformers.processing_utils.Unpack[transformers.utils.TransformersKwargs] = {}
) -> transformers.modeling_outputs.BaseModelOutputWithPast
```

Run the Qwen3 decoder in padded BSHD or packed THD layout.

**Parameters:**

Token IDs `[B, S]` or packed local IDs `[T]`.

Optional padded mask `[B, S]`. THD uses packed
document boundaries instead.

Position IDs `[B, S]` or packed local IDs `[T]`.

Optional BSHD generation cache; unsupported for THD.

Alternative hidden inputs `[B, S, H]` or `[T, H]`.

Whether to update the BSHD KV cache.

Whether to request attention outputs.

Whether to retain per-layer hidden states.

Whether to return `BaseModelOutputWithPast`.

Optional BSHD cache positions `[S]`.

THD metadata including `qkv_format`, `cu_seqlens`,
`max_seqlen`, `cp_size`, and `cp_rank`.

**Returns:** `BaseModelOutputWithPast`

Decoder output with final states `[B, S, H]` or packed local

```python
class nemo_automodel.components.models.qwen3.model.Qwen3PreTrainedModel()
```

**Bases:** `PreTrainedModel`

Base class for the dense Qwen3 implementation.

```python
nemo_automodel.components.models.qwen3.model.ModelClass = Qwen3ForCausalLM
```

```python
nemo_automodel.components.models.qwen3.model.__all__ = ['Qwen3ForCausalLM']
```

```python
nemo_automodel.components.models.qwen3.model.check_model_inputs = get_check_model_inputs_decorator()
```