nemo_automodel.components.models.flint.config

Local configuration for Flint’s hybrid-attention MoE architecture.

Module Contents

Classes

Name	Description
`FlintConfig`	Architecture settings matching the source Flint checkpoint.

API

class nemo_automodel.components.models.flint.config.FlintConfig()

Bases: PreTrainedConfig

Architecture settings matching the source Flint checkpoint.

The local class removes dependence on checkpoint remote code. Full layers have no RoPE; sliding layers rotate all head channels with NeoX RoPE. The window counts the current token. router_aux_loss_coef is preserved as checkpoint metadata; the native training implementation requires an explicit zero value until the original auxiliary objective is specified.

attention_compute_dtype='fp16' is an explicit TE-only QKV/kernel-output boundary within otherwise FP32 projection compute. It does not change the bulk parameter dtype or enable autocast. Small-gradient training requires an appropriate loss-scaling policy; the default None retains native precision.

attention_bias

bool = False

attention_compute_dtype

str | None = None

attention_dropout

float | int = 0.0

bos_token_id

int | None = None

decoder_sparse_step

int = 1

eos_token_id

int | list[int] | None = None

head_dim

int = 128

hidden_act

str = 'silu'

hidden_size

int = 2048

initializer_range

float = 0.02

intermediate_size

int = 6144

keys_to_ignore_at_inference

= ['past_key_values']

layer_types

list[str] | None = None

max_position_embeddings

int = 262144

mlp_only_layers

list[int] | None = None

model_type

= 'flint'

moe_intermediate_size

int = 512

no_rope_layer_indices

list[int] | None = None

norm_topk_prob

bool = True

num_attention_heads

int = 32

num_experts

int = 256

num_experts_per_tok

int = 6

num_hidden_layers

int = 49

num_key_value_heads

int = 4

output_router_logits

bool = False

pad_token_id

int | None = None

residual_dtype

str | None = None

rms_norm_eps

float = 1e-06

rope_parameters

RopeParameters | dict[str, object] | None = None

router_aux_loss_coef

float = 0.0

router_score_function

str = 'sigmoid'

schedule_num_hidden_layers

int | None = None

shared_expert_intermediate_size

int = 1024

sliding_window

int | None = 513

softmax_type

str = 'learnable'

tie_word_embeddings

bool = False

use_cache

bool = False

use_sliding_window

bool = True

vocab_size

int = 150144

nemo_automodel.components.models.flint.config.FlintConfig.__post_init__(
    kwargs: object = {}
) -> None

Resolve schedules and upstream RoPE fields, then validate the contract.

nemo_automodel.components.models.flint.config.FlintConfig.build(
    backend: nemo_automodel.components.models.common.BackendConfig | None = None
) -> nemo_automodel.components.models.flint.model.FlintForCausalLM

Construct a native model without changing these declarative settings.

Parameters:

backend

BackendConfig | NoneDefaults to None

Runtime kernel choices; defaults to the verified eager/torch path.

Returns: FlintForCausalLM

An initialized native causal language model.

nemo_automodel.components.models.flint.config.FlintConfig.validate_layer_type() -> None

Validate the preserved source schedule independently of active prefix depth.