nemo_automodel.components.models.baichuan.configuration#
Module Contents#
Classes#
API#
- class nemo_automodel.components.models.baichuan.configuration.BaichuanConfig(
- vocab_size=125696,
- hidden_size=4096,
- intermediate_size=11008,
- num_hidden_layers=32,
- num_attention_heads=32,
- hidden_act='silu',
- max_position_embeddings=4096,
- initializer_range=0.02,
- rms_norm_eps=1e-06,
- use_cache=True,
- pad_token_id=0,
- bos_token_id=1,
- eos_token_id=2,
- tie_word_embeddings=False,
- z_loss_weight=0,
- **kwargs,
Bases:
transformers.configuration_utils.PretrainedConfigInitialization
- model_type#
‘baichuan’
- keys_to_ignore_at_inference#
[‘past_key_values’]