`bridge.recipes.llama.llama3`#

Module Contents#

Classes#

Llama3CommonKwargs

Typed options accepted by Llama3 family recipe helpers.

Functions#

`llama32_1b_pretrain_config`	Return a pre-training config for Llama 3.2 1B.
`llama32_3b_pretrain_config`	Return a pre-training config for Llama 3.2 3B.
`llama3_8b_pretrain_config`	Return a pre-training config for Llama 3 8B.
`llama3_8b_16k_pretrain_config`	Return a pre-training config for Llama 3 8B 16K.
`llama3_8b_64k_pretrain_config`	Return a pre-training config for Llama 3 8B 64K.
`llama3_8b_128k_pretrain_config`	Return a pre-training config for Llama 3 8B 128K.
`llama3_70b_pretrain_config`	Return a pre-training config for Llama 3 70B.
`llama3_70b_16k_pretrain_config`	Return a pre-training config for Llama 3 70B 16K.
`llama3_70b_64k_pretrain_config`	Return a pre-training config for Llama 3 70B 64K.
`llama31_8b_pretrain_config`	Return a pre-training config for Llama 3.1 8B.
`llama31_70b_pretrain_config`	Return a pre-training config for Llama 3.1 70B.
`llama31_405b_pretrain_config`	Return a pre-training config for Llama 3.1 405B.
`_llama3_common`	Create a pre-training configuration for Llama3 family models using a given HuggingFace path.

Data#

`SEQUENCE_LENGTH_16K`
`SEQUENCE_LENGTH_64K`
`SEQUENCE_LENGTH_128K`

API#

class bridge.recipes.llama.llama3.Llama3CommonKwargs#

Bases: typing_extensions.TypedDict

Typed options accepted by Llama3 family recipe helpers.

Initialization

Initialize self. See help(type(self)) for accurate signature.

hf_path: str#: None

dir: Optional[str]#: None

name: str#: None

data_paths: Optional[List[str]]#: None

data_args_path: Optional[str]#: None

train_data_path: Optional[List[str]]#: None

valid_data_path: Optional[List[str]]#: None

test_data_path: Optional[List[str]]#: None

per_split_data_args_path: Optional[str]#: None

mock: bool#: None

tensor_parallelism: int#: None

pipeline_parallelism: int#: None

pipeline_parallelism_dtype: Optional[torch.dtype]#: None

virtual_pipeline_parallelism: Optional[int]#: None

context_parallelism: int#: None

sequence_parallelism: bool#: None

use_megatron_fsdp: bool#: None

account_for_embedding_in_pipeline_split: bool#: None

account_for_loss_in_pipeline_split: bool#: None

train_iters: int#: None

global_batch_size: int#: None

micro_batch_size: int#: None

seq_length: int#: None

lr: float#: None

min_lr: float#: None

lr_warmup_iters: int#: None

lr_decay_iters: Optional[int]#: None

eval_interval: int#: None

save_interval: int#: None

use_null_tokenizer: bool#: None

precision_config: Optional[Union[megatron.bridge.training.mixed_precision.MixedPrecisionConfig, str]]#: None

comm_overlap_config: Optional[megatron.bridge.training.comm_overlap.CommOverlapConfig]#: None

bridge.recipes.llama.llama3.SEQUENCE_LENGTH_16K: int#: 16384

bridge.recipes.llama.llama3.SEQUENCE_LENGTH_64K: int#: 65536

bridge.recipes.llama.llama3.SEQUENCE_LENGTH_128K: int#: 131072

bridge.recipes.llama.llama3.llama32_1b_pretrain_config(

**user_kwargs: typing_extensions.Unpack[bridge.recipes.llama.llama3.Llama3CommonKwargs],

) → megatron.bridge.training.config.ConfigContainer#

Return a pre-training config for Llama 3.2 1B.