bridge.recipes.llama.h100.llama3

Module Contents

Functions

llama32_1b_pretrain_1gpu_h100_bf16_config	Return a pre-training config for Llama 3.2 1B.
llama32_3b_pretrain_1gpu_h100_bf16_config	Return a pre-training config for Llama 3.2 3B.
llama3_8b_pretrain_2gpu_h100_bf16_config	Return a pre-training config for Llama 3 8B.
llama3_8b_pretrain_16gpu_h100_bf16_16k_config	Return a pre-training config for Llama 3 8B 16K.
llama3_8b_pretrain_32gpu_h100_bf16_64k_config	Return a pre-training config for Llama 3 8B 64K.
llama3_8b_pretrain_64gpu_h100_bf16_128k_config	Return a pre-training config for Llama 3 8B 128K.
_llama3_8b_pretrain_2gpu_h100_low_precision	Return a low precision (FP8 Current Scaling/MXFP8/NVFP4) pre-training config for Llama 3 8B.
llama3_8b_pretrain_2gpu_h100_fp8mx_config	Return a MXFP8 pre-training config for Llama 3 8B on H100.
llama3_8b_pretrain_2gpu_h100_fp8cs_config	Return a FP8 current-scaling pre-training config for Llama 3 8B on H100.
llama3_8b_pretrain_2gpu_h100_nvfp4_config	Return a NVFP4 pre-training config for Llama 3 8B on H100.
llama3_70b_pretrain_32gpu_h100_bf16_config	Return a pre-training config for Llama 3 70B.
llama3_70b_pretrain_32gpu_h100_bf16_deterministic_config	Return a deterministic pre-training config for Llama 3 70B.
llama3_70b_pretrain_32gpu_h100_bf16_16k_config	Return a pre-training config for Llama 3 70B 16K.
llama3_70b_pretrain_256gpu_h100_bf16_64k_config	Return a pre-training config for Llama 3 70B 64K.
llama31_8b_pretrain_2gpu_h100_bf16_config	Return a pre-training config for Llama 3.1 8B.
llama31_70b_pretrain_32gpu_h100_bf16_config	Return a pre-training config for Llama 3.1 70B.
llama31_405b_pretrain_256gpu_h100_bf16_config	Return a pre-training config for Llama 3.1 405B.
llama31_405b_pretrain_256gpu_h100_bf16_deterministic_config	Return a deterministic pre-training config for Llama 3.1 405B.
llama32_1b_sft_1gpu_h100_bf16_config	Return a full SFT config for Llama 3.2 1B.
llama32_3b_sft_1gpu_h100_bf16_config	Return a full SFT config for Llama 3.2 3B.
llama3_8b_sft_2gpu_h100_bf16_config	Return a full SFT config for Llama 3 8B.
llama31_8b_sft_2gpu_h100_bf16_config	Return a full SFT config for Llama 3.1 8B.
llama3_70b_sft_32gpu_h100_bf16_config	Return a full SFT config for Llama 3 70B.
llama31_70b_sft_32gpu_h100_bf16_config	Return a full SFT config for Llama 3.1 70B.
llama31_405b_sft_128gpu_h100_bf16_config	Return a full SFT config for Llama 3.1 405B.
llama32_1b_peft_1gpu_h100_bf16_config	Return a PEFT config for Llama 3.2 1B.
llama32_3b_peft_1gpu_h100_bf16_config	Return a PEFT config for Llama 3.2 3B.
llama3_8b_peft_1gpu_h100_bf16_config	Return a PEFT config for Llama 3 8B.
llama31_8b_peft_1gpu_h100_bf16_config	Return a PEFT config for Llama 3.1 8B.
llama3_70b_peft_8gpu_h100_bf16_config	Return a PEFT config for Llama 3 70B.
llama31_70b_peft_8gpu_h100_bf16_config	Return a PEFT config for Llama 3.1 70B.
llama31_405b_peft_32gpu_h100_bf16_config	Return a PEFT config for Llama 3.1 405B.

Data

SEQUENCE_LENGTH_16K
SEQUENCE_LENGTH_64K
SEQUENCE_LENGTH_128K
__all__

API

bridge.recipes.llama.h100.llama3.SEQUENCE_LENGTH_16K: int

16384

bridge.recipes.llama.h100.llama3.SEQUENCE_LENGTH_64K: int

65536

bridge.recipes.llama.h100.llama3.SEQUENCE_LENGTH_128K: int

131072

bridge.recipes.llama.h100.llama3.llama32_1b_pretrain_1gpu_h100_bf16_config() → megatron.bridge.training.config.ConfigContainer

Return a pre-training config for Llama 3.2 1B.

Recommended parallelism: TP=1, PP=1, CP=1.

bridge.recipes.llama.h100.llama3.llama32_3b_pretrain_1gpu_h100_bf16_config() → megatron.bridge.training.config.ConfigContainer

Return a pre-training config for Llama 3.2 3B.

Recommended parallelism: TP=1, PP=1, CP=1.

bridge.recipes.llama.h100.llama3.llama3_8b_pretrain_2gpu_h100_bf16_config() → megatron.bridge.training.config.ConfigContainer

Return a pre-training config for Llama 3 8B.

Recommended parallelism: TP=1, PP=1, CP=2.

bridge.recipes.llama.h100.llama3.llama3_8b_pretrain_16gpu_h100_bf16_16k_config() → megatron.bridge.training.config.ConfigContainer

Return a pre-training config for Llama 3 8B 16K.

Recommended parallelism: TP=4, PP=2, CP=2, SP=True.

bridge.recipes.llama.h100.llama3.llama3_8b_pretrain_32gpu_h100_bf16_64k_config() → megatron.bridge.training.config.ConfigContainer

Return a pre-training config for Llama 3 8B 64K.

Recommended parallelism: TP=4, PP=2, CP=4, SP=True.

bridge.recipes.llama.h100.llama3.llama3_8b_pretrain_64gpu_h100_bf16_128k_config() → megatron.bridge.training.config.ConfigContainer

Return a pre-training config for Llama 3 8B 128K.

Recommended parallelism: TP=4, PP=2, CP=8, SP=True.

bridge.recipes.llama.h100.llama3._llama3_8b_pretrain_2gpu_h100_low_precision(

mixed_precision_recipe: str,

) → megatron.bridge.training.config.ConfigContainer

Return a low precision (FP8 Current Scaling/MXFP8/NVFP4) pre-training config for Llama 3 8B.

Parameters:

mixed_precision_recipe (str) –

The mixed precision recipe to use. Valid options are:

• ”bf16_with_mxfp8_mixed”

• ”bf16_with_fp8_current_scaling_mixed”

• ”bf16_with_nvfp4_mixed”

Returns:

The pre-training configuration for Llama 3 8B.

Return type:

ConfigContainer

bridge.recipes.llama.h100.llama3.llama3_8b_pretrain_2gpu_h100_fp8mx_config() → megatron.bridge.training.config.ConfigContainer

Return a MXFP8 pre-training config for Llama 3 8B on H100.

bridge.recipes.llama.h100.llama3.llama3_8b_pretrain_2gpu_h100_fp8cs_config() → megatron.bridge.training.config.ConfigContainer

Return a FP8 current-scaling pre-training config for Llama 3 8B on H100.

bridge.recipes.llama.h100.llama3.llama3_8b_pretrain_2gpu_h100_nvfp4_config() → megatron.bridge.training.config.ConfigContainer

Return a NVFP4 pre-training config for Llama 3 8B on H100.

bridge.recipes.llama.h100.llama3.llama3_70b_pretrain_32gpu_h100_bf16_config() → megatron.bridge.training.config.ConfigContainer

Return a pre-training config for Llama 3 70B.

Recommended parallelism: TP=4, PP=4, VPP=5, CP=2, SP=True with CommOverlap.

bridge.recipes.llama.h100.llama3.llama3_70b_pretrain_32gpu_h100_bf16_deterministic_config() → megatron.bridge.training.config.ConfigContainer

Return a deterministic pre-training config for Llama 3 70B.

Wraps :func:llama3_70b_pretrain_config and applies

Func:

~megatron.bridge.recipes.utils.determinism_utils.apply_determinism_overrides. Bit-exact reproducibility also requires the executor-side env vars set by PerfEnvPlugin(deterministic=True).

bridge.recipes.llama.h100.llama3.llama3_70b_pretrain_32gpu_h100_bf16_16k_config() → megatron.bridge.training.config.ConfigContainer

Return a pre-training config for Llama 3 70B 16K.

Recommended parallelism: TP=8, PP=2, CP=2, SP=True with CommOverlap.

bridge.recipes.llama.h100.llama3.llama3_70b_pretrain_256gpu_h100_bf16_64k_config() → megatron.bridge.training.config.ConfigContainer

Return a pre-training config for Llama 3 70B 64K.

Recommended parallelism: TP=8, PP=4, CP=8, SP=True with CommOverlap.

bridge.recipes.llama.h100.llama3.llama31_8b_pretrain_2gpu_h100_bf16_config() → megatron.bridge.training.config.ConfigContainer

Return a pre-training config for Llama 3.1 8B.

Recommended parallelism: TP=1, PP=1, CP=2.

bridge.recipes.llama.h100.llama3.llama31_70b_pretrain_32gpu_h100_bf16_config() → megatron.bridge.training.config.ConfigContainer

Return a pre-training config for Llama 3.1 70B.

Recommended parallelism: TP=4, PP=4, VPP=5, CP=2, SP=True with CommOverlap, seq=128K.

bridge.recipes.llama.h100.llama3.llama31_405b_pretrain_256gpu_h100_bf16_config() → megatron.bridge.training.config.ConfigContainer

Return a pre-training config for Llama 3.1 405B.

Recommended parallelism: TP=8, PP=8, VPP=2, CP=4, SP=True with CommOverlap, seq=128K.

bridge.recipes.llama.h100.llama3.llama31_405b_pretrain_256gpu_h100_bf16_deterministic_config() → megatron.bridge.training.config.ConfigContainer

Return a deterministic pre-training config for Llama 3.1 405B.

Wraps :func:llama31_405b_pretrain_config and applies

Func:

~megatron.bridge.recipes.utils.determinism_utils.apply_determinism_overrides. Bit-exact reproducibility also requires the executor-side env vars set by PerfEnvPlugin(deterministic=True).

bridge.recipes.llama.h100.llama3.llama32_1b_sft_1gpu_h100_bf16_config() → megatron.bridge.training.config.ConfigContainer

Return a full SFT config for Llama 3.2 1B.

Default parallelism: TP=1, PP=1

Returns:

ConfigContainer with all settings pre-configured for Llama 3.2 1B SFT.

bridge.recipes.llama.h100.llama3.llama32_3b_sft_1gpu_h100_bf16_config() → megatron.bridge.training.config.ConfigContainer

Return a full SFT config for Llama 3.2 3B.

Default parallelism: TP=1, PP=1

Returns:

ConfigContainer with all settings pre-configured for Llama 3.2 3B SFT.

bridge.recipes.llama.h100.llama3.llama3_8b_sft_2gpu_h100_bf16_config() → megatron.bridge.training.config.ConfigContainer

Return a full SFT config for Llama 3 8B.

Default parallelism: TP=2, PP=1

Returns:

ConfigContainer with all settings pre-configured for Llama 3 8B SFT.

bridge.recipes.llama.h100.llama3.llama31_8b_sft_2gpu_h100_bf16_config() → megatron.bridge.training.config.ConfigContainer

Return a full SFT config for Llama 3.1 8B.

Default parallelism: TP=2, PP=1

Returns:

ConfigContainer with all settings pre-configured for Llama 3.1 8B SFT.

bridge.recipes.llama.h100.llama3.llama3_70b_sft_32gpu_h100_bf16_config() → megatron.bridge.training.config.ConfigContainer

Return a full SFT config for Llama 3 70B.

Default parallelism: TP=8, PP=4

Returns:

ConfigContainer with all settings pre-configured for Llama 3 70B SFT.

bridge.recipes.llama.h100.llama3.llama31_70b_sft_32gpu_h100_bf16_config() → megatron.bridge.training.config.ConfigContainer

Return a full SFT config for Llama 3.1 70B.

Default parallelism: TP=8, PP=4

Returns:

ConfigContainer with all settings pre-configured for Llama 3.1 70B SFT.

bridge.recipes.llama.h100.llama3.llama31_405b_sft_128gpu_h100_bf16_config() → megatron.bridge.training.config.ConfigContainer

Return a full SFT config for Llama 3.1 405B.

Default parallelism: TP=8, PP=16, SP=True Total: 128 GPUs (16 nodes)

Returns:

ConfigContainer with all settings pre-configured for Llama 3.1 405B SFT.

bridge.recipes.llama.h100.llama3.llama32_1b_peft_1gpu_h100_bf16_config(

peft_scheme: str | megatron.bridge.peft.base.PEFT = 'lora',

) → megatron.bridge.training.config.ConfigContainer

Return a PEFT config for Llama 3.2 1B.

Default parallelism: TP=1, PP=1

Parameters:

peft_scheme – PEFT scheme - “lora”, “dora”, or a custom PEFT instance.

Returns:

ConfigContainer with all settings pre-configured for Llama 3.2 1B PEFT.

bridge.recipes.llama.h100.llama3.llama32_3b_peft_1gpu_h100_bf16_config(

peft_scheme: str | megatron.bridge.peft.base.PEFT = 'lora',

) → megatron.bridge.training.config.ConfigContainer

Return a PEFT config for Llama 3.2 3B.

Default parallelism: TP=1, PP=1

Parameters:

peft_scheme – PEFT scheme - “lora”, “dora”, or a custom PEFT instance.

Returns:

ConfigContainer with all settings pre-configured for Llama 3.2 3B PEFT.

bridge.recipes.llama.h100.llama3.llama3_8b_peft_1gpu_h100_bf16_config(

peft_scheme: str | megatron.bridge.peft.base.PEFT = 'lora',

) → megatron.bridge.training.config.ConfigContainer

Return a PEFT config for Llama 3 8B.

Default parallelism: TP=1, PP=1

Parameters:

peft_scheme – PEFT scheme - “lora”, “dora”, or a custom PEFT instance.

Returns:

ConfigContainer with all settings pre-configured for Llama 3 8B PEFT.

bridge.recipes.llama.h100.llama3.llama31_8b_peft_1gpu_h100_bf16_config(

peft_scheme: str | megatron.bridge.peft.base.PEFT = 'lora',

) → megatron.bridge.training.config.ConfigContainer

Return a PEFT config for Llama 3.1 8B.

Default parallelism: TP=1, PP=1

Parameters:

peft_scheme – PEFT scheme - “lora”, “dora”, or a custom PEFT instance.

Returns:

ConfigContainer with all settings pre-configured for Llama 3.1 8B PEFT.

bridge.recipes.llama.h100.llama3.llama3_70b_peft_8gpu_h100_bf16_config(

peft_scheme: str | megatron.bridge.peft.base.PEFT = 'lora',

) → megatron.bridge.training.config.ConfigContainer

Return a PEFT config for Llama 3 70B.

Default parallelism: TP=8, PP=1

Parameters:

peft_scheme – PEFT scheme - “lora”, “dora”, or a custom PEFT instance.

Returns:

ConfigContainer with all settings pre-configured for Llama 3 70B PEFT.

bridge.recipes.llama.h100.llama3.llama31_70b_peft_8gpu_h100_bf16_config(

peft_scheme: str | megatron.bridge.peft.base.PEFT = 'lora',

) → megatron.bridge.training.config.ConfigContainer

Return a PEFT config for Llama 3.1 70B.

Default parallelism: TP=8, PP=1

Parameters:

peft_scheme – PEFT scheme - “lora”, “dora”, or a custom PEFT instance.

Returns:

ConfigContainer with all settings pre-configured for Llama 3.1 70B PEFT.

bridge.recipes.llama.h100.llama3.llama31_405b_peft_32gpu_h100_bf16_config(

peft_scheme: str | megatron.bridge.peft.base.PEFT = 'lora',

) → megatron.bridge.training.config.ConfigContainer

Return a PEFT config for Llama 3.1 405B.

Default parallelism: TP=4, PP=8, VPP=8, SP=True Total: 32 GPUs (4 nodes)

Parameters:

peft_scheme – PEFT scheme - “lora”, “dora”, or a custom PEFT instance.

Returns:

ConfigContainer with all settings pre-configured for Llama 3.1 405B PEFT.

bridge.recipes.llama.h100.llama3.__all__

[‘llama31_405b_peft_32gpu_h100_bf16_config’, ‘llama31_405b_pretrain_256gpu_h100_bf16_config’, ‘llama…