bridge.recipes.deepseek.deepseek_v2_lite#

Module Contents#

Functions#

model_config

Configure the DeepSeek-V2-Lite (16B) model.

pretrain_config

Create a pre-training configuration for DeepSeek-V2-Lite (16B) model.

API#

bridge.recipes.deepseek.deepseek_v2_lite.model_config(
tensor_parallelism: int = 1,
pipeline_parallelism: int = 1,
pipeline_parallelism_dtype: Optional[torch.dtype] = None,
virtual_pipeline_parallelism: Optional[int] = None,
context_parallelism: int = 1,
expert_parallelism: int = 8,
sequence_parallelism: bool = False,
recompute_granularity: str = 'full',
recompute_method: str = 'uniform',
recompute_num_layers: int = 1,
) → megatron.bridge.models.deepseek.DeepSeekV2LiteModelProvider#

Configure the DeepSeek-V2-Lite (16B) model.

Parameters:

  • tensor_parallelism (int) – Degree of tensor model parallelism.

  • pipeline_parallelism (int) – Degree of pipeline model parallelism.

  • pipeline_parallelism_dtype (Optional[torch.dtype]) – Data type for pipeline parallelism.

  • virtual_pipeline_parallelism (Optional[int]) – Size of virtual pipeline parallelism.

  • context_parallelism (int) – Degree of context parallelism.

  • expert_parallelism (int) – Degree of expert model parallelism for MoE.

  • sequence_parallelism (bool) – Whether to use sequence parallelism.

  • recompute_granularity (str) – Granularity of activation recomputation.

  • recompute_method (str) – Method for activation recomputation.

  • recompute_num_layers (int) – Number of layers to recompute.

Returns:

Configuration for the DeepSeek-V2-Lite model.

Return type:

DeepSeekV2LiteModelProvider

bridge.recipes.deepseek.deepseek_v2_lite.pretrain_config(
dir: Optional[str] = None,
name: str = 'default',
data_paths: Optional[List[str]] = None,
data_args_path: Optional[str] = None,
train_data_path: Optional[List[str]] = None,
valid_data_path: Optional[List[str]] = None,
test_data_path: Optional[List[str]] = None,
per_split_data_args_path: Optional[str] = None,
mock: bool = False,
tensor_parallelism: int = 1,
pipeline_parallelism: int = 1,
pipeline_parallelism_dtype: Optional[torch.dtype] = None,
virtual_pipeline_parallelism: Optional[int] = None,
context_parallelism: int = 1,
expert_parallelism: int = 8,
sequence_parallelism: bool = False,
use_megatron_fsdp: bool = False,
train_iters: int = 1000000,
global_batch_size: int = 512,
micro_batch_size: int = 1,
seq_length: int = 4096,
lr: float = 0.0003,
min_lr: float = 3e-05,
lr_warmup_iters: int = 2000,
lr_decay_iters: Optional[int] = None,
precision_config: Optional[Union[megatron.bridge.training.mixed_precision.MixedPrecisionConfig, str]] = 'bf16_mixed',
comm_overlap_config: Optional[megatron.bridge.training.comm_overlap.CommOverlapConfig] = None,
recompute_granularity: str = 'full',
recompute_method: str = 'uniform',
recompute_num_layers: int = 1,
) → megatron.bridge.training.config.ConfigContainer#

Create a pre-training configuration for DeepSeek-V2-Lite (16B) model.

Parameters:

  • dir (Optional[str]) – Base directory for saving logs and checkpoints.

  • name (str) – Name of the pre-training run.

  • data_paths (Optional[List[str]]) – List of paths to dataset files. If None, mock data will be used.

  • data_args_path (Optional[str]) – Path to file containing data arguments.

  • train_data_path (Optional[List[str]]) – List of training data paths.

  • valid_data_path (Optional[List[str]]) – List of validation data paths.

  • test_data_path (Optional[List[str]]) – List of test data paths.

  • per_split_data_args_path (Optional[str]) – Path to JSON file with per-split data configuration.

  • mock (bool) – Whether to use mock data. If True, ignores data_paths.

  • tensor_parallelism (int) – Degree of tensor model parallelism.

  • pipeline_parallelism (int) – Degree of pipeline model parallelism.

  • pipeline_parallelism_dtype (Optional[torch.dtype]) – Data type for pipeline parallelism.

  • virtual_pipeline_parallelism (Optional[int]) – Size of virtual pipeline parallelism.

  • context_parallelism (int) – Degree of context parallelism to be passed to model_config.

  • expert_parallelism (int) – Degree of expert model parallelism for MoE.

  • sequence_parallelism (bool) – Whether to use sequence parallelism.

  • train_iters (int) – Total number of training iterations.

  • global_batch_size (int) – Global batch size for training.

  • micro_batch_size (int) – Micro batch size for training.

  • seq_length (int) – Sequence length for training data.

  • lr (float) – Learning rate.

  • min_lr (float) – Minimum learning rate for cosine decay.

  • lr_warmup_iters (int)

  • precision_config (Optional[Union[MixedPrecisionConfig, str]]) – Precision configuration for the model.

  • recompute_granularity (str) – Granularity of activation recomputation.

  • recompute_method (str) – Method for activation recomputation.

  • recompute_num_layers (int) – Number of layers to recompute.

Returns:

Configuration for pre-training.

Return type:

ConfigContainer