bridge.peft.multi_lora#

Multi-adapter LoRA model transform.

class:

MultiLoRA wraps target modules with multi-adapter LoRA layers. All per-adapter state (alpha, rank, weights, routing) lives on the layers and is managed by standalone functions in :mod:multi_lora_layers.

Module Contents#

Classes#

MultiLoRA

Multi-adapter LoRA transform.

Data#

logger

_EXPERT_SKIP_WARNED

API#

bridge.peft.multi_lora.logger#

‘getLogger(…)’

bridge.peft.multi_lora._EXPERT_SKIP_WARNED#

False

class bridge.peft.multi_lora.MultiLoRA#

Bases: megatron.bridge.peft.base.PEFT, megatron.bridge.peft.module_matcher.ModuleMatcher

Multi-adapter LoRA transform.

Parameters:

  • target_modules – Module names or wildcard patterns to apply multi-LoRA to.

  • n_adapters – Maximum number of concurrent adapter slots.

  • dim – LoRA max rank (bottleneck dimension for weight allocation).

  • alpha – Default LoRA scaling parameter.

  • dropout – Dropout probability for the adapter.

  • dropout_position – Where to apply dropout.

  • lora_A_init_method – Initialisation method for the A matrix.

  • lora_B_init_method – Initialisation method for the B matrix.

  • a2a_experimental – Enable experimental all-to-all communication.

  • lora_dtype – Data type for adapter weights.

  • normalize_moe_lora – Unsupported for multi-LoRA; see :meth:__call__.

  • share_expert_adapters – Unsupported for multi-LoRA; see :meth:__call__.

  • experts_shared_outer_loras – Unsupported for multi-LoRA; see :meth:__call__.

target_modules: List[str]#

‘field(…)’

n_adapters: int#

2

dim: int#

32

alpha: int#

32

dropout: float#

0.0

dropout_position: Literal[pre, post]#

‘pre’

lora_A_init_method: str#

‘xavier’

lora_B_init_method: str#

‘zero’

a2a_experimental: bool#

False

lora_dtype: Optional[torch.dtype]#

None

normalize_moe_lora: bool#

False

share_expert_adapters: bool#

False

experts_shared_outer_loras: bool#

False

__call__(model, training: bool = True)#

Apply multi-LoRA, then install MoE slot routing for wrapped expert linears.

transform(
module: torch.nn.Module,
name: Optional[str] = None,
prefix: Optional[str] = None,
) → torch.nn.Module#
adapter_key_filter(key) → bool#