nemo_automodel.components._peft.module_matcher#
Module Contents#
Classes#
Matches Modules to apply PEFT adapters on. |
Functions#
Return whether the pattern (target module to add LoRA) matches the key (model weight name). |
API#
- nemo_automodel.components._peft.module_matcher.wildcard_match(pattern, key)[source]#
Return whether the pattern (target module to add LoRA) matches the key (model weight name).
Example:#
>>> wildcard_match("*.layers.0.*.linear_qkv", "decoder.layers.0.self_attention.linear_qkv") True >>> wildcard_match("*.layers.0.*.linear_qkv", "decoder.layers.1.self_attention.linear_qkv") False
- class nemo_automodel.components._peft.module_matcher.ModuleMatcher[source]#
Matches Modules to apply PEFT adapters on.
- Parameters:
target_modules (List[str], optional) – A list of module names to apply LoRA to. Defaults to all linear layers [‘linear_qkv’, ‘linear_proj’, ‘linear_fc1’, ‘linear_fc2’]. - ‘linear_qkv’: Apply LoRA to the fused linear layer used for query, key, and value projections in self-attention. - ‘linear_proj’: Apply LoRA to the linear layer used for projecting the output of self-attention. - ‘linear_fc1’: Apply LoRA to the first fully-connected layer in MLP. - ‘linear_fc2’: Apply LoRA to the second fully-connected layer in MLP. Target modules can also contain wildcards. For example, you can specify target_modules=[’.layers.0..linear_qkv’, ‘.layers.1..linear_qkv’] to add LoRA to only linear_qkv on the first two layers.
exclude_modules (List[str], optional) – A list of module names to exclude from applying LoRA to.
match_all_linear (bool, optional) – Whether to match all linear layers.
is_causal_lm (bool, optional) – Whether the model is a causal language model.
- target_modules: List[str]#
‘field(…)’
- exclude_modules: List[str]#
‘field(…)’
- match_all_linear: bool#
‘field(…)’
- is_causal_lm: bool#
‘field(…)’