Adapters API#

Core#

Adapter Networks#

Adapter Strategies#

class nemo.core.classes.mixins.adapter_mixin_strategies.AbstractAdapterStrategy

Bases: ABC

forward(
input: torch.Tensor,
adapter: torch.nn.Module,
*,
module: AdapterModuleMixin,
)

Forward method that defines how the output of the adapter should be merged with the input, or if it should be merged at all.

Also provides the module that called this strategy - thereby allowing access to all other adapters in the calling module. This can be useful if one adapter is a meta adapter, that combines the outputs of various adapters. In such a case, the input can be forwarded across all other adapters, collecting their outputs, and those outputs can then be merged via some strategy. For example, refer to :

Parameters:

  • input – Original output tensor of the module, or the output of the previous adapter (if more than one adapters are enabled).

  • adapter – The adapter module that is currently required to perform the forward pass.

  • module – The calling module, in its entirety. It is a module that implements AdapterModuleMixin, therefore the strategy can access all other adapters in this module via module.adapter_layer.

Returns:

The result tensor, after one of the active adapters has finished its forward passes.

class nemo.core.classes.mixins.adapter_mixin_strategies.ReturnResultAdapterStrategy

Bases: AbstractAdapterStrategy

An implementation of an adapter strategy that simply returns the result of the adapter. Supports stochastic

forward(
input: torch.Tensor,
adapter: torch.nn.Module,
*,
module: AdapterModuleMixin,
)

A basic strategy, which simply returns the result of the adapter’s calculation as the output.

Parameters:

  • input – Original output tensor of the module, or the output of the previous adapter (if more than one adapters are enabled).

  • adapter – The adapter module that is currently required to perform the forward pass.

  • module – The calling module, in its entirety. It is a module that implements AdapterModuleMixin, therefore the strategy can access all other adapters in this module via module.adapter_layer.

Returns:

The result tensor, after one of the active adapters has finished its forward passes.

compute_output(
input: torch.Tensor | List[torch.Tensor] | Tuple[torch.Tensor] | Dict[str, Any],
adapter: torch.nn.Module,
*,
module: AdapterModuleMixin,
) → torch.Tensor

Compute the output of a single adapter to some input.

Parameters:

  • input – Original output tensor of the module, or the output of the previous adapter (if more than one adapters are enabled).

  • adapter – The adapter module that is currently required to perform the forward pass.

  • module – The calling module, in its entirety. It is a module that implements AdapterModuleMixin, therefore the strategy can access all other adapters in this module via module.adapter_layer.

Returns:

The result tensor, after one of the active adapters has finished its forward passes.

class nemo.core.classes.mixins.adapter_mixin_strategies.ResidualAddAdapterStrategy(
stochastic_depth: float = 0.0,
l2_lambda: float = 0.0,
)

Bases: AbstractAdapterStrategy

An implementation of residual addition of an adapter module with its input. Supports stochastic depth regularization.

forward(
input: torch.Tensor,
adapter: torch.nn.Module,
*,
module: AdapterModuleMixin,
)

A basic strategy, comprising of a residual connection over the input, after forward pass by the underlying adapter.

Parameters:

  • input – Original output tensor of the module, or the output of the previous adapter (if more than one adapters are enabled).

  • adapter – The adapter module that is currently required to perform the forward pass.

  • module – The calling module, in its entirety. It is a module that implements AdapterModuleMixin, therefore the strategy can access all other adapters in this module via module.adapter_layer.

Returns:

The result tensor, after one of the active adapters has finished its forward passes.

compute_output(
input: torch.Tensor,
adapter: torch.nn.Module,
*,
module: AdapterModuleMixin,
) → torch.Tensor

Compute the output of a single adapter to some input.

Parameters:

  • input – Original output tensor of the module, or the output of the previous adapter (if more than one adapters are enabled).

  • adapter – The adapter module that is currently required to perform the forward pass.

  • module – The calling module, in its entirety. It is a module that implements AdapterModuleMixin, therefore the strategy can access all other adapters in this module via module.adapter_layer.

Returns:

The result tensor, after one of the active adapters has finished its forward passes.

apply_stochastic_depth(
output: torch.Tensor,
input: torch.Tensor,
adapter: torch.nn.Module,
*,
module: AdapterModuleMixin,
)

Compute and apply stochastic depth if probability is greater than 0.

Parameters:

  • output – The result tensor, after one of the active adapters has finished its forward passes.

  • input – Original output tensor of the module, or the output of the previous adapter (if more than one adapters are enabled).

  • adapter – The adapter module that is currently required to perform the forward pass.

  • module – The calling module, in its entirety. It is a module that implements AdapterModuleMixin, therefore the strategy can access all other adapters in this module via module.adapter_layer.

Returns:

The result tensor, after stochastic depth has been potentially applied to it.

compute_auxiliary_losses(
output: torch.Tensor,
input: torch.Tensor,
adapter: torch.nn.Module,
*,
module: AdapterModuleMixin,
)

Compute any auxiliary losses and preserve it in the tensor registry.

Parameters:

  • output – The result tensor, after one of the active adapters has finished its forward passes.

  • input – Original output tensor of the module, or the output of the previous adapter (if more than one adapters are enabled).

  • adapter – The adapter module that is currently required to perform the forward pass.

  • module – The calling module, in its entirety. It is a module that implements AdapterModuleMixin, therefore the strategy can access all other adapters in this module via module.adapter_layer.

On this page