- class modulus.loss.loss.CausalLossNorm(ord: int = 2, eps: float = 1.0, n_chunks=10)[source]
Bases:
Loss
Causal loss function for pointwise data Computes the p-th order loss of each output tensor
- Parameters
ord (int) – Order of the loss. For example, ord=2 would be the L2 loss.
eps (float) – Causal parameter determining the slopeness of the temporal weights. “eps=1.0” would be default value.
n_chunks (int) – Number of chunks splitting the temporal domain evenly.
- forward(invar: Dict[str, Tensor], pred_outvar: Dict[str, Tensor], true_outvar: Dict[str, Tensor], lambda_weighting: Dict[str, Tensor], step: int) → Dict[str, Tensor][source]
Defines the computation performed at every call.
Should be overridden by all subclasses.
NoteAlthough the recipe for forward pass needs to be defined within this function, one should call the
Module
instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.
- class modulus.loss.loss.DecayedIntegralLossNorm(start_ord: int = 2, end_ord: int = 1, decay_steps: int = 1000, decay_rate: float = 0.95)[source]
Bases:
DecayedLossNorm
Loss function for integral data where the norm of the loss is decayed from a start value to an end value.
- Parameters
start_ord (int) – Order of the loss when current iteration is zero.
end_ord (int) – Order of the loss to decay to.
decay_steps (int) – Number of steps to take for each decay_rate.
decay_rate – The rate of decay from start_ord to end_ord. The current ord will be given by ord = start_ord - (start_ord - end_ord) * (1.0 - decay_rate**(current_step / decay_steps)).
- forward(list_invar: List[Dict[str, Tensor]], list_pred_outvar: List[Dict[str, Tensor]], list_true_outvar: List[Dict[str, Tensor]], list_lambda_weighting: List[Dict[str, Tensor]], step: int) → Dict[str, Tensor][source]
Defines the computation performed at every call.
Should be overridden by all subclasses.
NoteAlthough the recipe for forward pass needs to be defined within this function, one should call the
Module
instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.
- class modulus.loss.loss.DecayedLossNorm(start_ord: int = 2, end_ord: int = 1, decay_steps: int = 1000, decay_rate: float = 0.95)[source]
Bases:
Loss
Base class for decayed loss norm
- class modulus.loss.loss.DecayedPointwiseLossNorm(start_ord: int = 2, end_ord: int = 1, decay_steps: int = 1000, decay_rate: float = 0.95)[source]
Bases:
DecayedLossNorm
Loss function for pointwise data where the norm of the loss is decayed from a start value to an end value.
- Parameters
start_ord (int) – Order of the loss when current iteration is zero.
end_ord (int) – Order of the loss to decay to.
decay_steps (int) – Number of steps to take for each decay_rate.
decay_rate – The rate of decay from start_ord to end_ord. The current ord will be given by ord = start_ord - (start_ord - end_ord) * (1.0 - decay_rate**(current_step / decay_steps)).
- forward(invar: Dict[str, Tensor], pred_outvar: Dict[str, Tensor], true_outvar: Dict[str, Tensor], lambda_weighting: Dict[str, Tensor], step: int) → Dict[str, Tensor][source]
Defines the computation performed at every call.
Should be overridden by all subclasses.
NoteAlthough the recipe for forward pass needs to be defined within this function, one should call the
Module
instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.
- class modulus.loss.loss.IntegralLossNorm(ord: int = 2)[source]
Bases:
Loss
L-p loss function for integral data Computes the p-th order loss of each output tensor
- Parameters
ord (int) – Order of the loss. For example, ord=2 would be the L2 loss.
- forward(list_invar: List[Dict[str, Tensor]], list_pred_outvar: List[Dict[str, Tensor]], list_true_outvar: List[Dict[str, Tensor]], list_lambda_weighting: List[Dict[str, Tensor]], step: int) → Dict[str, Tensor][source]
Defines the computation performed at every call.
Should be overridden by all subclasses.
NoteAlthough the recipe for forward pass needs to be defined within this function, one should call the
Module
instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.
- class modulus.loss.loss.Loss[source]
Bases:
Module
Base class for all loss functions
- forward(invar: Dict[str, Tensor], pred_outvar: Dict[str, Tensor], true_outvar: Dict[str, Tensor], lambda_weighting: Dict[str, Tensor], step: int) → Dict[str, Tensor][source]
Defines the computation performed at every call.
Should be overridden by all subclasses.
NoteAlthough the recipe for forward pass needs to be defined within this function, one should call the
Module
instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.
- class modulus.loss.loss.LossL2(*args, **kwargs)[source]
Bases:
Function
- static backward(ctx, grad_output)[source]
Defines a formula for differentiating the operation with backward mode automatic differentiation (alias to the vjp function).
This function is to be overridden by all subclasses.
It must accept a context
ctx
as the first argument, followed by as many outputs as theforward()
returned (None will be passed in for non tensor outputs of the forward function), and it should return as many tensors, as there were inputs toforward()
. Each argument is the gradient w.r.t the given output, and each returned value should be the gradient w.r.t. the corresponding input. If an input is not a Tensor or is a Tensor not requiring grads, you can just pass None as a gradient for that input.The context can be used to retrieve tensors saved during the forward pass. It also has an attribute
ctx.needs_input_grad
as a tuple of booleans representing whether each input needs gradient. E.g.,backward()
will havectx.needs_input_grad[0] = True
if the first input toforward()
needs gradient computated w.r.t. the output.
- static forward(ctx, pred_outvar: Tensor, true_outvar: Tensor, lambda_weighting: Tensor, area: Tensor)[source]
Performs the operation.
This function is to be overridden by all subclasses.
It must accept a context ctx as the first argument, followed by any number of arguments (tensors or other types).
The context can be used to store arbitrary data that can be then retrieved during the backward pass. Tensors should not be stored directly on ctx (though this is not currently enforced for backward compatibility). Instead, tensors should be saved either with
ctx.save_for_backward()
if they are intended to be used inbackward
(equivalently,vjp
) orctx.save_for_forward()
if they are intended to be used for injvp
.
- class modulus.loss.loss.PointwiseLossNorm(ord: int = 2)[source]
Bases:
Loss
L-p loss function for pointwise data Computes the p-th order loss of each output tensor
- Parameters
ord (int) – Order of the loss. For example, ord=2 would be the L2 loss.
- forward(invar: Dict[str, Tensor], pred_outvar: Dict[str, Tensor], true_outvar: Dict[str, Tensor], lambda_weighting: Dict[str, Tensor], step: int) → Dict[str, Tensor][source]
Defines the computation performed at every call.
Should be overridden by all subclasses.
NoteAlthough the recipe for forward pass needs to be defined within this function, one should call the
Module
instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.
- class modulus.loss.aggregator.Aggregator(params, num_losses, weights)[source]
Bases:
Module
Base class for loss aggregators
- class modulus.loss.aggregator.GradNorm(params, num_losses, alpha=1.0, weights=None)[source]
Bases:
Aggregator
GradNorm for loss aggregation Reference: “Chen, Z., Badrinarayanan, V., Lee, C.Y. and Rabinovich, A., 2018, July. Gradnorm: Gradient normalization for adaptive loss balancing in deep multitask networks. In International Conference on Machine Learning (pp. 794-803). PMLR.”
- forward(losses: Dict[str, Tensor], step: int) → Tensor[source]
Weights and aggregates the losses using the gradNorm algorithm
- Parameters
losses (Dict[str, torch.Tensor]) – A dictionary of losses.
step (int) – Optimizer step.
- Returns
loss – Aggregated loss.
- Return type
torch.Tensor
- class modulus.loss.aggregator.HomoscedasticUncertainty(params, num_losses, weights=None)[source]
Bases:
Aggregator
Homoscedastic task uncertainty for loss aggregation Reference: “Reference: Kendall, A., Gal, Y. and Cipolla, R., 2018. Multi-task learning using uncertainty to weigh losses for scene geometry and semantics. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 7482-7491).”
- forward(losses: Dict[str, Tensor], step: int) → Tensor[source]
Weights and aggregates the losses using homoscedastic task uncertainty
- Parameters
losses (Dict[str, torch.Tensor]) – A dictionary of losses.
step (int) – Optimizer step.
- Returns
loss – Aggregated loss.
- Return type
torch.Tensor
- class modulus.loss.aggregator.LRAnnealing(params, num_losses, update_freq=1, alpha=0.01, ref_key=None, eps=1e-08, weights=None)[source]
Bases:
Aggregator
Learning rate annealing for loss aggregation References: “Wang, S., Teng, Y. and Perdikaris, P., 2020. Understanding and mitigating gradient pathologies in physics-informed neural networks. arXiv preprint arXiv:2001.04536.”, and “Jin, X., Cai, S., Li, H. and Karniadakis, G.E., 2021. NSFnets (Navier-Stokes flow nets): Physics-informed neural networks for the incompressible Navier-Stokes equations. Journal of Computational Physics, 426, p.109951.”
- forward(losses: Dict[str, Tensor], step: int) → Tensor[source]
Weights and aggregates the losses using the learning rate annealing algorithm
- Parameters
losses (Dict[str, torch.Tensor]) – A dictionary of losses.
step (int) – Optimizer step.
- Returns
loss – Aggregated loss.
- Return type
torch.Tensor
- class modulus.loss.aggregator.NTK(run_per_step: int = 1000, save_name: Optional[str] = None)[source]
Bases:
Module
- forward(constraints, ntk_weights, step)[source]
Defines the computation performed at every call.
Should be overridden by all subclasses.
NoteAlthough the recipe for forward pass needs to be defined within this function, one should call the
Module
instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.
- class modulus.loss.aggregator.Relobralo(params, num_losses, alpha=0.95, beta=0.99, tau=1.0, eps=1e-08, weights=None)[source]
Bases:
Aggregator
Relative loss balancing with random lookback Reference: “Bischof, R. and Kraus, M., 2021. Multi-Objective Loss Balancing for Physics-Informed Deep Learning. arXiv preprint arXiv:2110.09813.”
- forward(losses: Dict[str, Tensor], step: int) → Tensor[source]
Weights and aggregates the losses using the ReLoBRaLo algorithm
- Parameters
losses (Dict[str, torch.Tensor]) – A dictionary of losses.
step (int) – Optimizer step.
- Returns
loss – Aggregated loss.
- Return type
torch.Tensor
- class modulus.loss.aggregator.ResNorm(params, num_losses, alpha=1.0, weights=None)[source]
Bases:
Aggregator
Residual normalization for loss aggregation Contributors: T. Nandi, D. Van Essendelft, M. A. Nabian
- forward(losses: Dict[str, Tensor], step: int) → Tensor[source]
Weights and aggregates the losses using the ResNorm algorithm
- Parameters
losses (Dict[str, torch.Tensor]) – A dictionary of losses.
step (int) – Optimizer step.
- Returns
loss – Aggregated loss.
- Return type
torch.Tensor
- class modulus.loss.aggregator.SoftAdapt(params, num_losses, eps=1e-08, weights=None)[source]
Bases:
Aggregator
SoftAdapt for loss aggregation Reference: “Heydari, A.A., Thompson, C.A. and Mehmood, A., 2019. Softadapt: Techniques for adaptive loss weighting of neural networks with multi-part loss functions. arXiv preprint arXiv: 1912.12355.”
- forward(losses: Dict[str, Tensor], step: int) → Tensor[source]
Weights and aggregates the losses using the original variant of the softadapt algorithm
- Parameters
losses (Dict[str, torch.Tensor]) – A dictionary of losses.
step (int) – Optimizer step.
- Returns
loss – Aggregated loss.
- Return type
torch.Tensor
- class modulus.loss.aggregator.Sum(params, num_losses, weights=None)[source]
Bases:
Aggregator
Loss aggregation by summation
- forward(losses: Dict[str, Tensor], step: int) → Tensor[source]
Aggregates the losses by summation
- Parameters
losses (Dict[str, torch.Tensor]) – A dictionary of losses.
step (int) – Optimizer step.
- Returns
loss – Aggregated loss.
- Return type
torch.Tensor