Modulus Sym Loss

Bases: Loss

Causal loss function for pointwise data Computes the p-th order loss of each output tensor

Parameters

• ord (int) – Order of the loss. For example, ord=2 would be the L2 loss.

• eps (float) – Causal parameter determining the slopeness of the temporal weights. “eps=1.0” would be default value.

• n_chunks (int) – Number of chunks splitting the temporal domain evenly.

forward(invar: Dict[str, Tensor], pred_outvar: Dict[str, Tensor], true_outvar: Dict[str, Tensor], lambda_weighting: Dict[str, Tensor], step: int) → Dict[str, Tensor][source]

Define the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

Bases: DecayedLossNorm

Loss function for integral data where the norm of the loss is decayed from a start value to an end value.

Parameters

• start_ord (int) – Order of the loss when current iteration is zero.

• end_ord (int) – Order of the loss to decay to.

• decay_steps (int) – Number of steps to take for each decay_rate.

• decay_rate – The rate of decay from start_ord to end_ord. The current ord will be given by ord = start_ord - (start_ord - end_ord) * (1.0 - decay_rate**(current_step / decay_steps)).

forward(list_invar: List[Dict[str, Tensor]], list_pred_outvar: List[Dict[str, Tensor]], list_true_outvar: List[Dict[str, Tensor]], list_lambda_weighting: List[Dict[str, Tensor]], step: int) → Dict[str, Tensor][source]

Define the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

Bases: Loss

Base class for decayed loss norm

Bases: DecayedLossNorm

Loss function for pointwise data where the norm of the loss is decayed from a start value to an end value.

Parameters

• start_ord (int) – Order of the loss when current iteration is zero.

• end_ord (int) – Order of the loss to decay to.

• decay_steps (int) – Number of steps to take for each decay_rate.

• decay_rate – The rate of decay from start_ord to end_ord. The current ord will be given by ord = start_ord - (start_ord - end_ord) * (1.0 - decay_rate**(current_step / decay_steps)).

forward(invar: Dict[str, Tensor], pred_outvar: Dict[str, Tensor], true_outvar: Dict[str, Tensor], lambda_weighting: Dict[str, Tensor], step: int) → Dict[str, Tensor][source]

Define the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

Bases: Loss

L-p loss function for integral data Computes the p-th order loss of each output tensor

Parameters

ord (int) – Order of the loss. For example, ord=2 would be the L2 loss.

forward(list_invar: List[Dict[str, Tensor]], list_pred_outvar: List[Dict[str, Tensor]], list_true_outvar: List[Dict[str, Tensor]], list_lambda_weighting: List[Dict[str, Tensor]], step: int) → Dict[str, Tensor][source]

Define the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

Bases: Module

Base class for all loss functions

forward(invar: Dict[str, Tensor], pred_outvar: Dict[str, Tensor], true_outvar: Dict[str, Tensor], lambda_weighting: Dict[str, Tensor], step: int) → Dict[str, Tensor][source]

Define the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

Bases: Function

static backward(ctx, grad_output)[source]

Define a formula for differentiating the operation with backward mode automatic differentiation.

This function is to be overridden by all subclasses. (Defining this function is equivalent to defining the vjp function.)

It must accept a context ctx as the first argument, followed by as many outputs as the forward() returned (None will be passed in for non tensor outputs of the forward function), and it should return as many tensors, as there were inputs to forward(). Each argument is the gradient w.r.t the given output, and each returned value should be the gradient w.r.t. the corresponding input. If an input is not a Tensor or is a Tensor not requiring grads, you can just pass None as a gradient for that input.

The context can be used to retrieve tensors saved during the forward pass. It also has an attribute ctx.needs_input_grad as a tuple of booleans representing whether each input needs gradient. E.g., backward() will have ctx.needs_input_grad[0] = True if the first input to forward() needs gradient computed w.r.t. the output.

static forward(ctx, pred_outvar: Tensor, true_outvar: Tensor, lambda_weighting: Tensor, area: Tensor)[source]

Define the forward of the custom autograd Function.

This function is to be overridden by all subclasses. There are two ways to define forward:

Usage 1 (Combined forward and ctx):

Copy

Copied!

            

            
@staticmethod
def forward(ctx: Any, *args: Any, **kwargs: Any) -> Any:
    pass
        

• It must accept a context ctx as the first argument, followed by any number of arguments (tensors or other types).

• See combining-forward-context for more details

Usage 2 (Separate forward and ctx):

Copy

Copied!

            

            
@staticmethod
def forward(*args: Any, **kwargs: Any) -> Any:
    pass

@staticmethod
def setup_context(ctx: Any, inputs: Tuple[Any, ...], output: Any) -> None:
    pass
        

• The forward no longer accepts a ctx argument.

• Instead, you must also override the torch.autograd.Function.setup_context() staticmethod to handle setting up the ctx object. output is the output of the forward, inputs are a Tuple of inputs to the forward.

• See extending-autograd for more details

The context can be used to store arbitrary data that can be then retrieved during the backward pass. Tensors should not be stored directly on ctx (though this is not currently enforced for backward compatibility). Instead, tensors should be saved either with ctx.save_for_backward() if they are intended to be used in backward (equivalently, vjp) or ctx.save_for_forward() if they are intended to be used for in jvp.

Bases: Loss

L-p loss function for pointwise data Computes the p-th order loss of each output tensor

Parameters

ord (int) – Order of the loss. For example, ord=2 would be the L2 loss.

forward(invar: Dict[str, Tensor], pred_outvar: Dict[str, Tensor], true_outvar: Dict[str, Tensor], lambda_weighting: Dict[str, Tensor], step: int) → Dict[str, Tensor][source]

Define the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

Bases: Module

Base class for loss aggregators

Bases: Aggregator

GradNorm for loss aggregation Reference: “Chen, Z., Badrinarayanan, V., Lee, C.Y. and Rabinovich, A., 2018, July. Gradnorm: Gradient normalization for adaptive loss balancing in deep multitask networks. In International Conference on Machine Learning (pp. 794-803). PMLR.”

forward(losses: Dict[str, Tensor], step: int) → Tensor[source]

Weights and aggregates the losses using the gradNorm algorithm

Parameters

• losses (Dict[str, torch.Tensor]) – A dictionary of losses.

• step (int) – Optimizer step.

Returns

loss – Aggregated loss.

Return type

torch.Tensor

Bases: Aggregator

Homoscedastic task uncertainty for loss aggregation Reference: “Reference: Kendall, A., Gal, Y. and Cipolla, R., 2018. Multi-task learning using uncertainty to weigh losses for scene geometry and semantics. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 7482-7491).”

forward(losses: Dict[str, Tensor], step: int) → Tensor[source]

Weights and aggregates the losses using homoscedastic task uncertainty

Parameters

• losses (Dict[str, torch.Tensor]) – A dictionary of losses.

• step (int) – Optimizer step.

Returns

loss – Aggregated loss.

Return type

torch.Tensor

Bases: Aggregator

Learning rate annealing for loss aggregation References: “Wang, S., Teng, Y. and Perdikaris, P., 2020. Understanding and mitigating gradient pathologies in physics-informed neural networks. arXiv preprint arXiv:2001.04536.”, and “Jin, X., Cai, S., Li, H. and Karniadakis, G.E., 2021. NSFnets (Navier-Stokes flow nets): Physics-informed neural networks for the incompressible Navier-Stokes equations. Journal of Computational Physics, 426, p.109951.”

forward(losses: Dict[str, Tensor], step: int) → Tensor[source]

Weights and aggregates the losses using the learning rate annealing algorithm

Parameters

• losses (Dict[str, torch.Tensor]) – A dictionary of losses.

• step (int) – Optimizer step.

Returns

loss – Aggregated loss.

Return type

torch.Tensor

Bases: Module

forward(constraints, ntk_weights, step)[source]

Define the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

Bases: Aggregator

Relative loss balancing with random lookback Reference: “Bischof, R. and Kraus, M., 2021. Multi-Objective Loss Balancing for Physics-Informed Deep Learning. arXiv preprint arXiv:2110.09813.”

forward(losses: Dict[str, Tensor], step: int) → Tensor[source]

Weights and aggregates the losses using the ReLoBRaLo algorithm

Parameters

• losses (Dict[str, torch.Tensor]) – A dictionary of losses.

• step (int) – Optimizer step.

Returns

loss – Aggregated loss.

Return type

torch.Tensor

Bases: Aggregator

Residual normalization for loss aggregation Contributors: T. Nandi, D. Van Essendelft, M. A. Nabian

forward(losses: Dict[str, Tensor], step: int) → Tensor[source]

Weights and aggregates the losses using the ResNorm algorithm

Parameters

• losses (Dict[str, torch.Tensor]) – A dictionary of losses.

• step (int) – Optimizer step.

Returns

loss – Aggregated loss.

Return type

torch.Tensor

Bases: Aggregator

SoftAdapt for loss aggregation Reference: “Heydari, A.A., Thompson, C.A. and Mehmood, A., 2019. Softadapt: Techniques for adaptive loss weighting of neural networks with multi-part loss functions. arXiv preprint arXiv: 1912.12355.”

forward(losses: Dict[str, Tensor], step: int) → Tensor[source]

Weights and aggregates the losses using the original variant of the softadapt algorithm

Parameters

• losses (Dict[str, torch.Tensor]) – A dictionary of losses.

• step (int) – Optimizer step.

Returns

loss – Aggregated loss.

Return type

torch.Tensor

Bases: Aggregator

Loss aggregation by summation

forward(losses: Dict[str, Tensor], step: int) → Tensor[source]

Aggregates the losses by summation

Parameters

• losses (Dict[str, torch.Tensor]) – A dictionary of losses.

• step (int) – Optimizer step.

Returns

loss – Aggregated loss.

Return type

torch.Tensor