Streaming Statistics for Uncertainty Quantification#

An ensemble or a Monte Carlo dropout model produces several predictions for the same input. You can summarize those predictions without retaining every field. PhysicsNeMo provides tensor-mode Mean and Variance accumulators in physicsnemo.metrics.general.ensemble_metrics. They also accept ShardTensor inputs, so you can reduce ensemble members while keeping a large spatial field distributed.

The model or recipe still controls training, sampling, decoding, and application decisions. The accumulators only compute statistics. Refer to Uncertainty Quantification for choosing a prediction method and Guardrails for using scores in an application policy.

Set mode="tensor" explicitly; the default constructors retain legacy behavior.

Accumulate Predictions#

For a model ensemble, decode every member into the same units and point order before updating. This example uses synthetic fields and checks the result against a direct tensor calculation:

import torch
from physicsnemo.metrics.general.ensemble_metrics import Variance

stats = Variance(mode="tensor", correction=0)
for member in range(3):
    prediction = torch.arange(8, dtype=torch.float32) + member
    stats.update(prediction, dim=None)

mean = stats.mean
variance = stats.finalize()
std = stats.finalize(std=True)
torch.testing.assert_close(mean, torch.arange(8, dtype=torch.float32) + 1)
torch.testing.assert_close(variance, torch.full((8,), 2.0 / 3.0))

dim=None contributes one field per call. dim=0 instead treats axis zero as a batch of observations and removes that axis from the result. For example, an input of shape (members, points, channels) produces statistics of shape (points, channels). Choose correction=0 for the population variance of an equally weighted ensemble, or correction=1 for sample variance.

Updates detach tensors from their computation graphs. By default, accumulation promotes half precision to float32 and retains float64. The first successful update fixes the dtype; use dtype=torch.float64 to request it explicitly. Empty reads and insufficient observations for the requested correction raise errors. reset() starts a new accumulation. combine(other) combines disjoint observations of the same output elements and layout; it does not concatenate spatial partitions.

For concrete dropout, keep the model in evaluation mode and enable only its ConcreteDropout layers during the sampling loop. Feed each prediction into the same accumulator. The runnable streaming predictions example demonstrates ensembles, dropout, and GP heads, including restoring dropout modes.

Streaming keeps the statistics state proportional to one output field rather than to the number of predictions. It avoids stacking all outputs and lets statistics stay on the prediction device. It does not reduce the number of model evaluations, and a speed improvement depends on the workload.

Combine Gaussian Process Predictions#

A single Gaussian process (GP) head already returns a mean and a predictive variance. Use those directly. For an equally weighted mixture of GP predictions, total variance includes both the average component variance and the population variance of the component means:

\[\operatorname{Var}(Y) = \frac{1}{K}\sum_{k=1}^{K}\sigma_k^2 + \frac{1}{K}\sum_{k=1}^{K}(\mu_k-\bar{\mu})^2.\]

The following function accepts an iterator of (mean, variance) pairs from GP heads or any other models that supply these moments:

from physicsnemo.metrics.general.ensemble_metrics import Mean, Variance

def summarize_mixture(predictions):
    between = Variance(mode="tensor", correction=0)
    within = Mean(mode="tensor")
    for mean, variance in predictions:
        between.update(mean, dim=None)
        within.update(variance, dim=None)
    return between.mean, between.finalize() + within.finalize()

For VariationalGPHead, supply prediction.mean and prediction.variance from head.predict(features). For a field head, supply its mean and total predictive variance. These are variances, not standard deviations. Under an affine output transform with scale s, the mean scales by s and the variance by s**2.

The within-component term can already include epistemic and observation-noise variance. It is not necessarily an aleatoric term. For a single component, the between-component variance is zero and the result preserves that component’s predictive variance.

Compute Observables Before Their Statistics#

To estimate uncertainty in a derived quantity, such as drag or force, compute that quantity for each member and then accumulate its statistics. Averaging a pointwise variance field does not give the variance of a spatial average or integral. For a GP-based field observable, use cross-point covariance or joint predictive samples; pointwise variances alone cannot recover correlations.

The force statistics example computes forces with autograd before accumulating detached statistics. The application controls the gradient context and can still use the original forces in a training loss.

Distribute Members and Space#

Ordinary PyTorch tensors remain local in tensor mode, even when DistributedManager is initialized. A ShardTensor describes a logical global tensor and its distribution. Reductions communicate according to its placements and the observation axis. Spatial axes that you do not reduce remain sharded. There is no accumulator local() or synchronize() call.

Save the following as distributed_statistics.py and run it with torchrun --standalone --nproc-per-node=4 distributed_statistics.py. Each rank holds two members and three spatial points from a global (4, 6) field:

import torch
from torch.distributed.tensor import Replicate, Shard
from physicsnemo.distributed import DistributedManager
from physicsnemo.domain_parallel import ShardTensor
from physicsnemo.metrics.general.ensemble_metrics import Variance

DistributedManager.initialize()
dm = DistributedManager()
mesh = dm.initialize_mesh([2, 2], ["member", "space"])
member_rank, space_rank = mesh.get_coordinate()
member_ids = torch.arange(2, device=dm.device) + 2 * member_rank
point_ids = torch.arange(3, device=dm.device) + 3 * space_rank
local_values = member_ids[:, None].float() + point_ids[None, :]
values = ShardTensor.from_local(
    local_values, mesh, [Shard(0), Shard(1)],
    sharding_shapes={0: [(2, 3), (2, 3)], 1: [(2, 3), (2, 3)]},
)

stats = Variance(mode="tensor", correction=0)
stats.update(values, dim=0)
variance = stats.finalize()
assert variance.placements == (Replicate(), Shard(0))
torch.testing.assert_close(
    variance.to_local(), torch.full((3,), 1.25, device=dm.device)
)
DistributedManager.cleanup()

All ranks on the participating mesh must follow the same collective update schedule. When streaming unequal member counts, an exhausted rank contributes a zero-length shard of the global observation batch with matching shape metadata. It must not skip the update. to_local() reads a result’s local shard; it does not turn a distributed statistic into an independent statistic.

For simulations that evolve over time, the reduction axes define the question. Reduce simulations and space to compute root mean square at each time. Reduce simulations and stream over time to compute pointwise variance. Alternatively, retain time moments per simulation and compose them at the end using the law of total variance. Equal time counts allow equal weighting; unequal time counts need weighted composition. These choices have different state and communication costs.

The distributed moments example checks these compositions, including uneven and empty partitions, against direct references. It supports two-rank member-only or space-only layouts and a four-rank combined layout.

Use Moments in Conformal Calibration#

Conformal calibration uses held-out labeled data to turn predictions and a chosen error score into prediction intervals. The existing conformal APIs accept the final tensors directly. For example, the normalized-error score uses a predicted standard deviation as its scale:

from physicsnemo.experimental.uq.conformal import (
    NormalizedErrorScore, PointwiseCalibrator,
)

calibrator = PointwiseCalibrator(NormalizedErrorScore(), alpha=0.1)
for mean, variance, target in calibration_cases:
    calibrator.update(mean, target, aux={"sigma": variance.sqrt()})
predictor = calibrator.finalize()
lower, upper = predictor.predict_interval(
    query_mean, aux={"sigma": query_variance.sqrt()}
)

Here, each item in calibration_cases is one held-out case with aligned points and decoded units. Use the same mean and scale construction for calibration and inference. A GP mixture uses its total variance above. NormalizedErrorScore floors the scale internally; choose its scale_floor in the decoded output units. Coverage depends on the score, calibration sample, and exchangeability assumptions, not on the accumulator.

In a distributed workflow, a spatial shard is not a separate calibration case. The distributed calibration example gathers final mean and variance fields with full_tensor() on all ranks, then calibrates on rank zero. Gathering each complete case has a memory cost; the example does not make the conformal calibrator distributed.

Extend the Accumulator#

Mean and Variance compose TensorAccumulator with MeanRule and VarianceRule. The accumulator handles detachment, precision, layout checks, and state ownership. A rule defines summarize, merge, and result. For example, a weighted-mean rule retains a weighted sum and a weight sum. Refer to the runnable weighted mean example for a rule that works with ordinary tensors and ShardTensor.

A new rule must define a valid merge for independent observations and complete distributed reductions before nonlinear arithmetic. The accumulator does not supply a distributed algorithm for arbitrary rules. Histograms are a possible extension, not an additional built-in rule in this implementation.

Refer to the metrics API reference for the complete tensor-mode contracts.