Streaming Statistics for Uncertainty Quantification#
An ensemble or a Monte Carlo dropout model produces several predictions for the
same input. You can summarize those predictions without retaining every field.
PhysicsNeMo provides tensor-mode Mean and Variance accumulators in
physicsnemo.metrics.general.ensemble_metrics. They also accept
ShardTensor inputs, so you can reduce ensemble members while keeping a
large spatial field distributed.
The model or recipe still controls training, sampling, decoding, and application decisions. The accumulators only compute statistics. Refer to Uncertainty Quantification for choosing a prediction method and Guardrails for using scores in an application policy.
Set mode="tensor" explicitly; the default constructors retain legacy
behavior.
Accumulate Predictions#
For a model ensemble, decode every member into the same units and point order before updating. This example uses synthetic fields and checks the result against a direct tensor calculation:
import torch
from physicsnemo.metrics.general.ensemble_metrics import Variance
stats = Variance(mode="tensor", correction=0)
for member in range(3):
prediction = torch.arange(8, dtype=torch.float32) + member
stats.update(prediction, dim=None)
mean = stats.mean
variance = stats.finalize()
std = stats.finalize(std=True)
torch.testing.assert_close(mean, torch.arange(8, dtype=torch.float32) + 1)
torch.testing.assert_close(variance, torch.full((8,), 2.0 / 3.0))
dim=None contributes one field per call. dim=0 instead treats axis zero
as a batch of observations and removes that axis from the result. For example,
an input of shape (members, points, channels) produces statistics of shape
(points, channels). Choose correction=0 for the population variance of
an equally weighted ensemble, or correction=1 for sample variance.
Updates detach tensors from their computation graphs. By default, accumulation
promotes half precision to float32 and retains float64. The first
successful update fixes the dtype; use dtype=torch.float64 to request it
explicitly. Empty reads and insufficient observations for the requested
correction raise errors. reset() starts a new accumulation.
combine(other) combines disjoint observations of the same output elements
and layout; it does not concatenate spatial partitions.
For concrete dropout, keep the model in evaluation mode and enable only its
ConcreteDropout layers during the sampling loop. Feed each prediction into
the same accumulator. The runnable streaming predictions example
demonstrates ensembles, dropout, and GP heads, including restoring dropout
modes.
Streaming keeps the statistics state proportional to one output field rather than to the number of predictions. It avoids stacking all outputs and lets statistics stay on the prediction device. It does not reduce the number of model evaluations, and a speed improvement depends on the workload.
Combine Gaussian Process Predictions#
A single Gaussian process (GP) head already returns a mean and a predictive variance. Use those directly. For an equally weighted mixture of GP predictions, total variance includes both the average component variance and the population variance of the component means:
The following function accepts an iterator of (mean, variance) pairs from
GP heads or any other models that supply these moments:
from physicsnemo.metrics.general.ensemble_metrics import Mean, Variance
def summarize_mixture(predictions):
between = Variance(mode="tensor", correction=0)
within = Mean(mode="tensor")
for mean, variance in predictions:
between.update(mean, dim=None)
within.update(variance, dim=None)
return between.mean, between.finalize() + within.finalize()
For VariationalGPHead, supply prediction.mean and
prediction.variance from head.predict(features). For a field head,
supply its mean and total predictive variance. These are variances, not
standard deviations. Under an affine output transform with scale s, the
mean scales by s and the variance by s**2.
The within-component term can already include epistemic and observation-noise variance. It is not necessarily an aleatoric term. For a single component, the between-component variance is zero and the result preserves that component’s predictive variance.
Compute Observables Before Their Statistics#
To estimate uncertainty in a derived quantity, such as drag or force, compute that quantity for each member and then accumulate its statistics. Averaging a pointwise variance field does not give the variance of a spatial average or integral. For a GP-based field observable, use cross-point covariance or joint predictive samples; pointwise variances alone cannot recover correlations.
The force statistics example computes forces with autograd before accumulating detached statistics. The application controls the gradient context and can still use the original forces in a training loss.
Distribute Members and Space#
Ordinary PyTorch tensors remain local in tensor mode, even when
DistributedManager is initialized. A ShardTensor describes a logical
global tensor and its distribution. Reductions communicate according to its
placements and the observation axis. Spatial axes that you do not reduce
remain sharded. There is no accumulator local() or synchronize() call.
Save the following as distributed_statistics.py and run it with
torchrun --standalone --nproc-per-node=4 distributed_statistics.py. Each rank
holds two members and three spatial points from a global (4, 6) field:
import torch
from torch.distributed.tensor import Replicate, Shard
from physicsnemo.distributed import DistributedManager
from physicsnemo.domain_parallel import ShardTensor
from physicsnemo.metrics.general.ensemble_metrics import Variance
DistributedManager.initialize()
dm = DistributedManager()
mesh = dm.initialize_mesh([2, 2], ["member", "space"])
member_rank, space_rank = mesh.get_coordinate()
member_ids = torch.arange(2, device=dm.device) + 2 * member_rank
point_ids = torch.arange(3, device=dm.device) + 3 * space_rank
local_values = member_ids[:, None].float() + point_ids[None, :]
values = ShardTensor.from_local(
local_values, mesh, [Shard(0), Shard(1)],
sharding_shapes={0: [(2, 3), (2, 3)], 1: [(2, 3), (2, 3)]},
)
stats = Variance(mode="tensor", correction=0)
stats.update(values, dim=0)
variance = stats.finalize()
assert variance.placements == (Replicate(), Shard(0))
torch.testing.assert_close(
variance.to_local(), torch.full((3,), 1.25, device=dm.device)
)
DistributedManager.cleanup()
All ranks on the participating mesh must follow the same collective update
schedule. When streaming unequal member counts, an exhausted rank contributes
a zero-length shard of the global observation batch with matching shape
metadata. It must not skip the update. to_local() reads a result’s local
shard; it does not turn a distributed statistic into an independent statistic.
For simulations that evolve over time, the reduction axes define the question. Reduce simulations and space to compute root mean square at each time. Reduce simulations and stream over time to compute pointwise variance. Alternatively, retain time moments per simulation and compose them at the end using the law of total variance. Equal time counts allow equal weighting; unequal time counts need weighted composition. These choices have different state and communication costs.
The distributed moments example checks these compositions, including uneven and empty partitions, against direct references. It supports two-rank member-only or space-only layouts and a four-rank combined layout.
Use Moments in Conformal Calibration#
Conformal calibration uses held-out labeled data to turn predictions and a chosen error score into prediction intervals. The existing conformal APIs accept the final tensors directly. For example, the normalized-error score uses a predicted standard deviation as its scale:
from physicsnemo.experimental.uq.conformal import (
NormalizedErrorScore, PointwiseCalibrator,
)
calibrator = PointwiseCalibrator(NormalizedErrorScore(), alpha=0.1)
for mean, variance, target in calibration_cases:
calibrator.update(mean, target, aux={"sigma": variance.sqrt()})
predictor = calibrator.finalize()
lower, upper = predictor.predict_interval(
query_mean, aux={"sigma": query_variance.sqrt()}
)
Here, each item in calibration_cases is one held-out case with aligned
points and decoded units. Use the same mean and scale construction for
calibration and inference. A GP mixture uses its total variance above.
NormalizedErrorScore floors the scale internally; choose its
scale_floor in the decoded output units. Coverage depends on the score,
calibration sample, and exchangeability assumptions, not on the accumulator.
In a distributed workflow, a spatial shard is not a separate calibration case.
The distributed calibration example
gathers final mean and variance fields with full_tensor() on all ranks,
then calibrates on rank zero. Gathering each complete case has a memory cost;
the example does not make the conformal calibrator distributed.
Extend the Accumulator#
Mean and Variance compose TensorAccumulator with MeanRule and
VarianceRule. The accumulator handles detachment, precision, layout checks,
and state ownership. A rule defines summarize, merge, and result.
For example, a weighted-mean rule retains a weighted sum and a weight sum.
Refer to the runnable weighted mean example
for a rule that works with ordinary tensors and ShardTensor.
A new rule must define a valid merge for independent observations and complete distributed reductions before nonlinear arithmetic. The accumulator does not supply a distributed algorithm for arbitrary rules. Histograms are a possible extension, not an additional built-in rule in this implementation.
Refer to the metrics API reference for the complete tensor-mode contracts.