Comparison Functions

Module: polygraphy.comparator

class Threshold

Bases: object

Base class for a comparison threshold – the criterion a comparison’s metric(s) are checked against. A threshold knows the metric field(s) it applies to (metric_fields), how to turn a result’s metric values into a pass/fail verdict (passed), and how to format a human-readable summary line (describe).

Comparison functions produce thresholds (see BaseCompareFunc.thresholds_for) and results store them, so a saved result can be re-checked against a new threshold – and described – without the comparison function that produced it.

classmethod metric_fields()

Returns:

The names of the metric fields this threshold checks.

Return type:

List[str]

passed(metric_values)

Computes the pass/fail verdict for a set of metric values.

Parameters:

metric_values (Dict[str, float]) – A mapping of metric field name to value.

Returns:

bool

describe(metric_values)

Returns a (line, passed) human-readable summary of a single-metric check against these metric values, or None if this threshold has no single-metric summary line (e.g. SimpleThreshold).

class SimpleThreshold(check_error_stat, atol, rtol)

Bases: Threshold

Absolute/relative tolerances on a chosen error statistic – the threshold for the simple comparison (SimpleCompareFunc / OutputCompareResult).

Parameters:

• check_error_stat (str) – The error statistic to check (max/mean/median/quantile).

• atol (float) – The absolute tolerance.

• rtol (float) – The relative tolerance.

passed(metric_values)

Checks the chosen error statistic’s stored absolute/relative difference against the tolerances.

"elemwise" is not supported here: it is a per-element check with no scalar statistic, so it cannot be evaluated from summary statistics. Choose max/mean/median/quantile, or re-run the comparison from saved raw outputs.

class L2Threshold(threshold)

Bases: _MetricThreshold

Maximum allowed L2 norm (L2CompareFunc / L2Result).

Parameters:

threshold (float) – The value the metric is checked against.

class CosineSimilarityThreshold(threshold)

Bases: _MetricThreshold

Minimum required cosine similarity (CosineSimilarityCompareFunc / CosineSimilarityResult).

Parameters:

threshold (float) – The value the metric is checked against.

class PsnrThreshold(threshold)

Bases: _MetricThreshold

Minimum required PSNR (PsnrCompareFunc / PsnrResult).

Parameters:

threshold (float) – The value the metric is checked against.

class SnrThreshold(threshold)

Bases: _MetricThreshold

Minimum required SNR (SnrCompareFunc / SnrResult).

Parameters:

threshold (float) – The value the metric is checked against.

class LpipsThreshold(threshold)

Bases: _MetricThreshold

Maximum allowed LPIPS (PerceptualMetricsCompareFunc / PerceptualMetricsResult).

Parameters:

threshold (float) – The value the metric is checked against.

passed(metric_values)

Computes the pass/fail verdict for a set of metric values.

Parameters:

metric_values (Dict[str, float]) – A mapping of metric field name to value.

Returns:

bool

class CompareResult(thresholds=None)

Bases: object

Base class for the result of comparing a single output between two runners. Subclasses record their metric value(s); this base holds the thresholds (a Threshold) the result was checked against and the two compared output names (populated by run_comparison).

The verdict (bool(result)) and the per-metric summary (describe) are computed by the stored Threshold from the result’s metric values, so a saved result can be re-checked against a new threshold without the comparison function that produced it. OutputCompareResult with the elemwise check is the one exception: its per-element verdict cannot be reconstructed from the summary statistics, so it is stored.

set_thresholds(thresholds)

Updates the threshold this result is checked against (used by re-thresholding).

classmethod metric_fields()

Returns:

The names of the scalar metric fields this result type carries, taken from its associated Threshold class.

Return type:

List[str]

metric_values()

Returns:

The stored value of each metric field (see metric_fields).

Return type:

OrderedDict[str, float]

describe()

Returns a (line, passed) human-readable summary, or None if this result’s threshold has no single-metric summary line (e.g. simple).

class OutputCompareResult(max_absdiff, max_reldiff, mean_absdiff, mean_reldiff, median_absdiff, median_reldiff, quantile_absdiff, quantile_reldiff, thresholds=None, passed=None)

Bases: CompareResult

Represents the result of comparing a single output of a single iteration between two runners.

Records the statistics gathered during comparison and the threshold checked against.

Parameters:

• max_absdiff (float) – The minimum required absolute tolerance to consider the outputs equivalent.

• max_reldiff (float) – The minimum required relative tolerance to consider the outputs equivalent.

• mean_absdiff (float) – The mean absolute error between the outputs.

• mean_reldiff (float) – The mean relative error between the outputs.

• median_absdiff (float) – The median absolute error between the outputs.

• median_reldiff (float) – The median relative error between the outputs.

• quantile_absdiff (float) – The q-th quantile absolute error between the outputs.

• quantile_reldiff (float) – The q-th quantile relative error between the outputs.

• thresholds (SimpleThreshold) – The threshold checked against.

• passed (bool) – The stored verdict. Only set for the elemwise check, whose per-element verdict cannot be derived from the summary statistics; otherwise the verdict is computed by the threshold from the metrics.

set_thresholds(thresholds)

Updates the threshold this result is checked against (used by re-thresholding).

class PerceptualMetricsResult(lpips=None, thresholds=None)

Bases: CompareResult

Represents the result of comparing a single output using perceptual metrics between two runners.

Parameters:

• lpips (float) – The Learned Perceptual Image Patch Similarity score between the outputs. Lower values indicate more perceptually similar outputs. May be None if LPIPS computation failed.

• thresholds (LpipsThreshold) – The threshold checked against.

class L2Result(l2_norm, thresholds=None)

Bases: CompareResult

Represents the result of comparing a single output using the L2 norm (Euclidean distance) between two runners.

Parameters:

• l2_norm (float) – The L2 norm (Euclidean distance) between the outputs.

• thresholds (L2Threshold) – The threshold checked against.

class CosineSimilarityResult(cosine_similarity, thresholds=None)

Bases: CompareResult

Represents the result of comparing a single output using cosine similarity between two runners.

Parameters:

• cosine_similarity (float) – The cosine similarity between the outputs.

• thresholds (CosineSimilarityThreshold) – The threshold checked against.

class PsnrResult(psnr, thresholds=None)

Bases: CompareResult

Represents the result of comparing a single output using PSNR (Peak Signal-to-Noise Ratio) between two runners.

Parameters:

• psnr (float) – The Peak Signal-to-Noise Ratio between the outputs.

• thresholds (PsnrThreshold) – The threshold checked against.

class SnrResult(snr, thresholds=None)

Bases: CompareResult

Represents the result of comparing a single output using SNR (Signal-to-Noise Ratio) between two runners.

Parameters:

• snr (float) – The Signal-to-Noise Ratio between the outputs.

• thresholds (SnrThreshold) – The threshold checked against.

class BaseCompareFunc

Bases: object

Base class for comparison functors.

A comparison functor compares two IterationResult s and returns an OrderedDict[str, <ResultObject>] mapping output names to result objects (or anything convertible to a boolean) indicating whether the corresponding output matched. Instances are used as the compare_func argument to Comparator.compare_accuracy.

Subclass to define custom comparisons. The comparison criterion lives on the Threshold the functor produces (its passed/describe); a functor only computes metric values and, via thresholds_for, supplies a Threshold. To support average accuracy checks (Comparator.compare_accuracy(..., check_average=True)) and re-thresholding of saved results, set _RESULT_CLASS (the CompareResult subclass produced) and implement thresholds_for to return a Threshold; this is duck-typed (it does not require subclassing this class), but subclassing documents the protocol.

__call__(iter_result0, iter_result1)

Performs a per-iteration comparison.

Parameters:

• iter_result0 (IterationResult) – The result of the first runner.

• iter_result1 (IterationResult) – The result of the second runner.

Returns:

A mapping of output names to result objects indicating whether the corresponding output matched.

Return type:

OrderedDict[str, <ResultObject>]

thresholds_for(output_name)

Returns the (per-output) Threshold this functor checks against. The threshold carries the comparison criterion (passed/describe) and the metric fields it applies to, and is stored on the result. This is reachable through InvokeFromScript (which only forwards public names), so it is how --compare-func-script functors expose their thresholds.

Parameters:

output_name (str) – The name of the output being evaluated.

Returns:

Threshold

class SimpleCompareFunc(check_shapes=None, rtol=None, atol=None, fail_fast=None, find_output_func=None, check_error_stat=None, infinities_compare_equal=None, save_heatmaps=None, show_heatmaps=None, save_error_metrics_plot=None, show_error_metrics_plot=None, error_quantile=None)

Bases: BaseCompareFunc

Compares two IterationResults using absolute/relative tolerances on a chosen error statistic.

Instances are used as the compare_func argument to Comparator.compare_accuracy.

Parameters:

• check_shapes (bool) – Whether shapes must match exactly. If this is False, this function may permute or reshape outputs before comparison. Defaults to True.

• rtol (Union[float, Dict[str, float]]) –

  The relative tolerance to use when checking accuracy. This is expressed as a percentage of the second set of output values. For example, a value of 0.01 would check that the first set of outputs is within 1% of the second.

  This can be provided on a per-output basis using a dictionary. In that case, use an empty string (“”) as the key to specify default tolerance for outputs not explicitly listed. Defaults to 1e-5.

• atol (Union[float, Dict[str, float]]) – The absolute tolerance to use when checking accuracy. This can be provided on a per-output basis using a dictionary. In that case, use an empty string (“”) as the key to specify default tolerance for outputs not explicitly listed. Defaults to 1e-5.

• fail_fast (bool) – Whether the function should exit immediately after the first failure. Defaults to False.

• find_output_func (Callable(str, int, IterationResult) -> List[str]) – A callback that returns a list of output names to compare against from the provided IterationResult, given an output name and index from another IterationResult. The comparison function will always iterate over the output names of the first IterationResult, expecting names from the second. A return value of [] or None indicates that the output should be skipped.

• check_error_stat (Union[str, Dict[str, str]]) –

  The error statistic to check. Possible values are:

  • ”elemwise”: Checks each element in the output to determine if it exceeds both tolerances specified.

    The minimum required tolerances displayed in this mode are only applicable when just one type of tolerance is set. Because of the nature of the check, when both absolute/relative tolerance are specified, the required minimum tolerances may be lower.

  • ”max”: Checks the maximum absolute/relative errors against the respective tolerances. This is the strictest possible check.

  • ”mean” Checks the mean absolute/relative errors against the respective tolerances.

  • ”median”: Checks the median absolute/relative errors against the respective tolerances.

  • ”quantile”: Checks the quantile absolute/relative errors against the respective tolerances.

  This can be provided on a per-output basis using a dictionary. In that case, use an empty string (“”) as the key to specify default error stat for outputs not explicitly listed. Defaults to “elemwise”.

• infinities_compare_equal (bool) – If True, then matching +-inf values in the output have an absdiff of 0. If False, then matching +-inf values in the output have an absdiff of NaN. Defaults to False.

• save_heatmaps (str) – [EXPERIMENTAL] Path to a directory in which to save figures of heatmaps of the absolute and relative error. Defaults to None.

• show_heatmaps (bool) – [EXPERIMENTAL] Whether to display heatmaps of the absolute and relative error. Defaults to False.

• save_error_metrics_plot (str) – [EXPERIMENTAL] Path to a directory in which to save the error metrics plots. Defaults to None.

• show_error_metrics_plot (bool) – [EXPERIMENTAL] Whether to display the error metrics plot.

• error_quantile (Union[float, Dict[str, float]]) – Quantile error to compute when checking accuracy. This is expressed as a float in range [0, 1]. For example, error_quantile=0.5 is the median. Defaults to 0.99.

thresholds_for(output_name)

Returns the (per-output) Threshold this functor checks against. The threshold carries the comparison criterion (passed/describe) and the metric fields it applies to, and is stored on the result. This is reachable through InvokeFromScript (which only forwards public names), so it is how --compare-func-script functors expose their thresholds.

Parameters:

output_name (str) – The name of the output being evaluated.

Returns:

Threshold

__call__(iter_result0, iter_result1)

Performs a per-iteration comparison.

Parameters:

• iter_result0 (IterationResult) – The result of the first runner.

• iter_result1 (IterationResult) – The result of the second runner.

Returns:

A mapping of output names to result objects indicating whether the corresponding output matched.

Return type:

OrderedDict[str, <ResultObject>]

class IndicesCompareFunc(index_tolerance=None, fail_fast=None)

Bases: BaseCompareFunc

Compares two IterationResults containing indices, e.g. the outputs of a Top-K operation.

Instances are used as the compare_func argument to Comparator.compare_accuracy.

Compares two IterationResults containing indices, and can be used as the compare_func argument in Comparator.compare_accuracy. This can be useful to compare, for example, the outputs of a Top-K operation.

Outputs with more than one dimension are treated like multiple batches of values. For example, an output of shape (3, 4, 5, 10) would be treated like 60 batches (3 x 4 x 5) of 10 values each.

Parameters:

• index_tolerance (Union[int, Dict[str, int]]) –

  The tolerance to use when comparing indices. This is an integer indicating the maximum distance between values before it is considered a mismatch. For example, consider two outputs:

  output0 = [0, 1, 2]
  output1 = [1, 0, 2]
  

  With an index tolerance of 0, this would be considered a mismatch, since the positions of 0 and 1 are flipped between the two outputs. However, with an index tolerance of 1, it would pass since the mismatched values are only 1 spot apart. If instead the outputs were:

  output0 = [0, 1, 2]
  output1 = [1, 2, 0]
  

  Then we would require an index tolerance of 2, since the 0 value in the two outputs is 2 spots apart.

  When this value is set, the final ‘index_tolerance’ number of values are ignored for each batch. For example, with an index tolerance of 1, mismatches in the final element are not considered. If used with a Top-K output, you can compensate for this by instead using a Top-(K + index_tolerance).

  This can be provided on a per-output basis using a dictionary. In that case, use an empty string (“”) as the key to specify default tolerance for outputs not explicitly listed.

• fail_fast (bool) – Whether the function should exit immediately after the first failure. Defaults to False.

__call__(iter_result0, iter_result1)

Performs a per-iteration comparison.

Parameters:

• iter_result0 (IterationResult) – The result of the first runner.

• iter_result1 (IterationResult) – The result of the second runner.

Returns:

A mapping of output names to result objects indicating whether the corresponding output matched.

Return type:

OrderedDict[str, <ResultObject>]

class L2CompareFunc(l2_threshold=None, check_shapes=None, fail_fast=None, find_output_func=None)

Bases: _SingleMetricCompareFunc

Compares two IterationResults using the L2 norm (Euclidean distance).

Instances are used as the compare_func argument to Comparator.compare_accuracy.

Parameters:

• l2_threshold (Union[float, Dict[str, float]]) – Maximum allowed L2 norm. Per-output values can be given as a dictionary; use "" as the key for a default. Defaults to 1e-5.

• check_shapes (bool) – Whether shapes must match exactly. If False, outputs may be permuted or reshaped before comparison. Defaults to True.

• fail_fast (bool) – Whether to exit immediately after the first failure. Defaults to False.

• find_output_func (Callable(str, int, IterationResult) -> List[str]) – A callback that returns the names of the output(s) to compare against, given an output name from the first runner, its index, and the second runner’s IterationResult.

class CosineSimilarityCompareFunc(cosine_similarity_threshold=None, check_shapes=None, fail_fast=None, find_output_func=None)

Bases: _SingleMetricCompareFunc

Compares two IterationResults using cosine similarity.

Instances are used as the compare_func argument to Comparator.compare_accuracy.

Parameters:

• cosine_similarity_threshold (Union[float, Dict[str, float]]) – Minimum cosine similarity required (range -1 to 1). Per-output values can be given as a dictionary; use "" as the key for a default. Defaults to 0.997.

• check_shapes (bool) – Whether shapes must match exactly. If False, outputs may be permuted or reshaped before comparison. Defaults to True.

• fail_fast (bool) – Whether to exit immediately after the first failure. Defaults to False.

• find_output_func (Callable(str, int, IterationResult) -> List[str]) – A callback that returns the names of the output(s) to compare against, given an output name from the first runner, its index, and the second runner’s IterationResult.

class PsnrCompareFunc(psnr_threshold=None, check_shapes=None, fail_fast=None, find_output_func=None)

Bases: _SingleMetricCompareFunc

Compares two IterationResults using PSNR (Peak Signal-to-Noise Ratio).

Instances are used as the compare_func argument to Comparator.compare_accuracy.

Parameters:

• psnr_threshold (Union[float, Dict[str, float]]) – Minimum PSNR (dB) required. Per-output values can be given as a dictionary; use "" as the key for a default. Defaults to 30.0.

• check_shapes (bool) – Whether shapes must match exactly. If False, outputs may be permuted or reshaped before comparison. Defaults to True.

• fail_fast (bool) – Whether to exit immediately after the first failure. Defaults to False.

• find_output_func (Callable(str, int, IterationResult) -> List[str]) – A callback that returns the names of the output(s) to compare against, given an output name from the first runner, its index, and the second runner’s IterationResult.

class SnrCompareFunc(snr_threshold=None, check_shapes=None, fail_fast=None, find_output_func=None)

Bases: _SingleMetricCompareFunc

Compares two IterationResults using SNR (Signal-to-Noise Ratio).

Instances are used as the compare_func argument to Comparator.compare_accuracy.

Parameters:

• snr_threshold (Union[float, Dict[str, float]]) – Minimum SNR (dB) required. Per-output values can be given as a dictionary; use "" as the key for a default. Defaults to 20.0.

• check_shapes (bool) – Whether shapes must match exactly. If False, outputs may be permuted or reshaped before comparison. Defaults to True.

• fail_fast (bool) – Whether to exit immediately after the first failure. Defaults to False.

• find_output_func (Callable(str, int, IterationResult) -> List[str]) – A callback that returns the names of the output(s) to compare against, given an output name from the first runner, its index, and the second runner’s IterationResult.

class PerceptualMetricsCompareFunc(lpips_threshold=None, check_shapes=None, fail_fast=None, find_output_func=None)

Bases: _SingleMetricCompareFunc

Compares two IterationResults using perceptual metrics (LPIPS), targeting image-like data.

Instances are used as the compare_func argument to Comparator.compare_accuracy.

Compares two IterationResults using perceptual metrics (LPIPS), and can be used as the compare_func argument in Comparator.compare_accuracy.

This function specifically targets image-like data and uses perceptual similarity metrics that correlate better with human perception than traditional distance metrics.

Parameters:

• lpips_threshold (Union[float, Dict[str, float]]) – The maximum LPIPS (Learned Perceptual Image Patch Similarity) score allowed for outputs to be considered matching. Lower values indicate more perceptually similar outputs. Typical values are below 0.1. This can be provided on a per-output basis using a dictionary. In that case, use an empty string (“”) as the key to specify default threshold for outputs not explicitly listed. If None, a default value of 0.1 will be used.

• check_shapes (bool) – Whether shapes must match exactly. If this is False, this function may permute or reshape outputs before comparison. Defaults to True.

• fail_fast (bool) – Whether the function should exit immediately after the first failure. Defaults to False.

• find_output_func (Callable(str, int, IterationResult) -> List[str]) – A callback that returns a list of output names to compare against from the provided IterationResult, given an output name and index from another IterationResult. The comparison function will always iterate over the output names of the first IterationResult, expecting names from the second. A return value of [] or None indicates that the output should be skipped.

class CompareFunc

Bases: object

Provides functions that can be used to compare two IterationResult s.

static simple(check_shapes=None, rtol=None, atol=None, fail_fast=None, find_output_func=None, check_error_stat=None, infinities_compare_equal=None, save_heatmaps=None, show_heatmaps=None, save_error_metrics_plot=None, show_error_metrics_plot=None, error_quantile=None)

Deprecated: Use SimpleCompareFunc instead.

Creates a SimpleCompareFunc. See SimpleCompareFunc for a description of the arguments.

static indices(index_tolerance=None, fail_fast=None)

Deprecated: Use IndicesCompareFunc instead.

Creates an IndicesCompareFunc. See IndicesCompareFunc for a description of the arguments.

static perceptual_metrics(lpips_threshold=None, check_shapes=None, fail_fast=None, find_output_func=None)

Deprecated: Use PerceptualMetricsCompareFunc instead.

Creates a PerceptualMetricsCompareFunc. See PerceptualMetricsCompareFunc for a description of the arguments.