Comparator

Module: polygraphy.comparator

class Comparator

Bases: object

Compares inference outputs.

static run(runners, data_loader=None, warm_up=None, use_subprocess=None, subprocess_timeout=None, subprocess_polling_interval=None, save_inputs_path=None, save_outputs_path=None, streaming=None)

Runs the supplied runners.

By default (streaming=False) runs each runner to completion one at a time and returns a materialized RunResults holding every iteration. With streaming=True returns a generator that activates all runners simultaneously and yields one single-iteration RunResults per input, keeping memory roughly constant regardless of dataset size. The streaming form composes (optionally via Comparator.postprocess/validate) with Comparator.compare_accuracy.

Note

In streaming mode, consume each yielded RunResults (e.g. by comparing or saving it) before requesting the next; runner output buffers may be reused between iterations.

Parameters:

• runners (List[BaseRunner]) – A list of runners to run.

• data_loader (Sequence[OrderedDict[str, numpy.ndarray]]) –

  An iterable that yields Dict[str, numpy.ndarray] feed dicts. The number of iterations is determined by the number of items yielded.

  If the data loader exposes a settable input_metadata attribute (as the built-in DataLoader does), it is set to the first runner’s TensorMetadata before iteration so the loader can size dynamic-shape inputs. This has no effect for plain generators or lists.

  Defaults to an instance of DataLoader.

• warm_up (int) – The number of warm up runs to perform for each runner before timing. Defaults to 0.

• use_subprocess (bool) – Whether each runner should be run in a subprocess. This allows each runner to have exclusive access to the GPU. When using a subprocess, runners and loaders will never be modified.

• subprocess_timeout (int) – The timeout before a subprocess is killed automatically. This is useful for handling processes that never terminate. A value of None disables the timeout. Defaults to None.

• subprocess_polling_interval (int) – The polling interval, in seconds, for checking whether a subprocess has completed or crashed. In rare cases, omitting this parameter when subprocesses are enabled may cause this function to hang indefinitely if the subprocess crashes. A value of 0 disables polling. Defaults to 30 seconds.

• save_inputs_path (str) – Where to save the inputs used during inference. A path with a file extension (e.g. inputs.json) saves all iterations to a single JSON file; an extensionless path is treated as a directory holding one <index>.json file per iteration (the directory must be empty or not yet exist). Defaults to None (inputs are not saved).

• save_outputs_path (str) – Where to save the results, with the same file-vs-directory semantics as save_inputs_path. In streaming mode each iteration is written as produced (so an extensionless path keeps memory constant); otherwise the materialized results are saved on completion. Must not resolve to the same path as save_inputs_path. Defaults to None (results are not saved).

• streaming (bool) – Whether to return a lazy per-iteration generator instead of a materialized RunResults. warm_up and use_subprocess are not supported in this mode. Defaults to False.

Returns:

A RunResults by default, or a generator of single-iteration RunResults with streaming=True.

Return type:

Union[RunResults, Iterable[RunResults]]

static postprocess(run_results, postprocess_func)

Applies postprocess_func to every IterationResult in run_results.

Accepts a RunResults, a stream (e.g. from Comparator.run(streaming=True)), or a list of either. Returns the same shape: a RunResults in place, a lazy generator for a stream, or a new list.

Parameters:

• run_results (Union[RunResults, Iterable[RunResults], List]) – The run, stream, or list of runs to process.

• postprocess_func (Callable(IterationResult) -> IterationResult) – The function to apply.

Returns:

The post-processed run(s) in the same shape as run_results.

static compare_accuracy(run_results, fail_fast=False, comparisons=None, compare_func=None, check_average=False)

Compares inference outputs across runners and reports per-comparison accuracy.

Accepts a RunResults, a stream of single-iteration RunResults (e.g. from Comparator.run(streaming=True)), or a list of either; streaming runs are compared one iteration at a time with constant memory. Post-processing and validation are separate steps (Comparator.postprocess / Comparator.validate).

Parameters:

• run_results (Union[RunResults, Sequence[Union[RunResults, Iterable[RunResults]]]]) – The run(s) to compare.

• fail_fast (bool) – Whether to stop after the first failure. Defaults to False.

• comparisons (List[Tuple[int, int]]) – Runner index pairs to compare. Defaults to comparing each runner to the next.

• compare_func (Union[Callable, Sequence[Callable]]) – A comparison functor/callable (or list of them) mapping two IterationResult``s to an ``OrderedDict[str, bool]; each produces one AccuracyResult. Defaults to SimpleCompareFunc().

• check_average (bool) – Check the average of each metric across iterations against the threshold rather than each iteration (per-iteration results are still recorded). Requires a compare_func that produces an averageable metric (e.g. SimpleCompareFunc with a scalar check_error_stat, the single-metric functions, or PerceptualMetricsCompareFunc); IndicesCompareFunc and SimpleCompareFunc with elemwise are rejected. Mutually exclusive with fail_fast. Defaults to False.

Returns:

One AccuracyResult per comparison function. bool(results) is True only if every result passed, so if results: is a correct overall pass/fail check.

Return type:

AccuracyResults

static validate(run_results, check_inf=None, check_nan=None, fail_fast=None)

Checks output validity (for NaNs/Infs).

Parameters:

• run_results (Union[RunResults, Iterable[RunResults], List]) – A RunResults, a stream of single-iteration RunResults (e.g. from Comparator.run(streaming=True)), or a list of either. For a stream, returns a lazy pass-through that validates each iteration as it is consumed and aborts via G_LOGGER.critical on the first invalid value. fail_fast applies only to the materialized form.

• check_inf (bool) – Whether to fail on Infs. Defaults to False.

• check_nan (bool) – Whether to fail on NaNs. Defaults to True.

• fail_fast (bool) – Whether to fail after the first invalid value. Defaults to False.

Returns:

For a RunResults, True if all outputs were valid, False otherwise. For a stream, a lazy pass-through that aborts on an invalid value. For a list, a list of the above.