Comparator

Module: polygraphy.comparator

class Comparator[source]

Bases: object

Compares inference outputs.

static run(runners, data_loader=None, warm_up=None, use_subprocess=None, subprocess_timeout=None, subprocess_polling_interval=None, save_inputs_path=None)[source]

Runs the supplied runners sequentially.

Parameters:

  • runners (List[BaseRunner]) – A list of runners to run.

  • data_loader (Sequence[OrderedDict[str, numpy.ndarray]]) –

    A generator or iterable that yields a dictionary that maps input names to input numpy buffers. In the simplest case, this can be a List[Dict[str, numpy.ndarray]] .

    In case you don’t know details about the inputs ahead of time, you can access the input_metadata property in your data loader, which will be set to an TensorMetadata instance by this function. Note that this does not work for generators or lists.

    The number of iterations run by this function is controlled by the number of items supplied by the data loader.

    Defaults to an instance of DataLoader.

  • warm_up (int) – The number of warm up runs to perform for each runner before timing. Defaults to 0.

  • use_subprocess (bool) – Whether each runner should be run in a subprocess. This allows each runner to have exclusive access to the GPU. When using a subprocess, runners and loaders will never be modified.

  • subprocess_timeout (int) – The timeout before a subprocess is killed automatically. This is useful for handling processes that never terminate. A value of None disables the timeout. Defaults to None.

  • subprocess_polling_interval (int) – The polling interval, in seconds, for checking whether a subprocess has completed or crashed. In rare cases, omitting this parameter when subprocesses are enabled may cause this function to hang indefinitely if the subprocess crashes. A value of 0 disables polling. Defaults to 30 seconds.

  • save_inputs_path (str) – Path at which to save inputs used during inference. This will include all inputs generated by the provided data_loader, and will be saved as a JSON List[Dict[str, numpy.ndarray]].

Returns:

A mapping of runner names to the results of their inference. The ordering of runners is preserved in this mapping.

Return type:

RunResults

static postprocess(run_results, postprocess_func)[source]

Applies post processing to all the outputs in the provided run results. This is a convenience function to avoid the need for manual iteration over the run_results dictionary.

Parameters:

  • run_results (RunResults) – The result of Comparator.run().

  • postprocess_func (Callable(IterationResult) -> IterationResult) – The function to apply to each IterationResult.

Returns:

The updated run results.

Return type:

RunResults

static compare_accuracy(run_results, fail_fast=False, comparisons=None, compare_func=None)[source]
Parameters:

  • run_results (RunResults) – The result of Comparator.run()

  • fail_fast (bool) – Whether to exit after the first failure

  • comparisons (List[Tuple[int, int]]) – Comparisons to perform, specified by runner indexes. For example, [(0, 1), (1, 2)] would compare the first runner with the second, and the second with the third. By default, this compares each result to the subsequent one.

  • compare_func (Callable(IterationResult, IterationResult) -> OrderedDict[str, bool]) – A function that takes in two IterationResults, and returns a dictionary that maps output names to a boolean (or anything convertible to a boolean) indicating whether outputs matched. The order of arguments to this function is guaranteed to be the same as the ordering of the tuples contained in comparisons.

Returns:

A summary of the results of the comparisons. The order of the keys (i.e. runner pairs) is guaranteed to be the same as the order of comparisons. For more details, see the AccuracyResult docstring (e.g. help(AccuracyResult)).

Return type:

AccuracyResult

static validate(run_results, check_inf=None, check_nan=None, fail_fast=None)[source]

Checks output validity.

Parameters:

  • run_results (Dict[str, List[IterationResult]]) – The result of Comparator.run().

  • check_inf (bool) – Whether to fail on Infs. Defaults to False.

  • check_nan (bool) – Whether to fail on NaNs. Defaults to True.

  • fail_fast (bool) – Whether to fail after the first invalid value. Defaults to False.

Returns:

True if all outputs were valid, False otherwise.

Return type:

bool