Data Structures

Module: polygraphy.comparator

class IterationResult(outputs=None, runtime=None, runner_name=None)

Bases: polygraphy.common.interface.Interface

An ordered dictionary containing the result of a running a single iteration of a runner.

This maps output names to NumPy arrays, and preserves the output ordering from the runner.

NOTE: The POLYGRAPHY_ARRAY_SWAP_THRESHOLD_MB environment variable can be set to enable the arrays to be swapped to the disk.

Also includes additional fields indicating the name of the runner which produced the outputs, and the time required to do so.

Parameters

• outputs (Dict[str, np.array]) – The outputs of this iteration, mapped to their names.

• runtime (float) – The time required for this iteration, in seconds.

• runner_name (str) – The name of the runner that produced this output.

class RunResults(lst=None)

Bases: polygraphy.common.interface.Interface

Maps runner names to zero or more IterationResults.

Note: Technically, this is a List[Tuple[str, List[IterationResult]]], but includes helpers that make it behave like an OrderedDict that can contain duplicates.

items()

Creates a generator that yields Tuple[str, List[IterationResult]] - runner names and corresponding outputs.

keys()

Creates a generator that yields runner names (str).

values()

Creates a generator that yields runner outputs (List[IterationResult]).

update(other)

Updates the results stored in this instance.

Parameters

other (Union[Dict[str, List[IterationResult]], RunResults]) – A dictionary or RunResults instance from which to update this one.

static from_json(src)

Decode a JSON object and create an instance of this class.

Parameters

src (str) – The JSON representation of the object

Returns

The decoded instance

Return type

RunResults

Raises

PolygraphyException – If the JSON cannot be decoded to an instance of RunResults

static load(src)

Loads an instance of this class from a JSON file.

Parameters

src (Union[str, file-like]) – The path or file-like object to read from.

Returns

The decoded instance

Return type

RunResults

Raises

PolygraphyException – If the JSON cannot be decoded to an instance of RunResults

save(dest)

Encode this instance as a JSON object and save it to the specified path or file-like object.

Parameters

dest (Union[str, file-like]) – The path or file-like object to write to.

to_json()

Encode this instance as a JSON object.

Returns

A JSON representation of this instance.

Return type

str

class AccuracyResult(dct=None)

Bases: polygraphy.common.interface.Interface

An ordered dictionary including details about the result of Comparator.compare_accuracy.

More specifically, it is an OrderedDict[Tuple[str, str], List[OrderedDict[str, bool]]] which maps a runner pair (a tuple containing both runner names) to a list of dictionaries of booleans (or anything that can be converted into a boolean, such as an OutputCompareResult), indicating whether there was a match in the outputs of the corresponding iteration. The List[OrderedDict[str, bool]] is constructed from the dictionaries returned by compare_func in compare_accuracy.

For example, to see if there’s a match between runner0 and runner1 during the 1st iteration for an output called output0:

runner_pair = ("runner0", "runner1")
iteration = 0
output_name = "output0"
match = bool(accuracy_result[runner_pair][iteration][output_name])

If there’s a mismatch, you can inspect the outputs from the results of Comparator.run(), assumed here to be called run_results:

runner0_output = run_results["runner0"][iteration][output_name]
runner1_output = run_results["runner1"][iteration][output_name]

__bool__()

Whether all outputs matched for every iteration. You can use this function to avoid manually checking each output. For example:

if accuracy_result:
    print("All matched!")

Returns

bool

percentage(runner_pair=None)

Returns the percentage of iterations that matched for the given pair of runners, expressed as a decimal between 0.0 and 1.0.

Always returns 1.0 when the number of iterations is 0, or when there are no runner comparisons.

Parameters

runner_pair (Tuple[str, str]) – A pair of runner names describing which runners to check. Defaults to the first pair in the dictionary.

stats(runner_pair=None)

Returns the number of iterations that matched, mismatched, and the total number of iterations.

Parameters

runner_pair (Tuple[str, str]) – A pair of runner names describing which runners to check. Defaults to the first pair in the dictionary.

Returns

Number of iterations that matched, mismatched, and total respectively.

Return type

Tuple[int, int, int]