Data Structures

Module: polygraphy.comparator

class IterationResult(outputs=None, runtime=None, runner_name=None)[source]

Bases: Interface

An ordered dictionary containing the result of a running a single iteration of a runner.

This maps output names to arrays, and preserves the output ordering from the runner.

NOTE: The POLYGRAPHY_ARRAY_SWAP_THRESHOLD_MB environment variable can be set to enable the arrays to be swapped to the disk.

Also includes additional fields indicating the name of the runner which produced the outputs, and the time required to do so.

Parameters:

outputs (Dict[str, Union[np.array, torch.Tensor]]) – The outputs of this iteration, mapped to their names.
runtime (float) – The time required for this iteration, in seconds. Only used for logging purposes.
runner_name (str) – The name of the runner that produced this output. If this is omitted, a default name is generated.

class RunResults(lst=None)[source]

Bases: Interface

Maps runners to per-iteration outputs (in the form of a List[IterationResult]).

For example, if results is an instance of RunResults(), then to access the outputs of the first iteration from a specified runner, do:

iteration = 0
runner_name = "trt-runner"
outputs = results[runner_name][iteration]

# `outputs` is a `Dict[str, np.ndarray]`

Note: Technically, this is a List[Tuple[str, List[IterationResult]]], but includes helpers that make it behave like an OrderedDict that can contain duplicates.

items()[source]: Creates a generator that yields Tuple[str, List[IterationResult]] - runner names and corresponding outputs.

keys()[source]: Creates a generator that yields runner names (str).

values()[source]: Creates a generator that yields runner outputs (List[IterationResult]).

update(other)[source]

Updates the results stored in this instance.

Parameters:: other (Union[Dict[str, List[IterationResult]], RunResults]) – A dictionary or RunResults instance from which to update this one.

add(out_list, runtime=None, runner_name=None)[source]

A helper to create a List[IterationResult] and map it to the specified runner_name.

This method cannot be used to modify an existing entry.

Calling this method is equivalent to:

results[runner_name] = []
for out in out_list:
    results[runner_name].append(IterationResult(out, runtime, runner_name))

Parameters:

out_list (List[Dict[str, np.array]]) – One or more set of outputs where each output is a dictionary of output names mapped to NumPy arrays.
runtime (float) – The time required for this iteration, in seconds. Only used for logging purposes.
runner_name (str) – The name of the runner that produced this output. If this is omitted, a default name is generated.

static from_json(src)

Decode a JSON object and create an instance of this class.

Parameters:: src (str) – The JSON representation of the object
Returns:: The decoded instance
Return type:: RunResults
Raises:: PolygraphyException – If the JSON cannot be decoded to an instance of RunResults

static load(src)

Loads an instance of this class from a JSON file.

Parameters:: src (Union[str, file-like]) – The path or file-like object to read from.
Returns:: The decoded instance
Return type:: RunResults
Raises:: PolygraphyException – If the JSON cannot be decoded to an instance of RunResults

save(dest)

Encode this instance as a JSON object and save it to the specified path or file-like object.

Parameters:: dest (Union[str, file-like]) – The path or file-like object to write to.

to_json()

Encode this instance as a JSON object.

Returns:: A JSON representation of this instance.
Return type:: str

class AccuracyResult(dct=None)[source]

Bases: Interface

An ordered dictionary including details about the result of Comparator.compare_accuracy.

More specifically, it is an OrderedDict[Tuple[str, str], List[OrderedDict[str, bool]]] which maps a runner pair (a tuple containing both runner names) to a list of dictionaries of booleans (or anything that can be converted into a boolean, such as an OutputCompareResult), indicating whether there was a match in the outputs of the corresponding iteration. The List[OrderedDict[str, bool]] is constructed from the dictionaries returned by compare_func in compare_accuracy.

For example, to see if there’s a match between runner0 and runner1 during the 1st iteration for an output called output0:

runner_pair = ("runner0", "runner1")
iteration = 0
output_name = "output0"
match = bool(accuracy_result[runner_pair][iteration][output_name])

If there’s a mismatch, you can inspect the outputs from the results of Comparator.run(), assumed here to be called run_results:

runner0_output = run_results["runner0"][iteration][output_name]
runner1_output = run_results["runner1"][iteration][output_name]

__bool__()[source]

Whether all outputs matched for every iteration. You can use this function to avoid manually checking each output. For example:

if accuracy_result:
    print("All matched!")

Returns:: bool

percentage(runner_pair=None)[source]

Returns the percentage of iterations that matched for the given pair of runners, expressed as a decimal between 0.0 and 1.0.

Always returns 1.0 when the number of iterations is 0, or when there are no runner comparisons.

Parameters:: runner_pair (Tuple[str, str]) – A pair of runner names describing which runners to check. Defaults to the first pair in the dictionary.

stats(runner_pair=None)[source]

Returns the number of iterations that matched, mismatched, and the total number of iterations.

Parameters:: runner_pair (Tuple[str, str]) – A pair of runner names describing which runners to check. Defaults to the first pair in the dictionary.
Returns:: Number of iterations that matched, mismatched, and total respectively.
Return type:: Tuple[int, int, int]