Data Structures¶

Module: polygraphy.comparator

class IterationResult(outputs=None, runtime=None, runner_name=None)[source]¶

Bases: Interface

An ordered dictionary containing the result of a running a single iteration of a runner.

This maps output names to arrays, and preserves the output ordering from the runner.

NOTE: The POLYGRAPHY_ARRAY_SWAP_THRESHOLD_MB environment variable can be set to enable the arrays to be swapped to the disk.

Also includes additional fields indicating the name of the runner which produced the outputs, and the time required to do so.

Parameters:

outputs (Dict[str, Union[np.array, torch.Tensor]]) – The outputs of this iteration, mapped to their names.
runtime (float) – The time required for this iteration, in seconds. Only used for logging purposes.
runner_name (str) – The name of the runner that produced this output. If this is omitted, a default name is generated.

class RunResults(lst=None)[source]¶

Bases: Interface

Maps runners to per-iteration outputs (in the form of a List[IterationResult]).

For example, if results is an instance of RunResults(), then to access the outputs of the first iteration from a specified runner, do:

iteration = 0
runner_name = "trt-runner"
outputs = results[runner_name][iteration]

# `outputs` is a `Dict[str, np.ndarray]`

Note: Technically, this is a List[Tuple[str, List[IterationResult]]], but includes helpers that make it behave like an OrderedDict that can contain duplicates.

items()[source]¶: Creates a generator that yields Tuple[str, List[IterationResult]] - runner names and corresponding outputs.

keys()[source]¶: Creates a generator that yields runner names (str).

values()[source]¶: Creates a generator that yields runner outputs (List[IterationResult]).

update(other)[source]¶

Updates the results stored in this instance.

Parameters:: other (Union[Dict[str, List[IterationResult]], RunResults]) – A dictionary or RunResults instance from which to update this one.

add(out_list, runtime=None, runner_name=None)[source]¶

A helper to create a List[IterationResult] and map it to the specified runner_name.

This method cannot be used to modify an existing entry.

Calling this method is equivalent to:

results[runner_name] = []
for out in out_list:
    results[runner_name].append(IterationResult(out, runtime, runner_name))

Parameters:

out_list (List[Dict[str, np.array]]) – One or more set of outputs where each output is a dictionary of output names mapped to NumPy arrays.
runtime (float) – The time required for this iteration, in seconds. Only used for logging purposes.
runner_name (str) – The name of the runner that produced this output. If this is omitted, a default name is generated.

split()[source]¶

A generator that yields one single-iteration RunResults per iteration of this run (each containing a single IterationResult per runner).

Yields:: RunResults – One per iteration.

static concat(runs)[source]¶

Concatenates the iterations of several RunResults into one (the inverse of split). Runners are matched by name, so ragged single-iteration runs from split (which omits a runner that has run out of iterations) are inverted correctly.

Parameters:: runs (Iterable[RunResults]) – The runs to concatenate, in iteration order.
Returns:: The combined run.
Return type:: RunResults

static load_streaming(path, allow_dirs=True)[source]¶

A generator that lazily yields single-iteration RunResults from a saved source, one iteration at a time, so arbitrarily large runs can be streamed without loading everything into memory. This is the reading inverse of Comparator.run(..., save_outputs_path=...).

The path may be:

A directory, treated as a single run whose per-iteration *.json files (in index order) are each a RunResults.
A file containing a RunResults, split into one RunResults per iteration. The file is loaded fully into memory; for constant memory over large runs, stream a directory of per-iteration files instead.
A list of the above, whose iterations are chained into a single run.

Parameters:

path (Union[str, List[str]]) – A file or directory path, or a list of such paths.
allow_dirs (bool) – Whether a directory path is permitted. When False, a directory raises an error instead of being treated as a run of per-iteration files. Defaults to True.

Yields:

RunResults – One per iteration, each containing a single IterationResult per runner.

static from_json(src)¶

Decode a JSON object and create an instance of this class.

Parameters:: src (str) – The JSON representation of the object
Returns:: The decoded instance
Return type:: RunResults
Raises:: PolygraphyException – If the JSON cannot be decoded to an instance of RunResults

static load(src)¶

Loads an instance of this class from a JSON file.

Parameters:: src (Union[str, file-like]) – The path or file-like object to read from.
Returns:: The decoded instance
Return type:: RunResults
Raises:: PolygraphyException – If the JSON cannot be decoded to an instance of RunResults

save(dest)¶

Encode this instance as a JSON object and save it to the specified path or file-like object.

Parameters:: dest (Union[str, file-like]) – The path or file-like object to write to.

to_json()¶

Encode this instance as a JSON object.

Returns:: A JSON representation of this instance.
Return type:: str

class AccuracyResult(dct=None, aggregation='per_sample')[source]¶

Bases: Interface

An ordered dictionary holding the per-iteration comparison results for a single comparison function. Comparator.compare_accuracy returns an AccuracyResults (a list of these), one per comparison function.

More specifically, it is an OrderedDict[Tuple[str, str], List[OrderedDict[str, bool]]] which maps a runner pair (a tuple containing both runner names) to a list of dictionaries of booleans (or anything that can be converted into a boolean, such as an OutputCompareResult), indicating whether there was a match in the outputs of the corresponding iteration. The List[OrderedDict[str, bool]] is constructed from the dictionaries returned by compare_func in compare_accuracy.

For example, to see if there’s a match between runner0 and runner1 during the 1st iteration for an output called output0:

runner_pair = ("runner0", "runner1")
iteration = 0
output_name = "output0"
match = bool(accuracy_result[runner_pair][iteration][output_name])

If there’s a mismatch, you can inspect the outputs from the results of Comparator.run(), assumed here to be called run_results:

runner0_output = run_results["runner0"][iteration][output_name]
runner1_output = run_results["runner1"][iteration][output_name]

The accessors summarize a runner pair’s results in different ways:

stats: per-iteration counts (matched, mismatched, total).
average_results: per-output averaged result objects (each metric averaged across iterations, checked against its threshold).
describe_average: per-output (line, passed) summaries of the averaged-metric checks.

Parameters:

dct – Initial contents: a mapping of runner pair to per-iteration results.
aggregation (str) – Either "per_sample" (the default) or "average"; controls how bool(self) and the accuracy summary are computed.

__bool__()[source]¶

Whether all outputs matched. You can use this function to avoid manually checking each output. For example:

if accuracy_result:
    print("All matched!")

In the default (per-sample) aggregation mode, this is True only if all outputs matched for every iteration. In "average" aggregation mode (set when compare_accuracy is run with check_average=True), this is True only if every output’s averaged metrics pass.

Returns:: bool

percentage(runner_pair=None)[source]¶

Deprecated: Use stats instead.

Returns the fraction of iterations that matched for the given pair of runners, as a value between 0.0 and 1.0. Always returns 1.0 when there are no iterations or no runner comparisons.

Parameters:: runner_pair (Tuple[str, str]) – A pair of runner names describing which runners to check. Defaults to the first pair in the dictionary.
Returns:: float

stats(runner_pair=None)[source]¶

Returns the number of iterations that matched, mismatched, and the total number of iterations.

Note: This always reflects per-iteration (per-sample) results, regardless of the aggregation mode. For per-output averaged pass/fail results, see average_results.

Parameters:: runner_pair (Tuple[str, str]) – A pair of runner names describing which runners to check. Defaults to the first pair in the dictionary.
Returns:: Number of iterations that matched, mismatched, and total respectively.
Return type:: Tuple[int, int, int]

output_names()[source]¶

Returns:: The names of the compared outputs, in order of first appearance, across all runner pairs.
Return type:: List[str]

describe_average(runner_pair=None)[source]¶

Produces a per-output, human-readable description of each averaged-metric check, mirroring the per-iteration log lines (metric value, threshold, and PASSED/FAILED).

The descriptions are derived on demand from the averaged result objects (see average_results), which carry their own thresholds, so no comparison function is needed. Returns an empty mapping for comparisons whose result type has no single-metric description line (e.g. simple).

Parameters:: runner_pair (Tuple[str, str]) – A pair of runner names describing which runners to check. Defaults to the first pair in the dictionary.
Returns:: Maps each output name to a (line, passed) tuple describing this result’s single averaged metric.
Return type:: OrderedDict[str, Tuple[str, bool]]

average_results(runner_pair=None)[source]¶

Computes, per output, the result of averaging each metric across iterations and checking it against the thresholds. Derived on demand from the per-iteration results – which carry their own metric values and thresholds – so it needs no comparison function.

NaN metric values propagate into the average so the check fails (matching the per-sample behavior). If any iteration for an output carries no averageable metrics (such as the False from a shape mismatch), that output’s averaged comparison fails loudly, since averaging over only the comparable iterations would be misleading.

Parameters:: runner_pair (Tuple[str, str]) – A pair of runner names describing which runners to check. Defaults to the first pair in the dictionary.
Returns:: Maps each output name to the aggregated result object.
Return type:: OrderedDict[str, <ResultObject>]

reevaluate(thresholds, aggregation=None)[source]¶

Re-checks every stored result against new thresholds in place by updating each result’s stored thresholds (the stored metric values are unchanged); the verdict then re-derives from the metrics and the new thresholds. Does not build new result objects.

Parameters:

thresholds (Union[Threshold, Dict[str, Threshold]]) – The new Threshold to check against, applied to every output, or per-output thresholds as a dictionary (with "" as the key for a default).
aggregation (str) – If provided, overrides the aggregation mode ("per_sample" or "average").

Returns:

self.

Return type:

AccuracyResult

class AccuracyResults(lst=None)[source]¶

Bases: Interface

A list of AccuracyResult objects – one per comparison function – as returned by Comparator.compare_accuracy.

Behaves like a list (iteration, indexing, len), except bool(results) is True only if every result passed, so if results: is a correct overall pass/fail check (mirroring bool(AccuracyResult)).

Can be saved to and loaded from JSON via save/load. A loaded AccuracyResults carries the per-iteration metric values and saved verdicts; reevaluate re-checks them against new thresholds (see polygraphy check accuracy).

Note: This wraps its list in .lst (rather than subclassing list) so that the JSON encoder is actually invoked – json serializes list subclasses directly, bypassing custom encoders.

static from_json(src)¶

Decode a JSON object and create an instance of this class.

Parameters:: src (str) – The JSON representation of the object
Returns:: The decoded instance
Return type:: AccuracyResults
Raises:: PolygraphyException – If the JSON cannot be decoded to an instance of AccuracyResults

static load(src)¶

Loads an instance of this class from a JSON file.

Parameters:: src (Union[str, file-like]) – The path or file-like object to read from.
Returns:: The decoded instance
Return type:: AccuracyResults
Raises:: PolygraphyException – If the JSON cannot be decoded to an instance of AccuracyResults

save(dest)¶

Encode this instance as a JSON object and save it to the specified path or file-like object.

Parameters:: dest (Union[str, file-like]) – The path or file-like object to write to.

to_json()¶

Encode this instance as a JSON object.

Returns:: A JSON representation of this instance.
Return type:: str

output_names()[source]¶

Returns:: The union of compared output names across all contained results, in order of first appearance.
Return type:: List[str]

reevaluate(thresholds, aggregation=None)[source]¶

Re-checks the requested comparisons against new thresholds, in place.

Each provided threshold selects the saved comparison it applies to – matched by the metric fields it checks – and that comparison is re-evaluated against it. Saved comparisons that no provided threshold selects are dropped, so e.g. re-checking only l2 and cosine_similarity ignores any other metrics in the file. It is an error to provide a threshold that matches no saved comparison (you asked to check data that is not present).

Parameters:

thresholds (Union[Threshold, Dict[str, Threshold], Sequence[Union[Threshold, Dict[str, Threshold]]]]) – The new threshold(s) to check against. Each is a single Threshold or a per-output {output_name: Threshold} dictionary; order does not matter.
aggregation (str) – If provided, overrides the aggregation mode of every checked result.

Returns:

self.

Return type:

AccuracyResults