IterationResult(outputs=None, runtime=None, runner_name=None)¶
An ordered dictionary containing the result of a running a single iteration of a runner.
This maps output names to NumPy arrays, and preserves the output ordering from the runner.
POLYGRAPHY_ARRAY_SWAP_THRESHOLD_MBenvironment variable can be set to enable the arrays to be swapped to the disk.
Also includes additional fields indicating the name of the runner which produced the outputs, and the time required to do so.
outputs (Dict[str, np.array]) – The outputs of this iteration, mapped to their names.
runtime (float) – The time required for this iteration, in seconds.
runner_name (str) – The name of the runner that produced this output.
Maps runner names to zero or more IterationResults.
Note: Technically, this is a
List[Tuple[str, List[IterationResult]]], but includes helpers that make it behave like an OrderedDict that can contain duplicates.
Creates a generator that yields
Tuple[str, List[IterationResult]]- runner names and corresponding outputs.
Creates a generator that yields runner names (str).
Creates a generator that yields runner outputs (List[IterationResult]).
Updates the results stored in this instance.
Decode a JSON object and create an instance of this class.
Loads an instance of this class from a JSON file.
Encode this instance as a JSON object and save it to the specified path or file-like object.
dest (Union[str, file-like]) – The path or file-like object to write to.
Encode this instance as a JSON object.
A JSON representation of this instance.
- Return type
An ordered dictionary including details about the result of
More specifically, it is an
OrderedDict[Tuple[str, str], List[OrderedDict[str, bool]]]which maps a runner pair (a tuple containing both runner names) to a list of dictionaries of booleans (or anything that can be converted into a boolean, such as an
OutputCompareResult), indicating whether there was a match in the outputs of the corresponding iteration. The
List[OrderedDict[str, bool]]is constructed from the dictionaries returned by
For example, to see if there’s a match between
runner1during the 1st iteration for an output called
runner_pair = ("runner0", "runner1") iteration = 0 output_name = "output0" match = bool(accuracy_result[runner_pair][iteration][output_name])
If there’s a mismatch, you can inspect the outputs from the results of
Comparator.run(), assumed here to be called
runner0_output = run_results["runner0"][iteration][output_name] runner1_output = run_results["runner1"][iteration][output_name]
Whether all outputs matched for every iteration. You can use this function to avoid manually checking each output. For example:
if accuracy_result: print("All matched!")
Returns the percentage of iterations that matched for the given pair of runners, expressed as a decimal between 0.0 and 1.0.
Always returns 1.0 when the number of iterations is 0, or when there are no runner comparisons.
runner_pair (Tuple[str, str]) – A pair of runner names describing which runners to check. Defaults to the first pair in the dictionary.
Returns the number of iterations that matched, mismatched, and the total number of iterations.
Number of iterations that matched, mismatched, and total respectively.
- Return type
Tuple[int, int, int]