Comparator

Module: polygraphy.tools.args

class ComparatorRunArgs

Bases: BaseArgs

Comparator Inference: running inference via Comparator.run().

Depends on:

• DataLoaderArgs

• RunnerSelectArgs

• ComparatorCompareArgs

parse_impl(args)

Parses command-line arguments and populates the following attributes:

warm_up

The number of warm-up runs to perform.

Type:

int

use_subprocess

Whether to run each runner in a subprocess.

Type:

bool

save_inputs_path

The path at which to save input data.

Type:

str

save_outputs_path

The path at which to save output data.

Type:

str

class ComparatorCompareArgs(allow_postprocessing: bool = None, allow_validate: bool = None, allow_load_outputs: bool = None, allow_sequential_runners: bool = None)

Bases: BaseArgs

Comparator Comparisons: inference output comparisons.

Depends on:

• CompareFuncSimpleArgs

• CompareFuncIndicesArgs

• CompareFuncL2Args

• CompareFuncCosineSimilarityArgs

• CompareFuncPsnrArgs

• CompareFuncSnrArgs

• CompareFuncPerceptualMetricsArgs

• RunnerSelectArgs

• DataLoaderArgs

• ComparatorRunArgs

• ComparatorPostprocessArgs: if allow_postprocessing == True

Parameters:

• allow_postprocessing (bool) – Whether to post-processing of outputs before comparison. Defaults to True.

• allow_validate (bool) – Whether to allow the --validate option (checking inference outputs for NaNs/Infs). Defaults to True.

• allow_load_outputs (bool) – Whether to allow the --load-outputs option (loading saved runs to compare against). Defaults to True.

• allow_sequential_runners (bool) – Whether to allow the --sequential-runners option (running each runner to completion instead of streaming). Defaults to True.

The allow_* options for running/loading runs are disabled by tools that only operate on already-computed comparison results (e.g. polygraphy check accuracy).

parse_impl(args)

Parses command-line arguments and populates the following attributes:

no_shape_check

Whether to skip shape checks.

Type:

bool

validate

Whether to run output validation.

Type:

bool

load_outputs_paths

Path(s) of saved runs to load and compare against.

Type:

List[str]

save_accuracy_results_path

Path at which to save the accuracy comparison results.

Type:

str

fail_fast

Whether to fail fast.

Type:

bool

compare_funcs

The names of the comparison functions to use.

Type:

List[str]

compare_func_script

Path to a script defining a custom comparison function.

Type:

str

compare_func_name

The name of the function in the script that runs comparison.

Type:

str

check_average

Whether to check averaged metrics instead of per-iteration.

Type:

bool

streaming

Whether to stream all runners together one iteration at a time (the default; the inverse of –sequential-runners).

Type:

bool

add_to_script_impl(script, results_name)

Parameters:

results_name (str) – The variable holding the live run from Comparator.run – a materialized RunResults or, in streaming mode, a per-iteration stream – or None if there are no runners (data comes entirely from --load-outputs).

Returns:

The name of the variable holding the overall pass/fail status.

Return type:

str

build_compare_funcs()

Builds comparison function instances (rather than script variables) from the parsed args, for tools that compare imperatively rather than via a generated script – e.g. polygraphy check accuracy, which re-checks saved accuracy results against new thresholds.

This runs the same script generation as run (_build_compare_funcs) and returns the resulting instances, so the two paths cannot drift.

Returns:

List[BaseCompareFunc]