Data Loader

Module: polygraphy.comparator

class DataLoader(seed=None, iterations=None, input_metadata=None, val_range=None, data_loader_backend_module=None)[source]

Bases: object

Generates synthetic input data.

Parameters:

  • seed (int) – The seed to use when generating random inputs. Defaults to util.constants.DEFAULT_SEED.

  • iterations (int) – The number of iterations for which to supply data. Defaults to 1.

  • input_metadata (TensorMetadata) – A mapping of input names to their corresponding shapes and data types. This will be used to determine what shapes to supply for inputs with dynamic shape, as well as to set the data type of the generated inputs. If either dtype or shape are None, then the value will be automatically determined. For input shape tensors, i.e. inputs whose value describes a shape in the model, the provided shape will be used to populate the values of the inputs, rather than to determine their shape.

  • val_range (Union[Tuple[number], Dict[str, Tuple[number]]]) – A tuple containing exactly 2 numbers, indicating the minimum and maximum values (inclusive) the data loader should generate. If either value in the tuple is None, the default will be used for that value. If None is provided instead of a tuple, then the default values will be used for both the minimum and maximum. This can be specified on a per-input basis using a dictionary. In that case, use an empty string (“”) as the key to specify default range for inputs not explicitly listed. Defaults to (0.0, 1.0).

  • data_loader_backend_module (str) – A string denoting what module to use to construct the input data arrays. Currently supports “numpy” and “torch”. Defaults to “numpy”.

__getitem__(index)[source]

Generates random input data.

May update the DataLoader’s input_metadata attribute.

Parameters:

index (int) – Since this class behaves like an iterable, it takes an index parameter. Generated data is guaranteed to be the same for the same index.

Returns:

A mapping of input names to input numpy buffers.

Return type:

OrderedDict[str, Union[numpy.ndarray, torch.Tensor]]

class StreamingDataLoader(paths, allow_dirs=True)[source]

Bases: object

A data loader that lazily yields previously-saved inputs from disk, one iteration at a time. It can be used anywhere a data loader is accepted (e.g. Comparator.run).

Memory usage stays constant when each iteration is a separate file (a directory of per-iteration files, or many single-feed_dict files). A single file containing a JSON list of feed_dicts is loaded in full before its iterations are yielded.

Parameters:

  • paths (Union[str, os.PathLike, Sequence]) – One or more paths to load (a single path may be given directly instead of a one-element sequence). Each path may be a file – if it contains a JSON list, each element is one iteration; otherwise the whole file is a single iteration – or a directory whose *.json files (in index order) are each one iteration.

  • allow_dirs (bool) – Whether a directory path is permitted. When False, a directory raises an error instead of being treated as a run of per-iteration files. Defaults to True.