Runners

Module: polygraphy.backend.tf

class TfRunner(sess, timeline_dir=None, name=None)

Bases: polygraphy.backend.base.runner.BaseRunner

Runs inference using a TensorFlow session.

Parameters

• sess (Callable() -> Tuple[tf.Session, Sequence[str]]) – A callable that can supply a tuple containing a TensorFlow session and output names.

• timeline_dir (str) – Path to write a TensorFlow timeline. Note that profiling may affect execution time.

• name (str) – The human-readable name prefix to use for this runner. A runner count and timestamp will be appended to this prefix.

__enter__()

Activate the runner for inference. This may involve allocating GPU buffers, for example.

__exit__(exc_type, exc_value, traceback)

Deactivate the runner.

If the POLYGRAPHY_INTERNAL_CORRECTNESS_CHECKS environment variable is set to 1, this will also check that the runner was reset to its state prior to activation.

activate()

Activate the runner for inference. This may involve allocating GPU buffers, for example.

Generally, you should use a context manager instead of manually activating and deactivating. For example:

with RunnerType(...) as runner:
    runner.infer(...)

deactivate()

Deactivate the runner.

If the POLYGRAPHY_INTERNAL_CORRECTNESS_CHECKS environment variable is set to 1, this will also check that the runner was reset to its state prior to activation.

Generally, you should use a context manager instead of manually activating and deactivating. For example:

with RunnerType(...) as runner:
    runner.infer(...)

get_input_metadata()

Returns information about the inputs of the model. Shapes here may include dynamic dimensions, represented by None. Must be called only after activate() and before deactivate().

Returns

Input names, shapes, and data types.

Return type

TensorMetadata

infer(feed_dict, check_inputs=True)

Runs inference using the provided feed_dict.

Parameters

• feed_dict (OrderedDict[str, numpy.ndarray]) – A mapping of input tensor names to corresponding input NumPy arrays.

• check_inputs (bool) – Whether to check that the provided feed_dict includes the expected inputs with the expected data types and shapes.

Returns

A mapping of output tensor names to their corresponding NumPy arrays.

IMPORTANT: Runners may reuse these output buffers. Thus, if you need to save outputs from multiple inferences, you should make a copy with copy.deepcopy(outputs).

Return type

OrderedDict[str, numpy.ndarray]

last_inference_time()

Returns the total inference time required during the last call to infer().

Returns

The time in seconds, or None if runtime was not measured by the runner.

Return type

float