Runners
Module: polygraphy.backend.tf
- class TfRunner(sess, timeline_dir=None, name=None)[source]
Bases:
BaseRunner
Runs inference using a TensorFlow session.
- Parameters:
sess (Union[Tuple[tf.Session, Sequence[str]], Callable() -> Tuple[tf.Session, Sequence[str]]]) – A tuple containing a TensorFlow session and output names or a callable that returns one.
timeline_dir (str) – Path to write a TensorFlow timeline. Note that profiling may affect execution time.
name (str) – The human-readable name prefix to use for this runner. A runner count and timestamp will be appended to this prefix.
- __enter__()
Activate the runner for inference. For example, this may involve allocating CPU or GPU memory.
- __exit__(exc_type, exc_value, traceback)
Deactivate the runner. For example, this may involve freeing CPU or GPU memory.
- activate()
Activate the runner for inference. For example, this may involve allocating CPU or GPU memory.
Generally, you should use a context manager instead of manually activating and deactivating. For example:
with RunnerType(...) as runner: runner.infer(...)
- deactivate()
Deactivate the runner. For example, this may involve freeing CPU or GPU memory.
Generally, you should use a context manager instead of manually activating and deactivating. For example:
with RunnerType(...) as runner: runner.infer(...)
- get_input_metadata(use_numpy_dtypes=None)
Returns information about the inputs of the model. Shapes here may include dynamic dimensions, represented by
None
. Must be called only afteractivate()
and beforedeactivate()
.- Parameters:
use_numpy_dtypes (bool) – [DEPRECATED] Whether to return NumPy data types instead of Polygraphy
DataType
s. This is provided to retain backwards compatibility. In the future, this parameter will be removed and PolygraphyDataType
s will always be returned. These can be converted to NumPy data types by calling the numpy() method. Defaults to True.- Returns:
Input names, shapes, and data types.
- Return type:
- infer(feed_dict, check_inputs=True, *args, **kwargs)
Runs inference using the provided feed_dict.
Must be called only after
activate()
and beforedeactivate()
.NOTE: Some runners may accept additional parameters in infer(). For details on these, see the documentation for their infer_impl() methods.
- Parameters:
feed_dict (OrderedDict[str, numpy.ndarray]) – A mapping of input tensor names to corresponding input NumPy arrays.
check_inputs (bool) – Whether to check that the provided
feed_dict
includes the expected inputs with the expected data types and shapes. Disabling this may improve performance. Defaults to True.
- inference_time
The time required to run inference in seconds.
- Type:
float
- Returns:
A mapping of output tensor names to their corresponding NumPy arrays.
IMPORTANT: Runners may reuse these output buffers. Thus, if you need to save outputs from multiple inferences, you should make a copy with
copy.deepcopy(outputs)
.- Return type:
OrderedDict[str, numpy.ndarray]
- last_inference_time()
Returns the total inference time in seconds required during the last call to
infer()
.Must be called only after
activate()
and beforedeactivate()
.- Returns:
The time in seconds, or None if runtime was not measured by the runner.
- Return type:
float
- is_active
Whether this runner has been activated, either via context manager, or by calling
activate()
.- Type:
bool