Is this page helpful?

Working with Dynamic Shapes#

Dynamic Shapes is the ability to defer specifying some or all tensor dimensions until runtime. Dynamic shapes can be used through both the C++ and Python interfaces.

The following sections provide greater detail; however, here is an overview of the steps for building an engine with dynamic shapes:

  1. Specify each runtime dimension of an input tensor by using -1 as a placeholder for the dimension.

  2. Specify one or more optimization profiles at build time that specify the permitted range of dimensions for inputs with runtime dimensions and the dimensions for which the auto-tuner will optimize. For more information, refer to the Optimization Profiles section.

  3. To use the engine:

    1. Create an execution context from the engine, the same as without dynamic shapes.

    2. Specify one of the optimization profiles from step 2 that covers the input dimensions.

    3. Specify the input dimensions for the execution context. After setting input dimensions, you can get the output dimensions that TensorRT computes for the given input dimensions.

    4. Enqueue work.

To change the runtime dimensions, repeat steps 3b and 3c, which do not have to be repeated until the input dimensions change.