Manages the saved and loaded weights of a weight-stripped engine. More...
#include <NvInferRuntime.h>
Public Member Functions | |
| virtual | ~IWeightsManager () noexcept=0 |
| TRT_NODISCARD int64_t | getSize () const noexcept |
| Return the number of bytes written when saving or required when restoring the engine weights. More... | |
| TRT_NODISCARD bool | saveToBuffer (void *dst, int64_t dstSize, TensorLocation location, cudaStream_t stream) noexcept |
| Save the engine weights to an application buffer. More... | |
| TRT_NODISCARD bool | saveToStream (IStreamWriter &writer, cudaStream_t stream) noexcept |
| Save the engine weights to an application stream. More... | |
| TRT_NODISCARD bool | unload (cudaStream_t stream) noexcept |
| Unload the engine weights while preserving their virtual address range. More... | |
| TRT_NODISCARD bool | restoreFromBuffer (void const *src, int64_t srcSize, TensorLocation location, cudaStream_t stream) noexcept |
| Restore the engine weights by copying from an application buffer. More... | |
| TRT_NODISCARD bool | restoreFromVmmAllocation (uint64_t allocationHandle, int64_t offset, int64_t size, cudaStream_t stream) noexcept |
| Restore a range of engine weights by mapping an application CUDA VMM allocation. More... | |
Protected Attributes | |
| apiv::VWeightsManager * | mImpl |
Additional Inherited Members | |
 Protected Member Functions inherited from nvinfer1::INoCopy | |
| INoCopy ()=default | |
| virtual | ~INoCopy ()=default |
| INoCopy (INoCopy const &other)=delete | |
| INoCopy & | operator= (INoCopy const &other)=delete |
| INoCopy (INoCopy &&other)=delete | |
| INoCopy & | operator= (INoCopy &&other)=delete |
Detailed Description
Manages the saved and loaded weights of a weight-stripped engine.
The weight data produced by saveToBuffer() and saveToStream() is opaque. The application must preserve every byte unchanged. The data may be restored to the engine that produced it or to another engine deserialized from the same serialized engine with a compatible runtime and CUDA device.
The application owns the object and must destroy it before its ICudaEngine. Methods that change weight memory must not be used concurrently with inference, refit, serialization, execution-context creation, weight-streaming changes, or another IWeightsManager method for the same engine.
Constructor & Destructor Documentation
◆ ~IWeightsManager()
|
inlinepure virtualdefaultnoexcept |
Member Function Documentation
◆ getSize()
|
inlinenoexcept |
Return the number of bytes written when saving or required when restoring the engine weights.
The size includes any padding required by CUDA virtual-memory allocation granularity.
- Returns
- The weight data size in bytes.
◆ restoreFromBuffer()
|
inlinenoexcept |
Restore the engine weights by copying from an application buffer.
The weights must be unloaded or not yet loaded. TensorRT allocates physical memory and copies the saved weight data into it without running refit. The copy is ordered on stream, and src must remain valid until the operation completes.
- Parameters
-
src Source backup buffer. srcSize Source backup size in bytes; must equal getSize(). location Location of src.stream CUDA stream used to order the operation.
- Returns
- True on success, false otherwise.
◆ restoreFromVmmAllocation()
|
inlinenoexcept |
Restore a range of engine weights by mapping an application CUDA VMM allocation.
The weights must be unloaded or not yet fully initialized before the first call. This operation maps the allocation into the engine's reserved weights virtual address range without copying the saved weight data. Multiple calls may map nonoverlapping ranges from different allocations. The engine remains unavailable until the mappings cover the complete range [0, getSize()).
allocationHandle must identify a physical allocation created by cuMemCreate() on the engine's CUDA device. The allocation must be at least size bytes. Its contents beginning at physical allocation offset zero must match the range [offset, offset + size) of the weight data produced by saveToBuffer() or saveToStream(). Both offset and size must be multiples of the minimum allocation granularity returned by cuMemGetAllocationGranularity(). The range [offset, offset + size) must lie within [0, getSize()).
The application owns the allocation handle and must keep it valid until this call returns. After a successful call, the application may release the handle because TensorRT's mapping keeps the allocation alive. To reuse the allocation after unload() or engine destruction, the application must retain the handle.
- Parameters
-
allocationHandle Numeric value of the CUmemGenericAllocationHandle. offset Byte offset of the destination range within the saved weight data. size Size of the range in bytes. stream CUDA stream used to order the operation.
- Returns
- True on success, false otherwise.
◆ saveToBuffer()
|
inlinenoexcept |
Save the engine weights to an application buffer.
The weights must be loaded. The copy is ordered on stream, and dst must remain valid until the operation completes.
- Parameters
-
dst Destination buffer. dstSize Destination buffer size in bytes; must equal getSize(). location Location of dst.stream CUDA stream used to order the operation.
- Returns
- True on success, false otherwise.
◆ saveToStream()
|
inlinenoexcept |
Save the engine weights to an application stream.
The weights must be loaded. TensorRT writes exactly getSize() bytes in order. The data passed to IStreamWriter::write() is valid only for the duration of that call.
Each chunk is copied to host memory on stream. TensorRT synchronizes stream after each copy and before passing that chunk to IStreamWriter::write().
- Parameters
-
writer Destination stream writer. stream CUDA stream containing prior engine work.
- Returns
- True on success, false otherwise.
◆ unload()
|
inlinenoexcept |
Unload the engine weights while preserving their virtual address range.
The application must order all prior engine work on stream. This call synchronizes the stream before unmapping the physical memory. TensorRT preserves the virtual address range until the associated ICudaEngine is destroyed, allowing the weights to be restored later.
If the weights were restored from an application CUDA VMM allocation, TensorRT removes only its mapping. The application remains responsible for releasing any allocation handle it retained. The physical memory is reclaimed only after the mapping is removed and all allocation handles are released.
- Parameters
-
stream CUDA stream containing prior engine work.
- Returns
- True on success, false otherwise.
Member Data Documentation
◆ mImpl
|
protected |
The documentation for this class was generated from the following file:
