Generic interface for CUDA device memory allocation. More...
#include <nvneural/CudaTypes.h>
Public Member Functions | |
| virtual NeuralResult | allocateMemoryBlock (MemoryHandle *pHandleOut, size_t byteCount, MemorySemantic semantic, const char *pTrackingKey, const char *pTrackingSubkey) noexcept=0 |
| Allocates a memory block of the requested size and allows tracking of the memory block using a given key. More... | |
| virtual NeuralResult | allocateMemoryBlock (MemoryHandle *pHandleOut, std::size_t byteCount, MemorySemantic semantic) noexcept=0 |
| Allocates a new memory block and returns a handle to it. More... | |
| virtual const MemoryTrackingData * | getMemoryTrackingData (const char *pTrackingKey, const char *pTrackingSubkey) const noexcept=0 |
| Compiles and returns memory data for the given key. More... | |
| virtual NeuralResult | getMemoryTrackingKeys (IStringList **ppKeysOut) noexcept=0 |
| Returns an IStringList of the currently tracked keys. More... | |
| virtual NeuralResult | getMemoryTrackingSubkeys (const char *pTrackingKey, IStringList **ppKeysOut) noexcept=0 |
| Returns an IStringList of the subkeys of given tracking key. More... | |
| virtual NeuralResult | setMemoryTrackingKey (const char *pTrackingKey, const char *pTrackingSubkey) noexcept=0 |
| Sets a potential tracking key. More... | |
 Public Member Functions inherited from nvneural::ICudaMemoryAllocator | |
| virtual NeuralResult | allocateMemoryBlock (MemoryHandle *pHandleOut, std::size_t byteCount, MemorySemantic semantic) noexcept=0 |
| Allocates a new memory block and returns a handle to it. More... | |
| virtual NeuralResult | compactMemory () noexcept=0 |
| Signals the allocator to release unused memory blocks back to the system. More... | |
| virtual NeuralResult | freeMemoryBlock (MemoryHandle handle) noexcept=0 |
| Frees a memory block. More... | |
| virtual void * | getAddressForMemoryBlock (MemoryHandle handle) const noexcept=0 |
| Converts a memory handle to a GPU virtual address. More... | |
| virtual std::size_t | getSizeForMemoryBlock (MemoryHandle handle) const noexcept=0 |
| Returns the buffer size associated with a memory handle. More... | |
| virtual NeuralResult | lockMemoryBlock (MemoryHandle handle) noexcept=0 |
| Adds a lock to a preexisting memory block. More... | |
| virtual NeuralResult | unlockMemoryBlock (MemoryHandle handle) noexcept=0 |
| Removes a lock from a preexisting memory block. More... | |
| Public Member Functions inherited from nvneural::IRefObject | |
| virtual RefCount | addRef () const noexcept=0 |
| Increments the object's reference count. More... | |
| virtual const void * | queryInterface (TypeId interface) const noexcept=0 |
| This is an overloaded member function, provided for convenience. It differs from the above function only in what argument(s) it accepts. | |
| virtual void * | queryInterface (TypeId interface) noexcept=0 |
| Retrieves a new object interface pointer. More... | |
| virtual RefCount | release () const noexcept=0 |
| Decrements the object's reference count and destroy the object if the reference count reaches zero. More... | |
Static Public Attributes | |
| static const IRefObject::TypeId | typeID = 0xd350a56002e2a077ul |
| Interface TypeId for InterfaceOf purposes. | |
| Static Public Attributes inherited from nvneural::ICudaMemoryAllocator | |
| static const IRefObject::TypeId | typeID = 0x121e6098096e5c97ul |
| Interface TypeId for InterfaceOf purposes. | |
| Static Public Attributes inherited from nvneural::IRefObject | |
| static const TypeId | typeID = 0x14ecc3f9de638e1dul |
| Interface TypeId for InterfaceOf purposes. | |
Additional Inherited Members | |
| Public Types inherited from nvneural::IRefObject | |
| using | RefCount = std::uint32_t |
| Typedef used to track the number of active references to an object. | |
| using | TypeId = std::uint64_t |
| Every interface must define a unique TypeId. This should be randomized. | |
| Protected Member Functions inherited from nvneural::IRefObject | |
| virtual | ~IRefObject ()=default |
| A protected destructor prevents accidental stack-allocation of IRefObjects or use with other smart pointer classes like std::unique_ptr. | |
Detailed Description
Generic interface for CUDA device memory allocation.
This is a revision of ICudaMemoryAllocator. See ICudaMemoryAllocator for more information.
Member Function Documentation
◆ allocateMemoryBlock() [1/2]
|
pure virtualnoexcept |
Allocates a memory block of the requested size and allows tracking of the memory block using a given key.
It is best practice to use the layer's name as the key and the semantic as the subkey, if that granularity is needed.
The tracking key has two main parts, a "key" which is the higher level tracking key, like a layer name, and the "subkey" which is tracked under the main key, like the semantic and weight name. If a subkey is not specified during tracking, "Unspecified" will be used.
This memory must be freed with freeMemoryBlock.
If this function fails, pHandle will receive nullptr as a value.
- Parameters
-
pHandleOut Pointer receiving a MemoryHandle to the new memory. byteCount Number of bytes to allocate. semantic Description of how the buffer will be used pTrackingKey A string key, usually the layer's name, which allows tracking of this allocation's information. Use nullptr to disable specific tracking. pTrackingSubkey A string key, usually the memory semantic, which allows an extra level of granularity of tracking this allocation's information. Use nullptr to ignore. If used, a tracking key is necessary.
◆ allocateMemoryBlock() [2/2]
|
noexcept |
Allocates a new memory block and returns a handle to it.
The returned memory has a lock count of 1; see lockMemoryBlock for details.
- Parameters
-
pHandleOut Variable receiving a new MemoryHandle object byteCount Size of buffer to allocate semantic Description of how the buffer will be used
◆ getMemoryTrackingData()
|
pure virtualnoexcept |
Compiles and returns memory data for the given key.
The returned memory data is in bytes. A nullptr is returned if the input(s) are null or if the tracking data could not be found.
- Parameters
-
pTrackingKey A string key, usually the layer's name. pTrackingSubkey A string key, usually from getMemoryTrackingSubkey.
◆ getMemoryTrackingKeys()
|
pure virtualnoexcept |
Returns an IStringList of the currently tracked keys.
The order of the keys in the list is unspecified. This list is retrieved from the allocator, so high-performance/low-overhead allocators are allowed to ignore tracking keys and return an empty list.
Special keys: { network } : memory allocations of the entire network { untracked } : memory allocations where no tracking key was given
Failed result is returned for null input.
- Parameters
-
ppKeysOut Variable receiving a reference to a new IStringList of tracked keys. Caller must release the reference.
◆ getMemoryTrackingSubkeys()
|
pure virtualnoexcept |
Returns an IStringList of the subkeys of given tracking key.
The order of the keys in the list is unspecified. This list is retrieved from the allocator, so high-performance/low-overhead allocators are allowed to ignore tracking keys and return an empty list.
Special keys: { all } : memory allocations of the entire key (usually the layer's name)
Failed result is returned for null input(s).
- Parameters
-
pTrackingKey A string key, usually the layer's name. ppKeysOut Variable receiving a reference to a new IStringList of tracked subkeys. Caller must release the reference.
◆ setMemoryTrackingKey()
|
pure virtualnoexcept |
Sets a potential tracking key.
The tracking key has two main parts, a key and optional subkey, which are described in the function ICudaMemoryAllocator2::allocateMemoryBlock. This function sets a key that will be used when the INetworkBackend::allocateMemoryBlock is used to allocate memory which does not contain a tracking key parameter. This function should be called before any memory allocations for a tracking need and should be called after to clear the key (nullptr). Normally this is called during reshapes and evaluteForwards with the layer's name.
Failed result is returned when a prefix could not be set.
- Parameters
-
pTrackingKey A string key, usually the layer's name. pTrackingSubkey A string key, usually the memory semantic.
The documentation for this class was generated from the following file:
- Core/Inc/nvneural/CudaTypes.h
