nvneural::BaseLayer Class Reference

BaseLayer provides common implementations for most of ILayer. More...

#include <BaseLayer.h>

Inheritance diagram for nvneural::BaseLayer:

Public Member Functions
float	activationCoefficient (std::size_t coefficientIndex) const noexcept
	Retrieves the activation coefficient for the specified index. More...
ActivationFunctionId	activationFunction () const noexcept
	Retrieves the activation function attached to this layer. More...
NetworkBackendId	backendId () const noexcept
	Returns the backend ID associated with this layer implementation. More...
NeuralResult	getConstData (const void **ppOut, TensorFormat format, const ILayer *pRequestingLayer) const noexcept
	Retrieves read-only device-side memory for the layer's output.
NeuralResult	getCpuConstData (void *pOutBuffer, size_t bufferByteCount, size_t *pBytesCopied, TensorFormat format) const noexcept
	Retrieves read-only CPU-side memory for the layer's output. More...
NeuralResult	getData (void **ppOut, TensorFormat format, const ILayer *pRequestingLayer) noexcept
	Retrieves device-side memory for the layer's output.
NeuralResult	getInputLayers (ILayerList **ppInputLayers) const noexcept
	Retrieves the inputs for this layer. More...
TensorDimension	internalDimensions () const noexcept
	Retrieves the dimensions of the layer's output tensor as allocated internally. More...
bool	isAffected () const noexcept
	Returns the current status of the "affected" flag. More...
bool	isPermanent () const noexcept
	Returns the current status of the "permanent" flag. More...
NeuralResult	loadFromParameters (const IParameterNode *pParameters) noexcept
	Loads layer parameters from a serialized key-value representation. More...
const char *	name () const noexcept
	Retrieves the layer name. More...
NeuralResult	setActivationCoefficient (std::size_t coefficientIndex, float value) noexcept
	Sets an activation coefficient. More...
NeuralResult	setActivationFunction (ActivationFunctionId activationFunction) noexcept
	Sets the activation function attached to the layer. More...
NeuralResult	setAffected (bool affected) noexcept
	Sets or clears the "affected" flag on a layer's output tensor. More...
NeuralResult	setInputLayer (std::size_t index, ILayer *pLayer) noexcept
	Sets an input layer by index. More...
NeuralResult	setName (const char *pName) noexcept
	Sets the layer name. More...
NeuralResult	setNetworkRuntime (INetworkRuntime *pNetworkRuntime) noexcept
	Informs the layer it has been attached to a new network. More...
NeuralResult	setPermanent (bool permanent) noexcept
	Sets or clears the "permanent" flag on a layer's output tensor. More...
NeuralResult	setWeightsName (const char *pWeightsName) noexcept
	Sets the name used to identify this layer's weights. More...
TensorDimension	stepping () const noexcept
	Returns the internal storage stride consumed by this layer implementation. More...
size_t	tensorBufferSize () const noexcept
	Retrieve the size of the layer's output tensor buffer in bytes. More...
TensorFormat	tensorFormat () const noexcept
	Returns the tensor format consumed by this layer implementation. More...
size_t	tensorInternalBufferSize () const noexcept
	Retrieves the dimensions of the layer's output tensor as allocated internally. More...
TensorDimension	weightsDimensions (const char *pWeightsName, WeightsQuery queryType) const noexcept
	Retrieves the tensor dimension of a layer's named weight input. More...
const char *	weightsName () const noexcept
	Retrieves the name used to identify this layer's weights. More...
Public Member Functions inherited from nvneural::ILayer
virtual TensorDimension	dimensions () const noexcept=0
	Retrieves the dimensions of the layer's output tensor. More...
virtual NeuralResult	evaluateForward () noexcept=0
	Performs forward evaluation for this layer. More...
virtual NeuralResult	reshape () noexcept=0
	Initializes (or reinitializes) the layer implementation with the current set of parameters. More...
virtual const char *	serializedType () const noexcept=0
	Retrieves the layer type. More...
Public Member Functions inherited from nvneural::IRefObject
virtual RefCount	addRef () const noexcept=0
	Increments the object's reference count. More...
virtual const void *	queryInterface (TypeId interface) const noexcept=0
	This is an overloaded member function, provided for convenience. It differs from the above function only in what argument(s) it accepts.
virtual void *	queryInterface (TypeId interface) noexcept=0
	Retrieves a new object interface pointer. More...
virtual RefCount	release () const noexcept=0
	Decrements the object's reference count and destroy the object if the reference count reaches zero. More...

Protected Member Functions
NeuralResult	allocateMemoryBlock (MemoryHandle *pHandle, size_t byteCount) noexcept
	Allocates a memory block of the requested size. More...
NeuralResult	allocateMemoryBlock (MemoryHandle *pHandle, size_t byteCount, const char *pTrackingKey) noexcept
	Allocates a memory block of the requested size and allows tracking of the memory block using a user-defined key. More...
void *	data (TensorFormat format)
	Returns a pointer to the output tensor memory in the provided format.
ILayer *	inputLayer (size_t index)
	This is an overloaded member function, provided for convenience. It differs from the above function only in what argument(s) it accepts.
const ILayer *	inputLayer (size_t index) const
	Returns the Nth input layer attached to this layer. More...
size_t	inputLayerCount () const
	Returns the number of input layers assigned to this object. More...
TensorDimension	loadedWeightsSize (const ILayer *pWeightsLayer, const char *pWeightsName) const
	Returns the size of already-loaded weights data, whether using a layer or IWeightsLoader. More...
RefPtr< INetworkBackend >	networkBackend () const
	Returns the INetworkBackend object associated with the implementation that most closely matches the current network.
INetworkRuntime *	networkRuntime () const
	Returns the INetworkRuntime object most recently set with setNetworkRuntime.
virtual void	onImplementationChanged () const
	Callback for derived classes to know that the "preferred implementation" has changed. More...
void	registerImplementation (NetworkBackendId backendId, TensorDataType elementType, TensorDataLayout layout, int penalty)
	Registers an implementation for multi-format layers. More...
void	requestReshape () noexcept
	Marks this layer as in need of reshape. More...
template<typename... Indices>
bool	verifyInputConnected (Indices... indices) const noexcept
	Test if inputs represented by indices are connected, and log errors if not. More...
const void *	weightsData (const char *pWeightsName) const
	Returns a pointer to device-side memory containing the indicated weights data. More...
Protected Member Functions inherited from nvneural::IRefObject
virtual	~IRefObject ()=default
	A protected destructor prevents accidental stack-allocation of IRefObjects or use with other smart pointer classes like std::unique_ptr.

Layer Helpers
enum class	LoadResult : std::uint16_t { Success = 0 , Failure = 1 , NonNumericInput = 2 , CouldNotTranslate = 3 }
	This enum describes a result from the load2ElementParameter function. More...
static LoadResult	load2ElementParameter (const std::vector< std::string > &inArray, size_t &outElementX, size_t &outElementY, size_t inFill, bool inFillFirstElement=false)
	Helper class that takes a vector of strings and loads 2 size_t elements. More...

Additional Inherited Members
Public Types inherited from nvneural::IRefObject
using	RefCount = std::uint32_t
	Typedef used to track the number of active references to an object.
using	TypeId = std::uint64_t
	Every interface must define a unique TypeId. This should be randomized.
Static Public Attributes inherited from nvneural::ILayer
static const IRefObject::TypeId	typeID = 0xcd07ffbfd74d9b11ul
	Interface TypeId for InterfaceOf purposes.
Static Public Attributes inherited from nvneural::IRefObject
static const TypeId	typeID = 0x14ecc3f9de638e1dul
	Interface TypeId for InterfaceOf purposes.

Detailed Description

BaseLayer provides common implementations for most of ILayer.

This class does not derive from RefObjectBase. Derived classes should consume these classes as in the following example:

class MyConcatLayer :
    public refobj::RefObjectBase<
        refobj::Implements<BaseLayer>,
        refobj::IndirectlyImplements<ILayer, BaseLayer>,
        refobj::Implements<IConcatenationLayer>>
{
    ...
};
class MyRegularLayer :
    public refobj::RefObjectBase<
        refobj::Implements<BaseLayer>>,
        refobj::IndirectlyImplements<ILayer, BaseLayer>>
{
    ...
};

This provides access to all interface methods without inheriting core interfaces such as ILayer or IRefObject more than once.

This class prefers not to own permanent references to its connected classes. Use of RefPtr<> or similar classes that call IRefObject::addReference can lead to object cycles and result in memory leaks.

For layers that wish to support multiple backends or tensor formats within a single class (useful for tuning, though will typically result in larger executable sizes), you should register your actually-supported tensor formats with a call to registerImplementation during object construction. At runtime, call tensorFormat to see the "closest match" tensor format to the network's current preferred tensor format.

Member Enumeration Documentation

LoadResult

enum nvneural::BaseLayer::LoadResult : std::uint16_t

strongprotected

This enum describes a result from the load2ElementParameter function.

Enumerator
Success	Parsing succeeded.
Failure	Parsing failed.
NonNumericInput	An input component contained a nondigit.
CouldNotTranslate	An input component could not be parsed as a number.

Member Function Documentation

activationCoefficient()

float BaseLayer::activationCoefficient ( std::size_t  coefficientIndex ) const

virtualnoexcept

Retrieves the activation coefficient for the specified index.

The meaning of this value is activation-specific. If no setActivationCoefficient call was previously issued for this index, the layer should return a default value of zero.

Parameters

coefficientIndex

Coefficient index

Implements nvneural::ILayer.

activationFunction()

ActivationFunctionId BaseLayer::activationFunction ( ) const

virtualnoexcept

Retrieves the activation function attached to this layer.

Should return the most recent value passed to setActivationFunction.

Implements nvneural::ILayer.

allocateMemoryBlock() [1/2]

NeuralResult BaseLayer::allocateMemoryBlock ( MemoryHandle *  pHandle, size_t  byteCount  )

protectednoexcept

Allocates a memory block of the requested size.

Uses the layer name as the tracking key, see allocateMemoryBlock below.

Parameters

pHandle	[out] Pointer receiving a MemoryHandle to the new memory
byteCount	Number of bytes to allocate

allocateMemoryBlock() [2/2]

NeuralResult BaseLayer::allocateMemoryBlock ( MemoryHandle *  pHandle, size_t  byteCount, const char *  pTrackingKey  )

protectednoexcept

Allocates a memory block of the requested size and allows tracking of the memory block using a user-defined key.

It is best practice to use the layer's name at this key.

Parameters

pHandle	[out] Pointer receiving a MemoryHandle to the new memory
byteCount	Number of bytes to allocate
pTrackingKey	A string key, usually the layer's name, which allows tracking of this allocation's information, nullptr to disable

backendId()

NetworkBackendId BaseLayer::backendId ( ) const

virtualnoexcept

Returns the backend ID associated with this layer implementation.

Networks will reshape tensors as necessary in order to match the return value.

Implements nvneural::ILayer.

getCpuConstData()

NeuralResult BaseLayer::getCpuConstData ( void *  pOutBuffer, size_t  bufferByteCount, size_t *  pBytesCopied, TensorFormat  format  ) const

virtualnoexcept

Retrieves read-only CPU-side memory for the layer's output.

This function is intended for debugging purposes. Element count of the buffer should be dimensions().elementCount(), multiplied by the size of the desired element type.

To check buffer sizes without preallocating memory, pass nullptr in pOutBuffer and zero in bufferByteCount. pBytesCopied will then receive the optimal size for pOutBuffer. Other uses of nullptr are considered errors.

pBytesCopied is not modified if this function fails.

Parameters

pOutBuffer	Memory region receiving the layer output data
bufferByteCount	Size of pOutBuffer in bytes
pBytesCopied	Optional output variable receiving the number of bytes copied into pOutBuffer
format	Format for the output buffer

Implements nvneural::ILayer.

getInputLayers()

NeuralResult BaseLayer::getInputLayers ( ILayerList **  ppInputLayers ) const

virtualnoexcept

Retrieves the inputs for this layer.

Parameters

ppInputLayers

Pointer receiving a reference to an ILayerList object.

Layer may create a new list or return an incremented reference to a cached object; this is an implementation detail.

In either case, caller is responsible for releasing its reference when done with *ppInputLayers.

Implements nvneural::ILayer.

inputLayer()

const ILayer * BaseLayer::inputLayer ( size_t  index ) const

protected

Returns the Nth input layer attached to this layer.

Exists to avoid needless round-tripping through ILayerList.

Returns

nullptr if no layer with this index has been assigned

inputLayerCount()

size_t BaseLayer::inputLayerCount ( ) const

protected

Returns the number of input layers assigned to this object.

Warning

Note that some layers in this range may be nullptr.

internalDimensions()

TensorDimension BaseLayer::internalDimensions ( ) const

virtualnoexcept

Retrieves the dimensions of the layer's output tensor as allocated internally.

Internal allocations may have additional padding or alignment restrictions.

See also

stepping()

Implements nvneural::ILayer.

isAffected()

bool BaseLayer::isAffected ( ) const

virtualnoexcept

Returns the current status of the "affected" flag.

By default, layers should be "affected" after initialization.

Implements nvneural::ILayer.

isPermanent()

bool BaseLayer::isPermanent ( ) const

virtualnoexcept

Returns the current status of the "permanent" flag.

By default, layers should not be "permanent."

Implements nvneural::ILayer.

load2ElementParameter()

static LoadResult nvneural::BaseLayer::load2ElementParameter ( const std::vector< std::string > &  inArray, size_t &  outElementX, size_t &  outElementY, size_t  inFill, bool  inFillFirstElement = false  )

inlinestaticprotected

Helper class that takes a vector of strings and loads 2 size_t elements.

Parameters

inArray	Vector of strings to load, only uses first 2 elements, if only 1 element, will use fill value as 2nd element
outElementX	First resulting size_t element
outElementY	Second resulting size_t element
inFill	If there is any issue with an element, like 1 element inArray, this will be the filler
inFillFirstElement	If this optional flag is true, the the inFill is disregarded, and the 1st element is used as the fill value

Returns

Failure is the only return in which the out elements are untouched, other returns are the result of the last issue encountered if not Success

loadedWeightsSize()

TensorDimension BaseLayer::loadedWeightsSize ( const ILayer *  pWeightsLayer, const char *  pWeightsName  ) const

protected

Returns the size of already-loaded weights data, whether using a layer or IWeightsLoader.

Parameters

pWeightsLayer	An optional pointer to a layer to be used preferentially for weights
pWeightsName	A weights identifier suitable for IWeightsLoader

Returns

Size of the loaded weights, or TensorDimension{0} if both are unavailable

loadFromParameters()

NeuralResult BaseLayer::loadFromParameters ( const IParameterNode *  pParameters )

virtualnoexcept

Loads layer parameters from a serialized key-value representation.

Tools such as ConverenceNG call this during network construction. Deployed applications may use this function themselves (often with a container class such as StringMapParameterNode) or may use layer-specific object interfaces; either approach is valid.

Implements nvneural::ILayer.

name()

const char * BaseLayer::name ( ) const

virtualnoexcept

Retrieves the layer name.

Returns

Layer name; pointer must be valid until object is destroyed or setName is called.

Implements nvneural::ILayer.

onImplementationChanged()

void BaseLayer::onImplementationChanged ( ) const

protectedvirtual

Callback for derived classes to know that the "preferred implementation" has changed.

Overriding this function is optional; it is provided purely for convenience.

Warning

The registerImplementation function invalidates the cached implementation; the next call to backendId, dimensions, or tensorFormat will cause this callback to trigger.

registerImplementation()

void BaseLayer::registerImplementation ( NetworkBackendId  backendId, TensorDataType  elementType, TensorDataLayout  layout, int  penalty  )

protected

Registers an implementation for multi-format layers.

Supports a "penalty" concept to help in cases of tie-breaking between competing implementations. In general, these are the rules for picking an implementation in backendId or tensorFormat:

1. Always choose an identical backend. CUDA-CUDA, DXML-DXML, etc.
2. If identical backends aren't possible, pick the CPU backend on the assumption that it costs only 1 trip to sysmem.
3. Choose an identical tensor format if possible. FP16/FP16, etc.
4. FUTURE: Choose a tensor format with fewer precision implications. Examples:
   • If trying to match FP16, choose FP32 over FP8. Don't narrow the data type.
   • If trying to match FP32, choose FP16 over FP8. Narrow the data type as little as possible.
   • If trying to match FP8, choose FP16 over FP32. If widening the data type, widen as little as possible to save memory.
5. Choose an identical tensor layout if possible.
6. Choose the implementation with a lower "penalty" value.
7. Choose in a deterministic-but-unspecified fashion for final tiebreaking.

Warning

This function is not thread-safe. Do not perform other operations on this layer while this function is executed.

Parameters

backendId	Backend ID to support
elementType	Element type to support
layout	Tensor layout to support
penalty	Penalty points to apply for this implementation

requestReshape()

void BaseLayer::requestReshape ( )

protectednoexcept

Marks this layer as in need of reshape.

Safe to call during parameter-setting methods where the layer may not yet have been attached to a network with INetwork::pushLayer.

setActivationCoefficient()

NeuralResult BaseLayer::setActivationCoefficient ( std::size_t  coefficientIndex, float  value  )

virtualnoexcept

Sets an activation coefficient.

The meaning of this value is activation-specific.

Parameters

coefficientIndex	Coefficient index
value	Value to store

Implements nvneural::ILayer.

setActivationFunction()

NeuralResult BaseLayer::setActivationFunction ( ActivationFunctionId  activationFunction )

virtualnoexcept

Sets the activation function attached to the layer.

Repeated calls to this function replace the activation rather than stacking them.

Parameters

activationFunction

New activation function ID

Implements nvneural::ILayer.

setAffected()

NeuralResult BaseLayer::setAffected ( bool  affected )

virtualnoexcept

Sets or clears the "affected" flag on a layer's output tensor.

Layers that are "affected" have pending computation work and will be reevaluated as necessary during inference. Networks will clear the "affected" flag after successful inference.

Layers should mark themselves as "affected" in response to parameter changes that would be visible during inference. Examples of such changes include loadFromParameters and the IRuntimeOptionsLayer::setRuntimeOptionValue callback.

Parameters

affected

New value for the "affected" flag

Implements nvneural::ILayer.

setInputLayer()

NeuralResult BaseLayer::setInputLayer ( std::size_t  index, ILayer *  pLayer  )

virtualnoexcept

Sets an input layer by index.

Input indices are explicitly allowed to be sparse in order to support optional inputs; usually such inputs are used to provide weights from other parts of the graph instead of going out to IWeightsLoader.

Parameters

index	Index of the input to assign
pLayer	Layer to assign as an input

Implements nvneural::ILayer.

setName()

NeuralResult BaseLayer::setName ( const char *  pName )

virtualnoexcept

Sets the layer name.

Layer names are unique within a network.

Parameters

pName

New layer name; layer must copy this data.

Implements nvneural::ILayer.

setNetworkRuntime()

NeuralResult BaseLayer::setNetworkRuntime ( INetworkRuntime *  pNetworkRuntime )

virtualnoexcept

Informs the layer it has been attached to a new network.

INetwork::pushLayer implementations will call this function for you; host applications do not normally need to call this function.

Parameters

pNetworkRuntime

New owning network

Implements nvneural::ILayer.

setPermanent()

NeuralResult BaseLayer::setPermanent ( bool  permanent )

virtualnoexcept

Sets or clears the "permanent" flag on a layer's output tensor.

Layers that are "permanent" are kept resident in memory and their tensor buffers are not reused. Setting too many layers as "permanent" will have negative effects on memory utilization.

Parameters

permanent

New value for the "permanent" flag

Implements nvneural::ILayer.

setWeightsName()

NeuralResult BaseLayer::setWeightsName ( const char *  pWeightsName )

virtualnoexcept

Sets the name used to identify this layer's weights.

Parameters

pWeightsName

New layer name to use for weights instead of name()

Implements nvneural::ILayer.

stepping()

TensorDimension BaseLayer::stepping ( ) const

virtualnoexcept

Returns the internal storage stride consumed by this layer implementation.

Output tensors are padded to be multiples of this value.

Implements nvneural::ILayer.

tensorBufferSize()

size_t BaseLayer::tensorBufferSize ( ) const

virtualnoexcept

Retrieve the size of the layer's output tensor buffer in bytes.

This result is defined as the number of elements in dimensions() multiplied by the size of an individual element (e.g., 4 for FP32).

Implements nvneural::ILayer.

tensorFormat()

TensorFormat BaseLayer::tensorFormat ( ) const

virtualnoexcept

Returns the tensor format consumed by this layer implementation.

Networks will reshape tensors as necessary in order to match the return value.

Implements nvneural::ILayer.

tensorInternalBufferSize()

size_t BaseLayer::tensorInternalBufferSize ( ) const

virtualnoexcept

Retrieves the dimensions of the layer's output tensor as allocated internally.

Internal allocations may have additional padding or alignment restrictions. The result is defined as the number of elements in internalDimensions() multiplied by the size of an individual element (e.g., 4 for FP32).

Implements nvneural::ILayer.

verifyInputConnected()

template<typename... Indices>

bool nvneural::BaseLayer::verifyInputConnected ( Indices...  indices ) const

inlineprotectednoexcept

Test if inputs represented by indices are connected, and log errors if not.

Parameters

indices

Input indices to test

Returns

True if all inputs are connected, false otherwise

weightsData()

const void * BaseLayer::weightsData ( const char *  pWeightsName ) const

protected

Returns a pointer to device-side memory containing the indicated weights data.

Parameters

pWeightsName

A weights identifier suitable for IWeightsLoader

Returns

A pointer to the weights data in the current configuration's tensor format, or nullptr on error

weightsDimensions()

TensorDimension BaseLayer::weightsDimensions ( const char *  pWeightsName, WeightsQuery  queryType  ) const

virtualnoexcept

Retrieves the tensor dimension of a layer's named weight input.

Parameters

pWeightsName	Name of a weights object being queried
queryType	Type of dimensions being queried; see WeightsQuery for details

Implements nvneural::ILayer.

weightsName()

const char * BaseLayer::weightsName ( ) const

virtualnoexcept

Retrieves the name used to identify this layer's weights.

By default, this should be the name assigned with setName. Alternate weights names allow layer aliasing and namespacing within subnetworks.

Implements nvneural::ILayer.

---

The documentation for this class was generated from the following files:

• Layer/Inc/BaseLayer.h
• Layer/Src/BaseLayer.cpp