This code snippet demonstrates the how the DNN module is typically used. Note that error handling is left out for clarity.
Initialize network from file.
If the model has been generated on DLA using --useDLA
option with tensorrt_optimization tool, the processor type should be either DW_PROCESSOR_TYPE_DLA_0
or DW_PROCESSOR_TYPE_DLA_1
depending on which DLA engine the inference should take place. Otherwise, the processor type should always be DW_PROCESSOR_TYPE_GPU
.
contextHandle
is assumed to be a previously initialized dwContextHandle_t
.
Check that the loaded network has the expected number of inputs and outputs.
uint32_t numInputs = 0;
uint32_t numOutputs = 0;
if (numInputs != 1) {
std::cerr << "Expected a DNN with one input blob." << std::endl;
return -1;
}
if (numOutputs != 2) {
std::cerr << "Expected a DNN with two output blobs." << std::endl;
return -1;
}
Ask the DNN about the order of the input and output blobs. The network is assumed to contain the input blob "data_in" and output blobs "data_out1" and "data_out2".
uint32_t inputIndex = 0;
uint32_t output1Index = 0;
uint32_t output2Index = 0;
Initialize host and device memory to hold the inputs and outputs of the network.
std::vector<float32_t*> dnnInputs(numInputs, nullptr);
std::vector<float32_t*> dnnOutputs(numOutputs, nullptr);
std::vector<float32_t> dnnInputHost;
std::vector<std::vector<float32_t>> dnnOutputHost(numOutputs);
cudaMalloc(&dnnInputs[inputIndex],
sizeof(
float32_t) * numInputElements);
dnnInputHost.resize(numInputElements);
cudaMalloc(&dnnOutputs[output1Index],
sizeof(
float32_t) * numElements1);
cudaMalloc(&dnnOutputs[output2Index],
sizeof(
float32_t) * numElements2);
dnnOutputHost[output1Index].resize(numElements1);
dnnOutputHost[output2Index].resize(numElements2);
Copy DNN input from host buffers to device, then perform DNN inference and copy results back. All operations are performed asynchronously with the host code.
cudaMemcpyAsync(dnnInputs[inputIndex], dnnInputHost.data(),
sizeof(
float32_t) * numInputElements, cudaMemcpyHostToDevice);
cudaMemcpyAsync(dnnOutputHost[output1Index].data(), dnnOutputs[output1Index],
sizeof(
float32_t) * numElements1, cudaMemcpyDeviceToHost);
cudaMemcpyAsync(dnnOutputHost[output2Index].data(), dnnOutputs[output2Index],
sizeof(
float32_t) * numElements2, cudaMemcpyDeviceToHost);
otherUsefulWork();
cudaDeviceSynchronize();
Finally, free previously allocated memory.
cudaFree(dnnInputs[inputIndex]);
cudaFree(dnnOutputs[output1Index]);
cudaFree(dnnOutputs[output2Index]);
For more information see: