Writing Python bindings for a C++ Operator
For convenience while maintaining high performance, operators written in C++ can be wrapped in Python. The general approach uses Pybind11 to concisely create bindings that provide a familiar, Pythonic experience to application authors.
While we provide some utilities to simplify part of the process, this section is designed for advanced developers, since the wrapping of the C++ class using pybind11 is mostly manual and can vary between each operator.
The existing Pybind11 documentation is good and it is recommended to read at least the basics on wrapping functions and classes. The material below will assume some basic familiarity with Pybind11, covering the details of creation of the bindings of a C++ Operator
. As a concrete example, we will cover creation of the bindings for ToolTrackingPostprocessorOp from Holohub as a simple case and then highlight additional scenarios that might be encountered.
There are several examples of bindings on Holohub in the operators folder. The subset of operators that provide a Python wrapper on top of a C++ implementation will have any C++ headers and sources together in a common folder, while any corresponding Python bindings will be in a “python” subfolder (see the tool_tracking_postprocessor folder layout, for example).
There are also several examples of bindings for the built-in operators of the SDK. Unlike on Holohub, for the SDK, the corresponding C++ headers and sources of an operator are stored under separate directory trees.
Creating a PyToolTrackingPostprocessorOp trampoline class
In a C++ file (tool_tracking_postprocessor.cpp in this case), create a subclass of the C++ Operator class to wrap. The general approach taken is to create a Python-specific class that provides a constructor that takes a Fragment*
, an explicit list of the operators parameters with default values for any that are optional, and an operator name. This constructor needs to setup the operator as done in Fragment::make_operator
, so that it is ready for initialization by the GXF executor. We use the convention of prepending “Py” to the C++ class name for this (so, PyToolTrackingPostprocessorOp
in this case). :
Listing 18 tool_tracking_post_processor/python/tool_tracking_post_processor.cpp
class PyToolTrackingPostprocessorOp : public ToolTrackingPostprocessorOp {
public:
/* Inherit the constructors */
using ToolTrackingPostprocessorOp::ToolTrackingPostprocessorOp;
// Define a constructor that fully initializes the object.
PyToolTrackingPostprocessorOp(
Fragment* fragment, const py::args& args, std::shared_ptr<Allocator> device_allocator,
std::shared_ptr<Allocator> host_allocator, float min_prob = 0.5f,
std::vector<std::vector<float>> overlay_img_colors = VIZ_TOOL_DEFAULT_COLORS,
std::shared_ptr<holoscan::CudaStreamPool> cuda_stream_pool = nullptr,
const std::string& name = "tool_tracking_postprocessor")
: ToolTrackingPostprocessorOp(ArgList{Arg{"device_allocator", device_allocator},
Arg{"host_allocator", host_allocator},
Arg{"min_prob", min_prob},
Arg{"overlay_img_colors", overlay_img_colors},
}) {
if (cuda_stream_pool) { this->add_arg(Arg{"cuda_stream_pool", cuda_stream_pool}); }
add_positional_condition_and_resource_args(this, args);
name_ = name;
fragment_ = fragment;
spec_ = std::make_shared<OperatorSpec>(fragment);
setup(*spec_.get());
}
};
This constructor will allow providing a Pythonic experience for creating the operator. Specifically, the user can pass Python objects for any of the parameters without having to explicitly create any holoscan::Arg
objects via holoscan.core.Arg
. For example, a standard Python float can be passed to min_prob
and a Python list[list[float]]
can be passed for overlay_img_colors
(Pybind11 handles conversion between the C++ and Python types). Pybind11 will also take care of conversion of a Python allocator class like holoscan.resources.UnboundedAllocator
or holoscan.resources.BlockMemoryPool
to the underlying C++ std::shared_ptr<holoscan::Allocator>
type. The arguments device_allocator
and host_allocator
correspond to required Parameters of the C++ class and can be provided from Python either positionally or via keyword while the Parameters min_prob
and overlay_img_colors
will be optional keyword arguments. cuda_stream_pool
is also optional, but is only conditionally passed as an argument to the underlying ToolTrackingPostprocessorOp
constructor when it is not a nullptr
.
For all operators, the first argument should be
Fragment* fragment
and is the fragment the operator will be assigned to. In the case of a single fragment application (i.e. not a distributed application), the fragment is just the application itself.An (optional)
const std::string& name
argument should be provided to enable the application author to set the operator’s name.The
const py::args& args
argument corresponds to the*args
notation in Python. It is a set of 0 or more positional arguments. It is not required to provide this in the function signature, but is recommended in order to enable passing additional conditions such as aCountCondition
orPeriodicCondtion
as positional arguments to the operator. The call below toadd_positional_condition_and_resource_args(this, args);
uses a helper function defined in operator_util.hpp to add any
Condition
orResource
arguments found in the list of positional arguments.The other arguments all correspond to the various parameters (
holoscan::Parameter
) that are defined for the C++ToolTrackingPostProcessorOp
class.All other parameters except
cuda_stream_pool
are passed directly in the argument list to the parentToolTrackingPostProcessorOp
class. The parameters present on the C++ operator can be seen in its header here with default values taken from thesetup
method of the source file here. Note thatCudaStreamHandler
is a utility that will add a parameter of typeParameter<std::shared_ptr<CudaStreamPool>>
.The
cuda_stream_pool
argument is only conditionally added if it was notnullptr
(Python’sNone
). This is done viaif (cuda_stream_pool) { this->add_arg(Arg{"cuda_stream_pool", cuda_stream_pool}); }
instead of passing it as part of the
holoscan::ArgList
provided to theToolTrackingPostprocessorOp
constructor call above.
The remaining lines of the constructor
name_ = name;
fragment_ = fragment;
spec_ = std::make_shared<OperatorSpec>(fragment);
setup(*spec_.get());
are required to properly initialize it and should be the same across all operators. These correspond to equivalent code within the Fragment::make_operator method.
Defining the Python module
For this operator, there are no other custom classes aside from the operator itself, so we define a module using PYBIND11_MODULE
as shown below with only a single class definition. This is done in the same tool_tracking_postprocessor.cpp file where we defined the PyToolTrackingPostprocessorOp
trampoline class.
The following header will always be needed.
#include <pybind11/pybind11.h>
namespace py = pybind11;
using pybind11::literals::operator""_a;
Here, we typically also add defined the py
namespace as a shorthand for pybind11
and indicated that we will use the _a
literal (it provides a shorthand notation when defining keyword arguments).
Often it will be necessary to include the following header if any parameters to the operator involve C++ standard library containers such as std::vector
or std::unordered_map
.
#include <pybind11/stl.h>
This allows pybind11 to cast between the C++ container types and corresponding Python types (Python dict
/ C++ std::unordered_map
, for example).
Listing 19 tool_tracking_post_processor/python/tool_tracking_post_processor.cpp
PYBIND11_MODULE(_tool_tracking_postprocessor, m) {
py::class_<ToolTrackingPostprocessorOp,
PyToolTrackingPostprocessorOp,
Operator,
std::shared_ptr<ToolTrackingPostprocessorOp>>(
m,
"ToolTrackingPostprocessorOp",
doc::ToolTrackingPostprocessorOp::doc_ToolTrackingPostprocessorOp_python)
.def(py::init<Fragment*,
const py::args& args,
std::shared_ptr<Allocator>,
std::shared_ptr<Allocator>,
float,
std::vector<std::vector<float>>,
std::shared_ptr<holoscan::CudaStreamPool>,
const std::string&>(),
"fragment"_a,
"device_allocator"_a,
"host_allocator"_a,
"min_prob"_a = 0.5f,
"overlay_img_colors"_a = VIZ_TOOL_DEFAULT_COLORS,
"cuda_stream_pool"_a = py::none(),
"name"_a = "tool_tracking_postprocessor"s,
doc::ToolTrackingPostprocessorOp::doc_ToolTrackingPostprocessorOp_python);
} // PYBIND11_MODULE NOLINT
If you are implementing the python wrapping in Holohub, the
<module_name>
passed toPYBIND_11_MODULE
must match_<CPP_CMAKE_TARGET>
as covered above.If you are implementing the python wrapping in a standalone CMake project,the
<module_name>
passed toPYBIND_11_MODULE
must match the name of the module passed to the pybind11-add-module CMake function.
Using a mismatched name in PYBIND_11_MODULE
will result in failure to import the module from Python.
The order in which the classes are specified in the py::class_<>
template call is important and should follow the convention shown here. The first in the list is the C++ class name (ToolTrackingPostprocessorOp
) and second is the PyToolTrackingPostprocessorOp
class we defined above with the additional, explicit constructor. We also need to list the parent Operator
class so that all of the methods such as start
, stop
, compute
, add_arg
, etc. that were already wrapped for the parent class don’t need to be redefined here.
The single .def(py::init<...
call wraps the PyToolTrackingPostprocessorOp
constructor we wrote above. As such, the argument types provided to py::init<>
must exactly match the order and types of arguments in that constructor’s function signature. The subsequent arguments to def
are the names and default values (if any) for the named arguments in the same order as the function signature. Note that the const py::args& args
(Python *args
) argument is not listed as these are positional arguments that don’t have a corresponding name. The use of py::none()
(Python’s None
) as the default for cuda_stream_pool
corresponds to the nullptr
in the C++ function signature. The “_a” literal used in the definition is enabled by the following declaration earlier in the file.
The final argument to .def
here is a documentation string that will serve as the Python docstring for the function. It is optional and we chose here to define it in a separate header as described in the next section.
Documentation strings
Prepare documentation strings (const char*
) for your python class and its parameters.
Below we use a PYDOC
macro defined in the SDK and available in HoloHub as a utility to remove leading spaces. In this case, the documentation code is located in header file tool_tracking_post_processor_pydoc.hpp, under a custom holoscan::doc::ToolTrackingPostprocessorOp
namespace. None of this is required, you just need to make any documentation strings available for use as an argument to the py::class_
constructor or method definition calls.
Listing 20 tool_tracking_post_processor/python/tool_tracking_post_processor_pydoc.hpp
#include "../macros.hpp"
namespace holoscan::doc {
namespace ToolTrackingPostprocessorOp {
// PyToolTrackingPostprocessorOp Constructor
PYDOC(ToolTrackingPostprocessorOp_python, R"doc(
Operator performing post-processing for the endoscopy tool tracking demo.
**==Named Inputs==**
in : nvidia::gxf::Entity containing multiple nvidia::gxf::Tensor
Must contain input tensors named "probs", "scaled_coords" and "binary_masks" that
correspond to the output of the LSTMTensorRTInfereceOp as used in the endoscopy
tool tracking example applications.
**==Named Outputs==**
out_coords : nvidia::gxf::Tensor
Coordinates tensor, stored on the host (CPU).
out_mask : nvidia::gxf::Tensor
Binary mask tensor, stored on device (GPU).
Parameters
----------
fragment : Fragment
The fragment that the operator belongs to.
device_allocator : ``holoscan.resources.Allocator``
Output allocator used on the device side.
host_allocator : ``holoscan.resources.Allocator``
Output allocator used on the host side.
min_prob : float, optional
Minimum probability (in range [0, 1]). Default value is 0.5.
overlay_img_colors : sequence of sequence of float, optional
Color of the image overlays, a list of RGB values with components between 0 and 1.
The default value is a qualitative colormap with a sequence of 12 colors.
cuda_stream_pool : ``holoscan.resources.CudaStreamPool``, optional
`holoscan.resources.CudaStreamPool` instance to allocate CUDA streams.
Default value is ``None``.
name : str, optional
The name of the operator.
)doc")
} // namespace ToolTrackingPostprocessorOp
} // namespace holoscan::doc
We tend to use NumPy-style docstrings for parameters, but also encourage adding a custom section at the top that describes the input and output ports and what type of data is expected on them. This can make it easier for developers to use the operator without having to inspect the source code to determine this information.
Configuring with CMake
We use CMake to configure pybind11 and build the bindings for the C++ operator you wish to wrap. There are two approaches detailed below, one for HoloHub (recommended), one for standalone CMake projects.
To have your bindings built, ensure the CMake code below is executed as part of a CMake project which already defines the C++ operator as a CMake target, either built in your project (with add_library
) or imported (with find_package
or find_library
).
We provide a CMake utility function named pybind11_add_holohub_module in HoloHub to facilitate configuring and building your python bindings.
In our skeleton code below, a top-level CMakeLists.txt which already defined the tool_tracking_postprocessor
target for the C++ operator would need to do add_subdirectory(tool_tracking_postprocessor)
to include the following CMakeLists.txt. The pybind11_add_holohub_module
lists that C++ operator target, the C++ class to wrap, and the path to the C++ binding source code we implemented above. Note how the module name provided as the first argument to PYPBIND11_MODULE needs to match _<CPP_CMAKE_TARGET>
(_tool_tracking_postprocessor_op
in this case).
Listing 21 tool_tracking_postprocessor/python/CMakeLists.txt
include(pybind11_add_holohub_module)
pybind11_add_holohub_module(
CPP_CMAKE_TARGET tool_tracking_postprocessor
CLASS_NAME "ToolTrackingPostprocessorOp"
SOURCES tool_tracking_postprocessor.cpp
)
The key details here are that CLASS_NAME
should match the name of the C++ class that is being wrapped and is also the name that will be used for the class from Python. SOURCES
should point to the file where the C++ operator that is being wrapped is defined. The CPP_CMAKE_TARGET
name will be the name of the holohub package submodule that will contain the operator.
Note that the python subdirectory where this CMakeLists.txt resides is reachable thanks to the add_subdirectory(python)
in the CMakeLists.txt one folder above, but that’s an arbitrary opinionated location and not a required directory structure.
Follow the pybind11 documentation to configure your CMake project to use pybind11. Then, use the pybind11_add_module function with the cpp files containing the code above, and link against holoscan::core
and the library that exposes your C++ operator to wrap.
Listing 22 my_op_python/CMakeLists.txt
pybind11_add_module(my_python_module my_op_pybind.cpp)
target_link_libraries(my_python_module
PRIVATE holoscan::core
PUBLIC my_op
)
Example: in the SDK, this is done here.
The name chosen for CPP_CMAKE_TARGET
must also be used (along with a preceding underscore) as the module name passed as the first argument to the PYBIND11_MODULE macro in the bindings.
Note that there is an initial underscore prepended to the name. This is the naming convention used for the shared library and corresponding __init__.py
file that will be generated by the pybind11_add_holohub_module
helper function above.
If the name is specified incorrectly, the build will still complete, but at application run time an ImportError
such as the following would occur
[command] python3 /workspace/holohub/applications/endoscopy_tool_tracking/python/endoscopy_tool_tracking.py --data /workspace/holohub/data/endoscopy
Traceback (most recent call last):
File "/workspace/holohub/applications/endoscopy_tool_tracking/python/endoscopy_tool_tracking.py", line 38, in <module>
from holohub.tool_tracking_postprocessor import ToolTrackingPostprocessorOp
File "/workspace/holohub/build/python/lib/holohub/tool_tracking_postprocessor/__init__.py", line 19, in <module>
from ._tool_tracking_postprocessor import ToolTrackingPostprocessorOp
ImportError: dynamic module does not define module export function (PyInit__tool_tracking_postprocessor)
Importing the class in Python
When building your project, two files will be generated inside <build_or_install_dir>/python/lib/holohub/<CPP_CMAKE_TARGET>
(e.g. build/python/lib/holohub/tool_tracking_postprocessor/
):
the shared library for your bindings (
_tool_tracking_postprocessor_op.cpython-<pyversion>-<arch>-linux-gnu.so
)an
__init__.py
file that makes the necessary imports to expose this in python
Assuming you have export PYTHONPATH=<build_or_install_dir>/python/lib/
, you should then be able to create an application in Holohub that imports your class via:
from holohub.tool_tracking_postprocessor_op import ToolTrackingPostProcessorOp
Example: ToolTrackingPostProcessorOp
is imported in the Endoscopy Tool Tracking application on HoloHub here.
When building your project, a shared library file holding the python bindings and named my_python_module.cpython-<pyversion>-<arch>-linux-gnu.so
will be generated inside <build_or_install_dir>/my_op_python
(configurable with OUTPUT_NAME
and LIBRARY_OUTPUT_DIRECTORY
respectively in CMake).
From there, you can import it in python via:
import holoscan.core
import holoscan.gxf # if your c++ operator uses gxf extensions
from <build_or_install_dir>.my_op_python import MyOp
To imitate HoloHub’s behavior, you can also place that file alongside the .so file, name it __init__.py
, and replace <build_or_install_dir>.
by .
. It can then be imported as a python module, assuming <build_or_install_dir>
is a module under the PYTHONPATH
environment variable.
In this section we will cover other cases that may occasionally be encountered when writing Python bindings for operators.
Optional arguments
It is also possible to use std::optional
to handle optional arguments. The ToolTrackingProcessorOp
example above, for example, has a default argument defined in the spec for min_prob
.
constexpr float DEFAULT_MIN_PROB = 0.5f;
// ...
spec.param(
min_prob_, "min_prob", "Minimum probability", "Minimum probability.", DEFAULT_MIN_PROB);
In the tutorial for ToolTrackingProcessorOp
above we reproduced this default of 0.5 in both the PyToolTrackingProcessorOp
constructor function signature as well as the Python bindings defined for it. This carries the risk that the default could change at the C++ operator level without a corresponding change being made for Python.
An alternative way to define the constructor would have been to use std::optional
as follows
// Define a constructor that fully initializes the object.
PyToolTrackingPostprocessorOp(
Fragment* fragment, const py::args& args, std::shared_ptr<Allocator> device_allocator,
std::shared_ptr<Allocator> host_allocator, std::optional<float> min_prob = 0.5f,
std::optional<std::vector<std::vector<float>>> overlay_img_colors = VIZ_TOOL_DEFAULT_COLORS,
std::shared_ptr<holoscan::CudaStreamPool> cuda_stream_pool = nullptr,
const std::string& name = "tool_tracking_postprocessor")
: ToolTrackingPostprocessorOp(ArgList{Arg{"device_allocator", device_allocator},
Arg{"host_allocator", host_allocator},
}) {
if (cuda_stream_pool) { this->add_arg(Arg{"cuda_stream_pool", cuda_stream_pool}); }
if (min_prob.has_value()) { this->add_arg(Arg{"min_prob", min_prob.value() }); }
if (overlay_img_colors.has_value()) {
this->add_arg(Arg{"overlay_img_colors", overlay_img_colors.value() });
}
add_positional_condition_and_resource_args(this, args);
name_ = name;
fragment_ = fragment;
spec_ = std::make_shared<OperatorSpec>(fragment);
setup(*spec_.get());
}
where now that min_prob
and overlay_img_colors
are optional, they are only conditionally added as an argument to ToolTrackingPostprocessorOp when they have a value. If this approach is used, the Python bindings for the constructor should be updated to use py::none()
as the default as follows:
.def(py::init<Fragment*,
const py::args& args,
std::shared_ptr<Allocator>,
std::shared_ptr<Allocator>,
float,
std::vector<std::vector<float>>,
std::shared_ptr<holoscan::CudaStreamPool>,
const std::string&>(),
"fragment"_a,
"device_allocator"_a,
"host_allocator"_a,
"min_prob"_a = py::none(),
"overlay_img_colors"_a = py::none(),
"cuda_stream_pool"_a = py::none(),
"name"_a = "tool_tracking_postprocessor"s,
doc::ToolTrackingPostprocessorOp::doc_ToolTrackingPostprocessorOp_python);
C++ enum parameters as arguments
Sometimes, operators may use a Parameter with an enum type. It is necessary to wrap the C++ enum to be able to use it as a Python type when providing the argument to the operator.
The built-in holoscan::ops::AJASourceOp
is an example of a C++ operator that takes a enum Parameter (an NTV2Channel
enum).
The enum can easily be wrapped for use from Python via py::enum_
as shown here. It is recommended in this case to follow Python’s convention of using capitalized names in the enum.
(Advanced) Custom C++ classes as arguments
Sometimes it is necessary to accept a custom C++ class type as an argument in the operator’s constructor. In this case additional interface code and bindings will likely be necessary to support the type.
A relatively simple example of this is the DataVecMap
type used by InferenceProcessorOp
. In that case, the type is a structure that holds an internal std::map<std:string, std::vector<std::string>>
. The bindings are written to accept a Python dict (py::dict
) and a helper function is used within the constructor to convert that dictionary to the corresponding C++ DataVecMap
.
A more complicated case is the use of a InputSpec
type in the HolovizOp
bindings. This case involves creating Python bindings for classes InputSpec
and View
as well as a couple of enum types. To avoid the user having to build a list[holoscan.operators.HolovizOp.InputSpec]
directly to pass as the tensors
argument, an additional Python wrapper class was defined in the __init__.py
to allow passing a simple Python dict for the tensors
argument and any corresponding InputSpec classes are automatically created in its constructor before calling the underlying Python bindings class.
Customizing the C++ types a Python operator can emit or receive
In some instances, users may wish to be able to have a Python operator receive and/or emit a custom C++ type. As a first example, suppose we are wrapping an operator that emits a custom C++ type. We need any downstream native Python operators to be able to receive that type. By default the SDK is able to handle the needed C++ types for built in operators like std::vector<holoscan::ops::HolovizOp::InputSpec>
. The SDK provides an EmitterReceiverRegistry
class that 3rd party projects can use to register receiver
and emitter
methods for any custom C++ type that needs to be handled. To handle a new type, users should implement an emitter_receiver<T>
struct for the desired type as in the example below. We will first cover the general steps necessary to register such a type and then cover where some steps may be omitted in certain simple cases.
Step 1: define emitter_receiver::emit and emitter_receiver::receive methods
Here is an example for the built-in std::vector<holoscan::ops::HolovizOp::InputSpec>
used by HolovizOp
to define the input specifications for its received tensors.
#include <holoscan/python/core/emitter_receiver_registry.hpp>
namespace py = pybind11;
namespace holoscan {
/* Implements emit and receive capability for the HolovizOp::InputSpec type.
*/
template <>
struct emitter_receiver<std::vector<holoscan::ops::HolovizOp::InputSpec>> {
static void emit(py::object& data, const std::string& name, PyOutputContext& op_output) {
auto input_spec = data.cast<std::vector<holoscan::ops::HolovizOp::InputSpec>>();
py::gil_scoped_release release;
op_output.emit<std::vector<holoscan::ops::HolovizOp::InputSpec>>(input_spec, name.c_str());
return;
}
static py::object receive(std::any result, const std::string& name, PyInputContext& op_input) {
HOLOSCAN_LOG_DEBUG("py_receive: std::vector<HolovizOp::InputSpec> case");
// can directly return vector<InputSpec>
auto specs = std::any_cast<std::vector<holoscan::ops::HolovizOp::InputSpec>>(result);
py::object py_specs = py::cast(specs);
return py_specs;
}
};
}
This emitter_receiver
class defines a receive
method that takes a std::any
message and casts it to the corresponding Python list[HolovizOp.InputSpect]
object. Here the pybind11::cast
call works because we have wrapped the HolovizOp::InputSpec
class here.
Similarly, the emit
method takes a pybind11::object
(of type list[HolovizOp.InputSpect]
) and casts it to the corresponding C++ type, std::vector<holoscan::ops::HolovizOp::InputSpec>
. The conversion between std::vector
and a Python list is one of Pbind11’s built-in conversions (available as long as “pybind11/stl.h” has been included).
The signature of the emit
and receive
methods must exactly match the case shown here.
Step 2: Create a register_types method for adding custom types to the EmitterReceiverRegistry.
The bindings in this operators module, should define a method named register_types
that takes a reference to an EmitterReceiverRegistry
as its only argument. Within this function there should be a call to EmitterReceiverRegistry::add_emitter_receiver
for each type that this operator wished to register. The HolovizOp defines this method using a lambda function
// Import the emitter/receiver registry from holoscan.core and pass it to this function to
// register this new C++ type with the SDK.
m.def("register_types", [](EmitterReceiverRegistry& registry) {
registry.add_emitter_receiver<std::vector<holoscan::ops::HolovizOp::InputSpec>>(
"std::vector<HolovizOp::InputSpec>"s);
// array camera pose object
registry.add_emitter_receiver<std::shared_ptr<std::array<float, 16>>>(
"std::shared_ptr<std::array<float, 16>>"s);
// Pose3D camera pose object
registry.add_emitter_receiver<std::shared_ptr<nvidia::gxf::Pose3D>>(
"std::shared_ptr<nvidia::gxf::Pose3D>"s);
// camera_eye_input, camera_look_at_input, camera_up_input
registry.add_emitter_receiver<std::array<float, 3>>("std::array<float, 3>"s);
});
Here the following line registers the std::vector<holoscan::ops::HolovizOp::InputSpec>
type
that we wrote an emitter_receiver
for above.
registry.add_emitter_receiver<std::vector<holoscan::ops::HolovizOp::InputSpec>>(
"std::vector<HolovizOp::InputSpec>"s);
Internally the registry stores a mapping between the C++ std::type_index
of the type specified in the template argument and the emitter_receiver
defined for that type. The second argument is a string that the user can choose which is a label for the type. As we will see later, this label can be used from Python to indicate that we want to emit using the emitter_receiver::emit
method that was registered for a particular label.
Step 3: In the init.py file for the Python module defining the operator call register_types
To register types with the core SDK, we need to import the io_type_registry
class (of type EmitterReceiverRegistry
) from holoscan.core
. We then pass that class as input to the register_types
method defined in step 2 to register the 3rd party types with the core SDK.
from holoscan.core import io_type_registry
from ._holoviz import register_types as _register_types
# register methods for receiving or emitting list[HolovizOp.InputSpec] and camera pose types
_register_types(io_type_registry)
where we chose to import register_types
with an initial underscore as a common Python convention to indicate it is intended to be “private” to this module.
In some cases steps 1 and 3 as shown above are not necessary.
When creating Python bindings for an Operator on Holohub, the pybind11_add_holohub_module.cmake utility mentioned above will take care of autogenerating the __init__.py
as shown in step 3, so it will not be necessary to manually create it in that case.
For types for which Pybind11’s default casting between C++ and Python is adequate, it is not necessary to explicitly define the emitter_receiver
class as shown in step 1. This is true because there are a couple of default implementations for emitter_receiver<T>
and emitter_receiver<std::shared_ptr<T>>
that already cover common cases. The default emitter_receiver works for the std::vector<HolovizOp::InputSpec>
type shown above, which is why the code shown for illustration there is not found within the operator’s bindings. In that case one could immediately implement register_types
from step 2 without having to explicitly create an emitter_receiver
class.
An example where the default emitter_receiver
would not work is the custom one defined by the SDK for pybind11::dict
. In this case, to provide convenient emit of multiple tensors via passing a dict[holoscan::Tensor]
to op_output.emit
we have special handling of Python dictionaries. The dictionary is inspected and if all keys are strings and all values are tensor-like objects, a single C++ nvidia::gxf::Entity
containing all of the tensors as an nvidia::gxf::Tensor
is emitted. If the dictionary is not a tensor map, then it is just emitted as a shared pointer to the Python dict object. The emitter_receiver
implementations used for the core SDK are defined in emitter_receivers.hpp. These can serve as a reference when creating new ones for additional types.
Runtime behavior of emit and receive
After registering a new type, receive of that type on any input port will automatically be handled. This is because due to the strong typing of C++, any op_input.receive
call in an operator’s compute
method can find the registered receive
method that matches the std::type_index
of the type and use that to convert to a corresponding Python object.
Because Python is not strongly typed, on emit
, the default behavior remains emitting a shared pointer to the Python object itself. If we instead want to emit
a C++ type, we can pass a 3rd argument to op_output.emit
to specify the name that we used when registering the types via the add_emitter_receiver
call as above.
Example of emitting a C++ type
As a concrete example, the SDK already registers std::string
by default. If we wanted, for instance, to emit a Python string as a C++ std::string
for use by a downstream operator that is wrapping a C++ operator expecting string input, we would add a 3rd argument to the op_output.emit
call as follows
# emit a Python filename string on port "filename_out" using registered type "std::string"
my_string = "filename.raw"
op_output.emit(my_string, "filename_out", "std::string")
This specifies that the emit
method that converts to C++ std::string
should be used instead of the default behavior of emitting the Python string.
Another example would be to emit a Python List[float]
as a std::array<float, 3>
parameter as input to the camera_eye
, camera_look_at
or camera_up
input ports of HolovizOp
.
op_output.emit([0.0, 1.0, 0.0], "camera_eye_out", "std::array<float, 3>")
Only types registered with the SDK can be specified by name in this third argument to emit
.
Table of types registered by the core SDK
The list of types that are registered with the SDK’s EmitterReceiverRegistry
are given in the table below.
C++ Type |
name in the EmitterReceiverRegistry |
---|---|
holoscan::Tensor | “holoscan::Tensor” |
std::shared_ptr |
“PyObject” |
std::string | “std::string” |
pybind11::dict | “pybind11::dict” |
holoscan::gxf::Entity | “holoscan::gxf::Entity” |
holoscan::PyEntity | “holoscan::PyEntity” |
nullptr_t | “nullptr_t” |
CloudPickleSerializedObject | “CloudPickleSerializedObject” |
std::array |
“std::array |
std::shared_ptr |
“std::shared_ptr |
std::shared_ptr |
“std::shared_ptr |
std::vector |
“std::vector |
There is no requirement that the registered name match any particular convention. We generally used the C++ type as the name to avoid ambiguity, but that is not required.
The sections above explain how a register_types
function can be added to bindings to expand this list. It is also possible to get a list of all currently registered types, including those that have been registered by any additional imported 3rd party modules. This can be done via
from holoscan.core import io_type_registry
print(io_type_registry.registered_types())