DOCA Documentation v3.1.0

DPL Runtime Controller SDK

DPL applications are deployed to the NVIDIA® BlueField® networking platform (DPU or SuperNIC) using the P4Runtime API. However, tables annotated with nv_high_update_rate=true are managed exclusively through the dpl_rt_controller SDK.

Control Protocols​

The DPL runtime service implements two control plane protocols for managing data plane elements:

  • P4Runtime – Based on the open, gRPC-based standard (P4Runtime Spec)

  • NVIDIA Proprietary Protocol – Optimized for High Update Rate tables, this protocol operates over shared memory.

Note

P4Runtime controllers can run remotely on systems with TCP/IP access to the DPU where the dpl_rt_service container is active. The proprietary dpl_rt_controller must run locally on the DPU, as it attaches to shared memory created by the dpl_rt_service container.

The proprietary dpl_rt_controller SDK provides two controller interfaces:

  • dpl_p4rt_controller (gRPC-based) – Used to manage all data plane elements:

    • Configure pipeline (load P4 program)

    • Manage table entries (add/delete, read counters, read entries)

    • Perform packet I/O (send/receive packets from the P4Runtime server)

    • Receive idle timeout notifications (for tables supporting timeouts)

  • dpl_rt_controller (Shared Memory-based) – Used to control High Update Rate (HUR) table entries:

    • Add/delete operations

    • Counter updates

    • Receive idle timeout notifications (for tables supporting timeouts)

    • Limited to tables marked with nv_high_update_rate=true

The dpl_rt_controller can be optionally integrated with a P4Runtime controller to offer unified table management across both regular and high-update-rate tables.

Table Types​

The DOCA Pipeline Language (DPL) supports two categories of tables:

Regular P4 Table

  • Defined per standard P4Runtime specifications.

  • Managed via the gRPC-based P4Runtime controller.

  • Supports all match types supported by the DPL compiler.

High Update Rate Table

  • Declared like a regular table, with the nv_high_update_rate = true attribute. See High Update Rate Tables in the DOCA Target Architecture.

  • Managed via the proprietary dpl_rt_controller SDK.

  • Designed for optimized performance:

    • Eliminates gRPC overhead.

    • Avoids protobuf encoding.

    • Uses rule handles instead of hash-table lookups.

  • Restrictions:

    • Only exact match is supported; this is enforced by the DPL Compiler.

    • Inserting duplicate entries is not rejected by dpl_rtd. It is the responsibility of the user’s controller application to avoid inserting duplicates.

      • Each duplicate entry will receive a distinct dpl_rt_controller_entry handle. However, previously existing handles will be invalidated.

      • Deleting an invalidated handle will trigger a completion error. This error is silently ignored by dpl_rtd but is tracked in the dpl_rt_controller_table_statistics.deletion_errors counter for debugging purposes.

    • HUR tables cannot have constant entries defined in the DPL program.

    • HUR tables are only manageable via the proprietary dpl_rt_controller SDK:

      • No API for entry lookup. The user controller application must maintain its own map between rules and their dpl_rt_controller_entry handles.

      • HUR tables do not support the use of gRPC-based P4Runtime controllers for adding, deleting, or reading entries.

      • Non-HUR tables can continue to use P4Runtime for management.

SDK Libraries Overview

The DPL SDK includes the following components:

Header Files

Path

Purpose

/usr/include/dpl_p4rt_controller.hpp

gRPC-based P4Runtime controller

/usr/include/dpl_rt_controller.h

Proprietary controller for High Update Rate tables


Shared Libraries

Library

Description

libdpl_p4rt_proto.so

Auto-generated P4Runtime .proto bindings

libdpl_p4rt_controller.so

gRPC implementation for P4Runtime

libdpl_rt_controller.so

API implementation for shared-memory-based High Update Rate tables


Sample Applications

Sample controller applications can be found under:

Copy
Copied!
            

/opt/dpl_rt_controller/samples/

The DPL Runtime Controller SDK packages are pre-installed on the DPU operating system after installing a BFB.

Note

Currently, the DPL Runtime Controller SDK supports only Ubuntu 22.04 operating.

Note

The relevant packages are called:

  • dpl-rt-controller

  • dpl-rt-controller-dev

  • dpl-p4rt-controller

  • dpl-p4rt-controller-dev

  • dpl-rt-controller-samples

Warning

When loading DPL programs with High Update Rate (HUR) tables, HugePages are used to pre-allocate entry buffers. The required buffer size depends on table properties such as the number of keys and the total table size.

The dpl_dpu_setup.sh system preparation script (documented in the DPL Container Deployment guide under the section "Pulling the Container Resources and Scripts from NGC") allocates 2048 hugepages by default. This allocation may be insufficient for certain programs. In such cases, dpl_rtd fails to load the program and logs an error message that includes the suggestion: (Try increasing HugePages number).

To prevent this issue, run the dpl_dpu_setup.sh script with a higher --hugepages-num value appropriate for the expected workload.

For example, to load a HUR table with 16 million entries, use:

Copy
Copied!
            

sudo <PATH TO>/dpl_dpu_setup.sh --hugepages-num 4096

Copy
Copied!
            

namespace DPL_P4RT_Controller {   /* ----------------------------------------------------------------------- * Controller class for connecting to dpl_rtd over gRPC. * ----------------------------------------------------------------------- */ class Controller { public: /* * ----------------------------------------------------------------------- * gRPC APIs * ----------------------------------------------------------------------- */   /** * @brief Construct a new Controller object. * * Assigns default election IDs of (high = 0, low = 1). * * @param [in] device_id * The Device ID to connect to in the dpl_rtd. * @param [in] ipaddr * The address for connecting to dpl_rtd */ DOCA_EXPERIMENTAL Controller(uint32_t device_id, const char *ipaddr);   /** * @brief Construct a new Controller object with custom election IDs. * * @param [in] device_id * The Device ID to connect to in the dpl_rtd. * @param [in] ipaddr * The address for connecting to dpl_rtd * @param [in] election_id_high * High part of the 128bit election ID used to determine the primary controller. * @param [in] election_id_low * Low part of the 128bit election ID used to determine the primary controller. */ DOCA_EXPERIMENTAL Controller(uint32_t device_id, const char *ipaddr, uint64_t election_id_high, uint64_t election_id_low);   /** * @brief Destroy the Controller object * */ DOCA_EXPERIMENTAL ~Controller();   /** * @brief Create an unsecured connection to dpl_rtd. * * This requires that TLS authentication was not enabled on dpl_rtd P4RT_RPC_SERVER. * * Uses grpc::InsecureChannelCredentials(). * * @return doca_error_t : * DOCA_SUCCESS - in case of success. * Error code - in case of failure. */ DOCA_EXPERIMENTAL doca_error_t ConnectUnsecured(void);   /** * @brief Create a secured TLS connection to dpl_rtd. * * This requires that TLS authentication was properly enabled and configured on dpl_rtd P4RT_RPC_SERVER. * * Uses grpc::SslCredentials(). * * @param [in] ca_crt_path * Path to ca.crt (cacert). * @param [in] client_key_path * Path to client.key (cert). * @param [in] client_crt_path * Path to client.crt (private-key). * @return doca_error_t : * DOCA_SUCCESS - in case of success. * Error code - in case of failure. */ DOCA_EXPERIMENTAL doca_error_t ConnectSecured(const char *ca_crt_path, const char *client_key_path, const char *client_crt_path);   /** * @brief Get the Channel Stub object for communicating with the dpl_rtd. * * @return const p4::v1::P4Runtime::Stub* : * Valid pointer - in case of success. * NULL - in case of failure. */ DOCA_EXPERIMENTAL const p4::v1::P4Runtime::Stub *GetChannelStub();   /** * @brief Get the ClientReaderWriter object for communicating with the dpl_rtd. * * @return p4::v1::P4Runtime::Stub* : * Valid pointer - in case of success. * NULL - in case of failure. */ DOCA_EXPERIMENTAL grpc::ClientReaderWriter<p4::v1::StreamMessageRequest, p4::v1::StreamMessageResponse> *GetClientReaderWriter();   /* * ----------------------------------------------------------------------- * Utility methods * ----------------------------------------------------------------------- */   /** * @brief Load a program on the dpl_rtd. * * @param [in] p4info_path * Path to the compiled p4info.txt file of the program. * @param [in] blob_path * Path to the compiled dplconfig blob file of the program. * @return doca_error_t : * DOCA_SUCCESS - in case of success. * Error code - in case of failure. */ DOCA_EXPERIMENTAL doca_error_t LoadProgram(const char *p4info_path, const char *blob_path);   /** * @brief Get the p4::config::v1::P4Info of the loaded program. * * @return const p4::config::v1::P4Info* : * Valid pointer - in case of success. * NULL - in case of failure. */ DOCA_EXPERIMENTAL const p4::config::v1::P4Info *GetP4Info();   /* * ----------------------------------------------------------------------- * Infrastructure methods * ----------------------------------------------------------------------- */   /** * @brief Get gRPC operations callbacks for working on regular tables from the generic dpl_rt_controller APIs. * * @return struct dpl_rt_controller_grpc_ops* : * Valid pointer - in case of success. * NULL - in case of failure. */ DOCA_EXPERIMENTAL const struct dpl_rt_controller_grpc_ops *GetGrpcOps(void);   /** * @brief Get the defined Election IDs. */ DOCA_EXPERIMENTAL void GetElectionId(uint64_t *high, uint64_t *low);   /** * @brief Get the defined device_id. * * @param [out] device_id */ DOCA_EXPERIMENTAL uint32_t GetDeviceId(void);   private: uint32_t device_id_; const char *rtd_ipaddr_; uint64_t election_id_high_; uint64_t election_id_low_; std::shared_ptr<grpc::Channel> channel_; std::unique_ptr<p4::v1::P4Runtime::Stub> stub_; p4::config::v1::P4Info p4info_; std::unique_ptr<grpc::ClientReaderWriter<p4::v1::StreamMessageRequest, p4::v1::StreamMessageResponse>> stream_; grpc::ClientContext stream_context_; struct dpl_rt_controller_grpc_ops grpc_ops_;   doca_error_t Connect(std::shared_ptr<grpc::ChannelCredentials> creds); };   /* ----------------------------------------------------------------------- * Helpers for working with Bytestrings. * ----------------------------------------------------------------------- */   /** * @brief Template for getting a string object representing converted value to big-endian (i.e. network) byte-order. * * This must be used when setting match key and action parameter values on gRPC table entries. */ template <typename T> std::string GetBeByteString(T i, size_t bits) { // Ensure T is an integral type static_assert(std::is_integral<T>::value, "Template parameter must be an integral type");   if constexpr (sizeof(T) == sizeof(uint16_t)) { i = htons(i); } else if constexpr (sizeof(T) == sizeof(uint32_t)) { i = htonl(i); } else if constexpr (sizeof(T) == sizeof(uint64_t)) { if (*reinterpret_cast<const char *>("\0\x01") == 0) { // check for little endian const uint32_t high_part = htonl(static_cast<uint32_t>(i >> 32)); const uint32_t low_part = htonl(static_cast<uint32_t>(i & 0xFFFFFFFFLL)); i = (static_cast<uint64_t>(low_part) << 32) | high_part; i = i >> (64 - bits); } } // else: 8-bits or byte-array assumed already to be in big-endian (i.e. network) byte-order   size_t bytes = (bits + 7) / 8; return std::string(reinterpret_cast<char *>(&i), bytes); };   /** * @brief Template for getting a string object representing converted value to big-endian (i.e. network) byte-order. * * This must be used when setting match key and action parameter values on gRPC table entries. */ template <typename T> std::string GetBeByteString(T i) { return GetBeByteString(i, sizeof(i) * 8); }   /** * @brief Template for getting a string object from a big-endian array. * * @note Array is assumed to be in big-endian (i.e. network) byte-order, no byte-order conversion is performed. * * This must be used when setting match key and action parameter values on gRPC table entries. */ template <typename T, size_t N> std::string GetByteString(T (&addr)[N]) { return std::string(reinterpret_cast<const char *>(addr), N * sizeof(T)); }   /** * @brief Get Hex string representation of a byte string value. * * @param [in] value * Value to translate. * @return std::string : Hex string representation */ std::string ByteStringToHexString(const std::string &value);   }; // namespace DPL_P4RT_Controller

Copy
Copied!
            

/* Opaque structures. */ struct dpl_rt_controller_device; struct dpl_rt_controller_entry;   /** Possible event types. */ enum dpl_rt_controller_event_type { /** A new program with High Update Rate tables was loaded. */ DPL_EVENT_PROGRAM_LOADED, /** Program was unloaded; need to cleanup and reconnect once new program is loaded. */ DPL_EVENT_PROGRAM_UNLOADED, /** Entry addition failed. */ DPL_EVENT_ERROR_ENTRY_ADD, /** The dpl_rtd crashed; the controller must be restarted. */ DPL_EVENT_ERROR_DETACHED, };   /** * @brief Maximum length of a status messages. */ #define DPL_RT_MAX_STATUS_MESSAGE_LEN 128   struct dpl_rt_controller_event_data { /** Optional details message. */ char message[DPL_RT_MAX_STATUS_MESSAGE_LEN]; /** Event data for applicable types. */ union { struct { /** Entry that failed to be added. */ struct dpl_rt_controller_entry *entry; } entry_add; }; };   /** Event type and data. */ struct dpl_rt_controller_event { /** The device that raised the event. */ uint32_t device_id; /** Event type. */ enum dpl_rt_controller_event_type type; /** Event data. */ struct dpl_rt_controller_event_data data; };   /** Event handling callback prototype. */ typedef void (*dpl_rt_controller_event_cb_t)(struct dpl_rt_controller_event *event);   /** * @brief Attributes for attaching to dpl_rtd shared memory. */ struct dpl_rt_controller_context_attr { /** Events handling callback, to be called when events arise. */ dpl_rt_controller_event_cb_t event_cb; };   /** * @brief Attributes for connecting to a device. */ struct dpl_rt_controller_device_attr { /** The Device ID to connect to in the dpl_rtd. */ uint32_t device_id; /** Program hash from the DPL auto-generated header. * Required to manage High Update Rate, set to <PROGRAM_NAME>_HASH defined by the DPL compiler generated header. * Required when without_shm_support=false. */ const char *program_hash; /** When set to true, do not connect over SHM, create a device that supports only regular P4Runtime tables, * without the ability to manage High Update Rate tables. * Used to allow using the generic entry management APIs when Update Rate tables not required. */ bool without_shm_support; };   /** * @brief Counter data. */ struct dpl_rt_controller_counter_data { /** Counter packets count value. */ uint64_t packet_count; /** Counter bytes count value. */ uint64_t byte_count; };   /** * @brief The direction of the packet with respect to the TCP connection establishment. * The sender of the first SYN packet is the originator of the flow. * The receiver is the direction that provides the SYN-ACK reply. */ enum dpl_rt_controller_tcp_state_direction { DPL_TCP_STATE_DIRECTION_ORIGINAL = 0, DPL_TCP_STATE_DIRECTION_REPLY = 1, };   /** * @brief A TCP state (Connection Tracking) data. */ struct dpl_rt_controller_tcp_state_data { /** The direction of the last packet that hit the TCP state object. */ enum dpl_rt_controller_tcp_state_direction last_direction; /** TCP acknowledgement (ACK) number of the last packet that hit the TCP state object. */ uint32_t last_ack_num; /** TCP sequence (SEQ) number of the last packet that hit the TCP state object. */ uint32_t last_seq_num; };   /** * @brief Opaque type for passing p4::v1::TableEntry pointer. */ typedef void *p4_v1_table_entry_ptr_t;   /** * @brief Attributes for adding a table entry. * * Only applicable when table was defined with idle timeout support * (i.e.: `nv_support_timeout = true; nv_delayed_counter_stats = true;` in p4 program) */ struct dpl_rt_controller_add_entry_attr { /** Idle timeout value in nanoseconds. */ uint64_t idle_timeout_ns; };   /** * @brief gRPC function callbacks required to allow managing gRPC tables using the dpl_rt_controller APIs. */ struct dpl_rt_controller_grpc_ops { /** Opaque user data pointer, e.g for passing DPL_P4RT_Controller::Controller pointer. */ void *user_data; /** Callback for allocating p4::v1::TableEntry. */ doca_error_t (*grpc_entry_alloc)(void *user_data, uint32_t table_id, p4_v1_table_entry_ptr_t *entry); /** Callback for freeing p4::v1::TableEntry handler */ void (*grpc_entry_free)(void *user_data, p4_v1_table_entry_ptr_t entry); /** Callback for adding p4::v1::TableEntry to loaded program */ doca_error_t (*grpc_entry_add)(void *user_data, p4_v1_table_entry_ptr_t entry, struct dpl_rt_controller_add_entry_attr *attr); /** Callback for deleting p4::v1::TableEntry from loaded program */ doca_error_t (*grpc_entry_delete)(void *user_data, p4_v1_table_entry_ptr_t entry); /** Callback for reading DirectCounter of p4::v1::TableEntry */ doca_error_t (*grpc_entry_counter)(void *user_data, p4_v1_table_entry_ptr_t entry, uint64_t *byte_count, uint64_t *packet_count); /** Callback for reading DirectCounter of table's default entry */ doca_error_t (*grpc_default_entry_counter)(void *user_data, uint32_t table_id, uint64_t *byte_count, uint64_t *packet_count); /** Callback for modifying the idle timeout of a table entry */ doca_error_t (*grpc_entry_modify_idle_timeout)(void *user_data, p4_v1_table_entry_ptr_t entry, uint64_t new_idle_timeout_ns); };   /** * @brief Table statistics collected by the dpl_rtd and the controller. */ struct dpl_rt_controller_table_statistics_data { // -- Maintained by the dpl_rtd side: uint64_t insertions; // The total number of successful entries added to the table. uint64_t insertion_errors; // The total number of failed entries additions. uint64_t deletions; // The total number of entries successfully removed from the table. uint64_t deletion_errors; // The total number of failed entries deletions. uint64_t noops; // Del entry requested before it was added (both ADD and DEL flags set while entry in INIT). uint64_t pending_add_dels; // How many entry delete received while entry was still PENDING_ADD.   // -- Maintained by controller library side: uint64_t entry_add_errors; // When the entry_add function returns a failure. uint64_t entry_del_errors; // When the entry_del function returns a failure. };   /* ----------------------------------------------------------------------- * Context management methods. * ----------------------------------------------------------------------- */   /** * @brief Attach to dpl_rtd SHared Memory (SHM). * * @note This must be called first thing, once per process. * In case that th dpl_rtd crashes, the controller process must be restarted. * * @param [in] attr * Attributes for attaching. * @return doca_error_t : * DOCA_SUCCESS - in case of success. * DOCA_ERROR_UNSUPPORTED_VERSION - SHM API mismatch; the dpl_rtd and the controller library were compiled using * different shared memory API versions. * DOCA_ERROR_ALREADY_EXIST - in case that attach is called more than once. * DOCA_ERROR_INITIALIZATION - in case of initialization errors. * Error code - in case of failure. */ DOCA_EXPERIMENTAL doca_error_t dpl_rt_controller_attach(struct dpl_rt_controller_context_attr *attr);   /** * @brief Detach from dpl_rtd shared memory. * * @return doca_error_t : * DOCA_SUCCESS - in case of success. * Error code - in case of failure. */ DOCA_EXPERIMENTAL doca_error_t dpl_rt_controller_detach(void);   /** * @brief Creates a device context * * There two modes when creating a device: * 1. attr.without_shm_support = false: * - Connects to a device over shared memory, to support managing High Update Rate tables. * - Requires calling dpl_rt_controller_attach() first. * - Optional support for regular P4Runtime tables can be added by calling dpl_rt_controller_grpc_ops_set(). * 2. attr.without_shm_support = true: * - Used when user only wants use the generic APIs to manage regular P4Runtime tables. * - Does not connect to a device over shared memory; No support for managing High Update Rate tables. * - Requires calling dpl_rt_controller_grpc_ops_set(). * - No need to call dpl_rt_controller_attach() at all. * * @note Only one client can be connected to a device over shared memory. * * @param [in] attr * Attributes for connecting. * @param [out] device * Pointer to created device handle. * @return doca_error_t : * DOCA_SUCCESS - in case of success. * DOCA_ERROR_AGAIN - the device is not loaded with a program yet. * DOCA_ERROR_NOT_SUPPORTED - the device is loaded with a program that does not contain High Update Rate tables. * DOCA_ERROR_UNSUPPORTED_VERSION - program_hash mismatch; the device is loaded with a different program version. * DOCA_ERROR_IN_USE - another client is already connected to this device. * Error code - in case of other failures. */ DOCA_EXPERIMENTAL doca_error_t dpl_rt_controller_connect(struct dpl_rt_controller_device_attr *attr, struct dpl_rt_controller_device **device);   /** * @brief Cleanup and disconnect a device * * @note After disconnecting, the dpl_rtd will delete remaining High Update Rate table entries! * * @param [in] ctx * Device to disconnect. * @return doca_error_t : * DOCA_SUCCESS - in case of success. * Error code - in case of failure. */ DOCA_EXPERIMENTAL doca_error_t dpl_rt_controller_disconnect(struct dpl_rt_controller_device *device);   /* ----------------------------------------------------------------------- * Entry allocation methods. * ----------------------------------------------------------------------- */   /** * @brief Allocate a High Update Rate table entry handle from shared memory. * * @note Supported only for tables defined with attribute nv_high_update_rate=true in the program. * * @param [in] device * Device to allocate from. * @param [in] table_id * High Update Rate table ID to allocate entry from. * @param [out] entry * Pointer for providing the dpl_rt_controller_entry handler. * @param [out] opaque_entry * Opaque pointer to the C table entry structure backing this entry. * @return doca_error_t : * DOCA_SUCCESS - in case of success. * DOCA_ERROR_NO_MEMORY - in case of memory allocation failure. * Error code - in case of failure. */ DOCA_EXPERIMENTAL doca_error_t dpl_rt_controller_table_entry_alloc_shm(struct dpl_rt_controller_device *device, uint32_t table_id, struct dpl_rt_controller_entry **entry, void **opaque_entry);   /** * @brief Allocate a regular P4Runtime table entry handle using defined gRPC callbacks. * * @note Not supported for tables defined with attribute nv_high_update_rate=true in the program. * @note dpl_rt_controller_grpc_ops_set() must be called prior using this API. * * @param [in] device * Device to allocate from. * @param [in] table_id * Regular P4Runtime table ID to allocate entry from. * @param [out] entry * @param [out] opaque_entry * opaque pointer to p4::v1::TableEntry backing this entry. * Pointer for providing the dpl_rt_controller_entry handler. * @return doca_error_t : * DOCA_SUCCESS - in case of success. * DOCA_ERROR_NO_MEMORY - in case of memory allocation failure. * Error code - in case of failure. */ DOCA_EXPERIMENTAL doca_error_t dpl_rt_controller_table_entry_alloc_grpc(struct dpl_rt_controller_device *device, uint32_t table_id, struct dpl_rt_controller_entry **entry, p4_v1_table_entry_ptr_t *opaque_entry);   /** * @brief Free backing memory of entry handler. * * @note This is used only in error flows, e.g. after failure to add an entry. * * @param [in] entry * Entry to free. */ DOCA_EXPERIMENTAL void dpl_rt_controller_table_entry_free(struct dpl_rt_controller_entry *entry);   /* ----------------------------------------------------------------------- * Entry management methods. * ----------------------------------------------------------------------- */   /** * @brief Add a table entry. * * The API is asynchronous non-blocking when adding High Update Rate table entries. * Returning success means that the entry was sent to the dpl_rtd for insertion. * In case of a successful insertion, there is no further feedback from the dpl_rtd. * In case of a failed insertion, an event of type DPL_EVENT_ERROR_ENTRY_ADD is raised. * It is possible to verify the entry insertion status by checking the table statistics using the * dpl_rt_controller_table_statistics() API. * * The API is synchronous blocking when adding a regular table entries, as it is done using P4Runtime RPC message. * The returned status reflects whether the dpl_rtd inserted the entry successfully or not. * * @note Inserting duplicate entries to High Update Rate tables is not rejected by the dpl_rtd, it is the user * controller app responsibility to avoid inserting duplicate entries. * @note There is no API for entry lookup, it is the user controller app responsibility to maintain a map between the * rule and its dpl_rt_controller_entry handle. * * @param [in] entry * Entry to add. * @param [in] attr * Attributes for adding the entry, see dpl_rt_controller_add_entry_attr for more details. * @note Can be NULL if not needed. * @return doca_error_t : * DOCA_SUCCESS - in case of success. * DOCA_ERROR_AGAIN - in case posting the request failed due to full requests ring. * DOCA_ERROR_IN_PROGRESS - in case that another addition operation is still in progress. * DOCA_ERROR_EMPTY - in case entry does not exist (entry is already deleted/freed). * Error code - in case of failure. */ DOCA_EXPERIMENTAL doca_error_t dpl_rt_controller_table_entry_add(struct dpl_rt_controller_entry *entry, struct dpl_rt_controller_add_entry_attr *attr);   /** * @brief Delete a table entry. * * The API is asynchronous non-blocking when deleting High Update Rate table entries. * Returning success means that the entry was sent to the dpl_rtd for deletion. * In case of a successful deletion, there is no further feedback from the dpl_rtd. * In case of a failed deletion, the relevant statistic counters are incremented (no even is raised). * It is possible to verify the entry deletion status by checking the table statistics using the * dpl_rt_controller_table_statistics() API. * * The API is synchronous blocking when deleting a regular table entries, as it is done using P4Runtime RPC message. * The returned status reflects whether the dpl_rtd deleted the entry successfully or not. * * @note If a duplicate entries were inserted, once any such entry is deleted, the existing handles will be invalidated. * As such, a completion error will be triggered by entry deletion of the invalidated handle. The error is * silently ignored by the dpl_rtd and counted by dpl_rt_controller_table_statistics_data.deletion_errors counter * for debug purposes. * * @param [in] entry * Entry to delete. * @return doca_error_t : * DOCA_SUCCESS - in case of success. * DOCA_ERROR_AGAIN - in case posting the request failed due to full requests ring. * DOCA_ERROR_IN_PROGRESS - in case that another deletion operation is still in progress. * DOCA_ERROR_EMPTY - in case entry does not exist (entry is already deleted/freed). * Error code - in case of failure. */ DOCA_EXPERIMENTAL doca_error_t dpl_rt_controller_table_entry_delete(struct dpl_rt_controller_entry *entry);   /** * @brief Read entry direct counter data. * * The client application is responsible for synchronizing the entry_delete and entry_counter calls. * * @note Supported only if direct_counter was defined on the parent table in the program. * @note This API is always blocking. * * @param [in] entry * Entry to read its counter data. * @param [out] counter * Returned entry counter data. * @return doca_error_t : * DOCA_SUCCESS - in case of success. * DOCA_ERROR_AGAIN - in case posting the request failed due to full requests ring. * DOCA_ERROR_IN_PROGRESS - in case that another query is still in progress. * DOCA_ERROR_NOT_FOUND - in case direct_counter was not defined on the table in the program. * DOCA_ERROR_EMPTY - in case entry does not exist (entry is already deleted/freed). * Error code - in case of failure. */ DOCA_EXPERIMENTAL doca_error_t dpl_rt_controller_table_entry_counter(struct dpl_rt_controller_entry *entry, struct dpl_rt_controller_counter_data *counter);   /** * @brief Read table's default entry direct counter data. * * @note Supported only if direct_counter was defined on the parent table in the program. * @note This API is always blocking. * * @param [in] device * Device to read its default entry counter data. * @param [in] table_id * Table ID to read its default entry counter data. * Can be either a High Update Rate table ID or a regular P4Runtime table ID. * @param [out] counter * Returned entry counter data. * @return doca_error_t : * DOCA_SUCCESS - in case of success. * DOCA_ERROR_AGAIN - in case posting the request failed due to full requests ring. * DOCA_ERROR_IN_PROGRESS - in case that another query is still in progress. * DOCA_ERROR_NOT_FOUND - in case direct_counter was not defined on the table in the program. * Error code - in case of failure. */ DOCA_EXPERIMENTAL doca_error_t dpl_rt_controller_table_default_entry_counter(struct dpl_rt_controller_device *device, uint32_t table_id, struct dpl_rt_controller_counter_data *counter);   /** * @brief Get the opaque pointer to the C table entry structure backing this entry. * * The returned opaque pointer points to memory buffer for providing the table entry details (keys, params, etc), * which must be casted to the corresponding table entry C structure defined at the C header generated by the DPL * Compiler. * * @note Supported only for entries allocated using dpl_rt_controller_table_entry_alloc_shm(). * * @param [in] entry * dpl_rt_controller_entry * @return void** : * Valid pointer for providing the opaque entry structure pointer - in case of success. * NULL - in case of failure. */ DOCA_EXPERIMENTAL void *dpl_rt_controller_table_entry_get_shm(struct dpl_rt_controller_entry *entry);   /** * @brief Get the opaque pointer to p4::v1::TableEntry backing this entry. * * The returned opaque pointer points to a p4::v1::TableEntry for providing the table entry details (keys, params, etc). * * @note Supported only for entries allocated using dpl_rt_controller_table_entry_alloc_grpc(). * * @param [in] entry * dpl_rt_controller_entry * @return void** : * Valid pointer for providing the p4::v1::TableEntry pointer - in case of success. * NULL - in case of failure. */ DOCA_EXPERIMENTAL p4_v1_table_entry_ptr_t dpl_rt_controller_table_entry_get_grpc(struct dpl_rt_controller_entry *entry);   /* ----------------------------------------------------------------------- * gRPC integration methods. * ----------------------------------------------------------------------- */   /** * @brief Set P4Runtime controller gRPC callbacks for allowing managing regular table entries. * * @param [in] device * Device to set the gRPC callback on. * @param [in] ops * The set of user's gRPC callbacks. * @return doca_error_t : * DOCA_SUCCESS - in case of success. * Error code - in case of failure. */ DOCA_EXPERIMENTAL doca_error_t dpl_rt_controller_grpc_ops_set(struct dpl_rt_controller_device *device, const struct dpl_rt_controller_grpc_ops *ops);   /** * @brief Clear the currently set gRPC callbacks. * * @param [in] device * Device to clear its gRPC callbacks. */ DOCA_EXPERIMENTAL void dpl_rt_controller_grpc_ops_clear(struct dpl_rt_controller_device *device);   /* * ----------------------------------------------------------------------- * TCP state methods. * ----------------------------------------------------------------------- */   /** * @brief Get avaiable indexes in a TCP state object for adding new entries. * * Returns indexes that can be be used to add new entries using the TCP state object. * * @note The user application is responsible to make sure not to use the same index for multiple entries using the same * direction. * @note The user application is responsible to make sure not to reuse indexes after deleting entries using them. * For adding new entries, need to call dpl_rt_controller_tcp_state_get_entries() again to get a new avaiable * index. * @note The dpl_rtd service will automatically track and reset no longer used indexes when the entry is deleted. * * @param [in] tcp_state_id * TCP state object ID to get the number of entries. * @param [in, out] indexes * Array of indexes to return the available unused indexes. * @param [in] indexes_num * Requested number of indexes to return. * @param [out] indexes_num_ret * Actual number of indexes returned. * @return doca_error_t : * DOCA_SUCCESS - in case of success. * Error code - in case of failure. */ DOCA_EXPERIMENTAL doca_error_t dpl_rt_controller_tcp_state_get_entries(struct dpl_rt_controller_device *device, uint32_t tcp_state_id, uint32_t *indexes, uint32_t indexes_num, uint32_t *indexes_num_ret);   /** * @brief Query the TCP state for a given index. * * @param [in] device * Device containing the requested TCP state. * @param [in] tcp_state_id * TCP state object ID to query. * @param [in] index * Index to query. * @param [out] state * Pointer to struct dpl_rt_controller_tcp_state_data to return the state data. * @return doca_error_t : * DOCA_SUCCESS - in case of success. * Error code - in case of failure. */ DOCA_EXPERIMENTAL doca_error_t dpl_rt_controller_tcp_state_query(struct dpl_rt_controller_device *device, uint32_t tcp_state_id, uint32_t index, struct dpl_rt_controller_tcp_state_data *state);   /* * ----------------------------------------------------------------------- * Utility methods * ----------------------------------------------------------------------- */   /** * @brief Get table statistics. * * @note Supported only for tables defined with attribute nv_high_update_rate=true in the program. * @note After a new DPL program is loaded, the 'insertions' counter will have value 1 since a default table entry * was inserted to the High Update Rate table during loading the program. * * @param [in] device * Device containing the requested table. * @param [in] table_id * Table ID to get it's statistics. * @param [out] stats * Pointer to struct dpl_rt_controller_table_statistics_data to return the statistics. * @return doca_error_t : * DOCA_SUCCESS - in case of success. * Error code - in case of failure. */ DOCA_EXPERIMENTAL doca_error_t dpl_rt_controller_table_statistics(struct dpl_rt_controller_device *device, uint32_t table_id, struct dpl_rt_controller_table_statistics_data *stats);   /** * @brief Get string representation of a dpl_rt_controller_event_type. * * @param [in] type * Event type value to convert. * @return const char* : * String representation. */ DOCA_EXPERIMENTAL const char *dpl_rt_controller_event_type_to_str(enum dpl_rt_controller_event_type type);   /** * @brief Modify the idle timeout of a table entry. * * @param [in] entry * Entry to modify the idle timeout of. * @param [in] new_idle_timeout_ns * New idle timeout value in nanoseconds. * @return doca_error_t : * DOCA_SUCCESS - in case of success. * DOCA_ERROR_NOT_SUPPORTED - in case of failure. * DOCA_ERROR_EMPTY - in case entry does not exist (entry is already deleted/freed). * Error code - in case of failure. */ DOCA_EXPERIMENTAL doca_error_t dpl_rt_controller_table_entry_modify_idle_timeout(struct dpl_rt_controller_entry *entry, uint64_t new_idle_timeout_ns);   /** * @brief Get all entries with expired idle timeout on the device. * * This API will fill the stale_entry_arr array with pointers to stale entries on High Update Rate tables. * * @param [in] device * Device to query idle timeout for. * @param [in, out] stale_entry_arr * Array of stale entries to be filled - should be allocated and passed empty by user. * @param [in] entry_arr_size * Size of the stale_entry_arr array. * @param [out] stale_entries_num * Total number of stale entries found (even if exceeds entry_arr_size). * @return doca_error_t : * DOCA_SUCCESS - in case of success. * DOCA_ERROR_NOT_FOUND - in case that the device does not have any tables supporting idle timeout. * Error code - in case of failure. */ DOCA_EXPERIMENTAL doca_error_t dpl_rt_controller_idle_timeout_query(struct dpl_rt_controller_device *device, struct dpl_rt_controller_entry **stale_entry_arr, size_t entry_arr_size, size_t *stale_entries_num);

Efficient management of network device tables is essential for maintaining optimal performance and resource utilization. In dynamic environments, table entries may become stale or unused over time. To address this, idle timeout provides a proactive mechanism for table hygiene. This feature continuously monitors the activity of each table entry. If an entry is not accessed (or "hit") within a specified duration (the idle timeout period) it is marked as stale.

Limitations

Idle timeout support applies only to:

  • Tables that explicitly support idle timeout

  • Non-default entries

Usage

Info

See sections "DPL P4Runtime Controller Library" and " DPL Runtime Controller Library " for full details.

  • Creating an entry with idle timeout: Use the same flow as for entries without idle timeout, but set the idle_timeout_ns field in the dpl_rt_controller_add_entry_attr struct, and pass it to the entry addition function.

  • Modifying idle timeout: Use the dpl_rt_controller_table_entry_modify_idle_timeout API.

  • Receiving stale entry notifications:

    • For gRPC-based (regular) tables – stale entry notifications are sent via the gRPC interface.

    • For High Update Rate (HUR) tables – use dpl_rt_controller_idle_timeout_query to retrieve stale entries.

Handling of Stale Entries

It is the controller implementer's responsibility to manage stale entries. System-specific constraints often require tailored solutions. Consider the following caveats:

  • When using timeouts to delete entries, deletions should occur only within the timeout handling routine. Otherwise, the implementer must ensure protection against multiple deletions of the same entry.

  • In systems where timeout scanning and entry deletion/insertion occur in different threads, pointers returned by the timeout scanning function may point to invalid or different entries by the time they are processed. The implementer must address this.

    • A basic solution is to use a mutex to block deletions while scanning and handling timeouts.

  • The idle timeout query function does not cause process crashes, regardless of whether it is called from single-threaded or multi-threaded environments.

Sample controller applications for a given DPL programs are installed by the dpl-rt-controller-samples package at the /opt/dpl_rt_controller/samples/ directory.

Running the Sample

Warning

Building the sample applications requires meson utility. Make sure it is available on your system.

For installing meson on Ubuntu system, you can run:

Copy
Copied!
            

sudo apt update sudo apt install meson

Follow these steps to build and run a sample application:

  1. Connect to the DPU.

  2. Make sure the DPL Runtime Service is up and running. See DPL Container Deployment for more details.

  3. Navigate to the desired sample application directory (one of the folders under /opt/dpl_rt_controller/samples/ on the DPU).

  4. Compile the provided DPL program (e.g., hello_world.p4) found at the sample's directory. See Compiling DPL Applications for more details.

  5. Copy the DPL program compilation output directory (_out) to a local path on the DPU.

    Note

    The compilation output directory contains the required C header files (<program name>_dpl_shm.h and <program name>_dpl_shm_id.h) for compiling the sample controller application.

  6. Compile the C sample application:

    Copy
    Copied!
                

    cd /opt/dpl_rt_controller/samples/<sample_name> meson /tmp/build -Dsample_programs_out=<PATH_TO_DPL_PROGRAM_COMPILATION_OUT_FOLDER> ninja -C /tmp/build

    Note

    Replace the <PATH_TO_DPL_PROGRAM_COMPILATION_OUT_FOLDER> with the path to the local directory containing the DPL program compilation output folder (where the required C header files (<program name>_dpl_shm.h and <program name>_dpl_shm_id.h) are found).

    Info

    The binary dpl_sample_<sample_name> is created under /tmp/build/.

  7. Run the sample application using a privileged user. For example:

    Copy
    Copied!
                

    sudo /tmp/build/dpl_sample_basic 1000 <path to p4info.txt> <path-to-dplconfig>

Samples

Basic

This sample demonstrates how to manage entries on a High Update Rate table as well as a regular table.

The sample logic includes:

  1. Connecting to dpl_rtd over gRPC.

  2. Loading a DPL program.

  3. Connecting todpl_rtd over SHM (SHared Memory).

  4. Adding entry to a regular table.

  5. Adding entry to a High Update Rate table.

  6. Reading entries counter.

  7. Deleting entries.

  8. Displaying statistics.

  9. Disconnecting and destroying all structures.

References:

  • /opt/dpl_rt_controller/samples/basic/basic_sample.cc

  • /opt/dpl_rt_controller/samples/basic/basic_main.cc

  • /opt/dpl_rt_controller/samples/basic/hello_world.p4

  • /opt/dpl_rt_controller/samples/basic/meson_options.txt

  • /opt/dpl_rt_controller/samples/basic/meson.build

gRPC Only

This sample demonstrates how to use the generic dpl_rt_controller APIs to manage regular P4Runtime tables when High Update Rate tables are neither required nor defined in the DPL program.

The sample logic includes:

  1. Connecting to dpl_rtd over gRPC.

  2. Loading a DPL program.

  3. Creating dpl_rt_controller device without SHM support.

  4. Adding entry to a regular table.

  5. Reading entries counter.

  6. Deleting entries.

  7. Disconnecting and destroying all structures.

References:

  • /opt/dpl_rt_controller/samples/grpc_only/grpc_only_sample.cc

  • /opt/dpl_rt_controller/samples/grpc_only/grpc_only_main.cc

  • /opt/dpl_rt_controller/samples/grpc_only/grpc_only.p4

  • /opt/dpl_rt_controller/samples/grpc_only/meson_options.txt

  • /opt/dpl_rt_controller/samples/grpc_only/meson.build

Idle Timeout Sample

The sample logic includes:

  1. Connecting to dpl_rtd over gRPC.

  2. Loading a DPL program.

  3. Connecting to dpl_rtd over SHM (SHared Memory).

  4. Adding 2 entries to a regular table (gRPC) with very long timeout.

  5. Adding 2 entries to a High Update Rate table (SHM) very long timeout.

  6. Waiting for couple of seconds to see if entries expire (shouldn't).

  7. Querying for stale entries (should find any).

  8. Modifying idle timeout for one gRPC entry and one SHM entry.

  9. Querying for stale entries (should find two).

  10. Deleting all entries.

  11. Displaying statistics.

  12. Disconnecting and destroying all structures.

References:

  • /opt/dpl_rt_controller/samples/idle_timeout/idle_timeout_sample.cc

  • /opt/dpl_rt_controller/samples/idle_timeout/idle_timeout_main.cc

  • /opt/dpl_rt_controller/samples/idle_timeout/idle_timeout.p4

  • /opt/dpl_rt_controller/samples/idle_timeout/meson_options.txt

  • /opt/dpl_rt_controller/samples/idle_timeout/meson.build

TCP state (Connection Tracking) Sample

This sample demonstrates how to manage a TCP connection state, including adding entries to track connections and retrieving the connection's acknowledgement (ACK) number and sequence (SEQ) number of the last packet that hit the TCP state object.

The sample logic includes:

  1. Connecting to dpl_rtd over gRPC.

  2. Loading a DPL program.

  3. Connecting to dpl_rtd over SHM (SHared Memory).

  4. Setting up DPDK for packet processing and table entry management (addition/deletion).

  5. Printing statistics and current TCP state object info.

  6. Disconnecting and destroying all structures.

References:

  • /opt/dpl_rt_controller/samples/tcp_state/tcp_state.p4

  • /opt/dpl_rt_controller/samples/tcp_state/tcp_state_sample.cc

  • /opt/dpl_rt_controller/samples/tcp_state/tcp_state_main.cc

  • /opt/dpl_rt_controller/samples/tcp_state/meson_options.txt

  • /opt/dpl_rt_controller/samples/tcp_state/meson.build

© Copyright 2025, NVIDIA. Last updated on Sep 4, 2025.