Skip to navigation
API referenceModule APIClasses

hololink::module::RoceReceiverInterfaceV2

View as Markdown

RoceReceiverInterfaceV1, plus whether the receive path has failed.

A fatal event on the network adapter (a completion queue or queue pair error, or the device failing) stops a RoCE receiver for good: no frame arrives until it is started again. Through RoceReceiverInterfaceV1 alone, that looks like a receiver that is merely waiting for frames.

The receivers this module provides publish this interface next to RoceReceiverInterfaceV1. Resolve it with allow_null and fall back to RoceReceiverInterfaceV1 for a module that doesn’t publish it.

#include <hololink/module/roce_receiver.hpp>

Inherits from: hololink::module::RoceReceiverInterfaceV1 (public), hololink::module::Service< RoceReceiverInterfaceV2 > (public)

Methods

receive_path_failed

virtual bool hololink::module::RoceReceiverInterfaceV2::receive_path_failed()

Whether a fatal event has stopped the receive path.

get_next_frame() then returns false at once, and frames_ready() returns true so that a caller waiting for it isn’t left waiting. No frame arrives until start() is called again.

Returns: True once the receive path has failed.

start

virtual hololink_module_status_t hololink::module::RoceReceiverInterfaceV2::start(
    const std::string &ibv_name,
    unsigned ibv_port,
    uint64_t cu_buffer,
    size_t cu_buffer_size,
    size_t cu_frame_size,
    size_t cu_page_size,
    unsigned pages,
    size_t metadata_offset,
    const std::string &peer_ip,
    unsigned queue_size
)

Configure the receiver and network adapter for RDMA.

Frames are delivered into cu_buffer, which is a single contiguous region with enough space to accommodate the total number of frames indicated by the pages parameter; each frame is cu_page_size bytes apart; cu_page_size needs to include space for received frame metadata (and its alignment).

Returns: HOLOLINK_MODULE_OK or an error status.

Throws: std::runtime_error if the receiver cannot be brought up, for example the named device is not found or peer_ip is not a valid address.

Parameters

ibv_name
const std::string &

Name of the InfiniBand verbs device of the network adapter that receives the frames, e.g. “roceP5p3s0f0”.

ibv_port
unsigned

Port of that device to use.

cu_buffer
uint64_t

CUDA memory frames are delivered into, as a CUdeviceptr.

cu_buffer_size
size_t

Bytes available at cu_buffer.

cu_frame_size
size_t

Bytes of frame data expected for each received data frame: the HSB device is programmed to trigger an end-of-frame when this many bytes are received; but an earlier TLAST signal will trigger end-of-frame at an earlier point.

cu_page_size
size_t

Bytes from one page to the next.

pages
unsigned

Pages the device cycles through.

metadata_offset
size_t

Offset, in bytes from the start of a page, of the metadata block the device writes after the frame data.

peer_ip
const std::string &

IPv4 address of the device, in dotted-decimal form. The receiver accepts data only from this address.

queue_size
unsigned

Completed frames the receiver holds while the application is busy with an earlier one. When a frame completes with this many already waiting, the oldest waiting frame is discarded (see RoceReceiverFrameInfoV1::dropped).

close

virtual void hololink::module::RoceReceiverInterfaceV2::close()

End the run, returning blocking_monitor() to its caller.

Callable from any thread, including while blocking_monitor() runs. Does nothing if the receiver was never started.

blocking_monitor

virtual void hololink::module::RoceReceiverInterfaceV2::blocking_monitor()

Run the receiver until close() is called.

Frames arrive only while this runs, so it gets a thread of its own, and that thread carries the CUDA context the receiver was started with. Clients typically execute this in its own thread.

Throws: std::runtime_error if start() has not been called.

get_next_frame

virtual bool hololink::module::RoceReceiverInterfaceV2::get_next_frame(
    unsigned timeout_ms,
    RoceReceiverFrameInfoV1 &info
)

Take the next completed frame.

Waits up to timeout_ms for a frame. On successful reception of a frame, info is updated with various metadata about the data, which is present at the indicated address within the cu_buffer passed to start. Frames are returned in the order they are received.

Returns: True when a frame was taken; false when the timeout elapsed first.

Throws: std::runtime_error if start() has not been called.

Parameters

timeout_ms
unsigned

How long to wait for a frame, in milliseconds.

info
RoceReceiverFrameInfoV1 &

Describes the frame that was taken.

frames_ready

virtual bool hololink::module::RoceReceiverInterfaceV2::frames_ready()

Returns: False when get_next_frame() will block waiting for a new frame to arrive.

set_frame_ready

virtual void hololink::module::RoceReceiverInterfaceV2::set_frame_ready(
    std::function<void()> callback
)

Register a callback run each time a frame becomes available.

An application that cannot afford to wait in get_next_frame() is told from here that a frame is waiting, and takes it when it is ready to. The callback runs on the thread running blocking_monitor(). Register it before that starts; pass an empty std::function to stop being called. It stays registered across calls to start().

Parameters

callback
std::function<void()>

Run once per frame that becomes available.

get_qp_number

virtual uint32_t hololink::module::RoceReceiverInterfaceV2::get_qp_number()

The number of the queue pair that receives this channel’s frames.

The channel programs this into the device so it sends to the right queue pair.

Returns: The number. Only meaningful once start() has run.

Throws: std::runtime_error if start() has not been called.

get_rkey

virtual uint32_t hololink::module::RoceReceiverInterfaceV2::get_rkey()

The key the device presents when it writes a frame into the receiver’s memory.

Returns: The key. Only meaningful once start() has run.

Throws: std::runtime_error if start() has not been called.

external_frame_memory

virtual uint64_t hololink::module::RoceReceiverInterfaceV2::external_frame_memory()

The address, in the device’s view of the receiver’s memory, where the first page begins.

The device addresses page n at this value plus n times page_size(). This is not a CUDA address.

Returns: The address. Only meaningful once start() has run.

Throws: std::runtime_error if start() has not been called.

frame_size

virtual size_t hololink::module::RoceReceiverInterfaceV2::frame_size()

Bytes of frame data the device sends per frame.

Returns: The cu_frame_size start() was given.

page_size

virtual size_t hololink::module::RoceReceiverInterfaceV2::page_size()

Bytes from one page to the next.

Returns: The cu_page_size start() was given.

pages

virtual unsigned hololink::module::RoceReceiverInterfaceV2::pages()

Pages the device cycles through.

Returns: The pages start() was given.

configure

virtual void hololink::module::ConfigurableService<RoceReceiverInterfaceV1>::configure(
    const EnumerationMetadata &
)

Use metadata contents to initialize this instance.

This routine is guaranteed to be called the first time a call to get_service(metadata) is called on a specific service instance.

module

std::shared_ptr<Module> hololink::module::ServiceBase::module() const

The Module that this service was published from.

The service holds no strong reference to its Module, so this does not extend the Module’s lifetime.

Returns: The owning Module, or an empty pointer if the service has not been published or its Module no longer exists.

Static methods

get_service

static std::shared_ptr<RoceReceiverInterfaceV2> hololink::module::RoceReceiverInterfaceV2::get_service(
    const EnumerationMetadata &metadata,
    bool allow_null = false
)

Fetch the receiver for a data channel, as RoceReceiverInterfaceV1::get_service(metadata) does.

Returns: The receiver, configured by metadata on first use.

Throws: std::runtime_error when allow_null is false and the receiver cannot be resolved.

Parameters

metadata
const EnumerationMetadata &

Enumeration metadata naming the data channel.

allow_null
boolDefaults to false

Use true to return nullptr when the module doesn’t provide this interface; the default false raises a std::runtime_error.

for_each_type_id

static bool hololink::module::Service<RoceReceiverInterfaceV1>::for_each_type_id(
    Callback &&cb
)

Iterate over every type_id supported by this instance.

The callback signature is bool(const char*) — returning false halts the walk; true continues. Iteration is depth-first, so you’ll see the most specific implementation types first.