Struct KafkaSourceStageInterfaceProxy#

Struct Documentation#

struct KafkaSourceStageInterfaceProxy#

Interface proxy, used to insulate python bindings.

Public Static Functions

static std::shared_ptr<mrc::segment::Object<KafkaSourceStage>> init_with_single_topic(
mrc::segment::Builder &builder,
const std::string &name,
TensorIndex max_batch_size,
std::string topic,
uint32_t batch_timeout_ms,
std::map<std::string, std::string> config,
bool disable_commit,
bool disable_pre_filtering,
std::size_t stop_after = 0,
bool async_commits = true,
std::optional<pybind11::function> oauth_callback = std::nullopt
)#

Create and initialize a KafkaSourceStage, and return the result.

Parameters:

  • builder – : Pipeline context object reference

  • name – : Name of a stage reference

  • max_batch_size – : The maximum batch size for the messages batch.

  • topic – : Input kafka topic.

  • batch_timeout_ms – : Frequency of the poll in ms.

  • config – : Kafka consumer configuration.

  • disable_commit – : Enabling this option will skip committing messages as they are pulled off the server. This is only useful for debugging, allowing the user to process the same messages multiple times

  • disable_pre_filtering – : Enabling this option will skip pre-filtering of json messages. This is only useful when inputs are known to be valid json.

  • stop_after – : Stops ingesting after emitting stop_after records (rows in the table). Useful for testing. Disabled if 0

  • async_commits – : Asynchronously acknowledge consuming Kafka messages

  • oauth_callback – : Callback used when an OAuth token needs to be generated.

static std::shared_ptr<mrc::segment::Object<KafkaSourceStage>> init_with_multiple_topics(
mrc::segment::Builder &builder,
const std::string &name,
TensorIndex max_batch_size,
std::vector<std::string> topics,
uint32_t batch_timeout_ms,
std::map<std::string, std::string> config,
bool disable_commit,
bool disable_pre_filtering,
std::size_t stop_after = 0,
bool async_commits = true,
std::optional<pybind11::function> oauth_callback = std::nullopt
)#

Create and initialize a KafkaSourceStage, and return the result.

Parameters:

  • builder – : Pipeline context object reference

  • name – : Name of a stage reference

  • max_batch_size – : The maximum batch size for the messages batch.

  • topics – : Input kafka topics.

  • batch_timeout_ms – : Frequency of the poll in ms.

  • config – : Kafka consumer configuration.

  • disable_commit – : Enabling this option will skip committing messages as they are pulled off the server. This is only useful for debugging, allowing the user to process the same messages multiple times

  • disable_pre_filtering – : Enabling this option will skip pre-filtering of json messages. This is only useful when inputs are known to be valid json.

  • stop_after – : Stops ingesting after emitting stop_after records (rows in the table). Useful for testing. Disabled if 0

  • async_commits – : Asynchronously acknowledge consuming Kafka messages

  • oauth_callback – : Callback used when an OAuth token needs to be generated.