Struct KafkaSourceStageInterfaceProxy

Interface proxy, used to insulate python bindings.

Public Static Functions

static std::shared_ptr<mrc::segment::Object<KafkaSourceStage>> init(mrc::segment::Builder &builder, const std::string &name, TensorIndex max_batch_size, std::string topic, uint32_t batch_timeout_ms, std::map<std::string, std::string> config, bool disable_commit, bool disable_pre_filtering, TensorIndex stop_after = 0, bool async_commits = true)

Create and initialize a KafkaSourceStage, and return the result.

Parameters

• builder – : Pipeline context object reference

• name – : Name of a stage reference

• max_batch_size – : The maximum batch size for the messages batch.

• topic – : Input kafka topic.

• batch_timeout_ms – : Frequency of the poll in ms.

• config – : Kafka consumer configuration.

• disable_commit – : Enabling this option will skip committing messages as they are pulled off the server. This is only useful for debugging, allowing the user to process the same messages multiple times

• disable_pre_filtering – : Enabling this option will skip pre-filtering of json messages. This is only useful when inputs are known to be valid json.

• stop_after – : Stops ingesting after emitting stop_after records (rows in the table). Useful for testing. Disabled if 0

• async_commits – : Asynchronously acknowledge consuming Kafka messages