mdx.mtmc.stream.multiprocessor module

class MultiProcessor(config: AppConfig, data_transformer: DataTransformer, data_preprocessor: Preprocessor, kafka_message_broker: KafkaMessageBroker)

Bases: object

Module for multi-processing

Parameters:

• config (AppConfig) – configuration for the app

• data_transformer (DataTransformer) – data transformer

• data_preprocessor (Preprocessor) – data pre-processor

• kafka_message_broker (KafkaMessageBroker) – kafka message broker

multi_processor = MultiProcessor(config, data_transformer, data_preprocessor, kafka_message_broker)

consume_raw_messages_and_add_to_behavior_list(idx_process: int, behavior_shared_list: List[Behavior]) → None

Consumes raw messages and adds to a behavior list

Parameters:

• idx_process (int) – index of process

• behavior_shared_list (List[Behavior]) – shared list of behaviors

Returns:

None

multi_processor.consume_raw_messages_and_add_to_behavior_list(idx_process, behavior_shared_list)

process_raw_message_lists(message_lists: List[List[bytes]]) → List[Behavior]

Processes lists of raw messages

Parameters:

message_lists (List[List[bytes]]) – list of message lists

Returns:

list of behaviors

Return type:

List[Behavior]

behaviors = multi_processor.process_raw_message_lists(message_lists)