Class FileSourceStage#

Inheritance Relationships#

Base Type#

  • public mrc::pymrc::PythonSource< std::shared_ptr< MessageMeta > >

Class Documentation#

class FileSourceStage : public mrc::pymrc::PythonSource<std::shared_ptr<MessageMeta>>#

Load messages from a file. Source stage is used to load messages from a file and dumping the contents into the pipeline immediately. Useful for testing performance and accuracy of a pipeline.

Public Types

using base_t = mrc::pymrc::PythonSource<std::shared_ptr<MessageMeta>>#

Public Functions

FileSourceStage(
std::string filename,
int repeat = 1,
bool filter_null = true,
std::vector<std::string> filter_null_columns = {},
std::optional<bool> json_lines = std::nullopt
)#

Construct a new File Source Stage object.

Parameters:

  • filename – : Name of the file from which the messages will be read

  • repeat – : Repeats the input dataset multiple times. Useful to extend small datasets for debugging

  • json_lines – Whether to force json or jsonlines parsing