Class FileSourceStage
Defined in File file_source.hpp
Base Type
public mrc::pymrc::PythonSource< std::shared_ptr< MessageMeta > >
-
class FileSourceStage : public mrc::pymrc::PythonSource<std::shared_ptr<MessageMeta>>
Load messages from a file. Source stage is used to load messages from a file and dumping the contents into the pipeline immediately. Useful for testing performance and accuracy of a pipeline.
Public Types
- using base_t = mrc::pymrc::PythonSource<std::shared_ptr<MessageMeta>>
Public Functions
-
FileSourceStage(std::string filename, int repeat = 1, bool filter_null = true, std::vector<std::string> filter_null_columns = {}, std::optional<bool> json_lines = std::nullopt)
Construct a new File Source Stage object.
- Parameters
filename – : Name of the file from which the messages will be read
repeat – : Repeats the input dataset multiple times. Useful to extend small datasets for debugging
json_lines – Whether to force json or jsonlines parsing