morpheus.utils.directory_watcher.DirectoryWatcher
- class DirectoryWatcher(input_glob, watch_directory, max_files, sort_glob, recursive, queue_max_size, batch_timeout, should_stop_fn=None)[source]
Bases:
objectThis class is in responsible of polling for new files in the supplied input glob of directories and forwarding them on to the pipeline for processing.
- Parameters
- c
morpheus.config.Config Pipeline configuration instance.
- input_globstr
Input glob pattern to match files to read. For example,
/input_dir/*.jsonwould read all files with the ‘json’ extension in the directory input_dir.- watch_directorybool
The watch directory option instructs this stage to not close down once all files have been read. Instead it will read all files that match the ‘input_glob’ pattern, and then continue to watch the directory for additional files. Any new files that are added that match the glob will then be processed.
- max_files: int
Max number of files to read. Useful for debugging to limit startup time. Default value of -1 is unlimited.
- sort_globbool
If true the list of files matching
input_globwill be processed in sorted order.- recursive: bool
If true, events will be emitted for the files in subdirectories matching
input_glob.- queue_max_size: int
Maximum queue size to hold the file paths to be processed that match
input_glob.- batch_timeout: float
Timeout to retrieve batch messages from the queue.
- should_stop_fn: Callable[[], bool]
Function that returns a boolean indicating if the watcher should stop processing files.
- c
Methods
build_node(name, builder)Build and return the MRC source node - build_node(name, builder)[source]
Build and return the MRC source node