morpheus.utils.directory_watcher.DirectoryWatcher
- class DirectoryWatcher(input_glob, watch_directory, max_files, sort_glob, recursive, queue_max_size, batch_timeout)[source]
Bases:
objectThis class is in responsible of polling for new files in the supplied input glob of directories and forwarding them on to the pipeline for processing.
- Parameters
- c
morpheus.config.Config - input_globstr
- watch_directorybool
- max_files: int
- sort_globbool
- recursive: bool
- queue_max_size: int
- batch_timeout: float
Pipeline configuration instance.
Input glob pattern to match files to read. For example,
/input_dir/*.jsonwould read all files with the ‘json’ extension in the directory input_dir.The watch directory option instructs this stage to not close down once all files have been read. Instead it will read all files that match the ‘input_glob’ pattern, and then continue to watch the directory for additional files. Any new files that are added that match the glob will then be processed.
Max number of files to read. Useful for debugging to limit startup time. Default value of -1 is unlimited.
If true the list of files matching
input_globwill be processed in sorted order.If true, events will be emitted for the files in subdirectories matching
input_glob.Maximum queue size to hold the file paths to be processed that match
input_glob.Timeout to retrieve batch messages from the queue.
- c
Methods
build_node(name, builder)Build and return the MRC source node
- build_node(name, builder)[source]
Build and return the MRC source node