`utils.config_utils`#

Module Contents#

Functions#

`build_downloader`
`build_filter`
`build_filter_pipeline`

API#

utils.config_utils.build_downloader( downloader_config_file: str, default_download_dir: str | None = None, ) → tuple[nemo_curator.download.DocumentDownloader, nemo_curator.download.DocumentIterator, nemo_curator.download.DocumentExtractor, dict]#

utils.config_utils.build_filter( filter_config: dict, ) → nemo_curator.Filter | nemo_curator.ScoreFilter#

utils.config_utils.build_filter_pipeline( filter_config_file: str, ) → nemo_curator.Sequential#