nemo_curator.stages.text.download.arxiv.download

Module Contents

Name	Description
`ArxivDownloader`	Downloads Arxiv data from s3://arxiv/src/

class nemo_curator.stages.text.download.arxiv.download.ArxivDownloader(
    download_dir: str,
    verbose: bool = False
)

Bases: DocumentDownloader

Downloads Arxiv data from s3://arxiv/src/

nemo_curator.stages.text.download.arxiv.download.ArxivDownloader._download_to_path(
    url: str,
    path: str
) -> tuple[bool, str | None]

nemo_curator.stages.text.download.arxiv.download.ArxivDownloader._get_output_filename(
    url: str
) -> str