nemo_curator.stages.text.download.common_crawl.download

Module Contents

Classes

Name	Description
`CommonCrawlWARCDownloader`	Downloads WARC files from the Common Crawl to a local directory
`CommonCrawlWARCReader`	Reads WARC records directly from Common Crawl using HTTPS or S3 range requests.

Data

CC_BASE_URL

HTTP_OK

HTTP_PARTIAL_CONTENT

API

class nemo_curator.stages.text.download.common_crawl.download.CommonCrawlWARCDownloader(
    download_dir: str,
    use_aws_to_download: bool = False,
    verbose: bool = False
)

Bases: DocumentDownloader

Downloads WARC files from the Common Crawl to a local directory

nemo_curator.stages.text.download.common_crawl.download.CommonCrawlWARCDownloader._download_to_path(
    url: str,
    path: str
) -> tuple[bool, str | None]

Download a file to a temporary file.

Parameters:

url

str

URL to download

path

str

Local path to save file

Returns: bool

Tuple of (success, error_message). If success is True, error_message is None.

nemo_curator.stages.text.download.common_crawl.download.CommonCrawlWARCDownloader._get_output_filename(
    url: str
) -> str

Generate output filename from URL.

class nemo_curator.stages.text.download.common_crawl.download.CommonCrawlWARCReader(
    warc_filename_col: str = 'warc_filename',
    warc_record_offset_col: str = 'warc_record_offset',
    warc_record_length_col: str = 'warc_record_length',
    binary_content_col: str = 'binary_content',
    drop_failed: bool = True,
    max_workers: int = 16,
    timeout: int = 30,
    max_retries: int = 3,
    use_s3: bool | None = None,
    s3_bucket: str | None = None,
    s3_key_prefix: str | None = None
)

Bases: ProcessingStage[DocumentBatch, DocumentBatch]

Reads WARC records directly from Common Crawl using HTTPS or S3 range requests.

This stage fetches raw HTML content from Common Crawl’s public servers using byte-range requests.

_lock

= threading.Lock()

name

= 'CommonCrawlWARCReader'

s3_bucket

s3_key_prefix

use_s3

nemo_curator.stages.text.download.common_crawl.download.CommonCrawlWARCReader.__getstate__() -> dict[str, object]

nemo_curator.stages.text.download.common_crawl.download.CommonCrawlWARCReader.__setstate__(
    state: dict[str, object]
) -> None

nemo_curator.stages.text.download.common_crawl.download.CommonCrawlWARCReader._get_s3_client() -> object

Get or create a boto3 S3 client with double-checked locking.

Credentials, region, and endpoint are resolved entirely by boto3’s standard chain (AWS_* env vars, ~/.aws/config, instance profiles). Only connection-pool and retry settings are overridden.

nemo_curator.stages.text.download.common_crawl.download.CommonCrawlWARCReader._get_session() -> requests.Session

Get or create a requests session for connection pooling.

nemo_curator.stages.text.download.common_crawl.download.CommonCrawlWARCReader._read_warc_record(
    row: pandas.Series
) -> bytes | None

Fetch a single WARC record using HTTPS range request.

This method:

Fetches gzip-compressed WARC record bytes via HTTP range request
Decompresses the gzip content
Parses the WARC record format using warcio
Extracts and returns the HTTP response body (the actual content)

nemo_curator.stages.text.download.common_crawl.download.CommonCrawlWARCReader._read_warc_record_s3(
    row: pandas.Series
) -> bytes | None

Fetch a single WARC record using S3 range request (boto3).

nemo_curator.stages.text.download.common_crawl.download.CommonCrawlWARCReader._read_warc_records_batch(
    df_partition: pandas.DataFrame
) -> list[bytes | None]

Fetch multiple records in parallel using ThreadPoolExecutor.

nemo_curator.stages.text.download.common_crawl.download.CommonCrawlWARCReader._s3_key_from_filename(
    filename: str
) -> str

Derive S3 object key from the warc_filename column value.

Strips s3_key_prefix from the front of filename when present. E.g. prefix "crawl-data/" + filename "crawl-data/CC-MAIN-…" → key "CC-MAIN-…". With an empty prefix the filename is used as-is (the default for the AWS commoncrawl bucket).

nemo_curator.stages.text.download.common_crawl.download.CommonCrawlWARCReader.inputs() -> tuple[list[str], list[str]]

nemo_curator.stages.text.download.common_crawl.download.CommonCrawlWARCReader.outputs() -> tuple[list[str], list[str]]

nemo_curator.stages.text.download.common_crawl.download.CommonCrawlWARCReader.process(
    batch: nemo_curator.tasks.DocumentBatch
) -> nemo_curator.tasks.DocumentBatch

nemo_curator.stages.text.download.common_crawl.download.CC_BASE_URL = 'https://data.commoncrawl.org/'

nemo_curator.stages.text.download.common_crawl.download.HTTP_OK = 200

nemo_curator.stages.text.download.common_crawl.download.HTTP_PARTIAL_CONTENT = 206