> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/curator/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/curator/_mcp/server.

# nemo_curator.stages.text.download.common_crawl.download

## Module Contents

### Classes

| Name                                                                                                              | Description                                                                     |
| ----------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------- |
| [`CommonCrawlWARCDownloader`](#nemo_curator-stages-text-download-common_crawl-download-CommonCrawlWARCDownloader) | Downloads WARC files from the Common Crawl to a local directory                 |
| [`CommonCrawlWARCReader`](#nemo_curator-stages-text-download-common_crawl-download-CommonCrawlWARCReader)         | Reads WARC records directly from Common Crawl using HTTPS or S3 range requests. |

### Data

[`CC_BASE_URL`](#nemo_curator-stages-text-download-common_crawl-download-CC_BASE_URL)

[`HTTP_OK`](#nemo_curator-stages-text-download-common_crawl-download-HTTP_OK)

[`HTTP_PARTIAL_CONTENT`](#nemo_curator-stages-text-download-common_crawl-download-HTTP_PARTIAL_CONTENT)

### API

```python
class nemo_curator.stages.text.download.common_crawl.download.CommonCrawlWARCDownloader(
    download_dir: str,
    use_aws_to_download: bool = False,
    verbose: bool = False
)
```

**Bases:** `DocumentDownloader`

Downloads WARC files from the Common Crawl to a local directory

```python
nemo_curator.stages.text.download.common_crawl.download.CommonCrawlWARCDownloader._download_to_path(
    url: str,
    path: str
) -> tuple[bool, str | None]
```

Download a file to a temporary file.

**Parameters:**

**`url`** `str`

URL to download

---

**`path`** `str`

Local path to save file

---

**Returns:** `bool`

Tuple of (success, error\_message). If success is True, error\_message is None.

```python
nemo_curator.stages.text.download.common_crawl.download.CommonCrawlWARCDownloader._get_output_filename(
    url: str
) -> str
```

Generate output filename from URL.

```python
class nemo_curator.stages.text.download.common_crawl.download.CommonCrawlWARCReader(
    warc_filename_col: str = 'warc_filename',
    warc_record_offset_col: str = 'warc_record_offset',
    warc_record_length_col: str = 'warc_record_length',
    binary_content_col: str = 'binary_content',
    drop_failed: bool = True,
    max_workers: int = 16,
    timeout: int = 30,
    max_retries: int = 3,
    use_s3: bool | None = None,
    s3_bucket: str | None = None,
    s3_key_prefix: str | None = None
)
```

**Bases:** [ProcessingStage\[DocumentBatch, DocumentBatch\]](/nemo-curator/nemo_curator/stages/base#nemo_curator-stages-base-ProcessingStage)

Reads WARC records directly from Common Crawl using HTTPS or S3 range requests.

This stage fetches raw HTML content from Common Crawl's public servers
using byte-range requests.

**`_lock`** `= threading.Lock()`

---

**`name`** `= 'CommonCrawlWARCReader'`

---

**`s3_bucket`**

---

**`s3_key_prefix`**

---

**`use_s3`**

---

```python
nemo_curator.stages.text.download.common_crawl.download.CommonCrawlWARCReader.__getstate__() -> dict[str, object]
```

```python
nemo_curator.stages.text.download.common_crawl.download.CommonCrawlWARCReader.__setstate__(
    state: dict[str, object]
) -> None
```

```python
nemo_curator.stages.text.download.common_crawl.download.CommonCrawlWARCReader._get_s3_client() -> object
```

Get or create a boto3 S3 client with double-checked locking.

Credentials, region, and endpoint are resolved entirely by boto3's
standard chain (`AWS_*` env vars, `~/.aws/config`, instance
profiles).  Only connection-pool and retry settings are overridden.

```python
nemo_curator.stages.text.download.common_crawl.download.CommonCrawlWARCReader._get_session() -> requests.Session
```

Get or create a requests session for connection pooling.

```python
nemo_curator.stages.text.download.common_crawl.download.CommonCrawlWARCReader._read_warc_record(
    row: pandas.Series
) -> bytes | None
```

Fetch a single WARC record using HTTPS range request.

This method:

1. Fetches gzip-compressed WARC record bytes via HTTP range request
2. Decompresses the gzip content
3. Parses the WARC record format using warcio
4. Extracts and returns the HTTP response body (the actual content)

```python
nemo_curator.stages.text.download.common_crawl.download.CommonCrawlWARCReader._read_warc_record_s3(
    row: pandas.Series
) -> bytes | None
```

Fetch a single WARC record using S3 range request (boto3).

```python
nemo_curator.stages.text.download.common_crawl.download.CommonCrawlWARCReader._read_warc_records_batch(
    df_partition: pandas.DataFrame
) -> list[bytes | None]
```

Fetch multiple records in parallel using ThreadPoolExecutor.

```python
nemo_curator.stages.text.download.common_crawl.download.CommonCrawlWARCReader._s3_key_from_filename(
    filename: str
) -> str
```

Derive S3 object key from the warc\_filename column value.

Strips `s3_key_prefix` from the front of *filename* when present.
E.g. prefix `"crawl-data/"` + filename `"crawl-data/CC-MAIN-…"`
→ key `"CC-MAIN-…"`.  With an empty prefix the filename is used
as-is (the default for the AWS `commoncrawl` bucket).

```python
nemo_curator.stages.text.download.common_crawl.download.CommonCrawlWARCReader.inputs() -> tuple[list[str], list[str]]
```

```python
nemo_curator.stages.text.download.common_crawl.download.CommonCrawlWARCReader.outputs() -> tuple[list[str], list[str]]
```

```python
nemo_curator.stages.text.download.common_crawl.download.CommonCrawlWARCReader.process(
    batch: nemo_curator.tasks.DocumentBatch
) -> nemo_curator.tasks.DocumentBatch
```

```python
nemo_curator.stages.text.download.common_crawl.download.CC_BASE_URL = 'https://data.commoncrawl.org/'
```

```python
nemo_curator.stages.text.download.common_crawl.download.HTTP_OK = 200
```

```python
nemo_curator.stages.text.download.common_crawl.download.HTTP_PARTIAL_CONTENT = 206
```