> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/curator/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/curator/_mcp/server.

# nemo_curator.stages.text.download.common_crawl.url_generation

## Module Contents

### Classes

| Name                                                                                                                        | Description                    |
| --------------------------------------------------------------------------------------------------------------------------- | ------------------------------ |
| [`BaseCommonCrawlUrlGenerator`](#nemo_curator-stages-text-download-common_crawl-url_generation-BaseCommonCrawlUrlGenerator) | Get URLs for Common Crawl data |
| [`MainCommonCrawlUrlGenerator`](#nemo_curator-stages-text-download-common_crawl-url_generation-MainCommonCrawlUrlGenerator) | -                              |
| [`NewsCommonCrawlUrlGenerator`](#nemo_curator-stages-text-download-common_crawl-url_generation-NewsCommonCrawlUrlGenerator) | -                              |

### API

```python
class nemo_curator.stages.text.download.common_crawl.url_generation.BaseCommonCrawlUrlGenerator(
    start_snapshot_str: str,
    end_snapshot_str: str,
    data_prefix: str = 'https://data.commoncrawl.org',
    limit: int | None = None
)
```

Dataclass

Abstract

**Bases:** `URLGenerator`

Get URLs for Common Crawl data
Each concrete implementation must implement `_parse_datetime_from_snapshot_string` and `generate_path_urls`

**`data_prefix`** `str = 'https://data.commoncrawl.org'`

---

**`end_snapshot_str`** `str`

---

**`limit`** `int | None = None`

---

**`start_snapshot_str`** `str`

---

```python
nemo_curator.stages.text.download.common_crawl.url_generation.BaseCommonCrawlUrlGenerator.__post_init__()
```

```python
nemo_curator.stages.text.download.common_crawl.url_generation.BaseCommonCrawlUrlGenerator._parse_datetime_from_snapshot_string(
    snapshot_str: str,
    for_start: bool
) -> datetime.datetime
```

abstract

Parses a snapshot string (YYYY-WW or YYYY-MM) into a datetime object.

```python
nemo_curator.stages.text.download.common_crawl.url_generation.BaseCommonCrawlUrlGenerator._start_end_dates() -> tuple[datetime.date, datetime.date]
```

Parses the start and end snapshot strings into date objects.
For 'news' (YYYY-MM), the day is set to 1 for start\_date, and the last day of the month for end\_date
to ensure the full month is covered.

```python
nemo_curator.stages.text.download.common_crawl.url_generation.BaseCommonCrawlUrlGenerator.generate_data_urls(
    path_urls: str | list[str] | None = None
) -> list[str]
```

Fetches all relevant warc.paths.gz files, decompresses them,
and returns a list of all individual WARC file URLs.

```python
nemo_curator.stages.text.download.common_crawl.url_generation.BaseCommonCrawlUrlGenerator.generate_path_urls() -> list[str]
```

abstract

Generates the list of URLs pointing to warc.paths.gz files.

```python
nemo_curator.stages.text.download.common_crawl.url_generation.BaseCommonCrawlUrlGenerator.generate_urls() -> list[str]
```

Process the task and return a list of WARC URLs

```python
class nemo_curator.stages.text.download.common_crawl.url_generation.MainCommonCrawlUrlGenerator(
    start_snapshot_str: str,
    end_snapshot_str: str,
    data_prefix: str = 'https://data.commoncrawl.org',
    limit: int | None = None,
    index_prefix: str = 'https://index.commoncrawl....
)
```

Dataclass

**Bases:** [BaseCommonCrawlUrlGenerator](#nemo_curator-stages-text-download-common_crawl-url_generation-BaseCommonCrawlUrlGenerator)

**`_snapshot_index`** `list[dict]`

---

**`index_prefix`** `str = 'https://index.commoncrawl.org'`

---

```python
nemo_curator.stages.text.download.common_crawl.url_generation.MainCommonCrawlUrlGenerator._parse_datetime_from_snapshot_string(
    snapshot_str: str,
    for_start: bool
) -> datetime.datetime
```

```python
nemo_curator.stages.text.download.common_crawl.url_generation.MainCommonCrawlUrlGenerator.generate_path_urls() -> list[str]
```

```python
class nemo_curator.stages.text.download.common_crawl.url_generation.NewsCommonCrawlUrlGenerator(
    start_snapshot_str: str,
    end_snapshot_str: str,
    data_prefix: str = 'https://data.commoncrawl.org',
    limit: int | None = None
)
```

Dataclass

**Bases:** [BaseCommonCrawlUrlGenerator](#nemo_curator-stages-text-download-common_crawl-url_generation-BaseCommonCrawlUrlGenerator)

```python
nemo_curator.stages.text.download.common_crawl.url_generation.NewsCommonCrawlUrlGenerator._parse_datetime_from_snapshot_string(
    snapshot_str: str,
    for_start: bool
) -> datetime.datetime
```

```python
nemo_curator.stages.text.download.common_crawl.url_generation.NewsCommonCrawlUrlGenerator.generate_path_urls() -> list[str]
```