> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/curator/llms.txt.
> For full documentation content, see https://docs.nvidia.com/nemo/curator/llms-full.txt.
# nemo_curator.stages.text.download.common_crawl.url_generation
## Module Contents
### Classes
| Name | Description |
| --------------------------------------------------------------------------------------------------------------------------- | ------------------------------ |
| [`BaseCommonCrawlUrlGenerator`](#nemo_curator-stages-text-download-common_crawl-url_generation-BaseCommonCrawlUrlGenerator) | Get URLs for Common Crawl data |
| [`MainCommonCrawlUrlGenerator`](#nemo_curator-stages-text-download-common_crawl-url_generation-MainCommonCrawlUrlGenerator) | - |
| [`NewsCommonCrawlUrlGenerator`](#nemo_curator-stages-text-download-common_crawl-url_generation-NewsCommonCrawlUrlGenerator) | - |
### API
```python
class nemo_curator.stages.text.download.common_crawl.url_generation.BaseCommonCrawlUrlGenerator(
start_snapshot_str: str,
end_snapshot_str: str,
data_prefix: str = 'https://data.commoncrawl.org',
limit: int | None = None
)
```
Dataclass
Abstract
**Bases:** `URLGenerator`
Get URLs for Common Crawl data
Each concrete implementation must implement `_parse_datetime_from_snapshot_string` and `generate_path_urls`
```python
nemo_curator.stages.text.download.common_crawl.url_generation.BaseCommonCrawlUrlGenerator.__post_init__()
```
```python
nemo_curator.stages.text.download.common_crawl.url_generation.BaseCommonCrawlUrlGenerator._parse_datetime_from_snapshot_string(
snapshot_str: str,
for_start: bool
) -> datetime.datetime
```
abstract
Parses a snapshot string (YYYY-WW or YYYY-MM) into a datetime object.
```python
nemo_curator.stages.text.download.common_crawl.url_generation.BaseCommonCrawlUrlGenerator._start_end_dates() -> tuple[datetime.date, datetime.date]
```
Parses the start and end snapshot strings into date objects.
For 'news' (YYYY-MM), the day is set to 1 for start\_date, and the last day of the month for end\_date
to ensure the full month is covered.
```python
nemo_curator.stages.text.download.common_crawl.url_generation.BaseCommonCrawlUrlGenerator.generate_data_urls(
path_urls: str | list[str] | None = None
) -> list[str]
```
Fetches all relevant warc.paths.gz files, decompresses them,
and returns a list of all individual WARC file URLs.
```python
nemo_curator.stages.text.download.common_crawl.url_generation.BaseCommonCrawlUrlGenerator.generate_path_urls() -> list[str]
```
abstract
Generates the list of URLs pointing to warc.paths.gz files.
```python
nemo_curator.stages.text.download.common_crawl.url_generation.BaseCommonCrawlUrlGenerator.generate_urls() -> list[str]
```
Process the task and return a list of WARC URLs
```python
class nemo_curator.stages.text.download.common_crawl.url_generation.MainCommonCrawlUrlGenerator(
start_snapshot_str: str,
end_snapshot_str: str,
data_prefix: str = 'https://data.commoncrawl.org',
limit: int | None = None,
index_prefix: str = 'https://index.commoncrawl....
)
```
Dataclass
**Bases:** [BaseCommonCrawlUrlGenerator](#nemo_curator-stages-text-download-common_crawl-url_generation-BaseCommonCrawlUrlGenerator)
```python
nemo_curator.stages.text.download.common_crawl.url_generation.MainCommonCrawlUrlGenerator._parse_datetime_from_snapshot_string(
snapshot_str: str,
for_start: bool
) -> datetime.datetime
```
```python
nemo_curator.stages.text.download.common_crawl.url_generation.MainCommonCrawlUrlGenerator.generate_path_urls() -> list[str]
```
```python
class nemo_curator.stages.text.download.common_crawl.url_generation.NewsCommonCrawlUrlGenerator(
start_snapshot_str: str,
end_snapshot_str: str,
data_prefix: str = 'https://data.commoncrawl.org',
limit: int | None = None
)
```
Dataclass
**Bases:** [BaseCommonCrawlUrlGenerator](#nemo_curator-stages-text-download-common_crawl-url_generation-BaseCommonCrawlUrlGenerator)
```python
nemo_curator.stages.text.download.common_crawl.url_generation.NewsCommonCrawlUrlGenerator._parse_datetime_from_snapshot_string(
snapshot_str: str,
for_start: bool
) -> datetime.datetime
```
```python
nemo_curator.stages.text.download.common_crawl.url_generation.NewsCommonCrawlUrlGenerator.generate_path_urls() -> list[str]
```