> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/curator/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/curator/_mcp/server.

# nemo_curator.stages.text.download.base.url_generation

## Module Contents

### Classes

| Name                                                                                              | Description                                                                 |
| ------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------- |
| [`URLGenerationStage`](#nemo_curator-stages-text-download-base-url_generation-URLGenerationStage) | Stage that generates URLs from minimal input parameters.                    |
| [`URLGenerator`](#nemo_curator-stages-text-download-base-url_generation-URLGenerator)             | Abstract base class for URL generators - generates URLs from minimal input. |

### API

```python
class nemo_curator.stages.text.download.base.url_generation.URLGenerationStage(
    url_generator: nemo_curator.stages.text.download.base.url_generation.URLGenerator,
    limit: int | None = None
)
```

Dataclass

**Bases:** [ProcessingStage\[EmptyTask, FileGroupTask\]](/nemo-curator/nemo_curator/stages/base#nemo_curator-stages-base-ProcessingStage)

Stage that generates URLs from minimal input parameters.

This allows pipelines to start with URL generation (like Common Crawl).

**`limit`** `int | None = None`

---

**`resources`** `= Resources(cpus=0.5)`

---

**`url_generator`** `URLGenerator`

---

```python
nemo_curator.stages.text.download.base.url_generation.URLGenerationStage.__post_init__()
```

```python
nemo_curator.stages.text.download.base.url_generation.URLGenerationStage.inputs() -> tuple[list[str], list[str]]
```

Define input requirements - expects empty task.

```python
nemo_curator.stages.text.download.base.url_generation.URLGenerationStage.num_workers() -> int | None
```

```python
nemo_curator.stages.text.download.base.url_generation.URLGenerationStage.outputs() -> tuple[list[str], list[str]]
```

Define output - produces FileGroupTask with URLs.

```python
nemo_curator.stages.text.download.base.url_generation.URLGenerationStage.process(
    task: nemo_curator.tasks.EmptyTask
) -> list[nemo_curator.tasks.FileGroupTask]
```

Generate URLs and create FileGroupTasks.

**Parameters:**

**`task`** `EmptyTask`

Empty input task

---

**Returns:** `list[FileGroupTask]`

list\[FileGroupTask]: List of tasks containing URLs

```python
class nemo_curator.stages.text.download.base.url_generation.URLGenerator()
```

Abstract

Abstract base class for URL generators - generates URLs from minimal input.

```python
nemo_curator.stages.text.download.base.url_generation.URLGenerator.generate_urls() -> list[str]
```

abstract

Generate a list of URLs to download.