> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/curator/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/curator/_mcp/server.

# nemo_curator.stages.deduplication.id_generator

## Module Contents

### Classes

| Name                                                                                 | Description                                                              |
| ------------------------------------------------------------------------------------ | ------------------------------------------------------------------------ |
| [`IdGenerator`](#nemo_curator-stages-deduplication-id_generator-IdGenerator)         | Ray actor version of IdGenerator.                                        |
| [`IdGeneratorBase`](#nemo_curator-stages-deduplication-id_generator-IdGeneratorBase) | Base IdGenerator class without Ray decorator for testing and direct use. |

### Functions

| Name                                                                                                       | Description                   |
| ---------------------------------------------------------------------------------------------------------- | ----------------------------- |
| [`create_id_generator_actor`](#nemo_curator-stages-deduplication-id_generator-create_id_generator_actor)   | Create an id generator actor. |
| [`get_id_generator_actor`](#nemo_curator-stages-deduplication-id_generator-get_id_generator_actor)         | -                             |
| [`kill_id_generator_actor`](#nemo_curator-stages-deduplication-id_generator-kill_id_generator_actor)       | -                             |
| [`write_id_generator_to_disk`](#nemo_curator-stages-deduplication-id_generator-write_id_generator_to_disk) | -                             |

### Data

[`CURATOR_DEDUP_ID_STR`](#nemo_curator-stages-deduplication-id_generator-CURATOR_DEDUP_ID_STR)

[`CURATOR_ID_GENERATOR_ACTOR_NAME`](#nemo_curator-stages-deduplication-id_generator-CURATOR_ID_GENERATOR_ACTOR_NAME)

### API

```python
class nemo_curator.stages.deduplication.id_generator.IdGenerator()
```

**Bases:** [IdGeneratorBase](#nemo_curator-stages-deduplication-id_generator-IdGeneratorBase)

Ray actor version of IdGenerator.

```python
nemo_curator.stages.deduplication.id_generator.IdGenerator.wait() -> None
```

Function used by create\_id\_generator\_actor to make sure the actor is started.

```python
class nemo_curator.stages.deduplication.id_generator.IdGeneratorBase(
    start_id: int = 0,
    batch_registry: dict[str, tuple[int, int]] | None = None
)
```

Base IdGenerator class without Ray decorator for testing and direct use.

**`batch_registry`** `= batch_registry or {}`

---

```python
nemo_curator.stages.deduplication.id_generator.IdGeneratorBase.from_disk(
    filepath: str,
    storage_options: dict[str, typing.Any] | None = None
) -> nemo_curator.stages.deduplication.id_generator.IdGeneratorBase
```

classmethod

```python
nemo_curator.stages.deduplication.id_generator.IdGeneratorBase.get_batch_range(
    files: str | list[str] | None,
    key: str | None
) -> tuple[int, int]
```

```python
nemo_curator.stages.deduplication.id_generator.IdGeneratorBase.hash_files(
    filepath: str | list[str]
) -> str
```

```python
nemo_curator.stages.deduplication.id_generator.IdGeneratorBase.register_batch(
    files: str | list[str],
    count: int
) -> int
```

```python
nemo_curator.stages.deduplication.id_generator.IdGeneratorBase.to_disk(
    filepath: str,
    storage_options: dict[str, typing.Any] | None = None
) -> None
```

```python
nemo_curator.stages.deduplication.id_generator.create_id_generator_actor(
    filepath: str | None = None,
    storage_options: dict[str, typing.Any] | None = None
) -> None
```

Create an id generator actor.

**Parameters:**

**`filepath`** `str` — default: None

Path from where we want to load the id generator state json file.
If None, a new actor is created.

---

**`storage_options`** `dict[str, Any] | None` — default: None

Storage options to pass to fsspec.open.

---

```python
nemo_curator.stages.deduplication.id_generator.get_id_generator_actor() -> ray.actor.ActorHandle[nemo_curator.stages.deduplication.id_generator.IdGenerator]
```

```python
nemo_curator.stages.deduplication.id_generator.kill_id_generator_actor() -> None
```

```python
nemo_curator.stages.deduplication.id_generator.write_id_generator_to_disk(
    filepath: str,
    storage_options: dict[str, typing.Any] | None = None
) -> None
```

```python
nemo_curator.stages.deduplication.id_generator.CURATOR_DEDUP_ID_STR = '_curator_dedup_id'
```

```python
nemo_curator.stages.deduplication.id_generator.CURATOR_ID_GENERATOR_ACTOR_NAME = 'curator_deduplication_id_generator'
```