Curate TextProcess DataContent Processing

Adding Document IDs

View as Markdown

Add unique identifiers to each document in your text dataset.

How It Works

Document IDs are useful for:

  • Pipeline tracking - Monitor documents through processing stages
  • Dataset versioning - Identify documents across different versions

Usage

Basic Usage

from nemo_curator.stages.text.modules import AddId
# Initialize pipeline, read stage, etc.
# Add to your pipeline
pipeline.add_stage(AddId(id_field="doc_id"))

Configuration Options

# Customize ID generation
pipeline.add_stage(AddId(
    id_field="document_id",        # Field name for IDs
    id_prefix="corpus_v2",         # Optional prefix
    overwrite=True                 # Overwrite existing IDs
))

Parameters

ParameterTypeDefaultDescription
id_fieldstrRequiredField name where IDs will be stored
id_prefixstrNoneOptional prefix for IDs
overwriteboolFalseWhether to overwrite existing ID fields

ID Format

Generated IDs follow this pattern:

  • Without prefix: {task_uuid}_{index}
  • With prefix: {prefix}_{task_uuid}_{index}

Complete Example

from nemo_curator.core.client import RayClient
from nemo_curator.pipeline import Pipeline
from nemo_curator.stages.text.io.reader import JsonlReader
from nemo_curator.stages.text.modules import AddId
from nemo_curator.stages.text.io.writer import JsonlWriter
# Initialize Ray client
ray_client = RayClient()
ray_client.start()
# Create pipeline
pipeline = Pipeline(name="add_ids")
# Add stages
pipeline.add_stage(JsonlReader(file_paths="input/"))
pipeline.add_stage(AddId(id_field="doc_id", id_prefix="v1"))
pipeline.add_stage(JsonlWriter("output/"))
# Run pipeline
result = pipeline.run()
# Stop Ray client
ray_client.stop()

Alternative: Reader-Based ID Generation

For deduplication workflows, unique IDs are generated during data loading:

from nemo_curator.core.client import RayClient
from nemo_curator.pipeline import Pipeline
from nemo_curator.stages.deduplication.id_generator import create_id_generator_actor
from nemo_curator.stages.text.io.reader import JsonlReader
# Initialize Ray client
ray_client = RayClient()
ray_client.start()
pipeline = Pipeline(name="id_generator_example")
# Create ID generator
create_id_generator_actor()
# Reader generates IDs automatically
reader = JsonlReader(
    file_paths="data/",
    _generate_ids=True  # Adds '_curator_dedup_id' field
)
pipeline.add_stage(reader)
# Run pipeline
results = pipeline.run()
# Stop Ray client
ray_client.stop()
# Examine the first 5 rows of the first DocumentBatch
print(results[0].data.head())

This approach:

  • Generates monotonically increasing integer IDs
  • Required for some deduplication workflows
  • Persists ID state across pipeline runs

Error Handling

Existing ID field:

# This raises ValueError if 'doc_id' already exists
AddId(id_field="doc_id", overwrite=False)
# This overwrites existing field with warning
AddId(id_field="doc_id", overwrite=True)

Best Practices

  • Place early in pipeline - Add IDs after loading, before filtering
  • Use descriptive field names - doc_id, document_id, unique_id
  • Choose appropriate method:
    • Use AddId for general document tracking
    • Use ID generator for deduplication workflows

For deduplication workflows, see Deduplication.