DocumentationAPI Reference
DocumentationAPI Reference
  • API Reference
    • Overview
        • Nemo Curator
          • Backends
          • Config
          • Core
          • Metrics
          • Models
          • Package Info
          • Pipeline
          • Stages
            • Audio
            • Base
            • Client Partitioning
            • Deduplication
            • File Partitioning
            • Function Decorators
            • Image
            • Interleaved
            • Math
            • Resources
            • Synthetic
            • Text
            • Video
          • Tasks
          • Utils
    • Pipeline
    • ProcessingStage
    • CompositeStage
    • Resources
On this page
  • Subpackages
  • Submodules
API ReferenceFull Library ReferenceNemo CuratorNemo CuratorStages

nemo_curator.stages.deduplication

||View as Markdown|

Subpackages

  • nemo_curator.stages.deduplication.exact
  • nemo_curator.stages.deduplication.fuzzy
  • nemo_curator.stages.deduplication.semantic
  • nemo_curator.stages.deduplication.shuffle_utils

Submodules

  • nemo_curator.stages.deduplication.gpu_utils
  • nemo_curator.stages.deduplication.id_generator
  • nemo_curator.stages.deduplication.io_utils
Previous

nemo_curator.stages.client_partitioning

Next

nemo_curator.stages.deduplication.exact

NVIDIANVIDIA
Developer-friendly docs for your API
Privacy Policy | Manage My Privacy | Do Not Sell or Share My Data | Terms of Service | Accessibility | Corporate Policies | Product Security | Contact

Copyright © 2026, NVIDIA Corporation.

LogoLogoNeMo Curator