Infrastructure References
This section provides technical reference documentation for NeMo Curator’s infrastructure components that are used across all modalities (text, image, video).
Infrastructure Components
Memory Management
Optimize memory usage when processing large datasets partitioning batching monitoring
GPU Acceleration
Leverage NVIDIA GPUs for faster data processing cuda rmm performance
Resumable Processing
Continue interrupted operations across large datasets checkpoints recovery batching
Stage Worker Sizing
Configure fixed and autoscaling worker pools across Ray Data, Xenna, and Ray Actor Pool backends workers autoscaling stage-overrides
Container Environments
Available environments and configurations in NeMo Curator containers. Includes build arguments and video-specific environments. docker conda environments