nemo_curator.backends.ray_data.utils

Module Contents

Functions

Name	Description
`get_actor_compute_strategy_for_stage`	Get the Ray Data actor-pool compute strategy for a processing stage.
`is_actor_stage`	Check if the stage is an actor stage.

API

nemo_curator.backends.ray_data.utils.get_actor_compute_strategy_for_stage(
    stage: nemo_curator.stages.base.ProcessingStage
) -> ray.data.ActorPoolStrategy

Get the Ray Data actor-pool compute strategy for a processing stage.

Explicit stage num_workers requests a fixed-size actor pool. Otherwise, actor stages use Ray Data’s autoscaling pool and can optionally override min/max/initial workers through ray_stage_spec.

nemo_curator.backends.ray_data.utils.is_actor_stage(
    stage: nemo_curator.stages.base.ProcessingStage
) -> bool

Check if the stage is an actor stage.