nemo_curator.backends.ray_data.utils

Module Contents

Functions

Name	Description
`get_actor_compute_strategy_for_stage`	Get the Ray Data actor-pool compute strategy for a processing stage.
`get_configured_actor_pool_sizing_keys`	Return actor-pool sizing keys configured in a ray stage spec.
`is_actor_stage`	Check if the stage is an actor stage.

Data

ACTOR_POOL_SIZING_KEYS

API

nemo_curator.backends.ray_data.utils.get_actor_compute_strategy_for_stage(
    stage: nemo_curator.stages.base.ProcessingStage
) -> ray.data.ActorPoolStrategy

Get the Ray Data actor-pool compute strategy for a processing stage.

Explicit stage num_workers requests a fixed-size actor pool. Otherwise, actor stages use Ray Data’s autoscaling pool and can optionally override min/max/initial workers through ray_stage_spec.

nemo_curator.backends.ray_data.utils.get_configured_actor_pool_sizing_keys(
    ray_stage_spec: collections.abc.Mapping[str, object]
) -> list[str]

Return actor-pool sizing keys configured in a ray stage spec.

nemo_curator.backends.ray_data.utils.is_actor_stage(
    stage: nemo_curator.stages.base.ProcessingStage
) -> bool

Check if the stage is an actor stage.

nemo_curator.backends.ray_data.utils.ACTOR_POOL_SIZING_KEYS = (RayStageSpecKeys.MIN_WORKERS, RayStageSpecKeys.MAX_WORKERS, RayStageSpecKeys.IN...