backends.experimental.ray_data.utils

Module Contents

Functions

calculate_concurrency_for_actors_for_stage	Calculate concurrency if we want to spin up actors based on available resources and stage requirements.
is_actor_stage	Check if the stage is an actor stage.

API

backends.experimental.ray_data.utils.calculate_concurrency_for_actors_for_stage(

stage: nemo_curator.stages.base.ProcessingStage,

) → tuple[int, int] | int

Calculate concurrency if we want to spin up actors based on available resources and stage requirements.

Returns: int | tuple[int, int]: Number of actors to use int: Number of workers to use tuple[int, int]: tuple of min / max actors to use and number of workers to use

backends.experimental.ray_data.utils.is_actor_stage(

stage: nemo_curator.stages.base.ProcessingStage,

) → bool

Check if the stage is an actor stage.