nemo_curator.backends.ray_data.utils

Module Contents

Functions

Name	Description
`calculate_concurrency_for_actors_for_stage`	Calculate concurrency if we want to spin up actors based on available resources and stage requirements.
`is_actor_stage`	Check if the stage is an actor stage.

API

nemo_curator.backends.ray_data.utils.calculate_concurrency_for_actors_for_stage(
    stage: nemo_curator.stages.base.ProcessingStage,
    ignore_head_node: bool = False
) -> tuple[int, int] | int

Calculate concurrency if we want to spin up actors based on available resources and stage requirements.

Returns: tuple[int, int] | int

int | tuple[int, int]: Number of actors to use int: Number of workers to use tuple[int, int]: tuple of min / max actors to use and number of workers to use

nemo_curator.backends.ray_data.utils.is_actor_stage(
    stage: nemo_curator.stages.base.ProcessingStage
) -> bool

Check if the stage is an actor stage.