nemo_curator.backends.experimental.ray_actor_pool.adapter

Module Contents

Name	Description
`RayActorPoolStageAdapter`	Adapts ProcessingStage to Ray actors for use with ActorPool.

class nemo_curator.backends.experimental.ray_actor_pool.adapter.RayActorPoolStageAdapter(
    stage: nemo_curator.stages.base.ProcessingStage
)

Bases: BaseStageAdapter

Adapts ProcessingStage to Ray actors for use with ActorPool.

This adapter is designed to work with Ray’s ActorPool for better resource management and load balancing.

_batch_size

= self.stage.batch_size

nemo_curator.backends.experimental.ray_actor_pool.adapter.RayActorPoolStageAdapter.get_batch_size() -> int

Get the batch size for this stage.

nemo_curator.backends.experimental.ray_actor_pool.adapter.RayActorPoolStageAdapter.setup_on_node() -> None

Setup method for Ray actors.

Note: This method is not used in the current implementation since we use the Ray Data pattern of calling setup_on_node before actor creation.