nemo_curator.core.serve.ray_serve.backend

Module Contents

Classes

Name	Description
`RayServeBackend`	Ray Serve backend for `InferenceServer`.

API

class nemo_curator.core.serve.ray_serve.backend.RayServeBackend(
    server: nemo_curator.core.serve.server.InferenceServer
)

Bases: InferenceBackend

Ray Serve backend for InferenceServer.

nemo_curator.core.serve.ray_serve.backend.RayServeBackend._cleanup_failed_deploy() -> None

staticmethod

Best-effort cleanup after a failed deploy.

nemo_curator.core.serve.ray_serve.backend.RayServeBackend._deploy() -> None

Deploy models onto the connected Ray cluster.

nemo_curator.core.serve.ray_serve.backend.RayServeBackend._quiet_runtime_env() -> dict[str, typing.Any]

staticmethod

Return a runtime_env dict that suppresses per-request logs.

nemo_curator.core.serve.ray_serve.backend.RayServeBackend._reset_serve_client_cache() -> None

staticmethod

Reset Ray Serve’s cached controller client.

TODO: Remove this once https://github.com/ray-project/ray/issues/61608 is fixed.

nemo_curator.core.serve.ray_serve.backend.RayServeBackend._to_llm_config(
    model: nemo_curator.core.serve.ray_serve.config.RayServeModelConfig,
    quiet_runtime_env: dict[str, typing.Any] | None = None
) -> ray.serve.llm.LLMConfig

staticmethod

Translate a typed Ray Serve model config into LLMConfig.

nemo_curator.core.serve.ray_serve.backend.RayServeBackend.start() -> None

Connect to Ray, deploy the models, and detach the driver.

nemo_curator.core.serve.ray_serve.backend.RayServeBackend.stop() -> None

Reconnect to Ray and tear down Ray Serve.