> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/gym/llms.txt.
> For full documentation content, see https://docs.nvidia.com/nemo/gym/llms-full.txt.
# nemo_gym.server_utils
## Module Contents
### Classes
| Name | Description |
| ----------------------------------------------------------------------------------------- | ------------------------------------------------------------- |
| [`BaseServer`](#nemo_gym-server_utils-BaseServer) | All instances of BaseServer are queryable using ServerClient. |
| [`GlobalAIOHTTPAsyncClientConfig`](#nemo_gym-server_utils-GlobalAIOHTTPAsyncClientConfig) | - |
| [`HeadServer`](#nemo_gym-server_utils-HeadServer) | - |
| [`ProfilingMiddlewareConfig`](#nemo_gym-server_utils-ProfilingMiddlewareConfig) | - |
| [`ProfilingMiddlewareInputConfig`](#nemo_gym-server_utils-ProfilingMiddlewareInputConfig) | - |
| [`ServerClient`](#nemo_gym-server_utils-ServerClient) | - |
| [`ServerInstanceDisplayConfig`](#nemo_gym-server_utils-ServerInstanceDisplayConfig) | - |
| [`SimpleServer`](#nemo_gym-server_utils-SimpleServer) | - |
| [`UvicornLoggingConfig`](#nemo_gym-server_utils-UvicornLoggingConfig) | - |
### Functions
| Name | Description |
| ------------------------------------------------------------------------------------------------------------------------- | ------------------------------------ |
| [`get_global_aiohttp_client`](#nemo_gym-server_utils-get_global_aiohttp_client) | - |
| [`get_nemo_gym_fastapi_num_workers`](#nemo_gym-server_utils-get_nemo_gym_fastapi_num_workers) | - |
| [`get_response_json`](#nemo_gym-server_utils-get_response_json) | - |
| [`get_server_url`](#nemo_gym-server_utils-get_server_url) | - |
| [`global_aiohttp_client_exit`](#nemo_gym-server_utils-global_aiohttp_client_exit) | - |
| [`initialize_ray`](#nemo_gym-server_utils-initialize_ray) | Initialize ray cluster in a process. |
| [`is_global_aiohttp_client_request_debug_enabled`](#nemo_gym-server_utils-is_global_aiohttp_client_request_debug_enabled) | - |
| [`is_global_aiohttp_client_setup`](#nemo_gym-server_utils-is_global_aiohttp_client_setup) | - |
| [`is_nemo_gym_fastapi_entrypoint`](#nemo_gym-server_utils-is_nemo_gym_fastapi_entrypoint) | - |
| [`is_nemo_gym_fastapi_worker`](#nemo_gym-server_utils-is_nemo_gym_fastapi_worker) | - |
| [`maybe_ray_cluster_exit`](#nemo_gym-server_utils-maybe_ray_cluster_exit) | - |
| [`raise_for_status`](#nemo_gym-server_utils-raise_for_status) | - |
| [`request`](#nemo_gym-server_utils-request) | - |
| [`set_global_aiohttp_client`](#nemo_gym-server_utils-set_global_aiohttp_client) | - |
| [`set_is_nemo_gym_fastapi_entrypoint`](#nemo_gym-server_utils-set_is_nemo_gym_fastapi_entrypoint) | - |
| [`set_is_nemo_gym_fastapi_worker`](#nemo_gym-server_utils-set_is_nemo_gym_fastapi_worker) | - |
| [`set_nemo_gym_fastapi_num_workers`](#nemo_gym-server_utils-set_nemo_gym_fastapi_num_workers) | - |
### Data
[`DEFAULT_HEAD_SERVER_PORT`](#nemo_gym-server_utils-DEFAULT_HEAD_SERVER_PORT)
[`DISCONNECTED_CLIENT_OS_HELP_TEXT`](#nemo_gym-server_utils-DISCONNECTED_CLIENT_OS_HELP_TEXT)
[`DISCONNECTED_CLIENT_OS_PRINT_INTERVAL`](#nemo_gym-server_utils-DISCONNECTED_CLIENT_OS_PRINT_INTERVAL)
[`IS_NEMO_GYM_FASTAPI_ENTRYPOINT_KEY_NAME`](#nemo_gym-server_utils-IS_NEMO_GYM_FASTAPI_ENTRYPOINT_KEY_NAME)
[`IS_NEMO_GYM_FASTAPI_WORKER_KEY_NAME`](#nemo_gym-server_utils-IS_NEMO_GYM_FASTAPI_WORKER_KEY_NAME)
[`MAX_NUM_TRIES`](#nemo_gym-server_utils-MAX_NUM_TRIES)
[`NEMO_GYM_FASTAPI_NUM_WORKERS`](#nemo_gym-server_utils-NEMO_GYM_FASTAPI_NUM_WORKERS)
[`SESSION_ID_KEY`](#nemo_gym-server_utils-SESSION_ID_KEY)
[`ServerStatus`](#nemo_gym-server_utils-ServerStatus)
[`_GLOBAL_AIOHTTP_CLIENT`](#nemo_gym-server_utils-_GLOBAL_AIOHTTP_CLIENT)
[`_GLOBAL_AIOHTTP_CLIENT_REQUEST_DEBUG`](#nemo_gym-server_utils-_GLOBAL_AIOHTTP_CLIENT_REQUEST_DEBUG)
[`_NEMO_GYM_STARTED_RAY_CLUSTER`](#nemo_gym-server_utils-_NEMO_GYM_STARTED_RAY_CLUSTER)
[`_NUM_CLIENT_OS_ERROR`](#nemo_gym-server_utils-_NUM_CLIENT_OS_ERROR)
[`_NUM_SERVER_DISCONNECTED_ERROR`](#nemo_gym-server_utils-_NUM_SERVER_DISCONNECTED_ERROR)
### API
```python
class nemo_gym.server_utils.BaseServer()
```
**Bases:** `BaseModel`
All instances of BaseServer are queryable using ServerClient.
```python
nemo_gym.server_utils.BaseServer.load_config_from_global_config() -> nemo_gym.config_types.BaseRunServerInstanceConfig
```
classmethod
```python
class nemo_gym.server_utils.GlobalAIOHTTPAsyncClientConfig()
```
**Bases:** `BaseModel`
```python
class nemo_gym.server_utils.HeadServer()
```
**Bases:** [BaseServer](#nemo_gym-server_utils-BaseServer)
```python
nemo_gym.server_utils.HeadServer.get_server_instances() -> typing.List[dict]
```
```python
nemo_gym.server_utils.HeadServer.global_config_dict_yaml() -> str
```
async
```python
nemo_gym.server_utils.HeadServer.run_webserver() -> typing.Tuple[uvicorn.Server, threading.Thread, nemo_gym.server_utils.HeadServer]
```
classmethod
```python
nemo_gym.server_utils.HeadServer.set_server_instances(
instances: typing.List
) -> None
```
```python
nemo_gym.server_utils.HeadServer.setup_webserver() -> fastapi.FastAPI
```
```python
class nemo_gym.server_utils.ProfilingMiddlewareConfig()
```
**Bases:** [ProfilingMiddlewareInputConfig](#nemo_gym-server_utils-ProfilingMiddlewareInputConfig)
```python
class nemo_gym.server_utils.ProfilingMiddlewareInputConfig()
```
**Bases:** `BaseModel`
```python
class nemo_gym.server_utils.ServerClient()
```
**Bases:** `BaseModel`
```python
nemo_gym.server_utils.ServerClient._build_server_base_url(
server_config_dict: omegaconf.OmegaConf
) -> str
```
```python
nemo_gym.server_utils.ServerClient.get(
server_name: str,
url_path: str,
kwargs: typing.Unpack[aiohttp.client._RequestOptions] = {}
) -> aiohttp.ClientResponse
```
async
**Parameters:**
str
The name of the server you are trying to call.
str
The URL path in the server you are trying to call e.g. "/v1/responses".
```python
nemo_gym.server_utils.ServerClient.load_from_global_config(
head_server_config: typing.Optional[nemo_gym.config_types.BaseServerConfig] = None
) -> nemo_gym.server_utils.ServerClient
```
classmethod
```python
nemo_gym.server_utils.ServerClient.load_head_server_config() -> nemo_gym.config_types.BaseServerConfig
```
classmethod
```python
nemo_gym.server_utils.ServerClient.poll_for_status(
server_name: str
) -> nemo_gym.server_utils.ServerStatus
```
```python
nemo_gym.server_utils.ServerClient.post(
server_name: str,
url_path: str,
kwargs: typing.Unpack[aiohttp.client._RequestOptions] = {}
) -> aiohttp.ClientResponse
```
async
**Parameters:**
str
The name of the server you are trying to call.
str
The URL path in the server you are trying to call e.g. "/v1/responses".
```python
nemo_gym.server_utils.ServerClient.request(
server_name: str,
url_path: str,
method: str,
kwargs: typing.Unpack[aiohttp.client._RequestOptions] = {}
) -> aiohttp.ClientResponse
```
async
```python
class nemo_gym.server_utils.ServerInstanceDisplayConfig()
```
**Bases:** `BaseModel`
```python
class nemo_gym.server_utils.SimpleServer()
```
**Bases:** [BaseServer](#nemo_gym-server_utils-BaseServer)
```python
nemo_gym.server_utils.SimpleServer.get_session_middleware_key() -> str
```
```python
nemo_gym.server_utils.SimpleServer.prefix_server_logs() -> None
```
```python
nemo_gym.server_utils.SimpleServer.run_webserver() -> typing.Optional[fastapi.FastAPI]
```
classmethod
```python
nemo_gym.server_utils.SimpleServer.set_ulimit(
target_soft_limit: int = 65535
)
```
```python
nemo_gym.server_utils.SimpleServer.setup_exception_middleware(
app: fastapi.FastAPI
) -> None
```
```python
nemo_gym.server_utils.SimpleServer.setup_profiling(
app: fastapi.FastAPI,
profiling_config: nemo_gym.server_utils.ProfilingMiddlewareConfig
) -> None
```
```python
nemo_gym.server_utils.SimpleServer.setup_session_middleware(
app: fastapi.FastAPI
) -> None
```
```python
nemo_gym.server_utils.SimpleServer.setup_webserver() -> fastapi.FastAPI
```
abstract
```python
class nemo_gym.server_utils.UvicornLoggingConfig()
```
**Bases:** `BaseModel`
```python
nemo_gym.server_utils.get_global_aiohttp_client(
global_config_dict_parser_config: typing.Optional[nemo_gym.global_config.GlobalConfigDictParserConfig] = None,
global_config_dict_parser_cls: typing.Type[nemo_gym.global_config.GlobalConfigDictParser] = GlobalConfigDictParser
) -> aiohttp.ClientSession
```
```python
nemo_gym.server_utils.get_nemo_gym_fastapi_num_workers() -> int
```
```python
nemo_gym.server_utils.get_response_json(
response: aiohttp.ClientResponse
) -> typing.Any
```
async
```python
nemo_gym.server_utils.get_server_url(
server_name: str
) -> str
```
```python
nemo_gym.server_utils.global_aiohttp_client_exit()
```
```python
nemo_gym.server_utils.initialize_ray() -> None
```
Initialize ray cluster in a process.
We store the Ray address in the global config dict so that child processes can connect to it.
This avoids the need to start a new Ray cluster in each child process.
Note: This function will modify the global config dict - update `ray_head_node_address`
```python
nemo_gym.server_utils.is_global_aiohttp_client_request_debug_enabled() -> bool
```
```python
nemo_gym.server_utils.is_global_aiohttp_client_setup() -> bool
```
```python
nemo_gym.server_utils.is_nemo_gym_fastapi_entrypoint(
file: str
) -> bool
```
```python
nemo_gym.server_utils.is_nemo_gym_fastapi_worker() -> bool
```
```python
nemo_gym.server_utils.maybe_ray_cluster_exit()
```
```python
nemo_gym.server_utils.raise_for_status(
response: aiohttp.ClientResponse
) -> None
```
async
```python
nemo_gym.server_utils.request(
method: str,
url: str,
_internal: bool = False,
kwargs: typing.Unpack[aiohttp.client._RequestOptions] = {}
) -> aiohttp.ClientResponse
```
async
```python
nemo_gym.server_utils.set_global_aiohttp_client(
cfg: nemo_gym.server_utils.GlobalAIOHTTPAsyncClientConfig
) -> aiohttp.ClientSession
```
```python
nemo_gym.server_utils.set_is_nemo_gym_fastapi_entrypoint(
file: str
) -> None
```
```python
nemo_gym.server_utils.set_is_nemo_gym_fastapi_worker() -> None
```
```python
nemo_gym.server_utils.set_nemo_gym_fastapi_num_workers(
num_workers: int
) -> None
```
```python
nemo_gym.server_utils.DEFAULT_HEAD_SERVER_PORT = 11000
```
```python
nemo_gym.server_utils.DISCONNECTED_CLIENT_OS_HELP_TEXT = "We've run into this issue in two different scenarios previously:\n1. Too many o...
```
```python
nemo_gym.server_utils.DISCONNECTED_CLIENT_OS_PRINT_INTERVAL: int = 100
```
```python
nemo_gym.server_utils.IS_NEMO_GYM_FASTAPI_ENTRYPOINT_KEY_NAME = 'IS_NEMO_GYM_FASTAPI_ENTRYPOINT'
```
```python
nemo_gym.server_utils.IS_NEMO_GYM_FASTAPI_WORKER_KEY_NAME = 'IS_NEMO_GYM_FASTAPI_WORKER'
```
```python
nemo_gym.server_utils.MAX_NUM_TRIES = 3
```
```python
nemo_gym.server_utils.NEMO_GYM_FASTAPI_NUM_WORKERS = 'NEMO_GYM_FASTAPI_NUM_WORKERS'
```
```python
nemo_gym.server_utils.SESSION_ID_KEY = 'session_id'
```
```python
nemo_gym.server_utils.ServerStatus = Union[Literal['success'], Literal['connection_error'], Literal['timeout'], Liter...
```
```python
nemo_gym.server_utils._GLOBAL_AIOHTTP_CLIENT: Union[None, ClientSession] = None
```
```python
nemo_gym.server_utils._GLOBAL_AIOHTTP_CLIENT_REQUEST_DEBUG: bool = False
```
```python
nemo_gym.server_utils._NEMO_GYM_STARTED_RAY_CLUSTER: bool = False
```
```python
nemo_gym.server_utils._NUM_CLIENT_OS_ERROR: int = 0
```
```python
nemo_gym.server_utils._NUM_SERVER_DISCONNECTED_ERROR: int = 0
```