nemo_gym.server_runtime.uvicorn_reuse_port

Run a multi-worker Uvicorn server whose workers each listen on their own SO_REUSEPORT socket.

Uvicorn’s multi-worker mode hands every worker the same listening socket, and whichever worker wakes first accepts the next connection. Under load that spreads connections unevenly, and since clients keep connections alive, a few workers end up serving most of the traffic while the others idle. Here each worker binds its own socket to the server’s address with SO_REUSEPORT, so the Linux kernel spreads new connections evenly across the workers’ sockets.

The main process keeps Uvicorn’s usual role (spawning, health-checking and restarting workers). Its own socket reserves the address, also while workers are still starting, and is never listened on, so it takes no connections.

Module Contents

Classes

Name	Description
`_ReusePortUvicornConfig`	A Uvicorn config whose bound socket can share its address with the workers’ sockets.
`_ReusePortWorker`	A Uvicorn server that listens on its own SO_REUSEPORT socket instead of the inherited one.

Functions

Name	Description
`_bind_reuse_port_socket`	-
`reuse_port_supported`	SO_REUSEPORT load-balances connections across sockets on Linux only.
`run_with_reuse_port`	`uvicorn.run` for a multi-worker server, with one SO_REUSEPORT listening socket per worker.

Data

LOG

API

class nemo_gym.server_runtime.uvicorn_reuse_port._ReusePortUvicornConfig()

Bases: Config

A Uvicorn config whose bound socket can share its address with the workers’ sockets.

nemo_gym.server_runtime.uvicorn_reuse_port._ReusePortUvicornConfig.bind_socket() -> socket.socket

class nemo_gym.server_runtime.uvicorn_reuse_port._ReusePortWorker()

Bases: Server

A Uvicorn server that listens on its own SO_REUSEPORT socket instead of the inherited one.

nemo_gym.server_runtime.uvicorn_reuse_port._ReusePortWorker.startup(
    sockets: typing.Optional[typing.List[socket.socket]] = None
) -> None

async

nemo_gym.server_runtime.uvicorn_reuse_port._bind_reuse_port_socket(
    family: socket.AddressFamily,
    address: tuple,
    reuse_address: bool = True
) -> socket.socket

nemo_gym.server_runtime.uvicorn_reuse_port.reuse_port_supported() -> bool

SO_REUSEPORT load-balances connections across sockets on Linux only.

nemo_gym.server_runtime.uvicorn_reuse_port.run_with_reuse_port(
    uvicorn_kwargs: typing.Any = {}
) -> None

uvicorn.run for a multi-worker server, with one SO_REUSEPORT listening socket per worker.

Takes the same keyword arguments as uvicorn.run; app must be an import string and workers above 1.

nemo_gym.server_runtime.uvicorn_reuse_port.LOG = logging.getLogger('uvicorn.error')