nemo_gym.telemetry.startup

Where server startup time goes, per server, exported through nemo-lens.

Startup is split between two kinds of process, and neither can see all of it. The supervisor is the process running gym env start or gym eval run. It knows when it spawned each server and when that server first answered a health probe, but not what happened in between. Each server knows what it did after its interpreter started. It cannot see the venv setup that ran before the server process existed.

The two halves meet through environment variables set when the supervisor spawns a server. The supervisor sets the spawn time and a W3C traceparent that names its span for that server. The launch shell adds the time venv setup finished. The server turns those into its first two stages, venv_setup and interpreter_start, and times the rest itself. When the server starts accepting connections, it exports its stages as spans under the supervisor’s span, so one startup is one trace. The Uvicorn workers of a multi-worker server are separate processes that import the server again, and each reports its own stages the same way.

Each call to StageTimeline.mark closes the stage that began at the previous call. Stages are therefore contiguous and add up to the process’s total, and call sites need no extra indentation. Spans are created after the fact with explicit timestamps, because telemetry is only initialised partway through startup.

Nothing here may fail a server or the supervisor. Every telemetry step is guarded, and with telemetry off every method returns without exporting anything.

Module Contents

Classes

Name	Description
`ServerStartupTimeline`	A server process’s own startup stages.
`Stage`	One contiguous slice of startup, in wall-clock nanoseconds.
`StageTimeline`	Contiguous stages: each `mark` closes the stage that began at the previous mark.
`SupervisorStartup`	The supervisor’s view of one startup.
`_ServerRecord`	-

Functions

Name	Description
`_env_ns`	-
`_exporting_telemetry`	-
`_set_error`	-
`_stage_spans`	-
`_start_span`	-

Data

INTERPRETER_START_STAGE

OUTCOME_ATTRIBUTE

PROCESS_PID_ATTRIBUTE

STARTUP_SERVE_NS_ENV

STARTUP_SETUP_DONE_NS_ENV

STARTUP_SPAWN_NS_ENV

STARTUP_TRACEPARENT_ENV

UVICORN_STARTUP_STAGE

VENV_SETUP_STAGE

WORKER_START_STAGE

logger

API

class nemo_gym.telemetry.startup.ServerStartupTimeline(
    server_name: str,
    server_type: typing.Optional[str],
    worker: bool = False,
    enabled: bool = True
)

Bases: StageTimeline

A server process’s own startup stages.

A multi-worker server runs run_webserver in its main process and again in every Uvicorn worker. The main process times the stages up to the hand-off to Uvicorn. Each worker times its own spawn, import, and setup. Each process reports its own spans and metrics, with worker telling them apart. A process that serves requests reports when the app’s lifespan startup finishes. Its last stage, uvicorn_startup, therefore ends when it starts accepting connections.

nemo_gym.telemetry.startup.ServerStartupTimeline._emit_telemetry() -> None

nemo_gym.telemetry.startup.ServerStartupTimeline.announce_serve() -> None

Stamp the hand-off to Uvicorn into the environment for the workers it is about to spawn.

nemo_gym.telemetry.startup.ServerStartupTimeline.mark(
    name: str
) -> None

nemo_gym.telemetry.startup.ServerStartupTimeline.report() -> None

Emit spans and metrics once.

Later calls do nothing.

nemo_gym.telemetry.startup.ServerStartupTimeline.report_when_serving(
    app: typing.Any
) -> None

Report when app’s startup finishes, adding the uvicorn_startup stage.

This wraps the app’s lifespan, so a lifespan the server already set keeps running inside it. The stage ends after that lifespan’s startup, so work the app does there is part of the stage.

class nemo_gym.telemetry.startup.Stage(
    name: str,
    start_ns: int,
    end_ns: int
)

Dataclass

One contiguous slice of startup, in wall-clock nanoseconds.

duration_ms

float

end_ns

int

name

str

start_ns

int

class nemo_gym.telemetry.startup.StageTimeline(
    start_ns: typing.Optional[int] = None
)

Contiguous stages: each mark closes the stage that began at the previous mark.

_cursor_ns

= self.start_ns

end_ns

int

stages

List[Stage] = []

start_ns

= time_ns() if start_ns is None else start_ns

nemo_gym.telemetry.startup.StageTimeline.add(
    name: str,
    start_ns: int,
    end_ns: int
) -> None

Record a stage measured elsewhere and move the cursor to its end.

nemo_gym.telemetry.startup.StageTimeline.mark(
    name: str
) -> None

End the current stage here and name it.

A repeated name adds a second stage.

class nemo_gym.telemetry.startup.SupervisorStartup()

Bases: StageTimeline

The supervisor’s view of one startup.

It records the supervisor’s own stages and, for each server, when it was spawned and when it first answered a health probe.

_servers

Dict[str, _ServerRecord] = {}

nemo_gym.telemetry.startup.SupervisorStartup._close(
    outcome: str,
    error: typing.Optional[BaseException],
    end_ns: typing.Optional[int] = None
) -> None

nemo_gym.telemetry.startup.SupervisorStartup.abort(
    outcome: str,
    error: BaseException
) -> None

Close every open startup span after a startup that did not finish, so the failed run is still exported.

OpenTelemetry exports only spans that have ended. Without this, a startup that times out or fails would export no gym.startup span, and the spans servers already reported would point at a parent that was never exported. Each server that was not ready yet gets outcome too, and every closed span gets an error status. The supervisor stage that was running when startup ended is recorded as unfinished, so the stages still cover the whole startup.

nemo_gym.telemetry.startup.SupervisorStartup.begin_trace() -> None

Open the gym.startup span.

Call this once telemetry has been initialised.

nemo_gym.telemetry.startup.SupervisorStartup.finish() -> None

Close the gym.startup span with the supervisor’s own stages beneath it, after every server is ready.

nemo_gym.telemetry.startup.SupervisorStartup.server_env(
    server_name: str,
    server_type: str
) -> typing.Dict[str, str]

Return the environment for one server about to be spawned.

This also records the server’s spawn time.

nemo_gym.telemetry.startup.SupervisorStartup.server_ready(
    server_name: str
) -> None

Record that server_name first answered a health probe.

Later calls for the same server are ignored.

class nemo_gym.telemetry.startup._ServerRecord(
    server_type: str,
    spawn_ns: int,
    span: typing.Any = None,
    ready_ns: typing.Optional[int] = None
)

Dataclass

ready_ns

Optional[int] = None

server_type

str

spawn_ns

int

nemo_gym.telemetry.startup._env_ns(
    name: str
) -> typing.Optional[int]

nemo_gym.telemetry.startup._exporting_telemetry() -> typing.Optional[typing.Any]

nemo_gym.telemetry.startup._set_error(
    span: typing.Any,
    error: BaseException
) -> None

nemo_gym.telemetry.startup._stage_spans(
    telemetry: typing.Any,
    timeline: nemo_gym.telemetry.startup.StageTimeline,
    parent_context: typing.Any,
    prefix: str,
    attributes: typing.Dict[str, typing.Any]
) -> None

nemo_gym.telemetry.startup._start_span(
    telemetry: typing.Any,
    name: str,
    start_ns: int,
    parent_context: typing.Any,
    attributes: typing.Dict[str, typing.Any]
) -> typing.Any

nemo_gym.telemetry.startup.INTERPRETER_START_STAGE = 'interpreter_start'

nemo_gym.telemetry.startup.OUTCOME_ATTRIBUTE = 'nemo.gym.startup.outcome'

nemo_gym.telemetry.startup.PROCESS_PID_ATTRIBUTE = 'process.pid'

nemo_gym.telemetry.startup.STARTUP_SERVE_NS_ENV = 'NEMO_GYM_STARTUP_SERVE_NS'

nemo_gym.telemetry.startup.STARTUP_SETUP_DONE_NS_ENV = 'NEMO_GYM_STARTUP_SETUP_DONE_NS'

nemo_gym.telemetry.startup.STARTUP_SPAWN_NS_ENV = 'NEMO_GYM_STARTUP_SPAWN_NS'

nemo_gym.telemetry.startup.STARTUP_TRACEPARENT_ENV = 'NEMO_GYM_STARTUP_TRACEPARENT'

nemo_gym.telemetry.startup.UVICORN_STARTUP_STAGE = 'uvicorn_startup'

nemo_gym.telemetry.startup.VENV_SETUP_STAGE = 'venv_setup'

nemo_gym.telemetry.startup.WORKER_START_STAGE = 'worker_start'

nemo_gym.telemetry.startup.logger = logging.getLogger(__name__)