> This page is for version 0.6.0.
> For other versions, use one of these documentation indexes:
> - Main (default): https://docs.nvidia.com/nemo/gym/main/llms.txt
> - 0.7.0: https://docs.nvidia.com/nemo/gym/v0.7.0/llms.txt
> - 0.6.0: https://docs.nvidia.com/nemo/gym/v0.6.0/llms.txt
> - 0.5.1: https://docs.nvidia.com/nemo/gym/v0.5.1/llms.txt
> - 0.5.0: https://docs.nvidia.com/nemo/gym/v0.5.0/llms.txt
> - 0.4.0: https://docs.nvidia.com/nemo/gym/v0.4.0/llms.txt
> - 0.3.0: https://docs.nvidia.com/nemo/gym/v0.3.0/llms.txt
> - 0.2.1: https://docs.nvidia.com/nemo/gym/v0.2.1/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/gym/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/gym/_mcp/server.

# nemo_gym.telemetry.startup

Where server startup time goes, per server, exported through nemo-lens.

Startup is split between two kinds of process, and neither can see all of it.
The supervisor is the process running `gym env start` or `gym eval run`.
It knows when it spawned each server and when that server first answered a health probe, but not what happened in between.
Each server knows what it did after its interpreter started.
It cannot see the venv setup that ran before the server process existed.

The two halves meet through environment variables set when the supervisor spawns a server.
The supervisor sets the spawn time and a W3C `traceparent` that names its span for that server.
The launch shell adds the time venv setup finished.
The server turns those into its first two stages, `venv_setup` and `interpreter_start`, and times the rest itself.
When the server starts accepting connections, it exports its stages as spans under the supervisor's span, so one startup is one trace.
The Uvicorn workers of a multi-worker server are separate processes that import the server again, and each reports its own stages the same way.

Each call to `StageTimeline.mark` closes the stage that began at the previous call.
Stages are therefore contiguous and add up to the process's total, and call sites need no extra indentation.
Spans are created after the fact with explicit timestamps, because telemetry is only initialised partway through startup.

Nothing here may fail a server or the supervisor.
Every telemetry step is guarded, and with telemetry off every method returns without exporting anything.

## Module Contents

### Classes

| Name                                                                         | Description                                                                      |
| ---------------------------------------------------------------------------- | -------------------------------------------------------------------------------- |
| [`ServerStartupTimeline`](#nemo_gym-telemetry-startup-ServerStartupTimeline) | A server process's own startup stages.                                           |
| [`Stage`](#nemo_gym-telemetry-startup-Stage)                                 | One contiguous slice of startup, in wall-clock nanoseconds.                      |
| [`StageTimeline`](#nemo_gym-telemetry-startup-StageTimeline)                 | Contiguous stages: each `mark` closes the stage that began at the previous mark. |
| [`SupervisorStartup`](#nemo_gym-telemetry-startup-SupervisorStartup)         | The supervisor's view of one startup.                                            |
| [`_ServerRecord`](#nemo_gym-telemetry-startup-_ServerRecord)                 | -                                                                                |

### Functions

| Name                                                                       | Description |
| -------------------------------------------------------------------------- | ----------- |
| [`_env_ns`](#nemo_gym-telemetry-startup-_env_ns)                           | -           |
| [`_exporting_telemetry`](#nemo_gym-telemetry-startup-_exporting_telemetry) | -           |
| [`_set_error`](#nemo_gym-telemetry-startup-_set_error)                     | -           |
| [`_stage_spans`](#nemo_gym-telemetry-startup-_stage_spans)                 | -           |
| [`_start_span`](#nemo_gym-telemetry-startup-_start_span)                   | -           |

### Data

[`INTERPRETER_START_STAGE`](#nemo_gym-telemetry-startup-INTERPRETER_START_STAGE)

[`OUTCOME_ATTRIBUTE`](#nemo_gym-telemetry-startup-OUTCOME_ATTRIBUTE)

[`PROCESS_PID_ATTRIBUTE`](#nemo_gym-telemetry-startup-PROCESS_PID_ATTRIBUTE)

[`STARTUP_SERVE_NS_ENV`](#nemo_gym-telemetry-startup-STARTUP_SERVE_NS_ENV)

[`STARTUP_SETUP_DONE_NS_ENV`](#nemo_gym-telemetry-startup-STARTUP_SETUP_DONE_NS_ENV)

[`STARTUP_SPAWN_NS_ENV`](#nemo_gym-telemetry-startup-STARTUP_SPAWN_NS_ENV)

[`STARTUP_TRACEPARENT_ENV`](#nemo_gym-telemetry-startup-STARTUP_TRACEPARENT_ENV)

[`UVICORN_STARTUP_STAGE`](#nemo_gym-telemetry-startup-UVICORN_STARTUP_STAGE)

[`VENV_SETUP_STAGE`](#nemo_gym-telemetry-startup-VENV_SETUP_STAGE)

[`WORKER_START_STAGE`](#nemo_gym-telemetry-startup-WORKER_START_STAGE)

[`logger`](#nemo_gym-telemetry-startup-logger)

### API

```python
class nemo_gym.telemetry.startup.ServerStartupTimeline(
    server_name: str,
    server_type: typing.Optional[str],
    worker: bool = False,
    enabled: bool = True
)
```

**Bases:** [StageTimeline](#nemo_gym-telemetry-startup-StageTimeline)

A server process's own startup stages.

A multi-worker server runs `run_webserver` in its main process and again in every Uvicorn worker.
The main process times the stages up to the hand-off to Uvicorn.
Each worker times its own spawn, import, and setup.
Each process reports its own spans and metrics, with `worker` telling them apart.
A process that serves requests reports when the app's lifespan startup finishes.
Its last stage, `uvicorn_startup`, therefore ends when it starts accepting connections.

```python
nemo_gym.telemetry.startup.ServerStartupTimeline._emit_telemetry() -> None
```

```python
nemo_gym.telemetry.startup.ServerStartupTimeline.announce_serve() -> None
```

Stamp the hand-off to Uvicorn into the environment for the workers it is about to spawn.

```python
nemo_gym.telemetry.startup.ServerStartupTimeline.mark(
    name: str
) -> None
```

```python
nemo_gym.telemetry.startup.ServerStartupTimeline.report() -> None
```

Emit spans and metrics once.

Later calls do nothing.

```python
nemo_gym.telemetry.startup.ServerStartupTimeline.report_when_serving(
    app: typing.Any
) -> None
```

Report when `app`'s startup finishes, adding the `uvicorn_startup` stage.

This wraps the app's lifespan, so a lifespan the server already set keeps running inside it.
The stage ends after that lifespan's startup, so work the app does there is part of the stage.

```python
class nemo_gym.telemetry.startup.Stage(
    name: str,
    start_ns: int,
    end_ns: int
)
```

Dataclass

One contiguous slice of startup, in wall-clock nanoseconds.

**`duration_ms`** `float`

---

**`end_ns`** `int`

---

**`name`** `str`

---

**`start_ns`** `int`

---

```python
class nemo_gym.telemetry.startup.StageTimeline(
    start_ns: typing.Optional[int] = None
)
```

Contiguous stages: each `mark` closes the stage that began at the previous mark.

**`_cursor_ns`** `= self.start_ns`

---

**`end_ns`** `int`

---

**`stages`** `List[Stage] = []`

---

**`start_ns`** `= time_ns() if start_ns is None else start_ns`

---

```python
nemo_gym.telemetry.startup.StageTimeline.add(
    name: str,
    start_ns: int,
    end_ns: int
) -> None
```

Record a stage measured elsewhere and move the cursor to its end.

```python
nemo_gym.telemetry.startup.StageTimeline.mark(
    name: str
) -> None
```

End the current stage here and name it.

A repeated name adds a second stage.

```python
class nemo_gym.telemetry.startup.SupervisorStartup()
```

**Bases:** [StageTimeline](#nemo_gym-telemetry-startup-StageTimeline)

The supervisor's view of one startup.

It records the supervisor's own stages and, for each server, when it was spawned and when it first answered a health probe.

**`_servers`** `Dict[str, _ServerRecord] = {}`

---

```python
nemo_gym.telemetry.startup.SupervisorStartup._close(
    outcome: str,
    error: typing.Optional[BaseException],
    end_ns: typing.Optional[int] = None
) -> None
```

```python
nemo_gym.telemetry.startup.SupervisorStartup.abort(
    outcome: str,
    error: BaseException
) -> None
```

Close every open startup span after a startup that did not finish, so the failed run is still exported.

OpenTelemetry exports only spans that have ended.
Without this, a startup that times out or fails would export no `gym.startup` span, and the spans servers already reported would point at a parent that was never exported.
Each server that was not ready yet gets `outcome` too, and every closed span gets an error status.
The supervisor stage that was running when startup ended is recorded as `unfinished`, so the stages still cover the whole startup.

```python
nemo_gym.telemetry.startup.SupervisorStartup.begin_trace() -> None
```

Open the `gym.startup` span.

Call this once telemetry has been initialised.

```python
nemo_gym.telemetry.startup.SupervisorStartup.finish() -> None
```

Close the `gym.startup` span with the supervisor's own stages beneath it, after every server is ready.

```python
nemo_gym.telemetry.startup.SupervisorStartup.server_env(
    server_name: str,
    server_type: str
) -> typing.Dict[str, str]
```

Return the environment for one server about to be spawned.

This also records the server's spawn time.

```python
nemo_gym.telemetry.startup.SupervisorStartup.server_ready(
    server_name: str
) -> None
```

Record that `server_name` first answered a health probe.

Later calls for the same server are ignored.

```python
class nemo_gym.telemetry.startup._ServerRecord(
    server_type: str,
    spawn_ns: int,
    span: typing.Any = None,
    ready_ns: typing.Optional[int] = None
)
```

Dataclass

**`ready_ns`** `Optional[int] = None`

---

**`server_type`** `str`

---

**`spawn_ns`** `int`

---

```python
nemo_gym.telemetry.startup._env_ns(
    name: str
) -> typing.Optional[int]
```

```python
nemo_gym.telemetry.startup._exporting_telemetry() -> typing.Optional[typing.Any]
```

```python
nemo_gym.telemetry.startup._set_error(
    span: typing.Any,
    error: BaseException
) -> None
```

```python
nemo_gym.telemetry.startup._stage_spans(
    telemetry: typing.Any,
    timeline: nemo_gym.telemetry.startup.StageTimeline,
    parent_context: typing.Any,
    prefix: str,
    attributes: typing.Dict[str, typing.Any]
) -> None
```

```python
nemo_gym.telemetry.startup._start_span(
    telemetry: typing.Any,
    name: str,
    start_ns: int,
    parent_context: typing.Any,
    attributes: typing.Dict[str, typing.Any]
) -> typing.Any
```

```python
nemo_gym.telemetry.startup.INTERPRETER_START_STAGE = 'interpreter_start'
```

```python
nemo_gym.telemetry.startup.OUTCOME_ATTRIBUTE = 'nemo.gym.startup.outcome'
```

```python
nemo_gym.telemetry.startup.PROCESS_PID_ATTRIBUTE = 'process.pid'
```

```python
nemo_gym.telemetry.startup.STARTUP_SERVE_NS_ENV = 'NEMO_GYM_STARTUP_SERVE_NS'
```

```python
nemo_gym.telemetry.startup.STARTUP_SETUP_DONE_NS_ENV = 'NEMO_GYM_STARTUP_SETUP_DONE_NS'
```

```python
nemo_gym.telemetry.startup.STARTUP_SPAWN_NS_ENV = 'NEMO_GYM_STARTUP_SPAWN_NS'
```

```python
nemo_gym.telemetry.startup.STARTUP_TRACEPARENT_ENV = 'NEMO_GYM_STARTUP_TRACEPARENT'
```

```python
nemo_gym.telemetry.startup.UVICORN_STARTUP_STAGE = 'uvicorn_startup'
```

```python
nemo_gym.telemetry.startup.VENV_SETUP_STAGE = 'venv_setup'
```

```python
nemo_gym.telemetry.startup.WORKER_START_STAGE = 'worker_start'
```

```python
nemo_gym.telemetry.startup.logger = logging.getLogger(__name__)
```