> This page is for version v0.1.3. > For other versions, use one of these documentation indexes: > - Latest (v0.1.3) (default): https://docs.nvidia.com/openshell/latest/llms.txt > - Dev: https://docs.nvidia.com/openshell/dev/llms.txt > - v0.1.3: https://docs.nvidia.com/openshell/v0.1.3/llms.txt > - v0.1.2: https://docs.nvidia.com/openshell/v0.1.2/llms.txt > - v0.1.1: https://docs.nvidia.com/openshell/v0.1.1/llms.txt > - v0.1.0: https://docs.nvidia.com/openshell/v0.1.0/llms.txt > - v0.0.116: https://docs.nvidia.com/openshell/v0.0.116/llms.txt > For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/openshell/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/openshell/_mcp/server. # Gateway Metrics > Scrape Prometheus metrics from the OpenShell gateway, including supervisor session, relay capacity, and peer routing signals for multi-replica deployments. The gateway exports Prometheus metrics for API requests, database readiness, gateway interceptors, the OCSF log file, and multi-replica capacity. Use them to alert on saturation, to follow rollouts, and to choose autoscaling signals. ## Enable the Metrics Endpoint The gateway serves `GET /metrics` in the Prometheus text format on its own listener, separate from the API and health ports. When the listener is disabled, the gateway records no metrics. Enable it in one of these ways: * In `gateway.toml`, set `metrics_bind_address` under `[openshell.gateway]`, for example `metrics_bind_address = "0.0.0.0:9090"`. Refer to [Gateway Configuration](/how-it-works/gateways/configuration). * On the command line, pass `--metrics-port` or set `OPENSHELL_METRICS_PORT`. The standalone gateway binary defaults to `0`, which disables the endpoint. * In the Helm chart, set `service.metricsPort`, which defaults to `9090`. The chart renders the TOML key, a container port named `metrics`, and a `metrics` port on the gateway Service. Set it to `0` to disable the endpoint. A stopping gateway keeps serving metrics until the process exits. > **Warning** > > The metrics endpoint uses plaintext HTTP with no authentication. Anyone who > can reach the port can read the metrics. The Helm chart publishes the port on > the gateway Service, so a `LoadBalancer` or `NodePort` Service exposes it > outside the cluster. Keep the Service internal, and restrict the port to your > monitoring system. The following NetworkPolicy keeps the API and health ports open and accepts metrics scrapes only from the `monitoring` namespace. Adjust the ports if you changed `service.port`, `service.healthPort`, or `service.metricsPort`. ```yaml apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: openshell-gateway-metrics namespace: openshell spec: podSelector: matchLabels: app.kubernetes.io/name: openshell app.kubernetes.io/instance: openshell policyTypes: - Ingress ingress: - ports: - port: 8080 - port: 8081 - from: - namespaceSelector: matchLabels: kubernetes.io/metadata.name: monitoring ports: - port: 9090 ``` ## Scrape Each Replica Capacity metrics describe one gateway replica, so scrape every gateway pod rather than the Service, and keep the `namespace` and `pod` labels on each series. With the Prometheus Operator, a PodMonitor adds those labels automatically: ```yaml apiVersion: monitoring.coreos.com/v1 kind: PodMonitor metadata: name: openshell-gateway namespace: openshell spec: selector: matchLabels: app.kubernetes.io/name: openshell app.kubernetes.io/instance: openshell podMetricsEndpoints: - port: metrics path: /metrics interval: 30s ``` With a plain Prometheus configuration, use pod discovery and copy the labels yourself: ```yaml scrape_configs: - job_name: openshell-gateway kubernetes_sd_configs: - role: pod namespaces: names: [openshell] relabel_configs: - source_labels: [__meta_kubernetes_pod_label_app_kubernetes_io_instance] regex: openshell action: keep - source_labels: [__meta_kubernetes_pod_container_port_name] regex: metrics action: keep - source_labels: [__meta_kubernetes_namespace] target_label: namespace - source_labels: [__meta_kubernetes_pod_name] target_label: pod ``` To check one pod without Prometheus, read its metrics through the Kubernetes API server proxy: ```shell kubectl get --raw "/api/v1/namespaces/openshell/pods/:9090/proxy/metrics" \ | grep '^openshell_server_supervisor_sessions' ``` The API server connects to the pod from the control plane. A NetworkPolicy like the one above blocks that connection unless it also allows the control plane on the metrics port. `kubectl port-forward` reaches the pod through the kubelet instead, and NetworkPolicy does not block it. Start the port-forward: ```shell kubectl -n openshell port-forward pod/ 9090:9090 ``` Then read the metrics from another terminal: ```shell curl -s http://localhost:9090/metrics | grep '^openshell_server_supervisor_sessions' ``` ## Histograms and Summaries The multi-replica latency metrics are Prometheus histograms with `_bucket`, `_sum`, and `_count` series and buckets from 1 millisecond to 15 seconds, so `histogram_quantile` can aggregate them across replicas. The older `*_duration_seconds` metrics and the interceptor latency metric are summaries with `quantile` labels that each replica computes on its own. Do not average summary quantiles across pods. ## Multi-Replica Capacity Metrics These metrics describe how much work one replica holds and how the replicas interact. No metric carries a sandbox, channel, endpoint, or replica identifier, because the scrape target already identifies the replica. Supervisor sessions: | Metric | Type | Labels | Description | | -------------------------------------- | ----- | ------ | ---------------------------------------------------------------------------------------------------------------------------------------- | | `openshell_server_supervisor_sessions` | Gauge | None | Supervisor control sessions held by this replica. After a rollout settles, the sum across pods equals the number of connected sandboxes. | Relays: | Metric | Type | Labels | Description | | ----------------------------------------------- | --------- | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `openshell_server_relay_pending` | Gauge | None | Relay channels on this replica waiting for the supervisor to connect back, including channels opened for peer replicas. A channel whose caller gave up stays counted until it is claimed or cleaned up, up to about 40 seconds. | | `openshell_server_relay_pending_capacity` | Gauge | None | Pending relay limit per replica, `256`. | | `openshell_server_relay_rejected_total` | Counter | `reason` | Relay opens rejected at a limit. Rejected relays fail the client request, so use this counter, not client error codes, to detect saturation. | | `openshell_server_relay_expired_total` | Counter | None | Pending relays dropped because the supervisor did not connect back within 10 seconds. | | `openshell_server_relay_claim_duration_seconds` | Histogram | None | Time from opening a relay to the supervisor claiming it. | Routed requests include local relay setup and outbound requests to the replica that owns a sandbox's supervisor session: | Metric | Type | Labels | Description | | ------------------------------------------------ | --------- | ---------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------- | | `openshell_server_routed_request_attempts_total` | Counter | `operation`, `route`, `relay_kind`, `outcome`, `grpc_code` | Local relay setup and outbound peer attempts, counted once when they finish or are cancelled. Each retry counts separately. | | `openshell_server_peer_request_duration_seconds` | Histogram | `operation`, `outcome` | Latency of outbound peer requests only (`route="peer"`). For relays, until the owner's supervisor claimed the relay. | The labels take these values: * `operation` is `relay`, `report_provider_readiness`, `report_endpoint_status`, or `get_sandbox_provider_status`. Only relay setup includes the local route; the other operations count outbound peer calls, not local API handling. On the owning replica, `openshell_server_grpc_requests_total` records the matching `method`, such as `PeerRelay` for `relay`. * `relay_kind` is `ssh` or `tcp` for relays, and `none` for other operations. An omitted relay target counts as `ssh`, matching the protocol's compatibility behavior. TCP hostnames and ports are not labels. * `route` is `local` when this replica tries its own supervisor session, or `peer` when it tries another replica. A routed retry counts again, including a change from local to peer routing. Waiting for an owner does not count. Incoming `PeerRelay` requests do not increment this counter on the receiving replica; they appear in that replica's gRPC request counter instead. This counter does not measure active connections, unique client requests, or unfinished attempts. * `reason` is `replica_capacity` when the gateway replica's pending-relay budget is full, or `sandbox_capacity` when one sandbox's budget on that replica, 32 pending relays, is full. * `outcome` is `success`, `local_error`, or `remote_error`. A relay succeeds when the supervisor claims it; this does not measure the data stream. `local_error` means the attempt failed on this replica, and `remote_error` means the owning replica returned an error or the open connection to it failed. On `route="local"` this replica is the owner, so failures that appear as `remote_error` on the peer route, such as an unclaimed relay (`deadline_exceeded`) or a full relay budget (`resource_exhausted`), appear as `local_error`. A cancelled attempt records `local_error` with `grpc_code="cancelled"`, even when the owner was already handling it. * `grpc_code` is the lowercase gRPC status name, such as `ok`, `unavailable`, or `resource_exhausted`, from local setup, the local supervisor claim, or the peer RPC. For a peer relay, the gateway then reports a failure to the client as `UNAVAILABLE`. The other RPCs return the owner's status to the client unchanged. The label is not named `code` because `openshell_server_grpc_requests_total` uses that name for the numeric status code. Gauges and counters exist from startup. They start at `0`, except `openshell_server_relay_pending_capacity`, which starts at its limit. `openshell_server_routed_request_attempts_total` starts with seven `outcome="success", grpc_code="ok"` series: four relay kind and route combinations, and three other peer operations with `relay_kind="none"`. Other outcomes and codes, and every histogram, appear after their first sample, so write alert expressions that tolerate absent series. ## Request and Readiness Metrics The gateway also records every API request and its background database check: | Metric | Type | Labels | Description | | ------------------------------------------------------------ | ------- | ---------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `openshell_server_grpc_requests_total` | Counter | `method`, `code` | gRPC requests by method name and numeric status code, including internal peer requests. Streaming RPCs record the status known when the response headers are sent. | | `openshell_server_grpc_request_duration_seconds` | Summary | `method`, `code` | Time until the response headers are sent. For streaming RPCs this covers setup only. | | `openshell_server_http_requests_total` | Counter | `path`, `status` | HTTP requests on the API port. `path` is `/_ws_tunnel`, `/auth`, or `unknown`. | | `openshell_server_http_request_duration_seconds` | Summary | `path`, `status` | Duration of those HTTP requests. | | `openshell_server_readiness_database_healthy` | Gauge | None | `1` when the last background database check succeeded, otherwise `0`. | | `openshell_server_readiness_database_probe_duration_seconds` | Summary | `outcome` | Duration of the background database check. `outcome` is `success`, `db_error`, or `timeout`. | ## Gateway Interceptor Metrics Gateways with [gateway interceptors](/extensibility/gateway-interceptors) record their evaluations: | Metric | Type | Labels | Description | | ---------------------------------------------------------------------- | ------- | --------------------------------------- | ---------------------------------------------------------------------------- | | `openshell_gateway_interceptor_latency_seconds` | Summary | None | Duration of interceptor calls. | | `openshell_gateway_interceptor_evaluations_total` | Counter | `decision`, `interceptor`, `binding_id` | Interceptor evaluations by decision. | | `openshell_gateway_interceptor_patches_total` | Counter | `interceptor`, `binding_id` | Patches that interceptors applied. | | `openshell_gateway_interceptor_fail_open_total` | Counter | None | Interceptor failures that let the request continue. | | `openshell_gateway_interceptor_fail_closed_total` | Counter | None | Interceptor failures that rejected the request. | | `openshell_gateway_interceptor_post_commit_observation_failures_total` | Counter | `stage` | Failed post-commit observations. `stage` is `response_body` or `evaluation`. | ## OCSF Log Metrics Gateways that write an [OCSF JSONL file](/how-it-works/gateways/configuration#ocsf-jsonl-output) record its queue and writer: | Metric | Type | Labels | Description | | --------------------------------------------------- | ------- | -------- | ---------------------------------------------------------------------------- | | `openshell_ocsf_log_queued_total` | Counter | None | Records queued for the file. | | `openshell_ocsf_log_written_total` | Counter | None | Records written to the file. | | `openshell_ocsf_log_dropped_total` | Counter | `reason` | Records discarded, by bounded reason. | | `openshell_ocsf_log_writer_errors_total` | Counter | None | Failed writes, reopens, and retention cleanups. | | `openshell_ocsf_log_recovery_discarded_bytes_total` | Counter | None | Bytes of an incomplete trailing line removed when the writer opens the file. | | `openshell_ocsf_log_queue_records` | Gauge | None | Records waiting to be written. | | `openshell_ocsf_log_queue_bytes` | Gauge | None | Encoded bytes waiting to be written. | ## Next Steps * To monitor capacity and choose autoscaling signals for multiple gateway replicas, refer to [High Availability](/kubernetes/high-availability#monitor-capacity). * To configure the metrics listener in `gateway.toml`, refer to [Gateway Configuration](/how-it-works/gateways/configuration).