Gateway Metrics

The gateway exports Prometheus metrics for API requests, database readiness, gateway interceptors, the OCSF log file, and multi-replica capacity. Use them to alert on saturation, to follow rollouts, and to choose autoscaling signals.

Enable the Metrics Endpoint

The gateway serves GET /metrics in the Prometheus text format on its own listener, separate from the API and health ports. When the listener is disabled, the gateway records no metrics. Enable it in one of these ways:

• In gateway.toml, set metrics_bind_address under [openshell.gateway], for example metrics_bind_address = "0.0.0.0:9090". Refer to Gateway Configuration.
• On the command line, pass --metrics-port or set OPENSHELL_METRICS_PORT. The standalone gateway binary defaults to 0, which disables the endpoint.
• In the Helm chart, set service.metricsPort, which defaults to 9090. The chart renders the TOML key, a container port named metrics, and a metrics port on the gateway Service. Set it to 0 to disable the endpoint.

A stopping gateway keeps serving metrics until the process exits.

The following NetworkPolicy keeps the API and health ports open and accepts metrics scrapes only from the monitoring namespace. Adjust the ports if you changed service.port, service.healthPort, or service.metricsPort.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: openshell-gateway-metrics
  namespace: openshell
spec:
  podSelector:
    matchLabels:
      app.kubernetes.io/name: openshell
      app.kubernetes.io/instance: openshell
  policyTypes:
    - Ingress
  ingress:
    - ports:
        - port: 8080
        - port: 8081
    - from:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: monitoring
      ports:
        - port: 9090

Scrape Each Replica

Capacity metrics describe one gateway replica, so scrape every gateway pod rather than the Service, and keep the namespace and pod labels on each series. With the Prometheus Operator, a PodMonitor adds those labels automatically:

apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  name: openshell-gateway
  namespace: openshell
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: openshell
      app.kubernetes.io/instance: openshell
  podMetricsEndpoints:
    - port: metrics
      path: /metrics
      interval: 30s

With a plain Prometheus configuration, use pod discovery and copy the labels yourself:

scrape_configs:
  - job_name: openshell-gateway
    kubernetes_sd_configs:
      - role: pod
        namespaces:
          names: [openshell]
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_label_app_kubernetes_io_instance]
        regex: openshell
        action: keep
      - source_labels: [__meta_kubernetes_pod_container_port_name]
        regex: metrics
        action: keep
      - source_labels: [__meta_kubernetes_namespace]
        target_label: namespace
      - source_labels: [__meta_kubernetes_pod_name]
        target_label: pod

To check one pod without Prometheus, read its metrics through the Kubernetes API server proxy:

kubectl get --raw "/api/v1/namespaces/openshell/pods/<pod>:9090/proxy/metrics" \
  | grep '^openshell_server_supervisor_sessions'

The API server connects to the pod from the control plane. A NetworkPolicy like the one above blocks that connection unless it also allows the control plane on the metrics port. kubectl port-forward reaches the pod through the kubelet instead, and NetworkPolicy does not block it. Start the port-forward:

kubectl -n openshell port-forward pod/<pod> 9090:9090

Then read the metrics from another terminal:

curl -s http://localhost:9090/metrics | grep '^openshell_server_supervisor_sessions'

Histograms and Summaries

The multi-replica latency metrics are Prometheus histograms with _bucket, _sum, and _count series and buckets from 1 millisecond to 15 seconds, so histogram_quantile can aggregate them across replicas. The older *_duration_seconds metrics and the interceptor latency metric are summaries with quantile labels that each replica computes on its own. Do not average summary quantiles across pods.

Multi-Replica Capacity Metrics

These metrics describe how much work one replica holds and how the replicas interact. No metric carries a sandbox, channel, endpoint, or replica identifier, because the scrape target already identifies the replica.

Supervisor sessions:

Relays:

Routed requests include local relay setup and outbound requests to the replica that owns a sandbox’s supervisor session:

The labels take these values:

• operation is relay, report_provider_readiness, report_endpoint_status, or get_sandbox_provider_status. Only relay setup includes the local route; the other operations count outbound peer calls, not local API handling. On the owning replica, openshell_server_grpc_requests_total records the matching method, such as PeerRelay for relay.
• relay_kind is ssh or tcp for relays, and none for other operations. An omitted relay target counts as ssh, matching the protocol’s compatibility behavior. TCP hostnames and ports are not labels.
• route is local when this replica tries its own supervisor session, or peer when it tries another replica. A routed retry counts again, including a change from local to peer routing. Waiting for an owner does not count. Incoming PeerRelay requests do not increment this counter on the receiving replica; they appear in that replica’s gRPC request counter instead. This counter does not measure active connections, unique client requests, or unfinished attempts.
• reason is replica_capacity when the gateway replica’s pending-relay budget is full, or sandbox_capacity when one sandbox’s budget on that replica, 32 pending relays, is full.
• outcome is success, local_error, or remote_error. A relay succeeds when the supervisor claims it; this does not measure the data stream. local_error means the attempt failed on this replica, and remote_error means the owning replica returned an error or the open connection to it failed. On route="local" this replica is the owner, so failures that appear as remote_error on the peer route, such as an unclaimed relay (deadline_exceeded) or a full relay budget (resource_exhausted), appear as local_error. A cancelled attempt records local_error with grpc_code="cancelled", even when the owner was already handling it.
• grpc_code is the lowercase gRPC status name, such as ok, unavailable, or resource_exhausted, from local setup, the local supervisor claim, or the peer RPC. For a peer relay, the gateway then reports a failure to the client as UNAVAILABLE. The other RPCs return the owner’s status to the client unchanged. The label is not named code because openshell_server_grpc_requests_total uses that name for the numeric status code.

Gauges and counters exist from startup. They start at 0, except openshell_server_relay_pending_capacity, which starts at its limit. openshell_server_routed_request_attempts_total starts with seven outcome="success", grpc_code="ok" series: four relay kind and route combinations, and three other peer operations with relay_kind="none". Other outcomes and codes, and every histogram, appear after their first sample, so write alert expressions that tolerate absent series.

Request and Readiness Metrics

The gateway also records every API request and its background database check:

Gateway Interceptor Metrics

Gateways with gateway interceptors record their evaluations:

OCSF Log Metrics

Gateways that write an OCSF JSONL file record its queue and writer:

Next Steps

• To monitor capacity and choose autoscaling signals for multiple gateway replicas, refer to High Availability.
• To configure the metrics listener in gateway.toml, refer to Gateway Configuration.