Gateway Metrics

The gateway exports Prometheus metrics for API requests, database readiness, gateway interceptors, the OCSF log file, and multi-replica capacity. Use them to alert on saturation, to follow rollouts, and to choose autoscaling signals.

Enable the Metrics Endpoint

The gateway serves GET /metrics in the Prometheus text format on its own listener, separate from the API and health ports. When the listener is disabled, the gateway records no metrics. Enable it in one of these ways:

• In gateway.toml, set metrics_bind_address under [openshell.gateway], for example metrics_bind_address = "0.0.0.0:9090". Refer to Gateway Configuration.
• On the command line, pass --metrics-port or set OPENSHELL_METRICS_PORT. The standalone gateway binary defaults to 0, which disables the endpoint.
• In the Helm chart, set service.metricsPort, which defaults to 9090. The chart renders the TOML key, a container port named metrics, and a metrics port on the gateway Service. Set it to 0 to disable the endpoint.

A stopping gateway keeps serving metrics until the process exits.

The following NetworkPolicy keeps the API and health ports open and accepts metrics scrapes only from the monitoring namespace. Adjust the ports if you changed service.port, service.healthPort, or service.metricsPort.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: openshell-gateway-metrics
  namespace: openshell
spec:
  podSelector:
    matchLabels:
      app.kubernetes.io/name: openshell
      app.kubernetes.io/instance: openshell
  policyTypes:
    - Ingress
  ingress:
    - ports:
        - port: 8080
        - port: 8081
    - from:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: monitoring
      ports:
        - port: 9090

Scrape Each Replica

Capacity metrics describe one gateway replica, so scrape every gateway pod rather than the Service, and keep the namespace and pod labels on each series. With the Prometheus Operator, a PodMonitor adds those labels automatically:

apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  name: openshell-gateway
  namespace: openshell
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: openshell
      app.kubernetes.io/instance: openshell
  podMetricsEndpoints:
    - port: metrics
      path: /metrics
      interval: 30s

With a plain Prometheus configuration, use pod discovery and copy the labels yourself:

scrape_configs:
  - job_name: openshell-gateway
    kubernetes_sd_configs:
      - role: pod
        namespaces:
          names: [openshell]
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_label_app_kubernetes_io_instance]
        regex: openshell
        action: keep
      - source_labels: [__meta_kubernetes_pod_container_port_name]
        regex: metrics
        action: keep
      - source_labels: [__meta_kubernetes_namespace]
        target_label: namespace
      - source_labels: [__meta_kubernetes_pod_name]
        target_label: pod

To check one pod without Prometheus, read its metrics through the Kubernetes API server proxy:

kubectl get --raw "/api/v1/namespaces/openshell/pods/<pod>:9090/proxy/metrics" \
  | grep '^openshell_server_supervisor_sessions'

The API server connects to the pod from the control plane. A NetworkPolicy like the one above blocks that connection unless it also allows the control plane on the metrics port. kubectl port-forward reaches the pod through the kubelet instead, and NetworkPolicy does not block it. Start the port-forward:

kubectl -n openshell port-forward pod/<pod> 9090:9090

Then read the metrics from another terminal:

curl -s http://localhost:9090/metrics | grep '^openshell_server_supervisor_sessions'

Metric Names

Every metric name starts with openshell_, followed by the subsystem that records it:

Names follow Prometheus conventions. Counters end in _total, and names with a unit end in that unit, such as _seconds or _bytes. Metrics carry no gateway name label, so use your scrape configuration to tell installations apart.

Histograms and Summaries

The multi-replica latency metrics and the configuration build and apply durations are Prometheus histograms with _bucket, _sum, and _count series and buckets from 1 millisecond to 15 seconds, so histogram_quantile can aggregate them across replicas. The older *_duration_seconds metrics and the interceptor latency metric are summaries with quantile labels that each replica computes on its own. Do not average summary quantiles across pods.

Multi-Replica Capacity Metrics

These metrics describe how much work one replica holds and how the replicas interact. No metric carries a sandbox, channel, endpoint, or replica identifier, because the scrape target already identifies the replica.

Supervisor sessions:

Relays:

Routed requests include local relay setup and outbound requests to the replica that owns a sandbox’s supervisor session:

Mutation locks:

The labels take these values:

• operation is relay, report_provider_readiness, report_endpoint_status, or get_sandbox_provider_status. Only relay setup includes the local route; the other operations count outbound peer calls, not local API handling. On the owning replica, openshell_server_grpc_requests_total records the matching method, such as PeerRelay for relay.
• relay_kind is ssh or tcp for relays, and none for other operations. An omitted relay target counts as ssh, matching the protocol’s compatibility behavior. TCP hostnames and ports are not labels.
• route is local when this replica tries its own supervisor session, or peer when it tries another replica. A routed retry counts again, including a change from local to peer routing. Waiting for an owner does not count. Incoming PeerRelay requests do not increment this counter on the receiving replica; they appear in that replica’s gRPC request counter instead. This counter does not measure active connections, unique client requests, or unfinished attempts.
• reason is replica_capacity when the gateway replica’s pending-relay budget is full, or sandbox_capacity when one sandbox’s budget on that replica, 32 pending relays, is full.
• outcome is success, local_error, or remote_error. A relay succeeds when the supervisor claims it; this does not measure the data stream. local_error means the attempt failed on this replica, and remote_error means the owning replica returned an error or the open connection to it failed. On route="local" this replica is the owner, so failures that appear as remote_error on the peer route, such as an unclaimed relay (deadline_exceeded) or a full relay budget (resource_exhausted), appear as local_error. A cancelled attempt records local_error with grpc_code="cancelled", even when the owner was already handling it.
• grpc_code is the lowercase gRPC status name, such as ok, unavailable, or resource_exhausted, from local setup, the local supervisor claim, or the peer RPC. For a peer relay, the gateway then reports a failure to the client as UNAVAILABLE. The other RPCs return the owner’s status to the client unchanged. The label is not named code because openshell_server_grpc_requests_total uses that name for the numeric status code.
• scope is global, workspace, or sandbox.

Gauges and counters exist from startup. They start at 0, except openshell_server_relay_pending_capacity and openshell_server_mutation_lock_connections_capacity, which start at their limits. openshell_server_routed_request_attempts_total starts with seven outcome="success", grpc_code="ok" series: four relay kind and route combinations, and three other peer operations with relay_kind="none". Other outcomes and codes, and every histogram, appear after their first sample, so write alert expressions that tolerate absent series.

Request and Readiness Metrics

The gateway also records every API request and its background database check:

Configuration Delivery Metrics

With config_delivery_mode = "push", the gateway builds each supervisor’s configuration and sends it over the supervisor session. These metrics stay empty in poll mode, where GetSandboxConfig and GetSandboxProviderEnvironment in the request metrics carry the same work.

The labels take these values:

• Each configuration carries the sandbox configuration and the provider environment it names. Credentials are resolved only when that provider environment changed.
• On builds, outcome is ok, failed, or timeout. A failed build retries with a per-session backoff from 1 to 30 seconds. Each session also rebuilds its configuration every 30 seconds to repair a missed change.
• On deliveries, outcome is queued, coalesced (held until the supervisor answers its previous update), unchanged, no_active_session, unsupported_session, payload_too_large, stream_closed (the session’s stream stopped draining, so the session was closed for the supervisor to reconnect), owner_moved, or owner_lookup_failed.
• On apply_results_total and apply_duration_seconds, outcome is a ConfigApplyOutcome name, such as CONFIG_APPLY_OUTCOME_APPLIED.
• On peer notifications, outcome is accepted, stale_owner, unsupported_peer, peer_error, or timeout.

Gateway Interceptor Metrics

Gateways with gateway interceptors record their evaluations:

OCSF Log Metrics

Gateways that write an OCSF JSONL file record its queue and writer:

OCSF Log Export Metrics

Gateways with OCSF log export enabled record its queue and exporter:

Next Steps

• See the Observability Overview for traces and OCSF events.
• To monitor capacity and choose autoscaling signals for multiple gateway replicas, refer to High Availability.
• To configure the metrics listener in gateway.toml, refer to Gateway Configuration.