> This page is for version v0.1.8. > For other versions, use one of these documentation indexes: > - Latest (v0.1.8) (default): https://docs.nvidia.com/openshell/latest/llms.txt > - Dev: https://docs.nvidia.com/openshell/dev/llms.txt > - v0.1.8: https://docs.nvidia.com/openshell/v0.1.8/llms.txt > - v0.1.7: https://docs.nvidia.com/openshell/v0.1.7/llms.txt > - v0.1.6: https://docs.nvidia.com/openshell/v0.1.6/llms.txt > - v0.1.5: https://docs.nvidia.com/openshell/v0.1.5/llms.txt > - v0.1.4: https://docs.nvidia.com/openshell/v0.1.4/llms.txt > - v0.1.3: https://docs.nvidia.com/openshell/v0.1.3/llms.txt > - v0.1.2: https://docs.nvidia.com/openshell/v0.1.2/llms.txt > - v0.1.1: https://docs.nvidia.com/openshell/v0.1.1/llms.txt > - v0.1.0: https://docs.nvidia.com/openshell/v0.1.0/llms.txt > - v0.0.116: https://docs.nvidia.com/openshell/v0.0.116/llms.txt > For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/openshell/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/openshell/_mcp/server. # Observability Overview > The traces, metrics, and OCSF events that OpenShell produces, where each one goes, and how to enable it. OpenShell produces three kinds of observability signals: traces, metrics, and OCSF security events. This page describes what each signal is for and how to turn it on, and links to the page that covers it in detail. | Signal | Produced by | Destination | | ---------------------- | --------------------------------------------- | ------------------------------------------------------ | | Traces | Gateway, compute drivers, sandbox supervisors | OTLP collector | | Agent traces | Agents, relayed by sandbox supervisors | OTLP collector | | Metrics | Gateway | Prometheus scrape | | OCSF events, shorthand | Gateway, sandbox supervisors | Gateway console, `openshell logs`, supervisor log file | | OCSF events, JSONL | Gateway, sandbox supervisors | Local file | | OCSF events, OTLP | Gateway, sandbox supervisors | OTLP collector | The examples on this page are `gateway.toml` tables. In the Helm chart, set each table under `gatewayConfig`, as described in [Configure Chart Values](/openshell/v0.1.8/kubernetes/setup#configure-chart-values). ## OTLP Collectors Configure OTLP export in `gateway.toml`. The gateway exports its own signals and passes each collector's settings to the compute drivers and sandbox supervisors it starts. Supervisors export their own signals and relay the agent's traces directly, so each collector must be reachable from sandboxes as well as from the gateway. ![OpenShell OTLP collectors. The gateway and its compute drivers export spans to the trace collector, and the gateway exports its OCSF events to the OCSF collector. The gateway gives each sandbox supervisor its collector settings with the launch credentials. The supervisor exports its own spans and OCSF events to the same collectors. The network-isolated agent sends OTLP/HTTP to the supervisor's agent trace relay, which tags the spans with the sandbox ID and forwards them to the agent trace collector.](/openshell/_fern-files/openshell.docs.buildwithfern.com/18efe6af36d5dd6292330e8d19ad6183659011698daf295bfe332368bf0b6979/pages-v0.1.8/images/openshell-otlp-collectors.svg) `[openshell.gateway.otlp]` sets one collector for every signal. `[openshell.gateway.otlp.traces]`, `[openshell.gateway.otlp.ocsf]`, and `[openshell.gateway.otlp.agent]` can each send their signal to a different collector instead, such as a SIEM for OCSF events or an MLflow tracking server for agent traces. Each collector has its own transport settings: * OTLP over gRPC, the default, or HTTP/protobuf. * Request headers, such as an authorization token, set inline or read from a file. * TLS, verified against the system trust roots or a CA file, with an optional client certificate. Supervisors receive collector headers and TLS material with their launch credentials, never through the sandbox environment. The agent reaches the relay through the sandbox's mediated channel, so it needs no egress rule and never sees the agent trace collector's address or credentials. The gateway ignores the `OTEL_*` variables that would choose which signals are exported or where they go, and honors those that tune export, such as sampling. For every setting, refer to [OTLP Export](/openshell/v0.1.8/how-it-works/gateways/configuration#otlp-export) and [Tuning](/openshell/v0.1.8/how-it-works/gateways/configuration#tuning). ## Traces Traces show how the gateway handles each request, including the compute-driver work it performs, and how each sandbox supervisor starts up. Use them to find where time goes in an operation such as sandbox creation. To export traces, set an OTLP collector endpoint in `gateway.toml`: ```toml [openshell.gateway.otlp] endpoint = "https://backend:4317" ``` For span names, resource attributes, and gateway names, refer to [Traces](/openshell/v0.1.8/observability/traces). ## Agent Traces Agent traces are the spans that the agent itself produces, such as its model calls and tool use when it is instrumented with OpenTelemetry. Each sandbox supervisor relays them to the collector you choose. The relay is off by default. To send agent traces to MLflow, which needs OTLP/HTTP and an experiment header: ```toml [openshell.gateway.otlp.agent] enabled = true endpoint = "http://mlflow.mlops.svc:5000" protocol = "http/protobuf" headers = { x-mlflow-experiment-id = "0" } ``` While the relay is on, every workload process gets `OTEL_EXPORTER_OTLP_ENDPOINT` pointing at it, so an agent that uses a standard OpenTelemetry SDK exports without further configuration. Each relayed trace carries `openshell.telemetry.source=agent` and the sandbox's `openshell.sandbox.id`, so you can separate agent traces from OpenShell's own and join them with the sandbox's supervisor spans. For the workload environment, request limits, and delivery guarantees, refer to [Agent Trace Relay](/openshell/v0.1.8/how-it-works/gateways/configuration#agent-trace-relay). ## Metrics The gateway serves Prometheus metrics for API requests, readiness, gateway interceptors, OCSF output, and multi-replica capacity, on a listener separate from its API. Use them to alert on saturation and to choose autoscaling signals. To enable the listener, set `metrics_bind_address` in `gateway.toml`: ```toml [openshell.gateway] metrics_bind_address = "0.0.0.0:9090" ``` The port is an example. Until the listener is enabled, the gateway serves no metrics. The endpoint has no authentication, so restrict who can reach it. For metric names, deployment defaults, the catalog, and scrape configuration, refer to [Gateway Metrics](/openshell/v0.1.8/observability/gateway-metrics). ## OCSF Events OCSF events record security-relevant activity, such as network and HTTP policy decisions, process lifecycle, SSH sessions, and configuration changes. They use the [Open Cybersecurity Schema Framework](https://ocsf.io). Sandbox supervisors produce most of them, and the gateway produces events for its own decisions, such as approving or rejecting draft policy rules. The same events are available in three forms. ### Shorthand Logs Shorthand logs are always on. Each event appears as a one-line summary in the gateway console, in the sandbox log stream that `openshell logs` and the TUI read, and in the supervisor's log file. For example, this policy denial is the event a sandbox produces when `curl https://github.com` runs without a network policy that allows `github.com`: ```text [1791996125.789] [sandbox] [OCSF ] [ocsf] NET:OPEN [MED] DENIED /usr/bin/curl(2187) -> github.com:443 [reason:transparent_tcp_policy_denied] ``` To read the format, refer to [Sandbox Logging](/openshell/v0.1.8/observability/logging). To find logs for each compute driver, refer to [Accessing Logs](/openshell/v0.1.8/observability/accessing-logs). ### JSONL Files JSONL files hold the full OCSF record, one JSON object per line, for a log shipper to collect. The gateway and supervisors are enabled separately: ```toml # Gateway events, in gateway.toml. [openshell.gateway.ocsf_log] path = "/var/log/openshell/gateway-ocsf.jsonl" ``` ```shell # Supervisor events, for every sandbox. openshell settings set --global --key ocsf_json_enabled --value true ``` For the record structure, file locations, and rotation, refer to [OCSF JSON Export](/openshell/v0.1.8/observability/ocsf-json-export). ### OTLP Log Export OTLP log export sends each OCSF event to a collector as an OTLP log record that contains the full JSON event. It is off by default, even when trace export is on. Set `enabled = true` in `[openshell.gateway.otlp.ocsf]` to turn it on for the gateway and for every sandbox supervisor the gateway starts: ```toml [openshell.gateway.otlp] endpoint = "https://backend:4317" [openshell.gateway.otlp.ocsf] enabled = true ``` The records go to the `[openshell.gateway.otlp]` endpoint unless the table sets its own `endpoint`, such as a collector that feeds a SIEM. For the record format, collector authentication, delivery guarantees, and loss reporting, refer to [OCSF Log Export](/openshell/v0.1.8/how-it-works/gateways/configuration#ocsf-log-export). ## Next Steps * Learn how spans are named and identified in [Traces](/openshell/v0.1.8/observability/traces). * Learn how to [read the shorthand format](/openshell/v0.1.8/observability/logging). * Learn what anonymous usage data OpenShell collects in [Telemetry](/openshell/v0.1.8/observability/telemetry). It is separate from the signals on this page.