> This page is for version v0.1.7. > For other versions, use one of these documentation indexes: > - Latest (v0.1.7) (default): https://docs.nvidia.com/openshell/latest/llms.txt > - Dev: https://docs.nvidia.com/openshell/dev/llms.txt > - v0.1.7: https://docs.nvidia.com/openshell/v0.1.7/llms.txt > - v0.1.6: https://docs.nvidia.com/openshell/v0.1.6/llms.txt > - v0.1.5: https://docs.nvidia.com/openshell/v0.1.5/llms.txt > - v0.1.4: https://docs.nvidia.com/openshell/v0.1.4/llms.txt > - v0.1.3: https://docs.nvidia.com/openshell/v0.1.3/llms.txt > - v0.1.2: https://docs.nvidia.com/openshell/v0.1.2/llms.txt > - v0.1.1: https://docs.nvidia.com/openshell/v0.1.1/llms.txt > - v0.1.0: https://docs.nvidia.com/openshell/v0.1.0/llms.txt > - v0.0.116: https://docs.nvidia.com/openshell/v0.0.116/llms.txt > For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/openshell/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/openshell/_mcp/server. # Gateway Configuration File > Reference for the OpenShell gateway TOML configuration file (RFC 0003). The OpenShell gateway reads its configuration from a TOML file when `--config` or `OPENSHELL_GATEWAY_CONFIG` is set. When neither is set, the gateway reads `$XDG_CONFIG_HOME/openshell/gateway.toml` if that file exists. If no config file exists, the gateway starts from built-in defaults. Gateway process flags and gateway `OPENSHELL_*` environment variables override the file. Compute driver settings live in the driver TOML tables. See [RFC 0003](https://github.com/NVIDIA/OpenShell/blob/main/rfc/0003-gateway-configuration/README.md) for the full schema. ## Source Precedence ```text Gateway CLI flag > gateway OPENSHELL_* env var > TOML file > built-in default ``` For Helm deployments, see [Configure Chart Values](/openshell/v0.1.7/kubernetes/setup#configure-chart-values) for setting `gatewayConfig` and chart-owned Kubernetes resources. `database_url` is env-only. The loader rejects it when it appears in the file. When `OPENSHELL_DB_URL` is unset, the gateway stores its SQLite database under `$XDG_STATE_HOME/openshell/gateway/openshell.db`. On-disk SQLite databases run in WAL mode with `synchronous=FULL`: the gateway keeps `openshell.db-wal` and `openshell.db-shm` next to the database file, the file must live on a local filesystem, and a backup must use `sqlite3 openshell.db ".backup "` rather than copying `openshell.db` on its own. Every acknowledged write, including SSH session revocations, survives a power loss. The one exception is SSH session issuance, which skips the per-commit sync; a power loss can drop a just-issued session token, and the client must request a new one. ### Trusted runtime image overrides Docker, Podman, and Kubernetes consume two gateway-owned trusted runtime images. Set complete tagged or digest-pinned OCI references for the gateway process when you need to qualify exact artifacts or use a registry mirror: ```shell OPENSHELL_SANDBOX_RUNTIME_IMAGE=registry.example.com/openshell/sandbox@sha256: OPENSHELL_SUPERVISOR_IMAGE=registry.example.com/openshell/supervisor:1.2.3 ``` These settings follow the gateway's normal environment-over-file precedence: ```text process environment > [openshell.drivers.] TOML > compiled release default ``` `openshell-gateway config preflight` and gateway startup use the same effective references. Environment values are handled exactly like the corresponding TOML fields and passed to the selected container runtime; this feature does not add separate OCI-reference validation. Configure registry credentials in Docker, Podman, or Kubernetes image-pull secrets instead of embedding them in a reference. The variables affect only the built-in Docker, Podman, and Kubernetes drivers. They do not affect VM, MXC, or extension drivers. They are trusted gateway operator inputs and are never accepted from sandbox create requests. For Debian and RPM user services, place them in `~/.config/openshell/gateway.env`; the service supplies that file to both preflight and gateway startup. `name` assigns an operator-facing identity to the gateway installation. Set it with `[openshell.gateway].name`, `--name`, or `OPENSHELL_GATEWAY_NAME`. It defaults to `openshell`; the Helm chart defaults it to the chart fullname so all replicas in one installation share a name. Chart fullnames are only unique within their Kubernetes namespace, so set `server.name` explicitly when one collector receives telemetry from multiple namespaces or clusters. This identity is independent of client-side gateway aliases, TLS names, and `gateway_jwt.gateway_id`. ## Package-Managed Locations Package-managed gateways use either built-in defaults or a package-seeded TOML file. Set `OPENSHELL_GATEWAY_CONFIG` in the launch environment to use a different file. | Package | Gateway TOML location | | --------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Homebrew | `$XDG_CONFIG_HOME/openshell/gateway.toml` when it exists, otherwise the Homebrew prefix config such as `/opt/homebrew/var/openshell/gateway.toml`. | | Debian/Ubuntu | `$XDG_CONFIG_HOME/openshell/gateway.toml`, usually `~/.config/openshell/gateway.toml` for the systemd user service. | | Fedora/RHEL RPM | `$XDG_CONFIG_HOME/openshell/gateway.toml`, usually `~/.config/openshell/gateway.toml`; the systemd user service seeds this file from the packaged template on first start. | | snap | New installations use `$SNAP_USER_COMMON/.config/openshell/gateway.toml`, usually `~/snap/openshell/common/.config/openshell/gateway.toml`. Upgraded legacy installations retain `$SNAP_COMMON/gateway.toml`, usually `/var/snap/openshell/common/gateway.toml`. | The Fedora/RHEL RPM template leaves `[openshell.gateway].bind_address` unset. The gateway therefore uses its built-in `127.0.0.1:17670` primary listener. Host-networked Podman supervisors use that same loopback endpoint, so the primary listener does not need a wildcard address. Set `bind_address` explicitly only when clients must reach the primary multiplexed API through another interface. The Homebrew formula creates its prefix config without setting `bind_address`, so the gateway uses its built-in `127.0.0.1:17670` primary listener. Host-networked Docker and Podman supervisors reuse that listener when the desktop runtime exposes host loopback. A user config takes precedence. Homebrew and RPM upgrades migrate only exact package-generated schema-v1 defaults. Homebrew recognizes both its empty v1 prefix config and the affected IPv6-loopback variant. RPM recognizes the v1 file seeded by its systemd user service. Package upgrades never rewrite an edited file; migrate an edited v1 file manually with the steps below. ## Layout The file is rooted at `[openshell]`. Gateway-wide settings live under `[openshell.gateway]`. Each compute driver owns its own `[openshell.drivers.]` table. Credential drivers own `[openshell.credential_drivers.]` tables. Driver-specific values are never inherited from gateway scope. ```toml [openshell] version = 2 [openshell.gateway] # ... gateway-wide settings ... [openshell.gateway.tls] # ... gateway listener TLS ... [openshell.gateway.oidc] # ... JWT bearer auth ... [openshell.supervisor.network] # ... shared network-supervisor destination trust ... [openshell.drivers.kubernetes] # ... driver-specific settings ... [openshell.credential_drivers.kubernetes-secrets] # ... credential-driver-specific settings ... ``` The gateway selector is `compute_driver = ""`. It accepts one scalar driver name. Omit it to retain auto-detection. ## Additional Destination CAs Use the global supervisor network table to trust private certificate authorities for sandbox egress destinations: ```toml [openshell.supervisor.network] additional_ca_bundle = "/etc/openshell/certs/private-destination-ca.pem" ``` The path refers to one pre-assembled PEM bundle on the gateway host or in the gateway container. It must contain one or more PEM certificate blocks. Inline PEM, private keys, DER-only input, URLs, empty files, unrelated text, mixed certificate/key files, symlinks, and non-regular files are unsupported. The gateway reads the bundle once at startup, validates and canonicalizes it into a gateway-owned artifact, and fails startup with the field and source path when material is missing or unusable. It does not include certificate contents in diagnostics. The normalized certificates augment the default bundled, system, or native roots used for sandbox destination TLS. Normal certificate-chain and hostname verification remains enabled. A configured CA can authenticate every policy-permitted endpoint whose certificate chains to that CA and matches the requested hostname, so scope private CAs accordingly. This setting is separate from all other trust domains: * `proxy_ca_bundle` authenticates a configured corporate proxy and its TLS interception path. It still requires the corresponding proxy setting. * `guest_tls_*`, `OPENSHELL_TLS_CA`, and Kubernetes client TLS Secrets authenticate sandbox callbacks to the gateway. * `[openshell.gateway.tls]` configures the gateway listener. * OIDC, middleware, and interceptor CA settings authenticate their own gateway-side connections. Docker and Podman bind-mount the gateway-owned artifact, Kubernetes creates an immutable content-addressed managed ConfigMap, and VM stages its snapshot in the sandbox overlay. Each mounts it at `/etc/openshell/tls/additional/ca-bundle.pem` and starts the supervisor with `--additional-ca-bundle` for that path. Drivers verify their staged artifact before handing it to a runtime; the SHA-256 generation is an internal staging value, not a supervisor argument. The supervisor reads the mounted bundle and folds it into its system roots. Remote and custom compute drivers reject this setting because the compute-driver protocol does not carry the normalized material. ### Lifecycle and certificate rollover This is **global gateway configuration**, not a per-sandbox setting. Source files (or the Helm source ConfigMap) are read and normalized **only when the gateway starts**. Updating a source file or ConfigMap alone has no effect: OpenShell does not watch or hot-reload those sources, and it does not automatically restart sandbox consumers. A running supervisor retains the startup trust bundle it already loaded. To apply any addition, removal, or source update, the operator must: 1. Update the source files or source ConfigMap. 2. Restart or redeploy the gateway so it reads and normalizes the new source. 3. Explicitly stop and then start **every affected sandbox**. Stop/start retains the sandbox identity and its supported durable workspace; it does not create a replacement sandbox. The restarted supervisor consumes the bundle staged from the restarted gateway's snapshot. Source normalization, driver delivery, and supervisor validation failures fail closed: they do not silently fall back to a previous or default trust bundle. Use this sequence for a non-disruptive private-CA rollover: 1. Configure both the old and new roots. 2. Restart or redeploy the gateway, then stop/start every affected sandbox. 3. Rotate destination certificates to chain to the new root. 4. Remove the old root, then repeat the gateway restart/redeploy and stop/start sequence for every affected sandbox. For Helm, create an operator-owned ConfigMap with a `ca.crt` key and set `supervisor.network.additionalCaConfigMapName`. The chart mounts the source at `/etc/openshell-tls/additional-ca-source/ca.crt`, renders that path in the global table, and grants the Kubernetes driver ConfigMap `get` and `create` permissions for the selected workspace mode. These permissions cover the shared sandbox namespace in shared mode and are cluster-wide through the existing ClusterRoleBinding in managed and operator modes. Kubernetes RBAC cannot limit `create` by resource name or limit `get` to the digest-bearing name prefix. The chart does not grant ConfigMap `list`, `watch`, `patch`, `update`, or `delete` for this feature. Use a dedicated gateway service account and namespace-scoped bindings when this read boundary matters. Updating only the source ConfigMap does not trigger the gateway workload checksum, so explicitly restart or redeploy the gateway and then stop/start every affected sandbox. ## Migrate to schema version 2 Schema version 2 is an intentional breaking cutover. The gateway rejects files that omit `[openshell] version`, declare version 1, or declare an unsupported future version. To migrate an existing file: 1. Set `[openshell] version = 2`. 2. Replace `compute_drivers = [""]` with the scalar `compute_driver = ""`. Replace `--drivers` and `OPENSHELL_DRIVERS` with `--compute-driver` and `OPENSHELL_COMPUTE_DRIVER`. Package upgrades temporarily accept one non-empty `OPENSHELL_DRIVERS` value as a deprecated environment-only alias when it agrees with any canonical selection. Comma-delimited, invalid, or conflicting legacy values fail startup. The removed `--driver` and `--drivers` flags remain unsupported. 3. Move every compute-driver option into `[openshell.drivers.]`. Schema version 2 does not inherit driver defaults from `[openshell.gateway]`. Keep `guest_tls_ca` at gateway scope and remove `guest_tls_cert` and `guest_tls_key`. A TLS-enabled Docker, Podman, or VM gateway requires the gateway CA unless package-managed local TLS supplies it. When TLS is disabled, omit it. Kubernetes projects the gateway CA into supervisor Pods through `client_tls_secret_name` instead. 4. Rename Docker `sandbox_namespace` to `sandbox_label`, Podman `sandbox_ssh_socket_path` to `ssh_socket_path`, and VM `openshell_endpoint` to `grpc_endpoint`. 5. Use canonical image pull policies: `always`, `if_not_present`, `never`, or Podman-only `newer`. Kubernetes-style capitalization and Podman's `missing` spelling are rejected. 6. Remove zero sentinels. Omit `gateway_jwt.ttl_secs` to use non-expiring local sandbox session credentials and the 900-second typed extension-token default, omit Docker or Podman `sandbox_pids_limit` to use OpenShell's default limit of 2048, and omit Podman `health_check_interval_secs` to disable health checks. Explicit zero values are invalid. 7. Remove local Docker, Podman, or VM `grpc_endpoint` when the driver-derived callback is correct, or retain it as an explicit override. Kubernetes raw TOML requires an explicit endpoint; Helm derives one from the release's gateway Service. New VM root filesystems use an image-provided `sandbox` account when present and otherwise use UID/GID 1000. Existing persisted VM state retains its recorded or recoverable identity, including legacy 10001; the driver does not assign 10001 to an overlay without supporting state. Every `[openshell.drivers.]` entry must be a TOML table. The gateway validates driver-specific fields when it selects and constructs that driver; it does not deserialize unselected driver tables. Unknown or misplaced fields in the selected table fail startup instead of being silently ignored. ## Full Example A complete gateway configuration covering every section. Trim to the fields you need. ```toml # SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. # SPDX-License-Identifier: Apache-2.0 [openshell] version = 2 [openshell.gateway] name = "production-us-west" bind_address = "0.0.0.0:8080" health_bind_address = "0.0.0.0:8081" metrics_bind_address = "0.0.0.0:9090" log_level = "info" # Activate only after retiring incompatible gateways and outstanding driver work. enable_sandbox_updates = false # Connection ceiling for the gateway's database pool. Omit to use the backend # default: 10 for Postgres, 5 for on-disk SQLite. database_max_connections = 10 # Connection ceiling for the Postgres mutation lock pool. Omit to use 4. database_lock_max_connections = 4 # When omitted, the gateway auto-detects Kubernetes, then Podman, then Docker. # VM is never auto-detected and requires an explicit entry here. compute_driver = "kubernetes" # Optional external provider credential storage backend. Omit this key to use # the gateway's default encrypted database credential storage. credential_drivers = ["kubernetes-secrets"] ssh_session_ttl_secs = 3600 # Absolute image preparation and initial supervisor startup budget. # Applies to new attempts only; allowed values are 1 through 86400 seconds. image_preparation_timeout_seconds = 1800 # Reject invalid policy generations securely by default. Set # "retain_last_valid" only when availability takes priority. policy_validation_failure_mode = "fail_closed" # How supervisors receive configuration. "poll" keeps periodic polling. # "push" sends configuration over the supervisor session to supervisors that # apply it; supervisors that do not support streamed configuration keep polling. config_delivery_mode = "poll" # Subject Alternative Names baked into the gateway server certificate. # Wildcard DNS SANs (e.g. "*.dev.openshell.localhost") also enable sandbox # service URLs under that domain. server_sans = ["openshell", "*.dev.openshell.localhost"] # Allow plaintext HTTP routing for loopback sandbox service URLs. enable_loopback_service_http = true # Hostname format for exposed sandbox services. "hashed" makes service URLs # opaque one-label hashes (https://./) discriminated by # the installation name above, instead of the readable # ---- composite. Renaming the installation # changes every issued URL. service_hostname_format = "hashed" # Serve the WebSocket tunnel for CLI clients behind an authenticating edge proxy. enable_websocket_tunnel = false # Set true only for local plaintext gateways or trusted TLS termination. disable_tls = false # The supervisor gateway CA remains a gateway setting. TLS-enabled Docker, # Podman, and VM gateways require it unless package-managed local TLS supplies # it automatically. Omit it when TLS is disabled. Kubernetes projects the CA # from client_tls_secret_name instead. Driver tables must not repeat this field. guest_tls_ca = "/etc/openshell/certs/ca.pem" # Optional gRPC rate limit. Both values must be positive to enable the limit. # Reflection queries use a separate counter with the same count and window. # Set either value to 0, or omit both, to disable rate limiting. grpc_rate_limit_requests = 120 grpc_rate_limit_window_seconds = 60 # Optional exact provider-profile source composition. When omitted, the # gateway uses imported profiles only. provider_profile_sources = [ { type = "user" }, ] # Additional private destination roots are global supervisor settings, not # compute-driver settings. The gateway reads certificate-only PEM at startup. [openshell.supervisor.network] additional_ca_bundle = "/etc/openshell/certs/private-destination-ca.pem" # Operator-run supervisor middleware. The gRPC endpoint must be reachable from # both the gateway and sandbox supervisors. [[openshell.supervisor.middleware]] name = "local-content-guard" grpc_endpoint = "https://host.openshell.internal:50051" tls_ca_cert_path = "/etc/openshell/certs/content-guard-ca.pem" audience = "urn:openshell:middleware:local-content-guard" max_payload_bytes = 262144 timeout = "500ms" # Gateway listener TLS (distinct from the supervisor gateway CA). # client_ca_path is optional; omit it for HTTPS-only listeners that do not # verify client certificates. [openshell.gateway.tls] cert_path = "/etc/openshell/certs/gateway.pem" key_path = "/etc/openshell/certs/gateway-key.pem" client_ca_path = "/etc/openshell/certs/client-ca.pem" # Optional: SNI-based dual certificate for external (e.g. ACME) TLS. # external_cert_path = "/etc/openshell/certs/external.pem" # external_key_path = "/etc/openshell/certs/external-key.pem" # external_server_names = ["gateway.example.com"] [openshell.gateway.gateway_jwt] signing_key_path = "/etc/openshell/jwt/signing.pem" public_key_path = "/etc/openshell/jwt/public.pem" kid_path = "/etc/openshell/jwt/kid" gateway_id = "openshell" # Optional; defaults to 900 seconds. ttl_secs = 3600 [openshell.gateway.auth] allow_unauthenticated_users = false [openshell.gateway.mtls_auth] enabled = false operator_enabled = false # Opt-in: trusted OU=operator grants gateway-wide administration. # OTLP export. Omit this table entirely to disable it. [openshell.gateway.otlp] endpoint = "http://otel-collector.observability.svc:4317" service_name = "openshell-gateway" [openshell.gateway.oidc] issuer = "https://idp.example.com/realms/openshell" audience = "openshell-cli" jwks_ttl_secs = 3600 # Must be greater than zero. jwks_allowed_origins = ["https://keys.example.com"] # Default: issuer origin only. dangerously_allow_insecure_http = false # Development-only numeric-loopback escape hatch. roles_claim = "realm_access.roles" admin_role = "openshell-admin" user_role = "openshell-user" scopes_claim = "" [[openshell.gateway.interceptors]] name = "quota" grpc_endpoint = "unix:///run/openshell/interceptors/quota.sock" audience = "urn:openshell:interceptor:quota" order = 10 failure_policy = "fail_closed" binding_policy = "allowlist" timeout = "500ms" max_response_bytes = 1048576 max_patches = 32 [[openshell.gateway.interceptors.bindings]] rpc = "openshell.v1.OpenShell/CreateSandbox" phases = ["modify_operation", "validate"] failure_policy = "fail_closed" [[openshell.gateway.interceptors.bindings]] rpc = "openshell.v1.OpenShell/UpdateConfig" phases = ["validate"] [openshell.drivers.kubernetes] namespace = "openshell" # Required in raw TOML; Helm derives this from the gateway Service. grpc_endpoint = "https://openshell-gateway.openshell.svc:8080" default_image = "nvcr.io/nvidia/base/ubuntu:24.04" # Defaults to the gateway version; override to pin a specific build. # supervisor_image = "ghcr.io/nvidia/openshell/supervisor:" client_tls_secret_name = "openshell-client-tls" service_account_name = "openshell-sandbox" host_gateway_ip = "10.0.0.1" enable_user_namespaces = false sa_token_ttl_secs = 3600 [openshell.credential_drivers.kubernetes-secrets] namespace = "openshell" ``` Set `[openshell.gateway.mtls_auth] enabled = true` to map a verified client certificate to a CLI user identity. This gateway policy is independent of the compute driver and defaults on when a client CA is configured without OIDC. The listener allows bearer-only clients and validates any client certificate presented against the configured CA. Supervisors use `guest_tls_ca` to authenticate the gateway and sandbox bearer tokens to authenticate their RPCs. Set `[openshell.gateway.mtls_auth] operator_enabled = true`, `--enable-operator-auth true`, or `OPENSHELL_ENABLE_OPERATOR_AUTH=true` to enable direct mTLS operators. The default is false. TLS and a client CA are required. This setting is independent of `enabled` and OIDC. A verified certificate with the exact `OU=operator` grants user/platform-admin capabilities across all workspaces and access to `GetProviderCredentials`. Do not combine an operator certificate with an `Authorization` header. See [Authentication](/openshell/v0.1.7/how-it-works/gateways/authentication#mtls-operators) for the certificate trust boundary. With operator authentication enabled and user mTLS disabled, callers without a verified operator certificate receive `Unauthenticated` unless another configured authentication mechanism accepts them or `allow_unauthenticated_users = true`. The client-certificate handshake policy has no `require_client_auth` TOML field. Client certificates are optional at the transport layer; gateway RPC authorization enforces the configured user or sandbox identity. `[openshell.gateway.tls]` supports optional SNI-based dual-certificate mode for deployments that need separate internal and external server certificates. Set `external_cert_path` and `external_key_path` to point at the external (e.g. ACME/publicly-trusted) certificate and key. List the hostnames that should be served with the external certificate in `external_server_names`. Connections whose TLS SNI hostname matches one of those names receive the external certificate; all other connections (including those with no SNI) receive the primary internal certificate from `cert_path`/`key_path`. Both fields must be set together — providing only one is a configuration error. On Kubernetes with the Helm chart, the external certificate is managed automatically when `certManager.serverIssuerRef.name` is set; the chart populates these fields from the cert-manager-issued external server certificate. `[openshell.gateway] policy_validation_failure_mode` controls what sandbox supervisors do when a complete candidate policy fails runtime validation. The default, `fail_closed`, deactivates the previous network policy, closes relays pinned to it, and denies new egress until a valid generation loads. `retain_last_valid` leaves the previous valid generation active. Both modes reject the candidate atomically; startup keeps the workload unstarted until the effective policy and matching provider configuration pass admission. A rejected startup exposes `ConfigurationInvalid` and remains available for policy/provider repair in either mode. Gateway mutation paths that can preflight a known effective scope reject invalid candidates before persistence and leave the active policy unchanged regardless of this setting. Changing the value requires restarting the gateway so it can reload `gateway.toml` and distribute the new posture to sandbox supervisors. `[openshell.gateway.gateway_jwt] ttl_secs` controls generation-bound gateway-facing and Sandbox Protocol credentials minted for a sandbox session, plus typed extension JWTs. Omit it for non-expiring local sandbox session credentials: both session tokens carry `exp = 0`, supervisors skip periodic session-token renewal, and refresh responses omit their expiration timestamps. Typed extension JWTs retain a 900-second default when the field is omitted. Use omission only for local single-player Docker, Podman, or VM gateways. Explicit `0` is invalid. Kubernetes and other shared deployments should set a positive TTL; Helm renders `3600` seconds by default, and the gateway logs a warning when a Kubernetes gateway omits the field. Listener TLS and sandbox launch signing use separate keys. A working HTTPS listener or mTLS client certificate does not supply the signing key needed to start a VM sandbox. Configure `signing_key_path`, `public_key_path`, `kid_path`, and `gateway_id` in `[openshell.gateway.gateway_jwt]`, as shown above. For a local installation, `openshell-gateway generate-certs --output-dir ` writes a signing bundle alongside the TLS bundle. Without explicit `gateway_jwt` configuration, the gateway discovers `jwt/signing.pem`, `jwt/public.pem`, and `jwt/kid` under `OPENSHELL_LOCAL_TLS_DIR`, or under the OpenShell state directory's `tls/` directory when that variable is unset. Explicit signing configuration takes precedence over local discovery. The gateway rejects partial or invalid signing bundles at startup, including private and public keys that do not match. When the selected driver requires launch signing and no bundle is configured, sandbox creation fails before the driver validates or prepares the sandbox. The error names the signing configuration and discovery location; it does not print keys or tokens. The VM driver also validates launch material before resolving images or creating sandbox state. External compute drivers declare this requirement by adding `openshell.compute.launch-authentication` to their extension metadata's `required_capabilities`. This requires a gateway that understands the launch-signing preflight and supplies `SandboxLaunchAuthentication` on creation. Drivers that use another launch mechanism can omit the requirement. The separate `supports_sandbox_authentication` capability describes the `AuthenticateSandbox` RPC and does not require launch signing. The gateway refuses to start before binding any ports when no OIDC issuer, mTLS user authentication, operator authentication, or `gateway_jwt` is configured and `allow_unauthenticated_users` is unset or false. Configure an authentication mechanism, or explicitly opt in for trusted local development: ```toml [openshell.gateway.auth] allow_unauthenticated_users = true ``` `[openshell.gateway.auth] allow_unauthenticated_users = true` is an unsafe local-development and trusted-proxy escape hatch. It accepts user-facing CLI/API calls without OIDC or mTLS credentials while sandbox supervisors still authenticate with gateway-minted sandbox JWTs. Leave it false for shared and production gateways. ## Service Routing Sandbox service URLs default to a readable composite hostname: `https://----./` for named services and `https://--./` for the unnamed endpoint. The base domain is the first wildcard DNS SAN in `server_sans`; gateways without SANs use the built-in `openshell.localhost`. `service_hostname_format` selects the hostname format for exposed sandbox services when multiple gateways share one service base domain behind a single ingress and wildcard certificate. Identically named workspaces, sandboxes, and services on two such gateways produce identical composite hostnames, so the shared ingress cannot select the correct gateway. Set the format to `hashed` with `[openshell.gateway].service_hostname_format`, `--service-hostname-format`, or `OPENSHELL_SERVICE_HOSTNAME_FORMAT`. The field defaults to `readable` (the composite hostname). Its precedence is `--service-hostname-format` > `OPENSHELL_SERVICE_HOSTNAME_FORMAT` > TOML > default. In hashed mode the discriminator hashed into every service hostname is the gateway installation `name` — the operator- or platform-assigned value used for telemetry identity. It must match `[a-z0-9]([-a-z0-9]*[a-z0-9])?` and be at most 63 bytes, and it must be unique among all gateways sharing the service base domain; that uniqueness is an operational contract enforced by the deploying platform, since a single gateway cannot observe its siblings. `gateway_jwt.gateway_id` is unrelated: it carries the sandbox JWT issuer and Kubernetes ownership semantics and must not be reused here. With the format set, service URLs replace the readable composite label with a deterministic 16-character hash of the installation name, workspace, sandbox, and service name: `https://./` (for example, `https://enh47uqyj7bpyttz.apps.example.com/`, the label a gateway named `gateway-a` issues for workspace `default`, sandbox `my-sandbox`, service `web`). The hash stays one DNS label deep, so the existing single wildcard certificate keeps working, and it is unique per gateway, so exact-host ingress rules can route it unambiguously. The gateway still accepts requests for legacy composite hostnames while hashed format is configured. Service URLs are recomputed from the current configuration, not persisted. Changing the installation name, the hostname format, or both changes every hash URL issued before the change; previously issued URLs stop resolving until callers request a fresh URL. ## Sandbox Updates `[openshell.gateway] enable_sandbox_updates` defaults to `false`. It gates the separate `openshell.v1.SandboxUpdates/UpdateSandbox` RPC on the existing gateway listener. It gates image, command, environment, label, and annotation updates for stopped sandboxes. Upgrade every gateway before activation. Disabling the flag stops new updates; accepted updates still recover automatically. The equivalent gateway process flag and environment variable are `--enable-sandbox-updates true` and `OPENSHELL_ENABLE_SANDBOX_UPDATES=true`. Enable it only after completing the [activation checklist](/openshell/v0.1.7/kubernetes/high-availability#activate-sandbox-updates). Setting it to `true` attests that every gateway that can write the database is compatible and incompatible gateways cannot return. Keep it false throughout a rolling upgrade, including while outstanding operations from retired gateways settle. Gateway membership checks fail closed but do not prove retirement. PostgreSQL administrative and monitoring sessions do not block updates; connection `application_name` settings are preserved. For a schema-v2 file, set: ```toml [openshell.gateway] enable_sandbox_updates = true ``` The canonical Helm values form uses a dotted table key: ```yaml gatewayConfig: "openshell.gateway": enable_sandbox_updates: true ``` This is not an automatic mixed-version migration switch. Do not downgrade while an image update is pending or retained sandboxes require the private image revision protocol. SQLite supports one gateway process only; activation attests that the previous process has stopped, not that sharing a SQLite file enables HA. ## Database Pool `[openshell.gateway] database_max_connections` sets the connection ceiling for the gateway's database pool. One pool is shared by every database-backed RPC, so this value is also the gateway's ceiling on concurrent database work. When the pool is saturated, callers queue on connection acquisition and sqlx logs that the time to acquire a connection exceeded its slow threshold. The background readiness monitor draws from the same pool, so sustained saturation can push its database ping past the check timeout and flip `/readyz` to not-ready while the gateway is otherwise serving. Omit the key to use the backend default, which is unchanged from earlier releases: | Backend | Default ceiling | | ------------------ | ------------------------------------------------------------------------------------------------------------------------------ | | Postgres | 10 | | SQLite (on disk) | 5 | | SQLite (in memory) | 1, always. The database lives inside its single connection, so a configured value is ignored and the gateway logs that it was. | The equivalent flag and environment variable are `--db-max-connections` and `OPENSHELL_DB_MAX_CONNECTIONS`, and Helm deployments set the key under `gatewayConfig.openshell.gateway`. Values below `1` are rejected at startup: a zero-sized pool would block every acquisition rather than degrade. Postgres needs at least `2`, because the SSH identity lock keeps one data connection while it queries the pool, so the gateway rejects `1` there too. On Postgres, the gateway holds its mutation advisory locks on a separate lock pool, one connection per guarded mutation while it runs or waits. `[openshell.gateway] database_lock_max_connections` sets that pool's ceiling, `4` by default, so it bounds how many guarded mutations a replica runs or waits on in Postgres at once. When every lock connection is in use, further guarded mutations queue for one, and fail with `MUTATION_LOCK_TIMEOUT` only if none frees up before their 10-second lock wait runs out. The equivalent flag and environment variable are `--db-lock-max-connections` and `OPENSHELL_DB_LOCK_MAX_CONNECTIONS`, and Helm deployments set the key under `gatewayConfig.openshell.gateway`. Values below `1` are rejected at startup. SQLite has no lock pool, so the gateway ignores the key there and logs that it did. On Postgres, each replica opens both pools, so plan for `(database_max_connections + database_lock_max_connections) × replicaCount` connections, 14 per replica with the defaults, and keep that total below the server's own `max_connections` (and below any connection-pooler limit in front of it). Rollouts run extra pods; refer to [Size PostgreSQL Connections](/openshell/v0.1.7/kubernetes/high-availability#size-postgresql-connections). Raising either ceiling past what the server admits moves the failure from a slow acquire to a refused connection. On SQLite, writers are serialized regardless of pool size, so a larger ceiling buys read concurrency only. A gateway that needs more write throughput wants an external Postgres database (`server.externalDbSecret`), not a larger SQLite pool. ## OCSF JSONL Output `[openshell.gateway.ocsf_log]` enables a gateway-local file containing native OCSF events emitted by the gateway, including events without a sandbox association. Omit the table to disable it. Restart the gateway after changing this configuration. Gateway records use `[openshell.gateway] name` as `device.uid` and `device.name`, shared across replicas; `device.hostname` identifies the replica. Assign a distinct name to each installation collected together. The default name does not guarantee uniqueness, and renaming an installation changes its audit identity. ```toml [openshell.gateway.ocsf_log] path = "/var/log/openshell/gateway-ocsf.jsonl" rotation = "daily" max_files = 7 queue_capacity = 10000 queue_max_bytes = 16777216 ``` | Field | Default | Behavior | | ----------------- | ---------- | ------------------------------------------------------------------------------------------------------------------------------ | | `path` | Required | Nonempty file path; relative paths use the gateway's working directory. | | `schema_version` | None | Deprecated and ignored; records are native OCSF 1.8.0. The gateway logs a warning at startup when it is set. | | `rotation` | `daily` | Rotate on the first write of a new UTC day. `never` allows unbounded file growth. | | `max_files` | `7` | Positive number of rotated segments to retain, in addition to the active file. Rejects explicit use with `rotation = "never"`. | | `queue_capacity` | `10000` | Positive queued-record limit. Overflow discards incoming records. | | `queue_max_bytes` | `16777216` | Positive limit on queued encoded bytes. Records exceeding this limit are discarded. | Unknown fields are rejected. This is one file destination, not an exporter registry. To send the same events to a collector, use [OCSF log export](#ocsf-log-export). The existing `ocsf_json_enabled` sandbox setting remains independent. For Helm installations, set `server.ocsfLog.enabled` to render this table. The default `server.ocsfLog.path`, `/tmp/gateway-ocsf.jsonl`, is writable in the gateway container with either the StatefulSet or Deployment workload but does not persist across pod restarts. To keep records, mount a volume with `server.extraVolumes` and `server.extraVolumeMounts` and set the path on it. When replicas share a volume, add `subPathExpr: $(OPENSHELL_POD_NAME)` to the mount so each replica writes its own file. Give each gateway replica its own writable file and an external shipper read access to the containing directory. New files use owner-only permissions on Unix; arrange shipper access deliberately. Rotation renames the active file to a date- and UUID-suffixed sibling. The shipper must follow rotated files and checkpoint its progress. Do not use multiple writers or external copy-truncate rotation on this path. The gateway queues OCSF independently of `RUST_LOG`, excludes ordinary diagnostics, and preserves native event IDs. It writes up to 100 records per batch with a 500 ms batching interval. One additional bounded batch can be in flight. File errors do not stop sandbox execution: failed writes are not replayed, and subsequent records are discarded during reopen backoff. A restarted writer removes an incomplete trailing line before appending. Metrics include `openshell_ocsf_log_queued_total`, `openshell_ocsf_log_written_total`, `openshell_ocsf_log_queue_records`, `openshell_ocsf_log_queue_bytes`, `openshell_ocsf_log_dropped_total` (with a bounded `reason` label), and `openshell_ocsf_log_writer_errors_total`. Recovery reports discarded bytes through `openshell_ocsf_log_recovery_discarded_bytes_total`, not an invented lost-record count. Warnings report writer failures independently of this file. Allow enough retention and disk space for shipper outages. Graceful shutdown allows five seconds for draining; outstanding OS writes remain uncertain if they outlast that budget. Flush does not mean fsync. The file is a best-effort collection boundary, not durable acceptance, a replay protocol, or proof of a complete audit history. See [OCSF JSON export](/openshell/v0.1.7/observability/ocsf-json-export) for the separate sandbox-local output. ## OTLP Export `[openshell.gateway.otlp]` enables OpenTelemetry export over OTLP. Omit the table to disable export. Its `endpoint` is the collector for every signal the gateway exports and, on its own, enables trace export. `[openshell.gateway.otlp.traces]` sends spans to a different collector. [OCSF log export](#ocsf-log-export) is off until `[openshell.gateway.otlp.ocsf]` sets `enabled = true`, and the [agent trace relay](#agent-trace-relay) is off until `[openshell.gateway.otlp.agent]` sets `enabled = true`. Each can use its own collector. The gateway already uses the Rust `tracing` framework for structured logs sent to stdout and the sandbox log stream. Enabling this section adds an OpenTelemetry layer to the same tracing subscriber. It exports span trees to an OTLP collector without exporting, replacing, or redirecting the existing diagnostic log events. ```toml [openshell.gateway.otlp] endpoint = "http://otel-collector.observability.svc:4317" service_name = "openshell-gateway" # Optional: send spans to a different collector, over OTLP/HTTP. [openshell.gateway.otlp.traces] endpoint = "http://traces.observability.svc:4318" protocol = "http/protobuf" ``` Each enabled signal needs an endpoint, either in its own table or from `endpoint`. A table without its own `endpoint` uses the whole `[openshell.gateway.otlp]` collector, including its `protocol`, headers, and TLS files. A table with its own `endpoint` takes nothing else from `[openshell.gateway.otlp]`, so configured settings for one collector, such as its credentials, never reach another, and setting any of them in a table without an `endpoint` is a configuration error. Endpoints must be valid URIs without a query string: exporters send no query parameters, so an endpoint with one is a configuration error, and a collector that needs parameters must take them as headers. If an endpoint is malformed, the gateway logs the configuration error and continues with all OTLP export disabled. It does not connect at startup: an unreachable collector produces export failures, never a failure to serve. `protocol` selects the transport to a collector: `grpc`, the default, or `http/protobuf`. With `http/protobuf`, the endpoint is a base URL and each signal is posted to its path under it, `/v1/traces` or `/v1/logs`, as with `OTEL_EXPORTER_OTLP_ENDPOINT`. HTTP/JSON is not supported. Point a `grpc` endpoint at a collector's gRPC receiver, conventionally port `4317`, and an `http/protobuf` endpoint at its HTTP receiver, conventionally port `4318`. A URI alone cannot distinguish the two, so a mismatched endpoint is accepted at startup and then fails on export. Sandbox supervisors and standalone compute drivers must come from the same release as the gateway to use `http/protobuf`: an older supervisor exports over gRPC, and an older driver rejects the protocol setting. `headers` sets request headers sent with every export to a collector, as gRPC metadata or HTTP headers, such as an authorization token or a backend's routing header. Each name must be a valid HTTP header name set once, ignoring case; names the exporters set themselves, such as `content-type`, `host`, and `grpc-*`, and binary gRPC names ending in `-bin` are rejected. Each value must be non-empty printable ASCII; spaces are allowed. The gateway's trace and OCSF exporters also read `OTEL_EXPORTER_OTLP_TRACES_HEADERS` or `OTEL_EXPORTER_OTLP_LOGS_HEADERS`, or else `OTEL_EXPORTER_OTLP_HEADERS`, and a header set there replaces a configured header of the same name. In-process compute drivers and the VM driver send the gateway's trace headers, including those from the environment, with their own spans. A standalone compute driver sends only the headers in its own `OTEL_EXPORTER_OTLP_*HEADERS` variables. Supervisors send their trace collector's `headers` with their own spans, and the gateway's OCSF headers, including those from the environment, with their OCSF records. Trace headers from the gateway's environment do not reach supervisors, so set `headers` or `headers_file` on the trace collector when supervisor spans need them. The agent trace relay sends only its collector's `headers`, never ones from the environment, so its credentials can only be set with `headers` or `headers_file`. A changed header takes effect when the gateway restarts, and reaches a running sandbox only after that sandbox restarts. Put credentials in `headers_file` rather than `headers`. It reads a collector's headers from a file instead, such as a mounted Kubernetes Secret, so credentials stay out of the configuration file. The file holds one `name=value` header per line. Blank lines are skipped, and whitespace around each name and value is trimmed. The headers are validated like `headers`, the file must be UTF-8 and at most 64 KiB, and setting both `headers` and `headers_file` in one table is a configuration error. A relative path uses the gateway's working directory. The gateway reads the file at startup and fails to start if it is missing or invalid, so a changed file takes effect as a changed header does. Supervisors receive the gateway's headers but export to the endpoints their compute driver gives them. An external compute driver takes those endpoints from its own flags, so give it the same endpoints as the gateway, or supervisors send the gateway's credentials to a different collector. Supervisors and external compute drivers from an earlier release than the gateway reject launch credentials that carry trace or agent relay headers, so when the trace or agent collector sets `headers` or `headers_file`, upgrade them with or before the gateway. A gateway rolled back to an earlier release cannot read the stored launch credentials of sandboxes created with those headers, so delete and recreate those sandboxes after a rollback. Headers travel in cleartext unless the endpoint uses `https://`, so use TLS when they carry credentials over an untrusted network. ```toml [openshell.gateway.otlp] endpoint = "http://otel-collector.observability.svc:4317" headers_file = "/etc/openshell/otlp/headers" # authorization=Bearer [openshell.gateway.otlp.traces] endpoint = "http://traces.observability.svc:4318" protocol = "http/protobuf" headers = { x-mlflow-experiment-id = "0" } ``` Use an `https://` endpoint for TLS over either protocol. The gateway verifies the collector certificate against the system trust roots, or only against the PEM certificates in `ca_file` when it is set. Set `cert_file` and `key_file` together to present a PEM client certificate to a collector that requires one. The gateway reads these files at startup and fails to start if one is unreadable, larger than 64 KiB, or holds no usable PEM, if the key does not match the certificate, or if one is set for an `http://` endpoint. A relative path uses the gateway's working directory. Supervisors receive each distinct set of files once, however many tables share it, and the files they receive may total at most 192 KiB. A shared supervisor's launch credentials, these files and headers included, may total at most 256 KiB, so large headers alongside large TLS files can still stop sandboxes from launching on one. Compute drivers and supervisors use the same CA and client certificate for their own exports, and the agent trace relay uses its collector's. Supervisors receive the file contents with their launch credentials, never through the sandbox environment or file system, so the collector cannot tell a supervisor's exports from the gateway's by certificate. The private key is stored with each sandbox's launch credentials wherever its compute driver keeps them, such as a Kubernetes Secret. The VM driver receives the trace collector's file paths. A standalone compute driver reads its own with `--otlp-ca-file`, `--otlp-cert-file`, and `--otlp-key-file`. The files are read once. To rotate a certificate, restart the gateway; running sandboxes keep the previous certificate until they restart. Supervisors and external compute drivers from an earlier release than the gateway reject launch credentials that carry TLS material, and an earlier VM driver rejects the TLS file flags, so when a collector sets TLS files, upgrade them with or before the gateway. A gateway rolled back to an earlier release cannot read the stored launch credentials of sandboxes created with TLS material, so delete and recreate those sandboxes after a rollback. ```toml [openshell.gateway.otlp] endpoint = "https://otel-collector.observability.svc:4317" ca_file = "/etc/openshell/otlp/ca.pem" cert_file = "/etc/openshell/otlp/client.pem" key_file = "/etc/openshell/otlp/client-key.pem" ``` Each collector has its own TLS files. To trust a different CA for agent traces than for OpenShell's own spans and OCSF events, give `[openshell.gateway.otlp.agent]` its own `endpoint` and `ca_file`. Supervisors then receive both CAs and use each for its own collector: ```toml [openshell.gateway.otlp] endpoint = "https://otel-collector.observability.svc:4317" ca_file = "/etc/openshell/otlp/infra-ca.pem" [openshell.gateway.otlp.agent] enabled = true endpoint = "https://mlflow.mlops.svc:5000" protocol = "http/protobuf" ca_file = "/etc/openshell/otlp/mlflow-ca.pem" ``` For Helm deployments, store the files in a Secret, mount it with `server.extraVolumes` and `server.extraVolumeMounts`, and set the table's keys to the mounted paths through `gatewayConfig`. This example mounts a Secret named `otlp-tls` with the keys `ca.pem`, `client.pem`, and `client-key.pem`: ```yaml server: otlp: endpoint: https://otel-collector.observability.svc:4317 extraVolumes: - name: otlp-tls secret: secretName: otlp-tls extraVolumeMounts: - name: otlp-tls mountPath: /etc/openshell/otlp readOnly: true gatewayConfig: openshell.gateway.otlp: ca_file: /etc/openshell/otlp/ca.pem cert_file: /etc/openshell/otlp/client.pem key_file: /etc/openshell/otlp/client-key.pem ``` The OpenTelemetry SDK logs export failures after startup. Spans in a failed batch are dropped rather than retried. `service_name` sets the gateway's `service.name` resource attribute and defaults to `openshell-gateway`. The gateway passes the trace collector to compute drivers and sandbox supervisors, which export their own spans to it, so the trace collector must also be reachable from sandboxes. Spans for routine or high-volume operations use DEBUG level and are exported only when the gateway or sandbox log level is `debug`. For what is traced, span names, and resource attributes, refer to [Traces](/openshell/v0.1.7/observability/traces). For Helm deployments, set `server.otlp.endpoint` to render this table. The optional `server.otlp.serviceName` value overrides the gateway service name; driver service names remain fixed. Set the `traces`, `ocsf`, and `agent` tables through `gatewayConfig`, which renders each key as a TOML table: ```yaml gatewayConfig: openshell.gateway.otlp.traces: endpoint: https://traces-backend:4317 openshell.gateway.otlp.ocsf: enabled: true openshell.gateway.otlp.agent: enabled: true ``` The local `mise run helm:k3s:create` workflow installs a trace collector and UI, enables Agent Sandbox controller tracing on supported releases, and configures both the controller and the existing Skaffold deployment to export traces to it automatically. The local `gateway`, `gateway:docker`, `gateway:podman`, and `gateway:vm` tasks also set the gateway installation name in their generated configuration. Their defaults are driver-specific (`kubernetes-dev`, `docker-dev`, `podman-dev`, and `vm-dev`), and their documented `OPENSHELL_*_GATEWAY_NAME` overrides update both CLI registration and exported telemetry identity. Run `mise run helm:k3s:forward` to forward OTLP/gRPC to local port `4317` and the trace UI to local port `18888`. When Skaffold has deployed a Kubernetes gateway, the task also forwards it to local port `8090`; otherwise it continues with the collector ports only. The Skaffold run task registers the forwarded gateway under the worktree-specific k3d cluster name; select it with `openshell gateway select `. The local Podman, Docker, and VM gateway tasks export spans and OCSF log records to the forwarded receiver automatically. In-process compute drivers read their backend-specific settings from `[openshell.drivers.]`. An external driver's gateway table supplies only its `socket_path`; configure the external driver process itself through that binary's flags or environment variables. A driver that advertises `supports_sandbox_authentication` may authenticate an opaque bootstrap credential through the compute-driver protocol. It must return the same non-empty runtime identity from sandbox creation and credential authentication. Start requests include the previously recorded opaque identity so the driver can preserve the stable resource while rotating generation-specific compute. For `IssueSandboxToken`, the gateway verifies the sandbox exists and the driver/runtime identity matches the durable provisioning record before returning a generation-bound session JWT. The in-process Kubernetes driver reads `service_account_name`, `workspace_mode`, and namespace discovery from `[openshell.drivers.kubernetes]`; an external Kubernetes driver receives the equivalent values through its own CLI or environment contract. ### OCSF Log Export Set `enabled = true` in `[openshell.gateway.otlp.ocsf]` to export gateway OCSF events as OTLP log records. It defaults to `false`, so trace export alone never sends OCSF events. Records go to the table's `endpoint`, or to `[openshell.gateway.otlp]` `endpoint` when it is omitted, with the same resource attributes as spans, under the `openshell.ocsf` instrumentation scope. Use a separate endpoint to send OCSF events to a collector that feeds a SIEM while spans go elsewhere. Sandbox supervisors receive the same endpoint and protocol and export their own OCSF events there directly, so the endpoint must also be reachable from sandboxes. Supervisor records carry the supervisor's resource attributes, including `openshell.sandbox.id` and `openshell.gateway.name`. Supervisors also send the collector [headers](#otlp-export) that the gateway uses for OCSF export, so a collector that requires authentication accepts their records too. The gateway delivers headers with each sandbox's launch credentials, never through the sandbox environment. Other `OTEL_*` variables do not reach supervisors. ```toml [openshell.gateway.otlp.ocsf] enabled = true endpoint = "http://siem-collector.security.svc:4317" # optional ``` Each OCSF event becomes one log record: | Field | Value | | ---------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Body | The OCSF event as a JSON string. | | Timestamp | The OCSF `time`. The observed timestamp is when the gateway received the event. | | Severity | Unknown, Informational, and Other are INFO, Low is WARN, Medium is WARN2, High is ERROR, Critical is ERROR2, and Fatal is FATAL. The severity text is the OCSF severity name. | | Attributes | `ocsf.class_uid`, `ocsf.activity_id`, `ocsf.severity_id`, `ocsf.metadata.uid`, and, for events associated with a sandbox, `openshell.sandbox.id`. | `schema_version` in this table is deprecated and ignored; the gateway logs a warning at startup when it is set. Log export is independent of `RUST_LOG` and of `[openshell.gateway.ocsf_log]`; you can enable both. The gateway queues up to 10,000 records, and up to 16 MiB of records including the batch being exported, without blocking event producers. It exports a batch as soon as 100 records or 1 MiB are queued, or 500 ms after the first record of a batch arrives. A record larger than 1 MiB, such as an event for a request with a very long path, is dropped rather than exported. A failed batch is retried with backoff for up to four attempts, so delivery is at least once. Deduplicate on `ocsf.metadata.uid`. Each attempt can take up to the exporter timeout, 10 seconds by default, so an unresponsive collector can hold a batch for about 40 seconds while new records fill the queue. Records in a batch that still fails, records rejected because the queue is full, and oversized records are dropped and counted. The next exported batch then includes a WARN record with event name `openshell.ocsf.gap`, the number of dropped records in `openshell.ocsf.gap.dropped`, and the time of the first loss in `openshell.ocsf.gap.since_unix_ms`. During shutdown, the gateway attempts each remaining batch once within a five-second budget, then abandons any export still in progress. Records lost during shutdown produce no gap record. A collector can accept a batch while rejecting some of its records, an OTLP partial success. The gateway counts such a batch as exported, and the exporter logs the number of rejected records as a warning. Gap records and `openshell_ocsf_otlp_dropped_total` cover only records the gateway dropped, so check the collector's own metrics for records it rejected. Metrics include `openshell_ocsf_otlp_queued_total`, `openshell_ocsf_otlp_exported_total`, `openshell_ocsf_otlp_dropped_total` (with a `reason` label of `queue_capacity`, `queue_bytes`, `oversized`, `export_failed`, `serialization`, or `closed`), `openshell_ocsf_otlp_export_errors_total`, and `openshell_ocsf_otlp_retries_total`. ### Agent trace relay Set `enabled = true` in `[openshell.gateway.otlp.agent]` to have each sandbox forward the workload's own traces. It defaults to `false`, so trace export alone never changes the workload environment. Traces go to the table's collector, or to the `[openshell.gateway.otlp]` collector when it has no `endpoint`, and that collector must be reachable from sandboxes. Use `protocol = "http/protobuf"` for trace backends that accept only OTLP/HTTP, and `headers` for the ones that need them. For example, MLflow accepts traces directly when each request names an experiment: ```toml [openshell.gateway.otlp.agent] enabled = true endpoint = "http://mlflow.mlops.svc:5000" protocol = "http/protobuf" headers = { x-mlflow-experiment-id = "0" } ``` While the relay is active, the sandbox runtime sets `OTEL_EXPORTER_OTLP_ENDPOINT=http://192.0.0.8:4318` and `OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf` in every workload process, and sets `OTEL_EXPORTER_OTLP_COMPRESSION` and `OTEL_EXPORTER_OTLP_TRACES_COMPRESSION` to `none`, so a standard OpenTelemetry SDK exports without configuration. These are lowest-precedence defaults. They are not set when the sandbox creation request or an environment passed to `openshell sandbox exec` names `OTEL_EXPORTER_OTLP_ENDPOINT` or `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT`, or when the image `ENV` names one for the main process, since `openshell sandbox exec` sessions do not inherit the image environment, and an endpoint the workload sets itself, for example with a shell `export`, takes precedence over them. A workload that exports elsewhere does so subject to the sandbox's network policy, and its traces carry no relay attribution. Provider credentials cannot redirect traces. While the relay is off, nothing is injected. Because the generic endpoint is set, an SDK that also exports metrics or logs sends them to the relay, which answers with an empty OTLP success and discards them. To keep metrics and logs, point those exporters elsewhere with `OTEL_EXPORTER_OTLP_METRICS_ENDPOINT` and `OTEL_EXPORTER_OTLP_LOGS_ENDPOINT`. The address is a reserved label rather than a network destination: the supervisor serves it inside the sandbox boundary and forwards accepted batches to the collector over the configured `protocol`. The collector's address is never visible inside the sandbox. Every forwarded resource carries `openshell.sandbox.id`, with the same value as the supervisor's own spans, and `openshell.telemetry.source=agent`. Values the agent supplies for those two keys are replaced at the resource level; span and scope attributes pass through unchanged, so filter and join on the resource attributes. Filter on `openshell.telemetry.source` to separate agent traces from infrastructure traces, and join on `openshell.sandbox.id` to correlate them. Agents must export traces as OTLP/HTTP protobuf with compression disabled. The relay answers 415 to JSON bodies and to any `Content-Encoding` other than `identity`, 413 to bodies larger than 2 MiB, to requests with more than 512 `ResourceSpans` entries, and to requests whose resource attributes exceed 8192 in number or 256 KiB in total, 408 to requests whose body does not arrive within 30 seconds, and 503 with `Retry-After` while its buffer of 32 batches is full, which standard SDKs retry with backoff. Error responses carry a protobuf `google.rpc.Status` whose message states the reason. While the relay is off, connections to the relay address are refused immediately. Delivery is best-effort. The relay answers 200 once a batch is buffered and makes one export attempt per batch, so a batch is lost if the collector is unavailable at that moment. The supervisor logs the first failed export and a periodic summary of failures. Before the sandbox reports that its main process exited, and before a suspended sandbox's supervisor exits, the relay flushes buffered batches for up to 5 seconds, and skips that wait when its most recent export failed. It keeps serving `openshell sandbox exec` sessions after the main process exits and flushes once more when the sandbox is torn down. ### Tuning `[openshell.gateway.otlp]` decides whether and where to export. How the SDK exports is controlled by the standard OpenTelemetry environment variables, which the gateway reads through the SDK rather than mirroring as TOML keys. They apply to the gateway and its in-process compute drivers. The VM driver also receives the trace headers, and sandbox supervisors receive headers as described for [headers](#otlp-export). No other variable reaches supervisors, which receive only the endpoints, protocols, [headers, and TLS material](#otlp-export) of their collectors. The gateway honors these variables: | Variable | Effect | | ----------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------ | | `OTEL_SERVICE_NAME` | The gateway's `service.name` when `service_name` is not set in `[openshell.gateway.otlp]`. Compute-driver and supervisor service names are fixed. | | `OTEL_RESOURCE_ATTRIBUTES` | Additional resource attributes, such as `deployment.environment=prod`. | | `OTEL_TRACES_SAMPLER`, `OTEL_TRACES_SAMPLER_ARG` | Sampling strategy and ratio. Defaults to `parentbased_always_on`. | | `OTEL_BSP_SCHEDULE_DELAY`, `OTEL_BSP_MAX_QUEUE_SIZE`, `OTEL_BSP_MAX_EXPORT_BATCH_SIZE`, `OTEL_BSP_EXPORT_TIMEOUT` | Batch span processor tuning. | | `OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT`, `OTEL_SPAN_EVENT_COUNT_LIMIT`, `OTEL_SPAN_LINK_COUNT_LIMIT` | Per-span limits. | | `OTEL_EXPORTER_OTLP_HEADERS`, `OTEL_EXPORTER_OTLP_TIMEOUT` | Exporter transport tuning for every signal. | | `OTEL_EXPORTER_OTLP_COMPRESSION` | Set to `gzip` to compress exports of every signal. `zstd` is not supported. | | `OTEL_EXPORTER_OTLP_TRACES_HEADERS`, `OTEL_EXPORTER_OTLP_TRACES_COMPRESSION`, `OTEL_EXPORTER_OTLP_TRACES_TIMEOUT` | Exporter transport tuning for spans only. These override the general variables. | | `OTEL_EXPORTER_OTLP_LOGS_HEADERS`, `OTEL_EXPORTER_OTLP_LOGS_COMPRESSION`, `OTEL_EXPORTER_OTLP_LOGS_TIMEOUT` | Exporter transport tuning for OCSF log records only. These override the general variables. | | `SSL_CERT_FILE`, `SSL_CERT_DIR` | Replace the system trust roots for `https://` endpoints without a `ca_file`. They affect the gateway and the compute drivers it runs, not supervisors. | For Helm deployments, set these variables with `server.extraEnv`. Read credentials from a Secret so they stay out of the values file: ```yaml server: extraEnv: - name: OTEL_EXPORTER_OTLP_HEADERS valueFrom: secretKeyRef: name: otel-collector-auth key: headers ``` These variables have no effect. Only `[openshell.gateway.otlp]` decides which signals are exported and where they go, for two reasons: * OCSF events are security records, so where they go must be a deliberate choice. Platform tooling can add `OTEL_*` variables to a process, as the OpenTelemetry Operator does for annotated pods on Kubernetes. Such a variable must not start sending security records to a collector, redirect them, or turn them off. * The gateway passes its export settings to compute drivers and sandbox supervisors. Supervisors run in each sandbox and do not inherit the gateway's environment, so they could not follow a setting that exists only there. | Variable | Reason | | --------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `OTEL_EXPORTER_OTLP_ENDPOINT`, `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT`, `OTEL_EXPORTER_OTLP_LOGS_ENDPOINT` | Enablement has one source, `[openshell.gateway.otlp]`, so an environment variable cannot silently turn export on or redirect it. | | `OTEL_SDK_DISABLED`, `OTEL_TRACES_EXPORTER`, `OTEL_LOGS_EXPORTER`, `OTEL_METRICS_EXPORTER` | `[openshell.gateway.otlp]` decides which signals are exported. To disable export, omit it. | | `OTEL_EXPORTER_OTLP_PROTOCOL`, `OTEL_EXPORTER_OTLP_TRACES_PROTOCOL`, `OTEL_EXPORTER_OTLP_LOGS_PROTOCOL` | `protocol` in each table selects the transport, and supervisors follow the same setting. | | `OTEL_EXPORTER_OTLP_INSECURE`, `OTEL_EXPORTER_OTLP_CERTIFICATE`, `OTEL_EXPORTER_OTLP_CLIENT_CERTIFICATE`, `OTEL_EXPORTER_OTLP_CLIENT_KEY`, and their per-signal forms | The endpoint scheme selects TLS, and `ca_file`, `cert_file`, and `key_file` in each table set its trust and client certificate, so supervisors follow the same setting. | | `OTEL_PROPAGATORS` | The gateway always propagates W3C Trace Context. | | `OTEL_BLRP_SCHEDULE_DELAY`, `OTEL_BLRP_MAX_QUEUE_SIZE`, `OTEL_BLRP_MAX_EXPORT_BATCH_SIZE`, `OTEL_BLRP_EXPORT_TIMEOUT` | OCSF log export uses its own queue and batching, described in [OCSF Log Export](#ocsf-log-export). | | `OTEL_EXPORTER_OTLP_METRICS_*`, `OTEL_METRIC_EXPORT_INTERVAL` | The gateway serves metrics for Prometheus scraping and does not export them over OTLP. | To sample 10% of traces: ```shell OTEL_TRACES_SAMPLER=parentbased_traceidratio OTEL_TRACES_SAMPLER_ARG=0.1 ``` The gateway flushes buffered spans during shutdown, so spans from in-flight requests survive a `SIGTERM`. ## Supervisor Middleware Services Register operator-run supervisor middleware services with one or more `[[openshell.supervisor.middleware]]` entries. Registration is static and operator-owned; changing it requires restarting the gateway. ```toml [[openshell.supervisor.middleware]] name = "local-content-guard" grpc_endpoint = "https://host.openshell.internal:50051" tls_ca_cert_path = "/etc/openshell/certs/content-guard-ca.pem" audience = "urn:openshell:middleware:local-content-guard" max_payload_bytes = 262144 timeout = "500ms" ``` Each service advertises operation and phase pairs through `Describe`. A sandbox policy attaches the service by its registration `name`; the gateway operator controls which bindings that attachment can use. Registration names must be unique and cannot start with the reserved `openshell/` namespace. The name in the service manifest is for diagnostics only. | `binding_policy` | What the gateway and supervisor require | | ------------------- | ------------------------------------------------------------------------------------------------ | | `dynamic` (default) | Use all valid advertised bindings. Omit `bindings`. A service update can change which hooks run. | | `allowlist` | Require every entry in `bindings`. Ignore additional valid bindings the service advertises. | | `exact` | Require the service to advertise exactly the entries in `bindings`. | For example, add the following lines to the registration above to require a request hook even if the service also offers a response hook: ```toml binding_policy = "allowlist" bindings = ["http_request_pre_credentials"] ``` The other supported bindings are `http_response_pre_return` and `websocket_message_pre_credentials`. Use each binding at most once. To require the check on matching traffic, attach the service in sandbox policy and keep the default `on_error: fail_closed` behavior. Upgrade the gateway and all supervisors before relying on `allowlist` or `exact`. Older supervisors ignore these fields. A running supervisor that rejects a configuration update keeps its last accepted configuration; confirm the new configuration is active before depending on the change. The gateway connects to every registered service and validates `Describe` before it starts. The service must therefore be running before the gateway. Policy creation and full policy updates call `ValidateConfig`; an unavailable service or invalid middleware configuration rejects the policy before persistence. Startup also requires compatible [extension protocol metadata](/openshell/v0.1.7/extensibility/overview#protocol-negotiation). Upgrade legacy gateways and operator-run services together during a coordinated outage; neither mixed legacy/current pairing is supported. A service rejects a gateway that omits peer metadata, and a gateway rejects a service that omits metadata, uses another protocol major, or requires unsupported gateway capabilities. Protected `GetGatewayInfo` and `openshell gateway info` report the resulting non-secret snapshot. `max_payload_bytes` is the shared operator limit for inspectable logical payloads across the bindings authorized for this service. It caps HTTP request and response units, replacement bodies, and complete WebSocket text messages and replacements. Whole-response inspection uses it as the stage's total body limit. Streaming response inspection applies it to each unit. The value must be greater than zero, no larger than each binding's advertised `max_payload_bytes` capability, and no larger than the 4 MiB platform maximum. OpenShell rejects oversized values instead of silently clamping them. Binary WebSocket messages are not exposed to V1 middleware, so this field does not limit binary pass-through. Middleware gRPC servers should allow messages of at least 4 MiB plus 293 KiB so a maximum-size payload and its protobuf envelope fit on the transport. `timeout` is the operator-configured service-wide RPC timeout. It accepts the same compact duration syntax as gateway interceptors: an integer followed by `ms` or `s`, such as `500ms` or `2s`. Values must be between `10ms` and `30s`, inclusive. Omit the field to use the 500 ms platform default. A binding may advertise a shorter `timeout` in the `Describe` manifest, but it cannot extend the operator-configured deadline; OpenShell uses the smaller value. OpenShell validates both levels before accepting the service. The operator-configured service timeout applies to `Describe` and `ValidateConfig`. The effective binding timeout applies to HTTP request evaluation, HTTP response preflight and unit exchanges, WebSocket preflight, and each WebSocket message. Accepted streaming protocols have no connection-wide RPC deadline. The service `grpc_endpoint` supports plaintext `http://` and TLS `https://`. HTTPS uses the platform trust store unless `tls_ca_cert_path` names a certificate-only PEM bundle. OpenShell rejects bundles containing private keys, loads the certificates at gateway startup, and distributes only public certificates to sandbox supervisors; normal TLS hostname verification still applies. `audience` sets the exact audience for gateway-minted service tokens and defaults to `urn:openshell:extension:middleware:`. After authenticated `Describe` succeeds, OpenShell treats a non-empty manifest `expected_audience` as a consistency assertion and refuses to start when it differs from the configured audience. A strict verifier may reject an incorrect audience before returning the manifest. When `gateway_jwt` is configured, OpenShell attaches short-lived bearer credentials to gateway and supervisor calls and requires `https://`. A middleware endpoint must be reachable from sandbox supervisors, so Unix sockets are not an option here. Set `allow_insecure_transport = true` on a registration to keep a plaintext `http://` endpoint: OpenShell then attaches no credential, supervisors do not request one, and the gateway logs a warning naming the registration at every startup. mTLS client authentication, health checks, and runtime registration are not currently supported. The endpoint must be reachable from both the gateway and sandbox supervisors; use `host.openshell.internal` or another shared address that can be resolved in both places. See [Supervisor Middleware Configuration](/openshell/v0.1.7/extensibility/supervisor-middleware/configure) for attachment, failure, and operational guidance. ## Gateway Interceptors `[[openshell.gateway.interceptors]]` configures gateway-side interceptor services. The gateway calls each service's `Describe` RPC at startup, validates its declared OpenShell RPC bindings against the compiled service descriptor, and applies matching phases from a central gRPC middleware path. Interceptors can target only methods in the gateway's built-in allowlist of unary mutation RPCs. New RPCs are non-interceptable until they are deliberately added to that allowlist; adding one does not require handler-specific interceptor code. Request bodies are exposed as protobuf JSON objects. Fields marked secret in the protobuf schema are recursively omitted from requests and post-commit responses. Interceptors cannot patch an omitted field or a containing object. Each interceptor must also complete [extension protocol negotiation](/openshell/v0.1.7/extensibility/overview#protocol-negotiation) during `Describe`. Upgrade legacy gateways and interceptors together during a coordinated outage; neither mixed legacy/current pairing is supported. The interceptor rejects missing or incompatible gateway metadata, and the gateway rejects missing metadata, incompatible protocol majors, and unmet requirements before serving requests. HTTPS interceptor endpoints use the platform trust store by default. Set `tls_ca_cert_path` to a PEM certificate bundle for a private CA; normal TLS hostname verification still applies. `audience` sets the exact audience for gateway-minted service tokens and defaults to `urn:openshell:extension:interceptor:`. After authenticated `Describe` succeeds, the gateway treats a non-empty manifest `expected_audience` as a consistency assertion and refuses to start when it differs from the configured audience. A strict verifier may reject an incorrect audience before returning the manifest. When `gateway_jwt` is configured, network interceptors must use HTTPS and receive short-lived gateway-caller bearer credentials; local Unix sockets are also supported. Set `allow_insecure_transport = true` to keep a plaintext `http://` interceptor endpoint with no credential attached and a startup warning. ### Extension Token Verification Endpoints When `gateway_jwt` is configured the gateway publishes two unauthenticated documents that extension services use to verify OpenShell callers: | Path | Contents | | ----------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------- | | `/.well-known/jwks.json` | Single-key JWKS holding the Ed25519 public key and its `kid`. | | `/.well-known/openid-configuration` | OIDC-shaped discovery metadata: the exact expected `iss`, an absolute `jwks_uri`, and `EdDSA` as the only supported signing algorithm. | The discovery document is OIDC-shaped rather than OIDC-compliant: `issuer` is the gateway identity (`openshell-gateway:`), not the URL the document is served from. Verifiers compare `iss` against that value and must not infer trust from the document's location; the serving TLS connection is what authenticates the gateway. Both endpoints return `404` when `gateway_jwt` is not configured. `binding_policy` controls how the manifest and operator binding configuration combine: * `dynamic` enables valid manifest bindings and treats configured entries as optional narrowing overrides. This is the compatibility default. The gateway logs a startup warning because the interceptor controls its non-secret RPC authority. * `allowlist` enables only configured RPC selectors and phases. The gateway ignores and logs extra manifest declarations, but fails startup when the manifest omits a configured RPC or phase. * `exact` requires the configured and manifest RPC selectors and phases to match exactly. Bindings under `allowlist` and `exact` require `rpc` or `service` plus `method`, and a nonempty `phases` list. They match by RPC rather than manifest binding ID. Duplicate selectors, `id`, and `disabled` are invalid in these modes; omit a binding to disable it. Binding failure policy comes from the binding, then the configured service, then defaults to `fail_closed`. Manifest failure policies do not override strict-mode operator configuration. Bindings that include `post_commit` must resolve to `failure_policy = "fail_open"`; the gateway rejects fail-closed post-commit configuration at startup. Post-commit evaluation is observational. If response observation or interceptor evaluation fails after a handler succeeds, the gateway logs and counts the failure but returns the committed response unchanged. `provider_profile_sources` selects the exact ordered provider-profile source set. Omit it to use imported profiles only. Use `{ type = "user" }` and `{ type = "interceptor", name = "" }` entries to compose a catalog. An interceptor entry must reference a configured service whose manifest advertises `provider_profiles = true`. Selecting only an interceptor makes that catalog authoritative; include `{ type = "user" }` explicitly to compose imported profiles with it. Empty or duplicate source lists, unknown interceptor names, and duplicate normalized profile IDs fail closed. Source order controls collection and diagnostics, not override precedence. OpenShell ships no profiles inside the gateway binary. A gateway with nothing imported reaches ready and serves an empty catalog; import the profiles it needs with `openshell provider profile import`. The `builtin` source type was removed and a configuration that still names it is rejected at startup. For an authoritative interceptor catalog, select only that interceptor: ```toml [openshell.gateway] provider_profile_sources = [ { type = "interceptor", name = "provider-governance" }, ] ``` The gateway validates snapshot structure and provider-profile semantics. It treats a configured interceptor as the source trust boundary and does not verify signature, hash, or key annotations in profile payloads. `failure_policy` accepts `fail_closed` or `fail_open`. `timeout` accepts `ms` and `s` suffixes. In `dynamic` mode, binding overrides may select a manifest binding by `id`, `rpc`, or `service` plus `method`; they can disable a binding, narrow its phases, or override its failure policy. `image_pull_policy` is a shared driver setting with the canonical values `always`, `if_not_present`, `never`, and `newer`. Set it inside the relevant driver table. Drivers translate these values to their runtime APIs; `newer` is supported only by Podman and is rejected at Docker and Kubernetes startup. ## Credential Drivers Set `credential_drivers` only when the gateway should store provider secrets in an external credential backend. Provider secrets include injectable credentials and gateway-only refresh material such as OAuth refresh tokens, client secrets, and service-account private keys. OpenShell supports at most one enabled credential driver at a time. When `credential_drivers` is omitted, the gateway uses its default encrypted database credential storage. `credential_drivers = []` is invalid in the TOML file; omit the field for the default encrypted store, or select a backend such as `kubernetes-secrets` or `vault`. Credential driver tables are backend-owned and live under `[openshell.credential_drivers.]`. Built-in drivers default to in-tree transport, so they do not need a `transport` field. Use `transport = "uds"` with an absolute `socket_path` only for a remote gRPC driver over a Unix domain socket. ```toml [openshell.gateway.credential_storage] key_encryption_key_path = "/var/lib/openshell/credentials/key-encryption-key.bin" ``` For Kubernetes Secrets: ```toml [openshell.gateway] credential_drivers = ["kubernetes-secrets"] [openshell.credential_drivers.kubernetes-secrets] namespace = "openshell" ``` For Vault instead: ```toml [openshell.gateway] credential_drivers = ["vault"] [openshell.credential_drivers.vault] address = "https://vault.vault.svc.cluster.local:8200" ca_bundle = "/etc/openshell-tls/vault-ca/ca.crt" mount = "secret" kv_version = "2" auth_method = "kubernetes" role = "openshell-gateway" service_account_token_path = "/var/run/secrets/kubernetes.io/serviceaccount/token" ``` For the default encrypted database store, OpenShell stores provider credentials as JSON envelopes encrypted with AES-256-GCM in the gateway database. Each credential gets a random data-encryption key; the gateway wraps that key with a local key-encryption key. By default, the key-encryption key is created at `$XDG_STATE_HOME/openshell/gateway/credentials/key-encryption-key.bin` with owner-only permissions. Use `[openshell.gateway.credential_storage] key_encryption_key_env` instead of `key_encryption_key_path` to load a base64-encoded 32-byte key-encryption key from an environment variable. Back up the database and key-encryption key together; losing either makes stored credentials unrecoverable. In Kubernetes, the Helm chart creates a retained Secret containing the shared key-encryption key, injects it as `OPENSHELL_GATEWAY_CREDENTIAL_KEY_ENCRYPTION_KEY`, and renders `key_encryption_key_env` for the gateway when no external credential driver is enabled. Multi-replica deployments need every replica to use the same database and key-encryption key; the chart default handles the key-encryption key side. For GitOps and `helm template` workflows where `lookup` returns empty, the chart-generated KEK Secret gets a random value on every render, making credentials unrecoverable. Set `server.credentialStorage.existingSecret` to the name of a pre-provisioned Secret containing the key-encryption key under the `key-encryption-key` data key. When set, the chart skips KEK Secret generation and references the provided Secret directly. ```yaml server: credentialStorage: existingSecret: my-preprovisioned-kek-secret ``` For `kubernetes-secrets`, `namespace` sets where OpenShell-managed provider Secret objects are stored in every workspace mode. When omitted, the driver uses the in-cluster ServiceAccount namespace when available, otherwise `default`. The driver rejects credential handles that reference any other namespace. The Helm chart creates a Role granting the gateway access to all Secrets in the credential namespace because OpenShell-managed Secret names are dynamic SHA-256 hashes that cannot be restricted with `resourceNames`; the gateway ClusterRole grants no credential Secret permissions. Deploy credential Secrets in a dedicated namespace (`server.credentialDrivers.kubernetesSecrets.namespace`) to limit the RBAC blast radius. When the credential namespace is the release namespace, the Role also covers the gateway's own TLS and database Secrets. Set `server.credentialDrivers.kubernetesSecrets.createNamespace=true` to have the chart create the dedicated namespace. The chart retains it on uninstall so stored credentials survive, and leaves an existing namespace it does not own untouched. For `vault`, `address` points at the Vault service. Non-loopback endpoints must use HTTPS; plaintext HTTP is accepted only for `localhost` or an IP loopback address during local development. Vault requests do not follow redirects, preventing credentials from being replayed to a downgraded or substituted endpoint. HTTPS uses platform trust roots by default. Set `ca_bundle` to a certificate-only PEM bundle for a private Vault CA; the bundle augments platform roots and normal hostname verification remains enabled. `mount` and `kv_version` describe the KV engine where OpenShell-managed provider secrets are stored, and `auth_method = "kubernetes"` logs in with the gateway Pod's ServiceAccount token. For local or development validation, use `auth_method = "token_file"` with `token_path = "/path/to/token"`. Do not put literal Vault tokens in TOML. For Helm deployments, set `credentialDrivers.vault.caConfigMapName` to a ConfigMap containing the private CA bundle under the `ca.crt` key. The chart mounts that key and derives `ca_bundle` automatically. Provider records that already contain inline database credentials remain readable for upgrade compatibility. New provider create/update requests still submit credential values through the normal API, but the gateway stores those values through the active credential storage path and persists only handles. Before OpenShell 0.1.0, OpenShell does not automatically migrate inline refresh material or credential handles between drivers. Reconfigure refresh grants after an upgrade. Before changing credential drivers, remove affected credentials while the original driver is still available, then select the new driver and create them again. Do not run mixed gateway versions against the same refresh records. For remote credential drivers, set `transport = "uds"` with `socket_path`. Omit `command`, `args`, and `startup_timeout_secs` when another service manager prestarts the driver socket. Keep backend tokens out of TOML; point the driver at mounted token files or native identity mechanisms instead. The built-in `kubernetes-secrets` and `vault` drivers can also run out of process over UDS. Set `command` to the standalone driver binary and pass driver-specific settings through `args`; the gateway appends `--bind-socket ` when it launches the process. ```toml [openshell.gateway] credential_drivers = ["kubernetes-secrets"] [openshell.credential_drivers.kubernetes-secrets] transport = "uds" socket_path = "/run/openshell/credential-drivers/kubernetes-secrets.sock" command = "/usr/libexec/openshell/openshell-driver-kubernetes-secrets" args = ["--namespace", "openshell"] ``` ```toml [openshell.gateway] credential_drivers = ["vault"] [openshell.credential_drivers.vault] transport = "uds" socket_path = "/run/openshell/credential-drivers/vault.sock" command = "/usr/libexec/openshell/openshell-driver-vault" args = [ "--address", "https://vault.vault.svc.cluster.local:8200", "--ca-bundle", "/etc/openshell/vault/ca.pem", "--auth-method", "kubernetes", "--role", "openshell-gateway", ] ``` ## Gateway Peer Routing Multi-replica gateways relay session-bound traffic to the replica that owns a sandbox supervisor. The Helm chart sets these variables; set them manually only outside the chart. | Variable | Purpose | | -------------------------------------------------------------- | --------------------------------------------------------------------------------------- | | `OPENSHELL_PEER_ENDPOINT` | Address other replicas use to reach this one. Derived from the pod DNS name when unset. | | `OPENSHELL_PEER_SERVICE_NAME` | Peer Service used to derive the endpoint. | | `OPENSHELL_POD_NAME` / `OPENSHELL_POD_NAMESPACE` | Pod identity used for endpoint derivation and peer verification. | | `OPENSHELL_SERVICE_ACCOUNT_NAME` | ServiceAccount that peer tokens must present. | | `OPENSHELL_PEER_SERVICE_ACCOUNT_TOKEN_FILE` | Projected peer token path. Defaults to `/var/run/secrets/openshell-peer/token`. | | `OPENSHELL_PEER_TOKEN_AUDIENCE` | Required token audience. Defaults to `openshell-gateway-peer`. | | `OPENSHELL_PEER_POD_LABELS` | Comma-separated labels a calling pod must carry. | | `OPENSHELL_PEER_TOKEN_CACHE_TTL_SECS` | How long a verified peer identity is cached. Defaults to `60`. | | `OPENSHELL_PEER_TLS_CA_FILE` | CA bundle for peer connections. Falls back to the platform trust store. | | `OPENSHELL_PEER_TLS_CERT_FILE` / `OPENSHELL_PEER_TLS_KEY_FILE` | Client certificate and key for peer mTLS. Set both or neither. | | `OPENSHELL_PEER_TLS_SERVER_NAME` | Name verified on peer certificates. | The gateway refuses to start when a peer endpoint is configured on a multi-replica backend and peer authentication is unavailable, because the replica could not serve relays from its peers. That happens when the pod identity variables are missing, when the in-cluster Kubernetes client cannot be built, or when the gateway is not running in a cluster. A plaintext `http://` peer endpoint is also rejected when the gateway serves TLS; use it only when the gateway itself serves plaintext. ## Driver References ### External Resource Admission Every compute driver disables caller-supplied `template.driver_config` by default. An operator can enable it without disabling resource approval: ```toml [openshell.drivers.kubernetes] allow_driver_config = true [openshell.drivers.kubernetes.resource_admission] enabled = true [openshell.drivers.kubernetes.resource_admission.required_labels] "openshell.ai/sandbox-attachable" = "true" "openshell.ai/sandbox-attachable-workspace" = "${workspace}" ``` Use the same fields under the selected Docker, Podman, VM, MXC, or external driver table. Omitted admission settings use the labels above. A supplied map replaces the defaults, and every entry must match. An explicitly empty map is invalid while enforcement is enabled. `${workspace}` is the entire label value and resolves to the authorized OpenShell workspace name, not a Kubernetes namespace or caller label. Other substitutions and expressions are unsupported. At least one fixed label is required so intentionally shared resources still need explicit operator approval. The following label keys are reserved for driver-owned metadata and cannot be used in `required_labels`: `openshell.ai/managed-by`, `openshell.ai/gateway-id`, and `openshell.ai/sandbox-workspace`. The default admission key `openshell.ai/sandbox-attachable-workspace` is distinct from the reserved sandbox metadata key. The operator must label the referenced resource before use. Labels on a sandbox do not approve its attachments. For example, a PVC used by workspace `team-a` needs this metadata: ```yaml metadata: labels: openshell.ai/sandbox-attachable: "true" openshell.ai/sandbox-attachable-workspace: "team-a" ``` Kubernetes checks caller-selected PVCs, RuntimeClasses, and PriorityClasses through the OpenShell-owned Pod template. Docker and Podman check engine volume labels. Read-only mounts still require approval. Namespaces do not grant approval. Newly provisioned private storage and bootstrap material use driver ownership checks instead. Kubernetes PVC mount destinations must not overlap the driver-managed `/.openshell` tree or `/run`, including mounts using `sub_path`. Approval labels do not override these restrictions. Mount application data at paths such as `/data` or a directory below `/sandbox`. RuntimeClasses and PriorityClasses are shared operator infrastructure. They must match every fixed required label, but `${workspace}` entries do not apply to them. Kubernetes API-server and admission-webhook mutations of the live Pod are trusted cluster-operator behavior and are not used as Workspace User authorization inputs. Image-pull Secrets are selected by gateway configuration, not by sandbox callers, and do not participate in resource admission. In managed mode OpenShell creates an immutable copy of each configured Secret for every sandbox runtime generation and deletes it with that generation. Sandbox creation fails if a Secret with the generation name already exists. In shared and operator modes, the configured Secret must already exist in the sandbox namespace. The typed Kubernetes driver config does not support caller-selected Secret or ConfigMap volumes or environment references. Automatic service-account tokens and projected token volumes are prohibited. GPU device attachments are temporarily exempt from label admission. Existing GPU validation still applies. Explicit GPU device settings inside driver JSON still require `allow_driver_config = true`; public GPU-count requests do not. The exception does not approve unrelated host mounts or non-GPU devices. Raw host bind mounts and supplemental image mounts have no trusted label resolver and are rejected while admission is enabled. Publisher image labels cannot grant operator approval. Docker's existing unsupported image-mount restriction remains. MXC host filesystem grants likewise require disabling admission until a resolver exists. MXC's command config and VM upload workflows encoded in driver JSON require explicit driver-config opt-in. `allow_driver_config` applies to nonempty caller config, including saved templates and non-storage overrides. An absent config or `{}` does not require opt-in. Drivers recheck admitted OpenShell input attachments before launch and on restart; running Docker, Podman, and Kubernetes workloads are periodically revalidated. Confirmed revocation stops or suspends the workload. Metadata lookup outages block new launches but do not alone stop existing workloads. Revalidation uses the Docker observation loop (normally two seconds) or a 30-second Podman / Kubernetes interval, plus lookup and retry time. These are not instantaneous revocation guarantees. To explicitly opt out of label authorization: ```toml [openshell.drivers.kubernetes.resource_admission] enabled = false ``` This does not enable driver JSON or bypass existing path, ownership, namespace, or isolation validation. It relinquishes external-resource label protection for that driver. Restart the gateway and driver after policy changes. Only trusted operators should control approval labels, runtime APIs, and resource replacement; application admission is not atomic mount authorization. Before upgrading, inventory and approve legitimate external resources. Legacy sandbox state without verifiable admission provenance requires recreation; OpenShell does not automatically approve old attachments. Remove stale labels before reusing a deleted workspace name. A custom approval-only map permits cross-workspace sharing and should be an intentional operator decision. Standalone Kubernetes, Podman, and VM drivers accept the common policy as `--admission-config-json` or `OPENSHELL_DRIVER_ADMISSION_CONFIG_JSON`; `{}` uses secure defaults. Docker also accepts the fields in its driver TOML file. The gateway-managed VM process receives the selected gateway policy automatically. Operator-managed external processes must be configured separately. Their versioned `GetCapabilities.resource_admission_policy` acknowledgement must match the gateway policy; legacy drivers require an explicit admission opt-out. A changed acknowledgement blocks subsequent remote validation/create/start calls until gateway and driver configuration agree again. Helm exposes `server.drivers.kubernetes.allowDriverConfig`, `resourceAdmission.enabled`, and `resourceAdmission.requiredLabels`. The label map defaults to `null`, which uses code defaults without merging those keys into a custom map. The Helm chart grants PVC metadata reads only when `allowDriverConfig` is enabled. Split workspace releases must set the matching `gateway.allowDriverConfig=true` workspace-chart value. Never mark gateway database storage as attachable. Each example is a complete TOML file for one compute driver. The examples repeat `[openshell]` and `[openshell.gateway]` so they stay copyable, and the driver tables list the accepted driver-specific keys. Drivers receive only their own tables, and the gateway rejects unknown gateway and driver fields. Kubernetes configurations set `namespace`, `service_account_name`, and `enable_user_namespaces` in `[openshell.drivers.kubernetes]`. Docker configurations use `sandbox_label`; the legacy `sandbox_namespace` key is rejected. ### Kubernetes The gateway runs as a Pod and creates paired workload and supervisor Pods in another namespace. The gateway CA is projected from a Kubernetes Secret into supervisor Pods; user client certificate and key entries in that Secret are not exposed to either Pod. ```toml [openshell] version = 2 [openshell.gateway] bind_address = "0.0.0.0:8080" health_bind_address = "0.0.0.0:8081" metrics_bind_address = "0.0.0.0:9090" log_level = "info" compute_driver = "kubernetes" [openshell.gateway.tls] cert_path = "/etc/openshell-tls/server/tls.crt" key_path = "/etc/openshell-tls/server/tls.key" # client_ca_path is only rendered when server.tls.enableMtls is true (the # default). When enableMtls is false — required for BackendTLSPolicy — the # gateway runs HTTPS-only and this line is omitted by Helm. client_ca_path = "/etc/openshell-tls/client-ca/ca.crt" # When cert-manager serverIssuerRef is configured, these are populated by Helm: # external_cert_path = "/etc/openshell-tls/server-external/tls.crt" # external_key_path = "/etc/openshell-tls/server-external/tls.key" # external_server_names = ["gateway.example.com"] [openshell.drivers.kubernetes] # Workspace isolation mode. "shared" renders all sandboxes into a single # namespace. "managed" auto-creates a K8s namespace per workspace # (openshell-{gateway_id}-{workspace}). "operator" maps each workspace to a # pre-provisioned namespace discovered via label selector or drop-in file. workspace_mode = "shared" # Gateway identity used in managed-mode namespace naming. Defaults to the # gateway JWT gateway_id. Must be a DNS-1123 label. # gateway_id = "openshell" namespace = "agents" service_account_name = "openshell-sandbox" default_image = "nvcr.io/nvidia/base/ubuntu:24.04" image_pull_policy = "if_not_present" image_pull_secrets = ["regcred"] # Defaults to the gateway version; override to pin a specific build. # sandbox_runtime_image = "ghcr.io/nvidia/openshell/sandbox:" sandbox_runtime_image_pull_policy = "if_not_present" # Defaults to the gateway version; override to pin a specific build. # supervisor_image = "ghcr.io/nvidia/openshell/supervisor:" supervisor_image_pull_policy = "if_not_present" # Optional corporate forward proxy for policy-approved TLS egress. The sandbox # workload cannot select or override these settings. http:// and https:// proxy # endpoints are supported, for TLS CONNECT traffic only; plain HTTP egress # remains direct. `no_proxy` bypasses only the corporate proxy, never OpenShell # policy. # https_proxy = "http://proxy.corp.example:8080" # no_proxy = ".svc,.svc.cluster.local,10.96.0.0/12,10.244.0.0/16" # Proxy credentials must be an existing Secret in the sandbox namespace. The # key contains a `user:pass` value and is mounted only in the network # supervisor container, never in workload environment or command arguments. # proxy_auth_secret_name = "corporate-proxy-auth" # proxy_auth_secret_key = "credentials" # The gateway validates the Secret name/key syntax and their configuration # relationship at startup; it does not read the Secret from the Kubernetes API. # Kubernetes resolves the Secret when the Sandbox Pod starts. A missing key or # Secret prevents that Pod from starting; unreadable or malformed `user:pass` # content is validated fail-closed by the supervisor at startup and never # falls back to direct egress. # Proxy credentials mount only in the separately scheduled supervisor Pod. # Required with a credential Secret when the proxy URL is http://, because # Basic authentication is then cleartext on the connection to that proxy. An # https:// proxy carries the credential inside the verified TLS session and # needs no acknowledgement. # proxy_auth_allow_insecure = true # Last resort for hostname-filtering proxy ACLs. The proxy resolves the target, # so its ACL becomes part of the egress boundary for proxied connections. # proxy_connect_by_hostname = true # CA bundle trusted for the corporate proxy, as a path on the gateway Pod's # filesystem. Needed for an https:// proxy whose certificate is not publicly # trusted, and for a TLS-intercepting proxy that re-signs upstream # certificates. The gateway reads the file and stages it into each sandbox's # immutable supervisor bootstrap Secret, so the anchor stays in the gateway's # trust domain and cannot change underneath a running sandbox. Helm mounts it # from `upstreamProxy.caBundle.configMapName` and renders this path. Supply # only the proxy's own CA: public roots already come from the supervisor image # and its TLS stack, so a full merged trust bundle wastes the sandbox # boundary's control-frame budget on duplicated roots. # proxy_ca_bundle = "/etc/openshell-tls/proxy-ca/ca.crt" # Required in raw gateway TOML because `namespace` identifies sandbox # placement, not the gateway Service. Helm renders this from the release's # gateway Service name and namespace. grpc_endpoint = "https://openshell-gateway.openshell.svc:8080" ssh_socket_path = "/run/openshell/ssh.sock" client_tls_secret_name = "openshell-client-tls" host_gateway_ip = "10.0.0.1" enable_user_namespaces = false workspace_default_storage_size = "10Gi" # Kubernetes StorageClass for the workspace PVC. Empty (default) omits the # field, using the cluster's default StorageClass. Set this on clusters with no # default StorageClass, otherwise the workspace PVC stays Pending. # workspace_storage_class = "fast-ssd" # Kubernetes RuntimeClass applied to sandbox pods when the API request does # not specify one. Empty (default) = omit the field, using the cluster default. # default_runtime_class_name = "kata-containers" # Pod annotation keys that sandbox templates may set. Each entry is an exact # key or a `/*` wildcard. Empty (default) rejects every template # annotation. Keys under openshell.ai (OpenShell sandbox identity) and # agents.x-k8s.io (owned by the Agent Sandbox controller), including # subdomains, are reserved: templates cannot set them and the allowlist # cannot include them. # allowed_pod_annotations = ["example.com/*", "team.example.org/owner"] # Pod label keys that sandbox templates may set, in the same format and with # the same reserved domains. Empty (default) rejects every template label. # Labels select pods for NetworkPolicy rules and Services, so only allow keys # that no other rule in the sandbox namespace selects on. # allowed_pod_labels = ["app", "team.example.org/*"] # Kubelet clamps projected tokens below 600 seconds. The driver caps values at 86400. sa_token_ttl_secs = 3600 # Optional SPIFFE Workload API socket mounted into sandbox pods for dynamic # provider token grants. Use an absolute path under a dedicated directory; # shared roots such as /run, /var, /tmp, and /etc are rejected. # Supervisor-to-gateway auth still uses gateway JWTs. provider_spiffe_workload_api_socket_path = "/spiffe-workload-api/spire-agent.sock" # Explicit sandbox UID/GID for the supervisor container securityContext and # PVC init container. When unset, the driver auto-detects from OpenShift SCC # namespace annotations (openshift.io/sa.scc.uid-range) if present, falling # back to 1000 on non-OpenShift clusters. Any non-root Linux UID/GID is valid. # Helm sets these with server.sandboxUid and server.sandboxGid. # sandbox_uid = 1500 # sandbox_gid = 1500 # Operator-mode namespace discovery. At least one must be set when # workspace_mode = "operator". Both can be combined. # operator_namespace_label discovers namespaces matching a K8s label selector. # operator_namespace_label = "openshell.ai/workspace=true" # operator_namespace_file reads allowed namespaces from a JSON/YAML file # (hot-reloaded on change, e.g. via ConfigMap volume mount). # operator_namespace_file = "/etc/openshell/workspace-namespaces.json" [openshell.drivers.kubernetes.managed_ssh_ingress] enabled = true gateway_namespace = "openshell" gateway_pod_selector = { "app.kubernetes.io/name" = "openshell", "app.kubernetes.io/instance" = "openshell" } [openshell.drivers.kubernetes.sandbox_runtime] # TLS-protected boundary listener reached only by the paired control pod. boundary_port = 5500 ``` The Kubernetes driver always creates the workload `NetworkPolicy` before releasing a sandbox Pod. Your cluster CNI must enforce ingress and egress `NetworkPolicy` in every sandbox namespace; the Kubernetes API does not verify enforcement. In managed workspace mode, the Kubernetes driver copies each explicitly named `image_pull_secrets` Secret from `namespace` into the managed workspace namespace on sandbox creation. Shared and operator modes require the Secret to already exist in the sandbox namespace. For token-exchange provider profiles, the gateway also needs access to its own SPIFFE Workload API socket. In Helm deployments, set `server.providerTokenGrants.spiffe.enabled=true`; the chart mounts the socket into the gateway pod and sets `OPENSHELL_GATEWAY_SPIFFE_WORKLOAD_API_SOCKET`. The gateway verifies supervisor JWT-SVIDs with JWT bundles fetched from the SPIFFE Workload API, so this validation path does not require gateway access to the SPIRE OIDC discovery endpoint or its TLS CA. ### MXC The MXC driver runs native Windows workloads through `wxc-exec`. Use this configuration with a build from the [`windows` branch](https://github.com/NVIDIA/OpenShell/tree/windows). Keep `openshell-supervisor.exe` and `openshell-windows-sandbox.exe` beside the gateway. ```toml [openshell] version = 2 [openshell.gateway] bind_address = "127.0.0.1:17670" compute_driver = "mxc" # Required when gateway TLS is enabled. guest_tls_ca = "C:\\OpenShell\\certs\\ca.pem" [openshell.drivers.mxc] wxc_exec_path = "C:\\mxc\\wxc-exec.exe" backend = "process_container" # Administrator opt-in for per-sandbox commands and working directories. allow_driver_config = true etw_audit = true ``` `etw_audit` defaults to `false`. Enabling it requires an administrator account or membership in Windows Performance Log Users. See [OCSF JSON Export](/openshell/v0.1.7/observability/ocsf-json-export) for durable audit output. Follow the [Windows configuration guide](https://github.com/NVIDIA/OpenShell/blob/windows/docs/how-it-works/gateways/configuration.mdx#mxc) for host prerequisites, TLS setup, sandbox creation, and runtime limitations. ### Docker Each Docker sandbox uses a workload container with networking disabled and a host-networked supervisor container. Configure the supervisor gateway CA once under `[openshell.gateway]`; the gateway validates and injects it into the selected local driver. The supervisor authenticates RPCs with a sandbox bearer token. ```toml [openshell] version = 2 [openshell.gateway] bind_address = "127.0.0.1:17670" log_level = "info" compute_driver = "docker" # Gateway-owned CA injected into the selected local driver. guest_tls_ca = "/etc/openshell/certs/ca.pem" [openshell.drivers.docker] socket_path = "/var/run/docker.sock" default_image = "nvcr.io/nvidia/base/ubuntu:24.04" # Canonical values: always | if_not_present | never. `newer` is Podman-only. image_pull_policy = "if_not_present" # Value assigned to the openshell.sandbox_namespace label on sandbox containers. sandbox_label = "docker-dev" # host (default) or bridge networking for the supervisor. # Operator-owned; sandbox driver config cannot override this choice. supervisor_network_mode = "host" # Optional override; Docker's host address is discovered automatically. # host_gateway_ip = "172.29.0.254" # Optional override. Omit for automatic host or bridge callback selection. # grpc_endpoint = "https://gateway.example.com:17670" # Workload-side runtime. Defaults to the gateway version. # sandbox_runtime_image = "ghcr.io/nvidia/openshell/sandbox:" # Supervisor runtime. Defaults to the gateway version. # supervisor_image = "ghcr.io/nvidia/openshell/supervisor:" # Unsafe operator override. Host bind mounts, including Docker local-driver # bind-backed volumes, expose gateway-host paths inside sandboxes and can # negate OpenShell isolation and filesystem controls. enable_bind_mounts = false # Omit to use OpenShell's 2048-process default. Explicit 0 is invalid. sandbox_pids_limit = 2048 # Explicit supervisor-compatible default. RuntimeDefault requires Docker to # report AppArmor support; Localhost/ requires an operator-loaded profile. app_armor_profile = "Unconfined" # Corporate TLS egress proxy. These are supervisor argv settings, not workload # environment variables. Do not embed credentials in the URL. https_proxy = "https://proxy.corp.example:8443" no_proxy = ".svc.cluster.local,10.0.0.0/8" # Optional root-owned host file containing user:pass. An http:// proxy also # requires proxy_auth_allow_insecure = true as an explicit acknowledgement. proxy_auth_file = "/etc/openshell/secrets/proxy-auth" proxy_auth_allow_insecure = false # Last resort for proxy ACLs that require hostname CONNECT targets. The proxy # then performs DNS resolution and becomes part of the effective egress boundary. proxy_connect_by_hostname = false # Operator-owned gateway-host PEM bundle trusted for an HTTPS proxy and for # destination certificates re-signed by a TLS-intercepting proxy. proxy_ca_bundle = "/etc/openshell/tls/proxy-ca.pem" # Project a host Unix Workload API socket into the supervisor for provider # token exchange. The socket parent must be a dedicated absolute directory. provider_spiffe_workload_api_socket = "/run/spire/agent.sock" ``` Use `sandbox_label` for Docker configurations. `supervisor_network_mode` accepts `host` (the default), `bridge`, or `auto`. `auto` also selects host; the driver does not infer a mode from the workload runtime. The workload always uses `network=none`. Omit `grpc_endpoint` for automatic selection: host mode uses loopback; bridge uses `host.docker.internal` on Desktop or with a wildcard listener, otherwise the Linux listener's non-loopback address. Docker's host address is discovered automatically for bridge mode and gateway endpoints using `host.docker.internal`. Set `host_gateway_ip` only to override it. In bridge mode, host services and the configured OTLP collector endpoint must be reachable from containers; native Linux loopback-only listeners are unreachable. Host mode on Docker Desktop requires host networking enabled. Docker accepts `http://` and `https://` proxy URLs in explicit `scheme://host:port` form. `no_proxy` bypasses only the corporate proxy; OpenShell policy still applies. Proxy URLs cannot embed credentials. Supply a `user:pass` file with `proxy_auth_file`, and set `proxy_auth_allow_insecure = true` only to acknowledge that Basic credentials travel in cleartext to an `http://` proxy. `proxy_ca_bundle` requires `https_proxy`, but the configured proxy URL may use either scheme because a plain HTTP proxy can still intercept destination TLS. The file must be a readable, bounded PEM bundle containing a usable certificate authority. Docker validates it at gateway startup and again before each supervisor launch. Missing, unreadable, empty, oversized, malformed, and certificate-free bundles fail closed rather than falling back to default trust or direct egress. The gateway-host path is operator-owned and cannot be selected through sandbox environment, image contents, or `template.driver_config.docker`. The driver copies the validated contents into its supervisor-only named volume and passes the fixed path `/.openshell/supervisor/upstream-proxy-ca-bundle.pem` to the companion supervisor. The bundle extends trust for the TLS connection to an `https://` proxy, supervisor connections to re-signed upstream certificates, and the combined trust bundle exposed to workload processes. The gateway-host path is never mounted into or exposed to the workload container. ### Podman Each Podman sandbox uses two containers. The workload container runs `openshell-sandbox` with `network=none`; the supervisor container runs on the host network and initiates policy-approved upstream connections. A private volume carries their authenticated Unix-domain socket. Configure the supervisor gateway CA once under `[openshell.gateway]`; the gateway validates and injects it into the selected local driver. The supervisor authenticates RPCs with a sandbox bearer token. Caller `cdi_devices` entries must be valid CDI qualified names (`/=`). Host-device paths are rejected. See [GPU selection](/openshell/v0.1.7/how-it-works/sandboxes/overview#gpu-resources). ```toml [openshell] version = 2 [openshell.gateway] bind_address = "127.0.0.1:17670" log_level = "info" compute_driver = "podman" # Gateway-owned CA injected into the selected local driver. guest_tls_ca = "/etc/openshell/certs/ca.pem" [openshell.drivers.podman] network_name = "openshell" # Rootless socket path. For root Podman use /run/podman/podman.sock. # Omit to auto-detect: the driver probes for a responsive Podman socket, then # asks the podman CLI where its socket is, and fails to start if neither finds # one. Set this to pin a specific Podman machine instead. socket_path = "/run/user/1000/podman/podman.sock" default_image = "nvcr.io/nvidia/base/ubuntu:24.04" image_pull_policy = "if_not_present" # always | if_not_present | never | newer # Optional override. When omitted, Linux uses gateway loopback and Podman # Machine uses host.containers.internal. grpc_endpoint = "https://127.0.0.1:17670" # The gateway overwrites gateway_port from bind_address at runtime. gateway_port = 17670 # Trusted supervisor-side destination returned for host.openshell.internal. # Omit for the platform default: 127.0.0.1 on native Linux and # 192.168.127.254 on macOS Podman Machine. # host_gateway_ip = "192.168.127.254" ssh_socket_path = "/run/openshell/ssh.sock" stop_timeout_secs = 45 # Statically linked workload-side runtime. Defaults to the gateway version. # sandbox_runtime_image = "ghcr.io/nvidia/openshell/sandbox:" # Dynamically linked supervisor runtime. Defaults to the gateway version. # supervisor_image = "ghcr.io/nvidia/openshell/supervisor:" # Unsafe operator override. Host bind mounts, including Podman local-driver # bind-backed volumes, expose gateway-host paths inside sandboxes and can # negate OpenShell isolation and filesystem controls. enable_bind_mounts = false # Omit to use OpenShell's 2048-process default. Explicit 0 is invalid. sandbox_pids_limit = 2048 # Health check interval in seconds. Omit to disable health checks; explicit 0 # is invalid. Lower values detect readiness faster but increase process churn # (each check spawns a conmon subprocess). health_check_interval_secs = 10 # User namespace mode for sandbox containers. Omit to use the default. # Supported modes: auto, host, keep-id, no-map, private. # userns = "auto" # Explicit UID/GID mappings for userns = "private". Each entry is # "container_id:host_id:size". Required when mode is "private"; rejected # for other modes. Rootless Podman uses intermediate IDs (0:0:1, 1:1:65535); # rootful Podman uses absolute host IDs (0:1000:1, 1:100000:65536). # uidmap = ["0:0:1", "1:1:65535"] # gidmap = ["0:0:1", "1:1:65535"] # Corporate forward proxy for sandbox egress. When set, the in-container # supervisor chains policy-approved TLS tunnels through this proxy with HTTP # CONNECT instead of dialing destinations directly. Plain-HTTP requests are # not proxied and always dial the destination directly. http:// and https:// # proxy URLs in explicit scheme://host:port form are supported: the scheme and # port are both required, and a URL carrying a path, query, or fragment is # rejected rather than silently truncated. For an https:// proxy the supervisor # wraps the connection to the proxy in TLS before the CONNECT handshake, # verifying the proxy certificate against the built-in and system roots plus # the optional proxy_ca_bundle below. # NO_PROXY entries (hostnames, domain suffixes, IPs, CIDRs, each with an # optional :port qualifier) are dialed directly. A port-qualified entry only # bypasses that destination port. IP and CIDR entries also match hostnames # through their validated DNS resolution; such a match dials directly only # the resolved addresses inside the entry. This is an operator-owned egress # boundary: # sandbox and template environment cannot override it, and the conventional # HTTPS_PROXY/HTTP_PROXY/NO_PROXY variables a sandbox sets do not affect it. # # The CONNECT request sent to the proxy targets a validated resolved IP, # not the hostname, so the proxy performs no DNS resolution of its own and # the tunnel stays bound to the address that passed the sandbox's SSRF and # allowed_ips validation. The hostname still travels inside the tunnel (TLS # SNI, application Host), so destination servers behave normally. In # split-horizon networks, point the gateway host at the corporate resolver # so internal names validate to their internal addresses. If the proxy's # ACLs filter on hostnames and reject IP CONNECT targets, set # proxy_connect_by_hostname = true as a last resort: the proxy then # resolves the name itself, so a name that resolves differently at the # proxy (split-horizon DNS, rebinding) can reach destinations the sandbox # policy never approved, and the proxy's own ACLs become the effective # egress control for proxied TLS. # # Configuration is fail-closed: an invalid proxy URL is rejected at gateway # startup, setting no_proxy, proxy_auth_file, or proxy_connect_by_hostname # without a proxy URL is rejected as well, and a set-but-invalid value # reaching a sandbox (for example an unreadable or malformed auth file) is # fatal to that sandbox's supervisor instead of silently falling back to # direct or unauthenticated egress. # # Credentials must NOT be embedded in the URL (an inline user:pass@ is # rejected at startup, since it would be stored here and exposed in container # metadata). Instead point proxy_auth_file at a file containing "user:pass"; # the gateway delivers it to the supervisor through a root-only secret mount. # The credential must use the user:pass form (non-empty user, no control # characters); the same validation runs at sandbox-create time and in the # supervisor, so a credential accepted here is never rejected in-container. # Keep gateway.toml and the auth file owner-readable only (mode 0600). # # WARNING: with an http:// proxy the supervisor sends the credential as a # Proxy-Authorization: Basic header over the plain-TCP connection. Basic # auth is base64, not encryption: anyone on the network path between the # sandbox host and the proxy can recover the credential. Because of that, # proxy_auth_file with an http:// proxy requires the explicit # acknowledgement proxy_auth_allow_insecure = true; without it the # configuration is rejected at gateway startup. Only opt in when the path # to the proxy is a trusted network segment. For an https:// proxy the # credential is sent inside the verified TLS session, so the # acknowledgement is not required (but tolerated if set). # # proxy_ca_bundle points at a PEM CA bundle (on the gateway host) trusted for # the corporate proxy. A CA certificate is not secret, so the gateway # bind-mounts it read-only into the sandbox. It is trusted in two places: the # TLS handshake with an https:// proxy, and — because a TLS-intercepting proxy # (mitmproxy, squid ssl-bump) re-signs tunneled server certificates with the # same CA — the sandbox trust bundle and the supervisor's upstream # re-encryption, so intercepted upstream TLS keeps working and sandbox # workloads trust the re-signed certificates. It is valid with either an # http:// or https:// proxy and requires a proxy URL; the file must exist and # contain at least one certificate, or the sandbox fails closed at startup. # https_proxy = "http://proxy.corp.com:8080" # no_proxy = "*.svc.cluster.local,10.0.0.0/8" # proxy_auth_file = "/etc/openshell/secrets/proxy-auth" # proxy_auth_allow_insecure = true # Last resort for hostname-filtering proxy ACLs; see the warning above. # proxy_connect_by_hostname = true # Corporate CA trusted for an https:// proxy and TLS-intercepting proxies. # proxy_ca_bundle = "/etc/openshell/tls/proxy-ca.pem" # Project a host Workload API Unix socket into the supervisor, or use an # explicit container-reachable TCP endpoint, for provider token exchange. # provider_spiffe_workload_api_socket = "/run/spire/agent.sock" # provider_spiffe_workload_api_socket = "tcp:169.254.1.2:8081" # Omit app_armor_profile to preserve Podman's runtime-selected workload profile. # Explicit RuntimeDefault and Localhost/ require Podman to report # AppArmor support. # app_armor_profile = "RuntimeDefault" ``` Use `ssh_socket_path` for Podman configurations. The legacy `sandbox_ssh_socket_path` key is rejected. When `app_armor_profile` is omitted, OpenShell sends no override and Podman applies its runtime-selected profile. The setting applies to the workload container; the supervisor retains Podman's runtime-selected profile. ### MicroVM Each sandbox runs inside its own libkrun microVM managed by the standalone `openshell-driver-vm` subprocess. Use this driver when you want stronger isolation than container namespaces alone. The example uses explicit launch-signing paths. Point them at an existing signing bundle, or omit the `gateway_jwt` table to use local discovery. Listener and guest TLS certificates do not supply these signing credentials. ```toml [openshell] version = 2 [openshell.gateway] bind_address = "127.0.0.1:17670" log_level = "info" # VM is never auto-detected; an explicit entry here is required. compute_driver = "vm" # Gateway-owned CA injected into the selected local driver. guest_tls_ca = "/var/lib/openshell/guest-tls/ca.pem" # Sandbox launch signing is separate from listener and guest TLS. [openshell.gateway.gateway_jwt] signing_key_path = "/etc/openshell/jwt/signing.pem" public_key_path = "/etc/openshell/jwt/public.pem" kid_path = "/etc/openshell/jwt/kid" gateway_id = "openshell" [openshell.drivers.vm] state_dir = "/var/lib/openshell/vm" # Where the gateway looks for the openshell-driver-vm subprocess binary. driver_dir = "/usr/local/libexec/openshell" default_image = "nvcr.io/nvidia/base/ubuntu:24.04" # Optional override. When omitted, the gateway derives # https://host.openshell.internal: for the VM driver. grpc_endpoint = "https://host.openshell.internal:17670" # Empty falls back only to default_image. The gateway rejects the VM # configuration when both fields are empty; sandbox-requested images are never # bootstrap images. bootstrap_image = "nvcr.io/nvidia/base/ubuntu:24.04" krun_log_level = 1 vcpus = 2 mem_mib = 2048 overlay_disk_mib = 4096 # Resolved sandbox UID/GID for new rootfs /etc/passwd entries. # Defaults to the image's sandbox account, or 1000 when the account is absent; # matching GID is used if sandbox_gid is empty. Persisted overlays recover their # recorded identity, including 10001, rather than receiving a legacy fallback. # Values must fall within OpenShell's allowed non-root sandbox identity range. # sandbox_uid = 20001 # sandbox_gid = 20001 # Corporate forward proxy for sandbox egress. The keys, their semantics, and # the fail-closed contract are identical to the Podman driver above: only TLS # (CONNECT) egress is chained, plain-HTTP destination requests always dial # directly, credentials must come from proxy_auth_file rather than the URL, # an http:// proxy with credentials requires proxy_auth_allow_insecure, and # any present-but-invalid value is rejected at gateway startup rather than # degrading to a direct dial. proxy_auth_file is a path on the gateway host. # # The sandbox cannot select or override these settings. The driver passes them # only to the host supervisor. # # Reachability: a proxy on the corporate network needs no special address and # works on every VM sandbox. # # A proxy on the gateway host is addressed as # http://host.openshell.internal:. The host supervisor normalizes that # name to host loopback for both libkrun and QEMU sandboxes. # # Credentials and private CA material remain host-side with the supervisor. # https_proxy = "http://host.openshell.internal:8080" # no_proxy = "10.0.0.0/8,.internal.example" # proxy_auth_file = "/etc/openshell/secrets/proxy-auth" # An http:// proxy with proxy_auth_file requires this explicit acknowledgement: # proxy_auth_allow_insecure = true # Last resort for hostname-filtering proxy ACLs; see the Podman section above. # proxy_connect_by_hostname = true # Gateway-host PEM bundle trusted for an https:// proxy and for server # certificates re-signed by a TLS-intercepting proxy. Requires https_proxy. # proxy_ca_bundle = "/etc/openshell/tls/proxy-ca.pem" # VM guests cannot mount a host Workload API Unix socket. Configure only a # separately operated guest-reachable TCP listener and explicitly acknowledge # the exposure; host-only sockets are never exposed automatically. # provider_spiffe_workload_api_tcp_endpoint = "tcp:192.0.2.10:8081" # provider_spiffe_allow_guest_tcp = true # Unsafe operator override. Host bind mounts expose gateway-host paths inside # VM sandboxes and can negate OpenShell isolation and filesystem controls. # Requires allow_driver_config = true and # [openshell.drivers.vm.resource_admission] enabled = false; libkrun only. # enable_bind_mounts = false # Where the gateway stages rootfs tar archives for `--from ./rootfs.tar`. # Defaults to /rootfs-tar-staging. The gateway creates one # request-scoped subdirectory per staging slot and removes it after use. # rootfs_tar_staging_dir = "/var/lib/openshell/vm/rootfs-tar-staging" # Largest rootfs tar archive the driver accepts, in bytes. Defaults to 10 GiB. # Gzip archives are decompressed while staging, and the limit also bounds the # expanded tar. # rootfs_tar_max_bytes = 10737418240 ``` To generate a local bundle for discovery, run these commands in fish and retain the same environment when starting the gateway: ```shell set -gx OPENSHELL_LOCAL_TLS_DIR ~/.local/state/openshell/tls openshell-gateway generate-certs --output-dir "$OPENSHELL_LOCAL_TLS_DIR" ``` The command also installs local CLI mTLS credentials. For explicit signing configuration, set the example's signing paths to `jwt/signing.pem`, `jwt/public.pem`, and `jwt/kid` under that directory. The gateway loads those paths in preference to local discovery. Rootfs tar staging requires the gateway and the VM driver to share a filesystem and run as the same user. That holds for the managed VM driver, which the gateway starts as a subprocess. If you point `compute_driver_endpoints` at an externally managed `vm` socket owned by another user, the driver cannot read the gateway's staging directory and rootfs tar sources fail with a `FAILED_PRECONDITION` error; use a registry image reference instead. ### Extension Driver Extension drivers run outside the gateway and expose the `compute_driver.proto` gRPC service on a Unix socket. Use a non-reserved driver name; built-in names such as `vm`, `docker`, `podman`, and `kubernetes` cannot be selected through unmanaged socket endpoints. The selected driver name is the key used for driver-owned sandbox config such as `template.driver_config.`. ```toml [openshell] version = 2 [openshell.gateway] bind_address = "127.0.0.1:17670" log_level = "info" compute_driver = "kyma" [openshell.drivers.kyma] socket_path = "/run/openshell/kyma-compute-driver.sock" ``` ## Preflight package configuration \{#gateway-config-preflight} Before starting a package-managed gateway, validate the selected file without changing it: ```shell openshell-gateway config preflight --path ~/.config/openshell/gateway.toml ``` Without `--path`, the command validates a nonempty `OPENSHELL_GATEWAY_CONFIG`. Otherwise, it validates an existing XDG gateway config when one is discovered. When neither source selects a config, preflight validates the effective daemon arguments. An explicit missing path, a legacy schema-v1 file, invalid TOML, a symlink, or any nonregular file fails. Preflight merges the selected file with the current `OPENSHELL_*` environment and applies the daemon's read-only startup checks. These checks include selector and socket normalization, registered-driver selection and configuration, rate-limit pairs, TLS and mTLS relationships, interceptor registrations, and supervisor middleware registrations. When a selected file omits `compute_driver`, preflight validates each configured table for an auto-detectable driver without running the runtime detection probes, which can connect local sockets or launch discovery commands. It validates configured driver TLS requirements, including the gateway CA, without requiring package-generated certificate files to exist before certificate generation. It does not construct a compute driver or connect to a transport. A failed preflight always preserves the file; it never migrates, replaces, or rewrites configuration. Config preflight validates configuration without loading launch-signing keys, so a successful result does not confirm that the key files exist or form a usable pair. Gateway startup validates configured or discovered signing bundles. Sandbox creation checks the selected driver's launch-signing requirement before image preparation. To validate the exact daemon arguments that a wrapper will pass, place them after `--` instead of using `--path`: ```shell openshell-gateway config preflight -- --config /etc/openshell/gateway.toml --grpc-rate-limit-requests 100 --grpc-rate-limit-window-seconds 60 ``` Debian and Ubuntu run preflight from the systemd user unit before local certificate generation. The unit still loads the `gateway.env` environment file and starts the gateway with no configuration arguments. snap replays the exact effective daemon arguments through preflight. It gives a nonempty `OPENSHELL_GATEWAY_CONFIG` precedence; otherwise it validates and passes its canonical service-specific path only when that path exists in the filesystem. The legacy system service uses `$SNAP_COMMON/gateway.toml`; the user service uses `$SNAP_USER_COMMON/.config/openshell/gateway.toml`. A broken symlink is therefore rejected instead of being treated as absent. Package startup does not modify an operator-owned v1 file. Back it up, follow [Migrate to schema version 2](#migrate-to-schema-version-2), then validate the result explicitly before restarting the service: ```shell cp ~/.config/openshell/gateway.toml ~/.config/openshell/gateway.toml.v1.bak $EDITOR ~/.config/openshell/gateway.toml openshell-gateway config preflight --path ~/.config/openshell/gateway.toml systemctl --user restart openshell-gateway ``` ### Check local VM host tools For an explicitly selected local `vm` driver, preflight also checks a formatter (`mke2fs` or `mkfs.ext4`), `debugfs`, and `e2fsck`. Install e2fsprogs 1.43 or newer with your operating system's package manager. OpenShell does not install or manage these host dependencies. For example, on macOS: ```shell brew install e2fsprogs openshell-gateway config preflight -- --config ~/.config/openshell/gateway.toml --compute-driver vm ``` Run preflight with the account, configuration, working directory, and environment intended for the gateway. A successful check in an interactive shell does not check a service with a different `PATH`. For a service whose environment uses a private installation, reproduce its configured path explicitly: ```shell env PATH=/opt/company/e2fsprogs/sbin:/opt/company/e2fsprogs/bin:/usr/bin:/bin /usr/local/bin/openshell-gateway config preflight -- --config ~/.config/openshell/gateway.toml --compute-driver vm ``` Replace the executable and configuration paths with your installation's paths, and run this command as the service account. Preflight and VM image operations search inherited `PATH` entries, then the existing e2fsprogs `sbin` and `bin` directories under `/opt/homebrew/opt/e2fsprogs` and `/usr/local/opt/e2fsprogs`. They prefer `mke2fs` and try `mkfs.ext4` only when `mke2fs` is absent. A selected file that cannot run is an error; fix that installation or the service's `PATH` before rerunning the command. The command reports each selected executable path and version. Each tool receives only `-V`, with a five-second deadline and an 8 KiB output limit per stream. Missing, non-executable, outdated, or failing tools return a nonzero exit status with corrective guidance and the selected tool's error. The checks create no gateway state or images and do not start a driver or VM. Version checks confirm the installation; they do not test image creation or filesystem recovery. Other drivers do not require these VM tools. A VM configuration table alone does not select the VM driver. When a remote driver socket is configured, preflight reports that host tool checks were not performed; run the check on that host in the driver service's environment with a local VM configuration. ## Supervisor application listener Enable a separate application TLS listener for declared sandbox HTTP and WebSocket services. Application traffic reaches the supervisor directly and uses authentication supplied by your application or ingress edge. Gateway service ingress remains available independently. ```toml [openshell.supervisor.app_listener] enabled = true port = 9443 domain = "services.example.com" [openshell.drivers.kubernetes.sandbox_runtime.app_listener] tls_secret_name = "openshell-app-tls" ``` The listener defaults to disabled, with port `8443` and no domain. Port, domain, or driver options alone do not enable it. Use an unprivileged port distinct from the supervisor runtime and health ports. The common `port` controls both the listener and its Kubernetes endpoint. With the listener enabled, supervisors cap configuration polling delays at five seconds so healthy routes retain their 30-second freshness lease. Failed polls do not renew the lease; expiry closes affected application connections. Disabling the listener preserves the configured polling interval. On Kubernetes, provide a TLS Secret with `tls.crt` and `tls.key` matching `*.services.example.com` in each supervisor namespace, and allow your ingress clients to reach the app port through NetworkPolicy. The driver mounts the Secret only in supervisor containers. Restart supervisors after changing certificates. Dedicated and shared supervisors expose the app listener directly on their Pod IPs. Route application traffic to the supervisor owning the sandbox, using the configured app port. The driver creates no Kubernetes Service for application traffic; ingress integrations manage any routing resources they need. Existing supervisor processes retain their startup listener configuration until restarted. Expose services with the existing commands. For direct listener testing, an unnamed service on sandbox `demo` in workspace `default` uses SNI and Host `default--demo.services.example.com`; a service named `web` uses `default--demo--web.services.example.com`. Configure DNS or your ingress to reach the owning supervisor while preserving that hostname. Automatic discovery and application URL publication are separate work; current CLI URLs still identify the existing gateway path.