Operate DSX Agent Gateway

Monitor DSX Agent Gateway as one request path that includes the Kubernetes Gateway, dataplane Pods, rate-limit path, upstreams, and optional bridge. The deployment guide contains the installation and configuration procedures.

Find the Right Procedure

Use the following table to start with the symptom that you observe:

Check Gateway Status

Check Helm and Kubernetes status before testing a Model Context Protocol (MCP) request:

helm status dsx-agent-gateway --namespace dsx-agent-gateway
kubectl get gateway,httproute,agentgatewaybackend,agentgatewaypolicy \
  --namespace dsx-agent-gateway
kubectl get deployments,pods,services --namespace dsx-agent-gateway

Wait for controller programming after an installation or configuration change:

kubectl wait \
  --namespace dsx-agent-gateway \
  --for=condition=Programmed \
  gateway/dsx-agent-gateway \
  --timeout=5m

The Programmed condition confirms that the controller accepted and programmed the Gateway. It does not confirm JSON Web Key Set (JWKS) access, Valkey access, bridge connectivity, or MCP server availability. Inspect Gateway conditions when programming does not complete:

kubectl describe gateway dsx-agent-gateway --namespace dsx-agent-gateway
kubectl describe httproute dsx-agent-gateway-mcp-route \
  --namespace dsx-agent-gateway

Check Workload Health

Kubernetes readiness is the primary health signal for chart-managed workloads. Hub readiness requires a NATS connection and a completed shard-discovery cache. Leaf readiness requires a NATS connection. List Pods that are not ready:

kubectl get pods --namespace dsx-agent-gateway \
  --field-selector='status.phase!=Running'

The field selector does not identify a running Pod with an unready container. Use the full Pod view to identify those containers:

kubectl get pods --namespace dsx-agent-gateway
export DSX_AGENT_GATEWAY_POD='pod-name'
kubectl describe pod "${DSX_AGENT_GATEWAY_POD}" --namespace dsx-agent-gateway

Scale the Workloads

Change replica counts through Helm values so later releases preserve the intended scale. The chart uses these defaults:

Set resource requests, limits, node selectors, and tolerations for each chart-managed workload before increasing traffic.

runtimeClassName applies to gateway, rate-limit, and bridge Pods.

Observe Agent Gateway

Agent Gateway can expose Prometheus metrics for the dataplane, controller, rate-limit service, bridge, and bundled Valkey. It can also export OpenTelemetry traces for the dataplane, rate-limit service, and bridge. Refer to the configuration reference to connect these integrations to your observability platform.

After enabling metrics, confirm that your Prometheus installation discovers the monitor resources in the release namespace:

kubectl get podmonitor,servicemonitor --namespace dsx-agent-gateway

Enable the bundled Agent Gateway dashboard with labels that match your Grafana dashboard discovery configuration:

agentgateway:
  monitoring:
    grafanaDashboard:
      enabled: true
      labels:
        grafana_dashboard: "1"

After Grafana discovers the dashboard, search for Agent Gateway. Use its request rate, error rate, latency, and configuration status panels to assess dataplane traffic and identify changes that correlate with an incident.

Enable the bundled alert rules with an owning team:

observability:
  metrics:
    enabled: true
  alerts:
    enabled: true
    team: gateway-operations

Confirm that Prometheus discovers the resulting rules:

kubectl get prometheusrule --namespace dsx-agent-gateway

The bundled rules report these conditions:

• No Agent Gateway dataplane metrics for 10 minutes.
• No bundled controller metrics for 10 minutes.
• Unsynchronized Agent Gateway configuration for 5 minutes.
• A 5xx response rate above 5 percent for 10 minutes when request traffic is present.
• A p99 request latency above 5 seconds for 10 minutes when request traffic is present.
• xDS authorization failures during a 5-minute window.

To investigate a request with distributed tracing, search your trace backend by the service.name resource attribute. The chart uses dsx-agent-gateway-agentgateway for the dataplane, dsx-agent-gateway-ratelimit for rate limiting, and dsx-agentgateway-bridge-hub or dsx-agentgateway-bridge-leaf for bridge roles.

Inspect Logs

Every chart-deployed container writes logs to standard output or standard error. The Agent Gateway dataplane and controller use their native structured logs. The rate-limit service and bridge use JSON logs. Use component labels to select logs:

kubectl logs \
  --namespace dsx-agent-gateway \
  --selector gateway.networking.k8s.io/gateway-name=dsx-agent-gateway \
  --all-containers
kubectl logs \
  --namespace dsx-agent-gateway \
  --selector app.kubernetes.io/component=ratelimit \
  --all-containers
kubectl logs \
  --namespace dsx-agent-gateway \
  --selector app.kubernetes.io/component=dsx-agentgateway-bridge \
  --all-containers

Add --prefix when several Pods match one selector. Do not copy bearer tokens or credential values into support records.

Understand Failure Behavior

The chart defaults rateLimit.failureMode to FailOpen. This mode serves requests without tenant limit enforcement when the rate-limit path fails.

The following defaults determine how the request path responds to dependency failures:

Troubleshoot Gateway Programming

If the Gateway does not become Programmed, check these conditions:

1. Confirm that Gateway API v1.5.1 and Agent Gateway v1.4.1 custom resource definitions (CRDs) are installed.
2. Confirm that the Agent Gateway controller Deployment is available.
3. Inspect the Gateway, HTTPRoute, backend, and policy conditions.
4. Inspect controller logs for rejected resources or synchronization failures.
5. Render the chart with the active values to identify validation errors.

The helm template command requires a local clone of the DSX Exchange repository. Render the release configuration locally, from the repository root:

helm template dsx-agent-gateway deploy/dsx-agent-gateway \
  --namespace dsx-agent-gateway \
  --values dsx-agent-gateway-values.yaml

Troubleshoot Authentication and Target Access

If the gateway rejects a caller, check these settings:

• The token issuer exactly matches one configured provider issuer.
• At least one token audience matches a configured audience.
• The provider JWKS URL is reachable from the dataplane.
• The provider Common Expression Language (CEL) expression returns a nonempty tenant ID from verified claims.
• auth.cel.operatorTenantId exactly matches the intended operator tenant ID.
• The target name appears in auth.cel.unprivilegedTenantMCPs for a non-operator tenant.

The gateway discards caller-supplied tenant headers when it derives rate-limit and authorization identity.

Troubleshoot Missing MCP Targets

If a selector upstream does not produce a target, check these conditions:

1. Confirm that upstreams.<name>.namespace names the MCP Service namespace.
2. Confirm that serviceLabels match the intended Service.
3. Confirm that the Service port has appProtocol: agentgateway.dev/mcp.
4. Confirm that agentgateway.discoveryNamespaceSelectors includes the upstream namespace.
5. Confirm that the MCP server endpoints are ready.

If a static target is unavailable, verify its complete HTTP or HTTPS URL and response-header deadline. The configuration reference lists the supported upstream fields and defaults.

Troubleshoot Rate Limiting and Valkey

HTTP 429 responses indicate that a tenant exhausted its configured request budget. Use the following checks:

• Check rateLimit.tenantRequestsPerSecond and matching tenantOverrides before changing the limit.
• If the rate-limit Pods remain in initialization, verify the configured Valkey host, port, Service, endpoints, and network policy. The init container waits for Valkey before the application container starts.
• For an external Valkey deployment, verify availability and connectivity with the external owner.
• During a runtime outage, confirm whether rateLimit.failureMode matches the required availability policy.

Troubleshoot the Bridge

If a bridge Pod is not ready, check these conditions:

1. Confirm that bridge.nats.endpoint identifies an in-cluster NATS Service and port.
2. Confirm that the NATS Service has ready endpoints.
3. Confirm that bridge.nats.auth.mode is noauth or oauth.
4. For OAuth mode, confirm the issuer, scope, and credential Secret keys.
5. For NATS TLS, confirm the server name and certificate authority Secret.
6. For a leaf, confirm that bridge.shardId is a valid NATS subject token.

A hub with no completed shard discovery remains unready. Verify that each expected leaf connects to the same NATS deployment and uses a unique shard ID. The bridge emits completion records without logging caller bearer tokens. Inspect bridge logs and traces for NATS request timeouts, unknown shards, and upstream gateway errors.

Upgrade and Roll Back

Record the chart version, values file, CRD versions, and Helm revision before an upgrade. Review Agent Gateway and Valkey compatibility before changing pinned dependency versions. From the DSX Exchange repository root, upgrade with the same deploy/dsx-agent-gateway chart path, release name, and complete values file:

helm upgrade dsx-agent-gateway deploy/dsx-agent-gateway \
  --namespace dsx-agent-gateway \
  --values dsx-agent-gateway-values.yaml \
  --wait \
  --timeout 5m

Verify workload readiness, Gateway programming, target discovery, authentication, and a representative MCP call after the upgrade. Review available Helm revisions before a rollback:

helm history dsx-agent-gateway --namespace dsx-agent-gateway

Use Helm rollback only after confirming that the earlier chart supports the installed CRDs and external dependencies:

export DSX_AGENT_GATEWAY_REVISION='1'
helm rollback dsx-agent-gateway "${DSX_AGENT_GATEWAY_REVISION}" \
  --namespace dsx-agent-gateway \
  --wait \
  --timeout 5m

Helm rollback reapplies an earlier release revision. It does not roll back cluster-scoped CRDs, external resources, or Valkey data. Verify the same request path after a rollback.

Uninstall the Release

Uninstall the Helm release:

helm uninstall dsx-agent-gateway --namespace dsx-agent-gateway

This command removes resources owned by the Helm release. It leaves the namespace, cluster-scoped CRDs, Valkey data persistent volume claims, external workloads, and operator-owned resources. Review those retained resources separately before removing them.