Skip to navigation
NVKE telemetry pipeline (NKTP)Guides

Secure gateway export

View as Markdown

NVKE telemetry pipeline (NKTP) exports to its bundled LGTM backend over plaintext OTLP/gRPC at lgtm:4317 by default. LGTM already enables both gRPC and HTTP ingestion. This backend is intended only for local development. Configure TLS before sending logs, metrics and traces to a production backend.

Choose the protocol and endpoint supported by your backend for each signal. The Collector’s otlp exporter uses gRPC; otlp_http uses HTTP. Changing the port alone does not change the protocol. For example, direct ingestion into Prometheus or Loki requires OTLP/HTTP. LGTM accepts gRPC through its bundled Collector, which forwards metrics and logs to those HTTP endpoints internally.

This guide configures mTLS for the three gateway Collectors using gRPC. The example shows the overrides. Receiver TLS for connections into the gateways is configured separately.

mTLS with an existing certificate

As a prerequisite, you need to already have:

  • the CA certificate used to verify the backend server certificate;
  • a client certificate signed by a CA that the backend trusts; and
  • the private key corresponding to the client certificate.
  1. Create a secret containing all three files in the Helm release namespace.
kubectl create secret generic backend-tls \
  --namespace <release-namespace> \
  --from-file=ca.crt=/path/to/backend-server-ca.crt \
  --from-file=tls.crt=/path/to/client.crt \
  --from-file=tls.key=/path/to/client.key
  1. Mount the secret into each gateway, preserving its internal TLS volumes and mounts. Each gateway needs its internal server certificate for ingress, and its sidecar needs its internal client certificate to export to watchdog. The backend credentials belong on the main gateway Collector, not its sidecar.
gateway_logs:
  kubernetes:
    podSecurityContext:
      fsGroup: 10001
    volumes:
      - name: internal-tls-server
        secret:
          secretName: nktp-gateway-logs-server-tls
          items:
            - {key: tls.crt, path: tls.crt}
            - {key: tls.key, path: tls.key}
            - {key: ca.crt, path: ca.crt}
      - name: internal-tls-client
        secret:
          secretName: nktp-gateway-logs-client-tls
          items:
            - {key: tls.crt, path: tls.crt}
            - {key: tls.key, path: tls.key}
            - {key: ca.crt, path: ca.crt}
      - name: backend-tls
        projected:
          defaultMode: 0440
          sources:
            - secret:
                name: backend-tls
                items:
                  - key: ca.crt
                    path: ca.crt
                  - key: tls.crt
                    path: tls.crt
                  - key: tls.key
                    path: tls.key
    volumeMounts:
      - name: internal-tls-server
        mountPath: /etc/nktp/internal-tls/server
        readOnly: true
      - name: backend-tls
        mountPath: /etc/nktp/backend-tls
        readOnly: true

Note: fsGroup: 10001 matches the group used by NKTP’s default Collector image. It lets the non-root Collector read the root-owned Secret files through the group read bit in 0440. If you use a custom Collector image, set fsGroup to a group available to that image. Alternatively, omit both defaultMode and fsGroup to use Kubernetes’ default Secret mode (0644).

  1. Configure the exporter to use the mounted certificate.
gateway_logs:
  ...
  otelcol:
    config:
      exporters:
        otlp:
          endpoint: telemetry.example.com:4317
          tls:
            insecure: false
            ca_file: /etc/nktp/backend-tls/ca.crt
            cert_file: /etc/nktp/backend-tls/tls.crt
            key_file: /etc/nktp/backend-tls/tls.key
            reload_interval: 1m

Apply the volume, mount, and exporter settings to gateway_logs, gateway_metrics, and gateway_traces. Use each gateway’s own internal Secrets: nktp-gateway-<signal>-server-tls and nktp-gateway-<signal>-client-tls, where <signal> is logs, metrics, or traces. Keep the volume names and mount paths shown above. The complete values example configures all three.

Set insecure: false explicitly: the development defaults disable TLS, and Helm retains nested defaults when merging overrides. Configure the watchdog and its sidecar’s backend exporters too, if sending NKTP’s own telemetry to this backend.

For an HTTP backend, remove the default gRPC exporter with otlp: null, add otlp_http with an https:// endpoint and the same certificate settings, and replace otlp with otlp_http in each affected pipeline’s exporters list. Preserve the queue and retry settings from the default exporter as needed.

For server-authenticated TLS with a private CA but no client authentication, mount only ca.crt and configure only ca_file.

For failures, inspect gateway logs for x509, tls, handshake, unauthorized, or forbidden. Common causes are a missing intermediate CA, an endpoint hostname absent from the server certificate, an expired client certificate, or a client issuer the backend does not trust.

Automate client certificate with cert-manager

cert-manager does not ship with NKTP. It can issue and renew the gateway client certificate when an existing Issuer or ClusterIssuer represents a CA that the backend trusts. Creating an issuer does not by itself make the backend trust that CA.

  1. Ensure a ClusterIssuer or Issuer CR exists for a Certificate Authority that your backend trusts. For details, go to the cert-manager documentation.

  2. Create a Certificate CR in the nktp namespace.

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: nktp-gateway-client
  namespace: {{ namespace }}
spec:
  secretName: backend-tls
  issuerRef:
    kind: ClusterIssuer
    name: backend-client-ca
  usages:
    - client auth
  commonName: example.com
  privateKey:
    rotationPolicy: Always

cert-manager maintains tls.crt and tls.key in backend-tls. Mount that Secret in every gateway and retain cert_file, key_file, and reload_interval in the exporter configuration.