Observability

Observability Configuration

View as Markdown

This page provides guidance on configuring observability for self-hosted NVCF control-plane, including metrics, logging, and tracing.

Overview

Self-hosted NVCF control-plane observability enables users to monitor the health and performance of their NVCF deployment. The observability solution is designed to be:

  • Cloud-agnostic: Works in any Kubernetes environment (cloud provider, on-premises, or air-gapped)
  • Offline-capable: Fully functional in isolated networks without external dependencies
  • Bring-Your-Own (BYO): Integrates with your existing observability platforms
  • No vendor lock-in: Uses open standards (Prometheus, OpenTelemetry, OTLP)

The observability solution currently provides:

  • [Metrics Collection]: Prometheus-compatible metrics from all control-plane services
  • [Logging]: Logs emitted to stdout/stderr for easy collection
  • [Tracing]: Distributed tracing via OTLP to your collector
  • [Dashboards]: Reference Grafana dashboards for key metrics

Looking for a quick start? If you want to quickly deploy example observability components to explore metrics, logs, and dashboards, see self-hosted-example-dashboards.

The example deployments are designed for development and testing only, and are not suitable for production use. For production deployments, follow the guidance on this page to integrate with your own observability infrastructure.

Early Access Phase

NVCF self-hosted observability is currently in Early Access (EA). During EA, NVCF provides interfaces and documentation for you to integrate with your own observability backend:

What’s Provided:

  • Documented metrics for critical control-plane services
  • Example scrape targets for prometheus-operator ServiceMonitor configuration
  • Metrics exposed via Prometheus-compatible endpoints
  • Logs emitted to stdout/stderr for easy collection
  • Configuration and deployment documentation
  • Example dashboards for key metrics

Your Responsibility:

  • Deploy and manage your own observability backend (Prometheus, Grafana, Loki, Elasticsearch, etc.)
  • Configure metrics scraping from control-plane services
  • Deploy log collectors (e.g., Fluentd, Promtail, OTel Collector) to aggregate logs
  • Set up your preferred visualization and alerting tools

Control-Plane Services

The following control-plane services expose metrics and logs for monitoring:

Core NVCF Services:

  • NVCF API: Main API for function management and invocation
  • Invocation Service: Handles function invocation requests
  • SPOT Instance Service (SIS): Manages worker pod and cluster state
  • State Metrics Service: (Available in GA) Aggregates and exports NVCF-specific metrics

Supporting Services:

  • Cassandra (C*): Primary database for control-plane state
  • OpenBao/Vault: Secret management and S2S authentication
  • Encrypted Secrets Service (ESS): Function and account secrets
  • NATS Core: Pub/sub messaging
  • NATS JetStream: Persistent messaging

Worker Pod Components:

  • Utils Container: Proxy to NATS from user applications
  • Init Container: Setup and resource loading
  • Inference Container: Inference workload

Architecture

Metrics Collection

All control-plane services expose Prometheus-compatible metrics endpoints. You can scrape these metrics using:

  • Prometheus Operator: Create ServiceMonitor resources based on the provided scrape targets
  • Prometheus: Configure scrape targets manually
  • OpenTelemetry Collector: Use the Prometheus receiver

Metrics Documentation:

Detailed metrics documentation is available for each service, including metric names, types, labels, and descriptions. See the per-service metrics reference under the Metrics section.

Logging

Log Format:

  • All services emit logs to stdout/stderr (standard for Kubernetes)
  • Sensitive data redaction must be configured by the log collector

Log Collection:

You can collect logs using any Kubernetes-compatible log aggregator:

  • Fluentd or Fluent Bit
  • Promtail (for Loki)
  • Filebeat (for Elasticsearch)
  • OpenTelemetry Collector (filelog receiver)

System Logs:

System logs are available at standard UNIX locations and from the systemd journal.

Tracing (Available in GA)

Distributed tracing support via OpenTelemetry Protocol (OTLP) is planned for a future release:

  • Key flows will be instrumented with OpenTelemetry SDK
  • Traces will be exportable via OTLP (HTTP or gRPC)
  • Configurable sampling strategies
  • Support for any OTLP-compatible backend (Jaeger, Tempo, Zipkin, etc.)
  • Tracing is configurable via Helm values under global.observability.tracing

Configuration

You configure observability by integrating with your own backend:

Metrics Scraping

Use Prometheus Operator with the provided ServiceMonitor examples:

1# Example ServiceMonitor for NVCF API
2apiVersion: monitoring.coreos.com/v1
3kind: ServiceMonitor
4metadata:
5  name: nvcf-api
6  namespace: nvcf
7spec:
8  selector:
9    matchLabels:
10      app: nvcf-api
11  endpoints:
12  # Endpoint created based on the scrape target in the
13  # per-service metrics documentation
14  - port: metrics
15    interval: 30s
16    path: /metrics

Or configure Prometheus scrape targets manually in your prometheus.yml.

Log Collection

Deploy a log collector as a DaemonSet to ship logs to your backend:

$# Example: Deploy Promtail for Loki
$kubectl apply -f promtail-daemonset.yaml
$
$# Example: Deploy Fluentd or Fluent Bit
$kubectl apply -f fluentd-daemonset.yaml

Configure your log collector to:

  • Tail logs from all namespaces
  • Add metadata labels (pod name, namespace, service)
  • Forward to your log aggregation backend (Loki, Elasticsearch, etc.)

Tracing Configuration

Enable distributed tracing by setting Helm values under global.observability.tracing. The control-plane exports traces via OTLP to your own OTLP-compatible collector. Set collectorEndpoint, collectorPort, and collectorProtocol to match your collector’s address and protocol.

Helm overrides example:

1global:
2  observability:
3    tracing:
4      enabled: true
5      collectorEndpoint: "otel-collector-gateway-collector.observability.svc.cluster.local"
6      collectorPort: 4317
7      collectorProtocol: http

Configuration fields:

  • enabled: Set to true to enable OTLP trace export from control-plane services.
  • collectorEndpoint: DNS name or address of your OTLP collector (e.g., OpenTelemetry Collector, Jaeger collector). Use a Kubernetes service DNS name such as <service>.<namespace>.svc.cluster.local when the collector runs in-cluster.
  • collectorPort: Port on which the collector accepts OTLP traffic (e.g., 4317 for HTTP, 4318 for gRPC depending on your collector setup).
  • collectorProtocol: OTLP protocol: http or grpc. Must match how your collector is configured.

Ensure your collector is deployed and reachable from the NVCF control-plane namespace, and that it forwards traces to your backend (Jaeger, Tempo, Zipkin, or another OTLP-compatible system).

Dashboards

Reference Grafana dashboards are provided for control-plane services showing critical metrics for key services:

  • ESS (Encrypted Secrets Service)

  • Cassandra

  • Vault

  • Invocation Service

  • NVCF API

  • SIS (SPOT Instance Service)

  • Worker Pods (Utils Container, Init Container, Inference Container)

    • Note: Worker Pods are deployed in the backend cluster, not the control-plane cluster, but their configuration is globally controlled as part of the control-plane

  • State Metrics Service (Available in GA)

Dashboard Location:

Dashboards are provided in native Grafana JSON format for file-provisioning.

Load dashboards into Grafana by placing them in /etc/grafana/provisioning/dashboards/ on startup.

Published dashboards will be available in the nv-cloud-function-helpers public GitHub repository.

Troubleshooting

For troubleshooting common observability issues:

Metrics not appearing:

  1. Verify the service is exposing metrics:

    $# Port-forward to the service metrics port
    $kubectl port-forward -n nvcf svc/nvcf-api 8080:8080
    $
    $# In another terminal, curl the metrics endpoint
    $curl http://localhost:8080/metrics

  2. Check ServiceMonitor or scrape configuration:

    $# Verify ServiceMonitor exists
    $kubectl get servicemonitor -n nvcf
    $
    $# Check ServiceMonitor details
    $kubectl describe servicemonitor nvcf-api -n nvcf

  3. Verify network policies allow scraping:

    $# List network policies that might block traffic
    $kubectl get networkpolicy -n nvcf
    $
    $# Test connectivity from Prometheus namespace
    $kubectl run -n <prometheus-namespace> --rm -it debug \
    >  --image=curlimages/curl --restart=Never -- \
    >  curl http://nvcf-api.nvcf.svc.cluster.local:8080/metrics

  4. Check service logs for errors:

    $# Check for metrics-related errors
    $kubectl logs -n nvcf deployment/nvcf-api | grep -i metric

Logs not being collected:

  1. Verify log collector DaemonSet is running:

    $# Check DaemonSet status (e.g., for Fluentd/Fluent Bit)
    $# Note: Namespaces may be different depending on the log collector deployment
    $kubectl get daemonset -n logging
    $kubectl get pods -n logging -l app=fluent-bit

  2. Check collector can access pod logs:

    $# Verify log collector has proper volume mounts
    $kubectl describe daemonset fluent-bit -n logging | grep -A5 Mounts
    $
    $# Check collector logs for errors
    $kubectl logs -n logging -l app=fluent-bit --tail=50

  3. Verify log backend is reachable:

    $# Test connectivity to log backend (e.g. Loki)
    $kubectl run -n logging --rm -it debug \
    >  --image=curlimages/curl --restart=Never -- \
    >  curl -v http://loki.logging.svc.cluster.local:3100/ready

  4. Check for log redaction or filtering rules:

    $# Review collector configuration
    $kubectl get configmap fluent-bit-config -n logging -o yaml
    $
    $# Check if logs are being dropped
    $kubectl logs -n logging -l app=fluent-bit | grep -i "drop\|filter"

Security

Metrics Endpoints:

  • Metrics endpoints should be accessed over HTTP in-cluster only

    • Any external access should be SSL/TLS or mTLS secured with a reverse proxy or other ingress controller, or
    • Aggregated locally and exposed via a secured otel-collector

  • All sensitive log data should be redacted by the log collector (currently, this is the responsibility of the log collector, not the service)

  • User-provided observability backend should be properly secured with RBAC, TLS/SSL, and other security best practices.

Version Compatibility

NVCF self-hosted control-plane observability is compatible with:

  • Any official supported Kubernetes version
  • Any Prometheus-compatible metrics collection system
  • Any log aggregation system that can collect from Kubernetes stdout/stderr or read from the filesystem (depending on K8s cluster configuration)

For the latest compatibility information, see the release notes.