> This page is for version Development. > For other versions, use one of these documentation indexes: > - 1.0.3 (default): https://docs.nvidia.com/nvcf/1.0.3/llms.txt > - 1.0.2: https://docs.nvidia.com/nvcf/1.0.2/llms.txt > - 1.0.1: https://docs.nvidia.com/nvcf/1.0.1/llms.txt > - Development: https://docs.nvidia.com/nvcf/dev/llms.txt > For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/nvcf/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nvcf/_mcp/server. # Bring Your Own Observability Bring Your Own Observability (BYOO) exports logs, metrics, and traces from a tenant's functions and tasks to an observability backend that the tenant owns. Each workload pod gets an OpenTelemetry (OTel) Collector sidecar. The sidecar sends telemetry to the endpoints that the tenant registers with the NVCF API. This guide is for two audiences: - Operators enable BYOO on self-managed GPU clusters. - Tenants register telemetry endpoints and attach them to functions and tasks. ## When to use BYOO Use BYOO when a tenant needs workload telemetry in its own backend, such as Grafana Cloud, Datadog, Splunk, Azure Monitor, a Prometheus remote-write receiver, or any OpenTelemetry Protocol (OTLP) HTTP receiver. BYOO does not replace the operator's observability stack. Control-plane and cluster-level monitoring is installed by the observability stack. For details, refer to [Observability](/nvcf/dev/observability/observability). BYOO exports only the telemetry of the workloads that reference a telemetry endpoint. ## Enable BYOO on a cluster ### Feature gates NVCA (NVIDIA Cloud Functions Agent) controls BYOO with two feature gates: | Gate | Default | Effect | |---|---|---| | `BYOObservability` | Off | Adds the BYOO OTel Collector sidecar to workloads that reference telemetry endpoints. | | `BYOOFluentBit` | Off | Adds a Fluent Bit sidecar that reads container logs and forwards them to the collector. | The compute-plane stack turns on `BYOObservability` when `observability.profile` is `compute` or `all`. The `compute` profile is the default for the compute-plane stack. The `disabled` and `control` profiles do not turn on the gate. To turn on `BYOOFluentBit`, set `global.nvcaOperator.byooFluentBit.enabled` in the compute-plane environment file: ```yaml observability: profile: compute global: nvcaOperator: byooFluentBit: enabled: true ``` `BYOOFluentBit` needs `BYOObservability`. The stack fails to render when `byooFluentBit.enabled` is true and `BYOObservability` is off. Add other gates, or turn off a derived gate with a leading `-`, through `global.nvcaOperator.selfManaged.featureGateValues`. The stack merges these values with its default gates. A value such as `-BYOObservability` turns off the derived gate. For a standalone `nvca-operator` Helm release, list the gates in `selfManaged.featureGateValues`. The `--set` flag replaces the whole list, so include every gate you need. For more information, refer to [Managing Feature Flags](/nvcf/dev/compute-plane/cluster-configuration#managing-feature-flags). ### Turn telemetry off for a deployment The NVCF API accepts telemetry endpoints even where workload telemetry export is not available, so tenants can register endpoints that never receive anything. On such a deployment, turn telemetry off: ```yaml api: remoteConfig: configData: nvcf: telemetry: enabled: false nvctApi: remoteConfig: configData: nvct: telemetry: enabled: false ``` Each API reads its own setting, so set both. The NVCF API setting is also available as an environment variable, `api.env.NVCF_TELEMETRY_ENABLED: "false"`. The stack builds the NVCT API environment itself, so remote config is the only route for the task side. With telemetry off, registering an endpoint or attaching one to a function returns 400 `telemetry is not supported on this deployment`, and so does creating a task that attaches one. Deleting an endpoint still works, so endpoints registered before the change can be cleared. Each API reads the setting once at startup, so changing it needs a restart. The setting defaults to on, so a deployment that does not set it is unchanged. It is temporary and is removed once telemetry export is available everywhere. ### Sidecar sizing and collector options The `nvca-operator` chart exposes BYOO sizing under the `byoo` key. Leave a key unset to keep the NVCA built-in defaults. In the compute-plane stack, set the same keys under `global.nvcaOperator.byoo`. The stack passes them to the chart unchanged. | Value | Purpose | |---|---| | `byoo.resources` | Requests and limits for the BYOO OTel Collector container. | | `byoo.fluentbit.resources` | Requests and limits for the Fluent Bit container. | | `byoo.logChunking` | Log chunking overrides: `maxPayloadBytes` and `dryRun`. | | `byoo.otelCollector` | Structured collector rendering overrides, including [platform metrics federation](#platform-metrics-federation). | | `byoo.additionalResourceOverhead` | Cluster capacity reserved for BYOO and related DaemonSet overhead. | Example values for the `nvca-operator` release: ```yaml byoo: resources: requests: cpu: 100m memory: 256Mi limits: cpu: 500m memory: 512Mi fluentbit: resources: requests: cpu: 50m memory: 64Mi limits: cpu: 200m memory: 128Mi logChunking: maxPayloadBytes: 262144 ``` You can also tune the collector through `agentConfig.mergeConfig`. For examples, refer to [Agent Config Merging](/nvcf/dev/compute-plane/cluster-configuration#agent-config-merging). ### Platform metrics federation The collector can read platform metrics for its workload from a Prometheus-compatible `/federate` endpoint: worker, NVCA, cAdvisor, kube-state-metrics, and DCGM series. On self-managed clusters, federation is off unless you set `byoo.otelCollector.platformMetricsFederation`. The collector then exports only the telemetry that the workload sends, plus container logs when `BYOOFluentBit` is on. To turn it on, set `target` to the `host:port` of a server that serves `/federate` over HTTP. For the VictoriaMetrics backend of the observability stack, use `vmsingle..svc.cluster.local:8428`. ```yaml byoo: otelCollector: platformMetricsFederation: target: vmsingle.monitoring.svc.cluster.local:8428 ``` In the compute-plane stack, set the same block under `global.nvcaOperator.byoo`. Set `enabled: false` to keep federation off explicitly. The federation jobs select series by their `job`, `namespace`, `pod`, and `container` labels. Series that the target stores under other job names or labels are not exported. With the `compute` or `all` profile, the default monitors of the observability stack store these series under the job names that the collector selects: | Series | Job | Observability stack source | | --- | --- | --- | | Worker | `nvcf-worker` | `nvcf-default-monitors-worker` PodMonitor | | NVCA | `nvca` | `nvcf-default-monitors-nvca` ServiceMonitor | | DCGM | `nvidia-dcgm-exporter` | `nvcf-default-monitors-dcgm` PodMonitor, for pods that NVCA labels with `nvca.nvcf.nvidia.io/dcgm-metrics-present` | | cAdvisor | `kubelet` | Not scraped | | kube-state-metrics | `kube-state-metrics` | Not scraped | Keep `defaultMonitors.computePlane.worker.jobName` and `defaultMonitors.computePlane.dcgm.jobName` at these defaults when you use federation. To export cAdvisor or kube-state-metrics series, or DCGM series from another exporter, scrape them into the target under the job names in the table and keep the workload `namespace`, `pod`, and `container` labels. The DCGM job reads these labels with an `exported_` prefix, which a scrape adds when the exporter labels conflict with the target labels. The worker and DCGM job names require observability stack 1.1.0 or later, which ships `nvcf-default-monitors` 0.3.0. Observability stack 1.0.0 has no `jobName` setting. Its PodMonitors store these series under the default job, `/`, such as `monitoring/nvcf-default-monitors-worker` and `monitoring/nvcf-default-monitors-dcgm`. Federation then exports no worker or DCGM series, such as `nvcf_worker_service_request_total` or `DCGM_FI_DEV_GPU_UTIL`. Upgrade the observability stack to export them. The compute-plane stack pins NVCA and BYOO collector releases that support this setting. Older releases ignore it and federate from the default target described in [Federation scrape errors in the collector log](#federation-scrape-errors-in-the-collector-log). ### Sidecar images The compute-plane stack sets both sidecar images and passes them to the `nvca-operator` chart: | Sidecar | Stack value | Default image | |---|---|---| | BYOO OTel Collector | `global.nvcaOperator.byooOtelCollector` | `${global.image.registry}/${global.image.repository}/byoo-otel-collector`, with the tag in `imageTag` | | Fluent Bit | `global.nvcaOperator.byooFluentBit.image` | `docker.io/fluent/fluent-bit`, with the tag in `tag` | The collector follows `global.image`, so mirror it with the other stack images. Fluent Bit does not follow `global.image`. To pull it from a private registry or mirror, copy the image with the tag that the stack pins. Then set its repository in the compute-plane environment file. Set `byooOtelCollector.imageRepository` only when the collector is published outside the `global.image` path: ```yaml global: nvcaOperator: byooOtelCollector: imageRepository: //byoo-otel-collector byooFluentBit: image: registry: repository: /fluent-bit ``` If you mirror other tags, also set `byooOtelCollector.imageTag` and `byooFluentBit.image.tag`. For a standalone `nvca-operator` release, set `agent.byooOtelCollector.imageRepository` and `agent.byooOtelCollector.imageTag`. For Fluent Bit, set `agent.byooFluentBit.imageRepository`, including the registry, and `agent.byooFluentBit.imageTag`. Without `agent.byooFluentBit.imageRepository`, NVCA uses the upstream Fluent Bit image. The workload namespace must have pull access to each registry. For registry credentials, refer to [Working with Third-Party Registries](/nvcf/dev/self-managed/third-party-registries). ### Network access In self-hosted mode, NVCA removes NetworkPolicies from workload namespaces, so NVCA does not restrict collector egress. Your cluster network must still allow the workload pods to reach each telemetry endpoint. The collector listens inside the pod on these ports: | Port | Purpose | |---|---| | 14357 | OTLP gRPC receiver | | 14358 | OTLP HTTP receiver, also used by Fluent Bit | | 13133 | Health check | | 18888 | Collector internal metrics | ## Supported providers A telemetry endpoint has a provider, an endpoint, a set of signal types, and a secret. The BYOO collector renders an exporter only for the providers and signal types in this table. Telemetry registration returns 400 for any other provider or signal type. Secret updates return 400 for any other provider. A secret update checks the new secret against the stored signal types that the provider supports and ignores the others. An endpoint stored with an unsupported signal type can still rotate its secret. Function and task creation return 400 when they reference a telemetry for a signal type its provider does not support. The `secret.value` field is a string or a JSON object of string fields. The collector writes a string to one credential file. It writes each key of a JSON object to its own file. The `secret.name` is the telemetry name. An empty value counts as not set. | Provider | Logs | Metrics | Traces | `endpoint` | `secret.value` | |---|---|---|---|---|---| | `AZURE_MONITOR` | Yes | Yes | Yes | Azure Monitor ingestion endpoint | Object with `instrumentationKey`, `applicationId`, and `liveEndpoint`. | | `DATADOG` | Yes | Yes | Yes | Datadog site, for example `datadoghq.com` | String. The Datadog API key. | | `GRAFANA_CLOUD` | Yes | Yes | Yes | Grafana Cloud OTLP endpoint | Object with `instanceId` and `apiKey`. | | `KRATOS` | Yes | No | Yes | Full OTLP HTTP logs or traces URL, used as is | Logs: object with `collectorId`, `clientCert`, and `clientKey`. Traces: object with `clientCert` and `clientKey`, and optionally `caFile` and `collectorId`. Certificates and keys are PEM text. | | `KRATOS_THANOS` | No | Yes | No | Remote-write URL | Object with `clientCert` and `clientKey`, `caFile`, or all three. All values are PEM text. | | `OTEL_COLLECTOR` | Yes | Yes | Yes | OTLP HTTP base URL, for example `https://otel.example.com:4318` | String. Sent as `Authorization: Bearer `. | | `PROMETHEUS` | No | Yes | No | Remote-write URL | Object with `clientCert` and `clientKey`, `caFile`, or all three. All values are PEM text. | | `SERVICENOW` | No | No | Yes | OTLP gRPC endpoint | String. A token that starts with `otel_` is sent in the `otel-token` header. Any other token is sent in the `lightstep-access-token` header. | | `SPLUNK` | Yes | No | No | Splunk HTTP Event Collector (HEC) URL | String. The HEC token. | Use `OTEL_COLLECTOR` to send to any OTLP HTTP receiver, including your own OpenTelemetry Collector. The collector appends the standard OTLP signal paths, such as `/v1/logs`, to the base URL. `KRATOS` sends `collectorId` in the `collector-id` header. For `PROMETHEUS` and `KRATOS_THANOS`, a client certificate needs a client key. ## Register a telemetry endpoint Register endpoints with `nvcf-cli telemetry`. For CLI setup, refer to [Self-hosted CLI](/nvcf/dev/overview/cli). The commands need a credential with the `manage_telemetries` scope, which the default function API key includes. The account limit `maxTelemetries` caps the number of endpoints per account. Write the secret value to a file, in the shape that [Supported providers](#supported-providers) lists for the provider. A value that starts with `{` is sent as a JSON object. Then create the endpoint: ```bash nvcf-cli telemetry create \ --name my-otel-endpoint \ --endpoint https://otel.example.com:4318 \ --provider OTEL_COLLECTOR \ --type LOGS --type METRICS --type TRACES \ --secret-file otel-token.txt ``` The command prints the telemetry ID. `--name` is also the secret name and follows the `secret` rules in the field table below. `--protocol` defaults to `HTTP`. `--secret-file -` reads the value from standard input. The CLI rejects a provider, type, or secret shape that the collector cannot render. Manage endpoints with these commands: - `nvcf-cli telemetry list` lists endpoints, and `nvcf-cli telemetry get ` shows one. Neither returns the secret. - `nvcf-cli telemetry delete ` deletes an endpoint and its secret. The API rejects the request while a function references the endpoint. - `nvcf-cli telemetry update-secret --secret-file ` replaces the secret. It also needs the `update_secrets` scope, which the default function API key does not include. To generate a function API key that can update secrets, list the default scopes too. `--scopes` replaces them, and the new key replaces the saved function key: ```bash nvcf-cli api-key generate --for function \ --scopes invoke_function,list_functions,queue_details,list_functions_details,manage_telemetries,update_secrets ``` ### Use the API To call the API directly, create the endpoint with `POST /v2/nvcf/telemetries`. The token needs the `manage_telemetries` scope. Set `NVCF_API_URL` to the NVCF API URL of your Gateway. With DNS and HTTPS configured as described in [Production: DNS and HTTPS](/nvcf/dev/self-managed/gateway-routing#production-dns-and-https), use the API hostname: ```bash export NVCF_API_URL="https://api." curl -s -X POST "${NVCF_API_URL}/v2/nvcf/telemetries" \ -H "Authorization: Bearer ${NVCF_TOKEN}" \ -H "Content-Type: application/json" \ -d '{ "endpoint": "https://otel.example.com:4318", "protocol": "HTTP", "provider": "OTEL_COLLECTOR", "types": ["LOGS", "METRICS", "TRACES"], "secret": { "name": "my-otel-endpoint", "value": "" } }' | jq . ``` The Gateway from [Gateway quickstart](/nvcf/dev/self-managed/gateway-routing#gateway-quickstart) has no TLS listener. It serves the API on plain HTTP port 80 and routes by the `Host` header. On a test cluster with that Gateway, set `NVCF_API_URL="http://${GATEWAY_ADDR}"` and add `-H "Host: api.${GATEWAY_ADDR}"` to the command. Plain HTTP sends the API token and `secret.value` unencrypted. Use it only on a trusted network. | Field | Values | |---|---| | `endpoint` | Provider endpoint, as listed in [Supported providers](#supported-providers). | | `protocol` | `HTTP` or `GRPC`. | | `provider` | A provider from [Supported providers](#supported-providers). | | `types` | One or more of `LOGS`, `METRICS`, and `TRACES`. The provider must support each type. | | `secret` | `name` and `value`. The name is 1 to 48 characters. It starts with a letter or digit, followed by letters, digits, `_`, `.`, or `-`. It must be unique in the account. | The response includes a `telemetryId`. Use it to attach the endpoint to workloads. Other operations: - `GET /v2/nvcf/telemetries` lists endpoints. - `GET /v2/nvcf/telemetries/{telemetryId}` returns one endpoint. - `DELETE /v2/nvcf/telemetries/{telemetryId}` deletes an endpoint. - `PUT /v2/nvcf/secrets/telemetries/{telemetryId}` updates the secret. It needs the `update_secrets` scope instead of `manage_telemetries`. ## Attach endpoints to functions and tasks Each workload can reference one endpoint per signal. The endpoint must include that signal in its `types`, and its provider must support that signal. For functions, pass the IDs to `nvcf-cli function create`: ```bash nvcf-cli function create --input-file function.json \ --logs-telemetry-id "${TELEMETRY_ID}" \ --metrics-telemetry-id "${TELEMETRY_ID}" \ --traces-telemetry-id "${TELEMETRY_ID}" ``` For tasks, `nvcf-cli task create` accepts the same flags. In a task JSON file or API request, set the `telemetries` object: ```json { "telemetries": { "logsTelemetryId": "", "metricsTelemetryId": "", "tracesTelemetryId": "" } } ``` The function creation API uses the same `telemetries` object. `nvcf-cli function get` and `nvcf-cli task get` show the attached IDs under `Telemetries`. ## What is exported The collector sends each signal only to the endpoint attached for that signal. - Logs, metrics, and traces that the workload sends with OTLP to the collector on ports 14357 or 14358. - Container logs, when `BYOOFluentBit` is on. For container workloads, Fluent Bit reads the logs of the inference or task container. For Helm workloads, it reads the logs of the application pods in the workload namespace that run on the same node as the utils pod. - The collector's own `otelcol_*` metrics, with the workload metrics. - Worker, NVCA, DCGM, cAdvisor, and kube-state-metrics series for the workload, only when platform metrics federation is on. Federation is off by default on self-managed clusters. For more information, refer to [Platform metrics federation](#platform-metrics-federation). ## Security - The NVCF API stores the secret in the secrets service. The API does not return the secret value. - At workload start, the secret is delivered to the collector sidecar and written to files under `/etc/byoo-otel-collector/secrets`. The collector configuration references these files instead of embedding the values. - Use HTTPS or TLS endpoints so credentials and telemetry are encrypted in transit. ## Telemetry compatibility BYOO preserves valid upstream telemetry from the OTel Collector and cAdvisor. Upgrades of those components can change metric names, types, labels, and histogram buckets. For the policy and examples, refer to [BYOO Telemetry Compatibility](/nvcf/dev/compute-plane/byoo-telemetry-compatibility). ## Troubleshooting ### Sidecar image pull fails A workload pod shows `ImagePullBackOff` or `ErrImagePull` for the `byoo-otel-collector` or `fluentbit-logs` container. The cluster cannot pull the sidecar image from its default registry. Mirror the image and set its repository, as described in [Sidecar images](#sidecar-images). ### Federation scrape errors in the collector log The collector log shows scrape errors for its federation jobs, such as `nvcf-worker` and `nvca`. Platform metrics are not exported, except `up` series with the value 0 for the failing jobs. OTLP telemetry from the workload and container logs are not affected. These errors mean that the collector cannot reach its federation target. Read the collector log: ```bash kubectl logs -n -c byoo-otel-collector ``` NVCA checks a configured `target` at startup and every 5 minutes while `BYOObservability` is on. When the check fails, NVCA logs a warning, reports a `byoo-platform-metrics-federation` component with the error in its `/healthz` response, and sets `nvca_byoo_platform_metrics_federation_state{state="unreachable"}` to 1. Workloads still deploy. Check the target host and port, or set `enabled: false`. Read the NVCA log for the check result: ```bash kubectl logs -n nvca-system deploy/nvca -c agent | grep byoo-platform-metrics-federation ``` NVCA does not check the default target. The collector federates from it when federation is on without a `target`, for example with only `enabled: true`. NVCA and collector releases without the federation setting also use it. The default target is the `nvcf-byoo-prometheus-kube-prometheus` service in the `monitoring` namespace, which the self-managed stacks do not install. Set `target`, or set `enabled: false`. ### Container logs are missing Traces and OTLP metrics arrive, but container logs do not. Check that `BYOOFluentBit` is on, as described in [Feature gates](#feature-gates), and that the logs endpoint includes `LOGS` in `types`. Without the gate, only logs that the workload sends with OTLP are exported. ### Function or task fails with a BYOO error A function deployment or task that references telemetry endpoints fails when the target cluster does not have `BYOObservability` on. NVCA rejects the workload and reports it as failed. The function deployment moves to `ERROR`, and the task moves to `ERRORED`. The `error` field of the deployment or task `healthInfo` holds the reason: ```text Bring Your Own Observability (BYOO) is disabled on this cluster: the task requests telemetries, but these NVCA feature flags are not enabled: BYOObservability. Enable them on the cluster, or deploy without telemetries. ``` For a function, the message lists each required flag that is off: `BYOObservability`, and `UseFunctionTranslator`, which is on by default. NVCA does not retry a rejected workload. Turn on the gate with the `compute` or `all` profile, or add `BYOObservability` to `featureGateValues`. Then deploy the function or create the task again. You can also remove the telemetry references from the workload. To check the gates on a cluster, refer to [Verifying Feature Flags](/nvcf/dev/compute-plane/cluster-configuration#verifying-feature-flags).