Observability (Legacy Managed)

Overview

NVIDIA Cloud Functions provides a comprehensive observability solution through two main approaches:

1. NGC UI/CLI Observability

   1. Basic metrics in the Overview tab
   2. Log data with time ranges in the Logs tab
   3. Limited to NGC account holders
   4. Enabled by default for all functions or tasks

   See details on using the built-in observability features below.

2. External Observability Integration

   1. Send telemetry data to your organization’s observability platforms
   2. Support for logs, metrics, and traces
   3. Requires explicit configuration of telemetry endpoints

   See external-observability for detailed instructions.

NGC UI Observability

The NGC UI provides basic observability through three main tabs:

1. Overview Tab

   1. Provides real-time status and performance metrics
   2. Shows instance counts and request statistics
   3. Displays basic function or task information

2. Logs Tab

   1. Access to container and event logs
   2. Real-time log streaming capabilities
   3. Search and filtering functionality

3. Metrics Tab

   1. Detailed performance indicators
   2. Time-series data visualization
   3. Resource utilization trends

You can access these tabs by navigating to your function in the NGC UI. Each tab offers specific insights into your function’s operation and performance.

Overview

The Overview tab provides access to your function’s current status and performance metrics, offering real-time insights into your function’s operation and health.

1. Basic Function or Task Metrics

   1. Current function or task status (Running, Stopped, Error)
   2. Last updated timestamp
   3. Function or task version
   4. Runtime environment details

2. Instance Counts

   1. Active instances
   2. Pending instances
   3. Failed instances
   4. Historical instance trends

3. Request Statistics

   1. Total requests processed
   2. Current request rate
   3. Success/failure ratios
   4. Average response times

How to Access

1. Navigate to the Functions list page
2. Click on your function
3. The Overview tab is displayed by default
4. Use the refresh button to update the data

Logs

The Logs tab enables monitoring through detailed log access.

NVCF displays logs related to:

• Deployment stages

  • Function or Task Creation
  • Function or Task Deployment

• Function or task invocation logs

• Real-time logs (listed in the UI under the Live Tail tab)

  • For detailed information about real-time logging capabilities, see the NGC UI Logs tab described above.

Viewing Metrics

1. Navigate to the Functions list page
2. Click on your function
3. Select the Metrics tab

The Metrics view displays:

Summary Statistics

• Total Invocations - Number of function calls in the selected time period
• Average Inference Time - Mean processing time for function calls
• Total Instance Count - Current number of running instances
• Failures - Count of failed executions

Time Series Graphs

• Invocation Activity and Queue Depth - Shows request patterns and queued requests
• Average Inference Time - Processing duration trends
• Instances Over Time - Shows scaling behavior
• Success Rate - Function reliability metrics

Use the time range selector (e.g., Past 1 Hour) in the top right to adjust the view period.

External Observability

Configure external observability endpoints to monitor your NVIDIA Cloud Functions. By setting up telemetry endpoints, you can stream metrics (see appendix-b), logs, and traces to popular observability platforms like Grafana Cloud and Datadog. This extends beyond the basic metrics in NGC UI, giving you deeper insights into your functions’ performance.

Ports

The OpenTelemetry collector uses the following ports:

• OTLP (OpenTelemetry Protocol)

  • OTLP gRPC: Port 14357
  • OTLP HTTP: Port 14358

• Metrics

  • Port 18888 - Used for collector metrics

• Health Check

  • Port 13133 - Used for health check endpoint

Configuration

Telemetry endpoints can only be configured when creating a new function or deploying a new version. You cannot add a telemetry endpoint to an existing function deployment.

A Telemetry Endpoint is a configuration that specifies where telemetry data is sent. This is allowed for all functions or tasks to be configured to send telemetry data to an external observability platform.

1. Configure External Telemetry Endpoints

You can configure telemetry endpoints using either the web UI or the NGC CLI:

Web UI Method:

• Navigate to your NGC organization settings
• Select “Settings” in your Cloud Functions NGC organization
• Scroll to the bottom of the page
• Click “Add Telemetry Endpoint”

• Select your desired endpoint type (Grafana Cloud or Datadog)
• Configure the endpoint with the required credentials

$
ngc cloud-function telemetry-endpoint create --name grafana-cloud-metrics \
>
--type METRICS \
>
--provider GRAFANA_CLOUD \
>
--protocol HTTP \
>
--endpoint https://otlp-gateway-prod-us-west-0.grafana.net/otlp \
>
--key your-grafana-api-token

$
# Example
$
ngc cloud-function telemetry-endpoint create --name datadog-metrics \
>
--type METRICS \
>
--provider DATADOG \
>
--protocol HTTP \
>
--endpoint datadoghq.com \
>
--key your-datadog-api-key

CLI Method:

As an alternative to the web UI, you can use the NGC CLI to manage telemetry endpoints. Here are the basic CLI commands:

$
    # List existing telemetry endpoints
$
    ngc cloud-function telemetry-endpoint list
$
$
    # Create a new telemetry endpoint
$
    ngc cloud-function telemetry-endpoint create --name <endpoint-name> \
>
      --type <LOGS|METRICS> \
>
      --provider <GRAFANA_CLOUD|DATADOG> \
>
      --protocol <GRPC|HTTP> \
>
      --endpoint <endpoint-url> \
>
      --key <api-key>
$
$
    # Remove a telemetry endpoint
$
    ngc cloud-function telemetry-endpoint remove <endpoint-name>

1. Add Telemetry Endpoint to Function or Task

   Telemetry endpoints can only be configured when creating a new function or deploying a new version. You cannot add a telemetry endpoint to an existing function deployment.

   Web UI Method:

   When creating a new function or deploying a new version:

   • In the function creation/deployment form
   • Look for the Telemetry Endpoints section
   • Select the desired telemetry endpoint from the dropdown
   • Complete the rest of the function creation/deployment process

1. Verify Deployment

   After deploying the function with the telemetry endpoint, verify that the telemetry data is flowing correctly to your observability platform.

1. Delete a Function or Task and Remove Telemetry Endpoint

   To remove a telemetry endpoint, you must first cancel all deployments and remove all functions that use that endpoint. The endpoint cannot be removed while any functions are still using it, even if those functions are not currently deployed.

   Web UI method:

   1. Navigate to the Functions list page

   2. Click on the function you want to delete

   3. Navigate to the Deployments tab

   4. For each deployment:

      1. Click “Cancel Deployment” and confirm
      2. Wait for all deployments to be fully cancelled

   5. Navigate to the Settings tab

   6. Click “Delete Function” and confirm

   7. Verify the function is completely removed

   8. After all functions using the telemetry endpoint have been removed:

      1. Navigate to your NGC organization settings
      2. Select “Settings” in your Cloud Functions NGC organization
      3. Scroll to the Telemetry Endpoints section
      4. Find the endpoint you want to remove
      5. Click the delete icon next to the endpoint
      6. Confirm the deletion

   CLI method:

$
  # First, cancel all deployments for a function version
$
  ngc cloud-function function deploy remove <function-id>:<function-version-id>
$
$
  # Wait for deployments to be fully cancelled, then remove the function
$
  ngc cloud-function function remove <function-id>
$
$
  # After all functions using the telemetry endpoint have been removed, delete the endpoint
$
  ngc cloud-function telemetry-endpoint remove <endpoint-name>

When you select a telemetry endpoint, NVCF:

• Deploys a dedicated OpenTelemetry collector with your function or task
• Automatically configures authentication and endpoint connections
• Enables collection of metrics, logs, and traces from your function or task
• Directs telemetry data to your organization’s observability platform

Resource Management

In the pod for each function or task, an OpenTelemetry collector is deployed. This collector has automatic memory management and built-in resource protection to ensure reliable telemetry collection without impacting function or task performance. NVCF manages all resource allocation for the collector, so you don’t need to worry about resource configuration.

Security

NVCF ensures secure telemetry handling by storing credentials securely in the NGC Encrypted Secrets Store, Each collector only accesses its own function’s or task’s data, and authentication is handled automatically. Credentials are rotated securely to maintain security and integrity.

How to Set Up External Observability on a BYOC Cluster

BYOC Steps

• BYOC cluster registered with NVCA on 2.46.10+ version

  • See the NGC-Managed Clusters page for upgrade instructions.

• Ensure the Bring Your Own Observability cluster feature is enabled. If you are running a cluster agent version older than 2.50.0, refer to the Configuration page for managing feature flags.

Error Handling

If issues occur with telemetry collection:

• Your function or task continues to run normally
• Error messages are logged for troubleshooting
• Health status is monitored and reported
• Automatic retry logic handles temporary failures

The collector’s health can be monitored through:

• Status checks in the NGC UI
• Metrics in your observability platform
• Built-in health endpoints

Appendix A: Terminology

Appendix B: Available Metrics

The following metrics are collected through the OpenTelemetry collector deployed with your function when using External Observability and exported through your configured Telemetry Endpoints. The metrics exported depend on the Kubernetes deployment used by the function or task.

Key metrics include:

• Function or task invocation metrics
• Resource utilization metrics
• Platform metrics related to the function or task

CPU Metrics

Memory Metrics

Filesystem Metrics

Only present if the container is performing IO operations:

Network Metrics

Only present if the container is performing network operations:

Kubernetes State Metrics

Only present if helm-based function has a deployment k8s object:

Only present if helm-based function has a replicaset k8s object:

Only present if helm-based function has a stateful k8s object:

Only present if the helm-based function has a job/cronjob k8s object:

Only present if function has a configmap k8s object:

Only present if function has a secret k8s object:

Only present if function has a pod k8s object:

Only present if function/task helm deployments:

Only present if function/task helm defined an init container:

GPU Metrics

Always present for container and helm:

For detailed information about all available DCGM field IDs and GPU metrics, see the NVIDIA DCGM API Field IDs documentation.

NVCF Worker Service Metrics

Streaming metrics are only present for streaming functions.

NVCA Instance Type Metrics

Present for cluster management:

OpenTelemetry Collector Metrics

Always present for container and helm. The final list of metrics depends on telemetries received & exported by function/task:

Resource Attributes

All logs and metrics have the following attributes added to their metadata:

The platform metrics have the following attributes when available:

Appendix C: Adding Custom Application Metrics/Logs/Traces

You can export custom metrics/logs/traces to your external observability platform by sending them to the OpenTelemetry collector. Refer to the following table for the available environment variables that you can specify: