> This page is for version Development. > For other versions, use one of these documentation indexes: > - 1.0.3 (default): https://docs.nvidia.com/nvcf/1.0.3/llms.txt > - 1.0.2: https://docs.nvidia.com/nvcf/1.0.2/llms.txt > - 1.0.1: https://docs.nvidia.com/nvcf/1.0.1/llms.txt > - Development: https://docs.nvidia.com/nvcf/dev/llms.txt > For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/nvcf/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nvcf/_mcp/server. # Event Ledger Event Ledger records function deployment events from the NVCF control plane and its registered compute planes. Use it to find the latest deployment state for each workload context and inspect the events that led to that state. Event Ledger is an observation service. It does not schedule workloads, change deployment state, or replace application logs, metrics, or traces. ## How it works Event Ledger combines events from three producers: - The Spot Instance Service (SIS) publishes function instance lifecycle events from the control plane as CloudEvents. - The NVIDIA Cluster Agent (NVCA) OpenTelemetry Collector watches Kubernetes Pod and ICMSRequest events on each enabled compute plane and publishes them as OpenTelemetry Protocol (OTLP) log records. - The gRPC proxy publishes service lifecycle events as CloudEvents when its Event Ledger publisher is configured. The Event Ledger stores events from these producers under a namespace. For function events, the namespace is the function version ID. Each event also has a context that identifies the workload or resource that produced it. Event delivery is asynchronous. A newly produced event might not be visible immediately. ## Event model An Event Ledger record contains these fields: | Field | Description | | --- | --- | | `namespace` | Function version ID for function deployment events. | | `context` | Canonical, comma-separated set of fields that identifies one workload or resource. | | `event_name` | Producer-defined event name. | | `source` | Producer of the event, such as `nvidia-spot` or `nvca`. | | `timestamp` | Time reported by the producer. | | `details.body` | Producer payload, such as a Kubernetes event message or object. | | `details.attributes` | Additional producer attributes available for inspection and filtering. | | `created_at` | Time the event name was first stored for the context. | | `updated_at` | Time the stored event was last updated. | ### Context fields A context can contain any combination of these fields. The canonical context uses the order shown below: | Context field | Meaning | | --- | --- | | `cluster_id` | Registered compute-plane cluster ID. | | `deployment_id` | NVCF deployment ID. | | `gpu_specification_id` | GPU specification selected for the deployment. | | `instance_id` | Function instance or Kubernetes Pod name. | | `resource_id` | Producer-supplied resource identifier, such as an ICMSRequest ID. | For example: ```text cluster_id=cluster-1,deployment_id=5a3ef171-f6df-48cd-81db-c8096a6298d6,gpu_specification_id=gpu-a100,instance_id=function-abc-0 ``` Context matching is exact. Supply every field shown in the context when you read its events. If no context fields are supplied, Event Ledger returns only events that were stored without a context. It does not return events from every context. Context values can contain letters, numbers, and dashes. `instance_id` can also contain dots between non-empty segments. An invalid context value returns `400 Bad Request`. ### Event names SIS publishes these function instance lifecycle event names: | Event name | Meaning | | --- | --- | | `InstanceReady` | The function instance became ready. | | `InstanceDestroyed` | The function instance was destroyed. | | `InstanceRequestingTermination` | Termination of the function instance was requested. | The NVCA collector creates names from Kubernetes event reasons: | Event name form | Meaning | | --- | --- | | `Pod.` | A Pod-level Kubernetes event. | | `Pod..` | A Kubernetes event for a specific container in a Pod. | | `ICMSRequest.InstanceCreation[.]` | An instance creation request event. | | `ICMSRequest.InstanceStatusUpdate[.]` | An instance state change. | | `ICMSRequest.InstanceTermination[.]` | An instance termination request event. | For example, a running instance can produce `ICMSRequest.InstanceStatusUpdate.running`. Kubernetes reason values are producer-defined, so Pod event names vary by workload and cluster. NVCA events can include attributes such as `function_id`, `nca_id`, `icms_request_id`, `instance_type_name`, `failure_category`, `termination_cause`, `k8s.event.reason`, and Kubernetes object metadata. Use JSON output to inspect all attributes on an event. ## Storage semantics `stats get` returns one row per context. Each row contains the latest event for that context, and the summary counts contexts by their latest event name. Event Ledger stores one row for each unique combination of namespace, context, and event name. If the same event name occurs again for the same context, the new occurrence replaces the stored payload and timestamp. `created_at` remains the time that event name was first stored, while `updated_at` records the most recent update. As a result, `stages events list` returns the latest stored occurrence of each event name for a context, ordered by event timestamp. It is not an immutable log of every repeated occurrence. The v3 read endpoints do not paginate results. A read returns all stored event names for the selected context. The v3 API does not expose a retention duration. Plan Cassandra capacity and backups according to the event rate of the registered compute planes. ## Prerequisites Before reading events: 1. Enable and verify Event Ledger on the control plane. 2. Enable the NVCA collector on every compute plane whose Kubernetes events you want to record. 3. Configure `nvcf-cli` with the Event Ledger endpoint and gateway host. 4. Generate a dedicated Event Ledger API key. See [Enable and Operate Event Ledger](/nvcf/dev/self-managed/event-ledger) for the self-managed setup. ## Generate an API key Event Ledger uses a dedicated audience and does not fall back to the function or task API key. Generate a read key with the default `fnds:getEvents` scope: ```bash nvcf-cli api-key generate --for event-ledger ``` The CLI saves the key as its dedicated Event Ledger credential. It also reads `EVENT_LEDGER_API_KEY` from the environment. Running `nvcf-cli api-key generate` without `--for` generates function, task, and Event Ledger keys together. The default key lifetime is 24 hours. Use `--expires-in` to choose another duration. ## Inspect deployment state Start with `stats get`. The command accepts a function version ID or uses the version ID saved by a previous CLI function command: ```bash nvcf-cli stats get --function-version-id ``` Example output: ```text Function version: 38ff2c62-c76f-4f89-b944-f3aaf68bf62a Contexts: 2 ICMSRequest.InstanceStatusUpdate.running: 1 InstanceReady: 1 CONTEXT EVENT TIMESTAMP FIRST SEEN instance_id=function-abc-0 InstanceReady 2026-10-26T20:10:20Z 2026-10-26T20:09:01Z cluster_id=cluster-1,resource_id=request-123 ICMSRequest.InstanceStatusUpdate.running 2026-10-26T20:10:18Z 2026-10-26T20:08:55Z ``` Use `--event-filter` to keep contexts whose latest event has one of the exact event names: ```bash nvcf-cli stats get \ --function-version-id \ --event-filter InstanceReady,ICMSRequest.InstanceStatusUpdate.running ``` Use `--json` when scripting or when you need to inspect contexts without the text table: ```bash nvcf-cli stats get \ --function-version-id \ --json | jq '.contexts[]' ``` ## Read events for a context Copy the fields from one `stats get` context into `stages events list`. Include only the fields present in that context. For an SIS instance context: ```bash nvcf-cli stages events list \ --function-version-id \ --instance-id ``` For a Pod context that contains all four fields: ```bash nvcf-cli stages events list \ --function-version-id \ --cluster-id \ --deployment-id \ --gpu-specification-id \ --instance-id ``` For an ICMSRequest resource context: ```bash nvcf-cli stages events list \ --function-version-id \ --cluster-id \ --resource-id ``` Text output includes the timestamp, event name, severity, and message. JSON output also includes the source, attributes, body, `created_at`, and `updated_at` fields: ```bash nvcf-cli stages events list \ --function-version-id \ --cluster-id \ --resource-id \ --json | jq '.events[]' ``` Events are returned most recent first. ### Filter by an event attribute Use `--attribute-key` and `--attribute-value` together to keep events whose `details.attributes` entry has the requested value: ```bash nvcf-cli stages events list \ --function-version-id \ --cluster-id \ --deployment-id \ --gpu-specification-id \ --instance-id \ --attribute-key icms_request_id \ --attribute-value ``` Attribute filtering applies after the context is selected. It does not search other contexts in the namespace. Both flags are required, and comparison uses the string representation of the stored attribute value. ## HTTP API The CLI uses the Event Ledger v3 read API: | Method | Endpoint | Purpose | | --- | --- | --- | | `GET` | `/v3/ledger/namespace/{namespace}/stats` | Return the latest event for every context. | | `GET` | `/v3/ledger/namespace/{namespace}/events` | Return stored event names for one exact context. | The stats endpoint accepts `eventFilter`, a comma-separated list of exact event names. The optional `view=filtered` selects the filtered stats view when it is enabled in the service configuration. An unknown view returns `400`. Requesting the filtered view when it is disabled returns `404`. Use an Event Ledger API key as a bearer credential: ```bash curl -sS "${EVENT_LEDGER_URL}/v3/ledger/namespace/${FUNCTION_VERSION_ID}/stats" \ -H "Authorization: Bearer ${EVENT_LEDGER_API_KEY}" \ -H "Accept: application/json" ``` The response contains the latest event for every context: ```json { "namespace": "", "summary": { "total_contexts": 1, "by_event": { "ICMSRequest.InstanceStatusUpdate.running": 1 } }, "contexts": [ { "context": "cluster_id=cluster-1,resource_id=request-123", "event_name": "ICMSRequest.InstanceStatusUpdate.running", "timestamp": "2026-10-26T20:10:18Z", "first_seen": "2026-10-26T20:08:55Z" } ] } ``` The events endpoint accepts `instance_id`, `deployment_id`, `gpu_specification_id`, `cluster_id`, `resource_id`, `attribute_key`, and `attribute_value`. The context fields select one exact context. Send `attribute_key` and `attribute_value` together. ```bash curl -sS "${EVENT_LEDGER_URL}/v3/ledger/namespace/${FUNCTION_VERSION_ID}/events?cluster_id=cluster-1&resource_id=request-123" \ -H "Authorization: Bearer ${EVENT_LEDGER_API_KEY}" \ -H "Accept: application/json" ``` The response contains the latest stored occurrence of each event name for that context: ```json { "namespace": "", "context": "cluster_id=cluster-1,resource_id=request-123", "events": [ { "event_name": "ICMSRequest.InstanceStatusUpdate.running", "source": "nvca", "timestamp": "2026-10-26T20:10:18Z", "details": { "severity": "Info", "body": "Instance is running", "attributes": {} }, "created_at": "2026-10-26T20:08:55Z", "updated_at": "2026-10-26T20:10:18Z" } ] } ``` When sending requests to a gateway address instead of the Event Ledger DNS name, also set `Host: events.`. Event producers use these v3 endpoints: | Method | Endpoint | Content type | Producer authentication | | --- | --- | --- | --- | | `POST` | `/v3/ledger/cloudevents` | `application/cloudevents+json` or `application/cloudevents-batch+json` | Service bearer token or an authorized NVCA projected token. | | `POST` | `/v3/ledger/k8s-events` | `application/x-protobuf` OTLP logs, optionally gzip-compressed | Service bearer token or an authorized NVCA projected token. | `POST /v3/ledger/cloudevents` accepts one structured CloudEvent or a batch. `namespace` is required. A producer keyed by function version can send `functionVersionId` instead. `POST /v3/ledger/k8s-events` accepts an OTLP `ExportLogsServiceRequest`. Each log record requires the string attributes `event_name`, `namespace`, and `source`. The request can be gzip-compressed. Service credentials need the `fnds:createEvent` scope to write. A read key with `fnds:getEvents` cannot write, and a write credential cannot read unless it also has a read scope. An authorized NVCA projected token can write after Event Ledger verifies its cluster identity through SIS. The CLI does not wrap the producer endpoints. SIS and the NVCA collector configure publishing as part of the self-managed deployment. gRPC proxy publishing is opt-in; see [Enable and Operate Event Ledger](/nvcf/dev/self-managed/event-ledger). The API returns errors as `application/problem+json`. Common status codes are: | Status | Meaning | | --- | --- | | `400` | Invalid namespace, context, filter, or event payload. | | `401` | The bearer credential is missing, or a JWT fails validation. | | `403` | An opaque API key is unknown, expired, lacks the required scope, or lacks tenant access. | | `404` | The route or optional filtered stats view was not found. | | `413` | A decompressed OTLP request exceeded 10 MiB. | | `415` | The producer used an unsupported content type. | | `500` | Event Ledger could not read or write its database. | Self-managed deployments expose the v3 API. The legacy v1 and v2 APIs are disabled and are not part of the supported self-managed interface. ## Task events The NVCA collector can store task-related Kubernetes events under a task ID namespace. The current `stats` and `stages` CLI commands accept only a function version UUID, so they do not query task namespaces. Use `nvcf-cli task events ` for the NVCT task lifecycle API. That command and the Event Ledger read commands use different APIs and credentials. ## Limitations - Event Ledger does not store every repeated occurrence of an event name. - Context reads require an exact context match. - Attribute filters do not search across contexts. - The v3 read API does not paginate results. - The Event Ledger CLI commands currently query function version namespaces only.