Architecture

Overview

NVIDIA Attribution is two components plus a set of connector frameworks the Blueprint exposes to operators.

The Service does not query schedulers, read logs, or scrape metrics. The Blueprint does. The Service receives normalized observations over HTTP and returns a verdict. That separation is what lets operators run the Service centrally and the Blueprint at the edge of each cluster.

Proposed extensions are documented separately. See the Kueue Pod placement recorder reference for durable Pod-to-node placement before post-hoc Kueue attribution.

End-to-end flow for one workload

1. Discover. The Blueprint’s execution-context adapter (Slurm shells to sacct, Kueue shells to kubectl, Static hand-feeds one workload) returns one or more ExecutionContext objects.
2. Collect. For each context, every configured log and metric collector runs collect(ctx) and returns zero or more Observation objects. Each Observation carries a Source enum value (applog, syslog, node_metrics) and an entry_timeseries body; the entry shape is in Observation entry format.
3. Open attributor. POST /v2/attributor creates a per-context handle. On a duplicate-session 409, the connector raises AttributorConflictError with the original HTTP error as its cause. The Blueprint preserves the existing session, records that context as conflicted in the run statistics and summaries, and continues with other contexts. Retry after the existing session’s owner finishes and deletes it.
4. Submit observations. Each Observation goes via POST /v2/attributor/{id}/observation. Individual failures are logged and skipped. A duplicate-observation 409 counts as delivered. Per-connector delivery outcomes land in ingestion_reports.
5. Get verdict. GET /v2/attributor/{id}/attribution returns the raw verdict.
6. Clean up. DELETE /v2/attributor/{id} for the handle created by this run so later runs don’t collide. A failed create does not grant ownership of an existing handle.
7. Enrich. The Blueprint’s result.enrich_attribution_result() maps the service verdict to (terminal_state, sub_domain) via taxonomy.py, and attaches a ConnectorReport per collector.
8. Write. Each configured output sink (stdout, sqlite) persists the enriched result.

The orchestrator exposes conflict counts in last_run_stats["conflicted"] and records every attributed or conflicting context in last_run_summaries. Each summary identifies the context, job, and attempt, with a status of attributed or conflicted. Attributed entries include verdict fields; conflict entries include the existing attributor_id and do not represent an attribution verdict. attributed means a verdict was obtained; the succeeded counter still requires that at least one output sink accepted it.

The CLI prints these summaries and conflict totals for each run and across multiple configuration files. Its exit policy is unchanged: a configuration with attempted contexts but no successful output exits nonzero; partial success or no discovered contexts exits zero. Scheduling systems that need to detect partial conflicts must inspect the summaries; a zero exit code alone does not mean every context was attributed. With --strict, the CLI also exits nonzero when any attributed context’s evidence_status is not complete. See the session lifecycle guidance for retry and abandoned-session handling.

Connector frameworks

The Blueprint’s configuration model is patterned on Open MPI’s Modular Component Architecture (MCA). Frameworks group components by role. Each framework registers at import time.

Three configuration surfaces feed one canonical settings tree: CLI flags (--connector eca slurm ids "1,5,7"), env vars (NVAB_ECA_SLURM_IDS=...), and TOML ([eca.slurm] with ids = "1,5,7"). Precedence: CLI > env > TOML > defaults.

Output schema

The Service returns a raw verdict. The Blueprint enriches it with normalized fields and connector telemetry. Grounded against src/attribution_blueprint/result.py and taxonomy.py.

terminal_state and sub_domain are mapped from the service verdict via taxonomy.py. The mapping is deterministic. Downstream consumers should branch on these normalized fields.

connector_reports is the collector coverage record for a run. A collector that raised mid-run reports status: "error" with a reason. A collector that returned zero observations reports observation_count: 0. Observation submission failures after collection are logged and skipped by the Blueprint, but they are not represented in connector_reports. Downstream consumers treating verdicts as evidence should inspect this list and the run logs.

Data sources (V1)

Source names are wire-format enum values the Service expects. Each log or metric connector instance produces exactly one Source. A backend serving multiple Sources is instantiated once per Source so failure isolation lives at the instance boundary.

Auth

The Blueprint’s connectors/auth/ package provides two HTTP auth schemes shared by telemetry connectors (Loki, PromQL):

• Token-based. Bearer token in the Authorization header. Token issuance happens through whatever identity system the operator already runs.
• mTLS. Mutual TLS. Client and server certificates configured per connector instance.

The Blueprint attribution client calls the Attribution Service with service_url and timeout_seconds; it does not attach bearer-token or mTLS credentials to Service requests. If a deployment needs auth in front of the Service API, terminate it outside the Service process, for example at an ingress, gateway, or sidecar.

Output backends (V1)

• stdout. Prints the enriched result as JSON. Used for one-shot CLI runs and for piping into other tools.
• sqlite. Persists the result into a local SQLite database. The schema lives in blueprint/docs/output_schema.md and carries three tables (schema_version, execution_context, attribution_run). The read-side nvidia-attribution-blueprint-web dashboard queries this database.

Multiple sinks can be configured per run. Per-sink failures are isolated. The architecture supports additional sinks (Elasticsearch, Kafka, downstream HTTP) via the same OutputConnector Protocol. None ship at V1.