Skip to navigation

Operations

View as Markdown
View cross-stack controls in the interactive architecture

Operations and resilience require a complete operating model across facilities, infrastructure, cloud-native platform, AI services, data paths, applications, and agents. Named owners must maintain the platform securely, remediate vulnerabilities, respond to incidents, preserve continuity, and recover service and trust state.

Explore observability ecosystem solutions that provide signals for operations and resilience.

Ownership Model

Assign accountable owners across the following operational surfaces:

SurfaceAccountable Responsibilities
Facilities and hardwarePower, cooling, rack, component health, replacement, firmware, and vendor escalation. Refer to Hardware authority and OEM handoff.
Network and storageCapacity, latency, errors, isolation, data availability, backup, and recovery. Refer to AI Factory Observability Guide.
Cloud-Native PlatformCluster health, upgrades, policy, operators, scheduling, tenancy, and runtime security. Refer to GPU Operator troubleshooting.
AI platformModel lifecycle, inference, orchestration, gateway, connectors, artifacts, and service objectives. Refer to AI platform responsibilities.
Applications and agentsQuality, outcomes, tool authority, state, release controls, human review, and incident containment. Refer to NeMo Relay observability.

Telemetry Requirements

Correlate:

  • Workload quality, latency, throughput, concurrency, and errors.
  • Agent plans, model requests, retrieval, tool calls, outcomes, approvals, and faults.
  • Service saturation, queues, routing, cache behavior, retries, and dependency health.
  • Kubernetes scheduling, quota, eviction, network, storage, and operator state.
  • GPU, CPU, memory, interconnect, fabric, storage, thermal, power, and facility signals.
  • Security policy decisions, credentials, data access, artifact provenance, and audit events.

Use the AI Factory Observability Guide and current product documentation as implementation sources.

Organization-Owned Change and Recovery Controls

The reference design does not prescribe one change-management framework. Each deploying organization selects its framework and approval policy, but production operation still requires the following minimum control points.

1

Classify the Change

Identify affected contracts, owners, dependencies, risks, and rollback conditions.
2

Test Representative Behavior

Exercise normal demand, contention, dependency failure, capacity loss, and recovery.
3

Promote Controlled State

Use reviewed desired state and immutable artifacts through environment promotion.
4

Read Back the Deployment

Confirm live configuration, artifact identity, health, telemetry, and workload behavior.
5

Retain Recovery Options

Keep tested rollback, backup, restore, break-glass, and escalation procedures.

Review Triggers

Review architecture and operations when workloads, trust boundaries, tenant models, products, support status, data paths, scale, facility constraints, critical vulnerabilities, or recovery objectives materially change.