> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/ai-enterprise/ai-factory/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/ai-enterprise/ai-factory/_mcp/server.

# Operations

> Day-two service ownership, observability, capacity, change, incident, upgrade, rollback, backup, and recovery guidance.

Operations and resilience require a complete operating model across facilities, infrastructure, cloud-native platform, AI services, data paths, applications, and agents. Named owners must maintain the platform securely, remediate vulnerabilities, respond to incidents, preserve continuity, and recover service and trust state.

Explore [observability ecosystem solutions](/ai-enterprise/ai-factory/ecosystem-integrations/solution-catalog?facet=observability) that provide signals for operations and resilience.

## Ownership Model

Assign accountable owners across the following operational surfaces:

| Surface                 | Accountable Responsibilities                                                                                                                                                                                               |
| ----------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Facilities and hardware | Power, cooling, rack, component health, replacement, firmware, and vendor escalation. Refer to [Hardware authority and OEM handoff](/ai-enterprise/ai-factory/core-reference-design/enterprise-hardware-ra-foundation).                             |
| Network and storage     | Capacity, latency, errors, isolation, data availability, backup, and recovery. Refer to [AI Factory Observability Guide](https://docs.nvidia.com/enterprise-reference-architectures/observability-guide/latest/).          |
| Cloud-Native Platform   | Cluster health, upgrades, policy, operators, scheduling, tenancy, and runtime security. Refer to [GPU Operator troubleshooting](https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/troubleshooting.html). |
| AI platform             | Model lifecycle, inference, orchestration, gateway, connectors, artifacts, and service objectives. Refer to [AI platform responsibilities](/ai-enterprise/ai-factory/core-reference-design/ai-platform).                                            |
| Applications and agents | Quality, outcomes, tool authority, state, release controls, human review, and incident containment. Refer to [NeMo Relay observability](https://docs.nvidia.com/nemo/relay/about-nemo-relay/overview).                     |

## Telemetry Requirements

Correlate:

* Workload quality, latency, throughput, concurrency, and errors.
* Agent plans, model requests, retrieval, tool calls, outcomes, approvals, and faults.
* Service saturation, queues, routing, cache behavior, retries, and dependency health.
* Kubernetes scheduling, quota, eviction, network, storage, and operator state.
* GPU, CPU, memory, interconnect, fabric, storage, thermal, power, and facility signals.
* Security policy decisions, credentials, data access, artifact provenance, and audit events.

Use the [AI Factory Observability Guide](https://docs.nvidia.com/enterprise-reference-architectures/observability-guide.pdf) and current product documentation as implementation sources.

## Organization-Owned Change and Recovery Controls

The reference design does not prescribe one change-management framework. Each deploying organization selects its framework and approval policy, but production operation still requires the following minimum control points.

#### Classify the Change

Identify affected contracts, owners, dependencies, risks, and rollback conditions.

#### Test Representative Behavior

Exercise normal demand, contention, dependency failure, capacity loss, and recovery.

#### Promote Controlled State

Use reviewed desired state and immutable artifacts through environment promotion.

#### Read Back the Deployment

Confirm live configuration, artifact identity, health, telemetry, and workload behavior.

#### Retain Recovery Options

Keep tested rollback, backup, restore, break-glass, and escalation procedures.

## Review Triggers

Review architecture and operations when workloads, trust boundaries, tenant models, products, support status, data paths, scale, facility constraints, critical vulnerabilities, or recovery objectives materially change.