Skip to navigation

AI Platform

View as Markdown
View AI Platform in the interactive architecture

The AI platform turns accelerated infrastructure into reusable services for model and application teams. The core defines capabilities and interfaces. Implementations select supported products and versions.

Explore AI platform ecosystem solutions.

Build and Customize

The lifecycle must support:

  • Approved model, dataset, prompt, tool, and evaluation inputs.
  • Reproducible training or customization jobs.
  • Immutable artifacts with provenance and promotion state.
  • Separation of development, testing, and production authority.
  • Rollback or withdrawal when a release no longer meets requirements.

NVIDIA implementation options include NVIDIA NeMo for model workflows. The broader NVIDIA Agent Toolkit includes NeMo Fabric for agent integration and execution, and NeMo Relay for runtime telemetry and policy integration. Select supported components that satisfy the architectural contract.

Evaluate and Promote Continuously

Evaluation is not a one-time development gate. Apply quality, safety, security, policy, and operational evaluation throughout the lifecycle:

  • Evaluate candidates against representative task, safety, security, and policy scenarios before promotion.
  • Preserve the evaluation set, configuration, result, owner, and asset versions as linked evidence.
  • Correlate production outcomes, agent traces, drift, incidents, latency, and cost with the deployed asset version.
  • Re-evaluate when models, prompts, tools, data, policy, runtimes, or dependent services change.
  • Hold, roll back, or withdraw releases that no longer meet approved thresholds.

NeMo Evaluator is an implementation option for evaluation. NeMo Relay supplies agent telemetry for evaluation and observability systems. AgentOps and Observability supply production evidence. Named owners retain promotion and risk decisions.

Optimize and Serve

Inference services expose consistent model discovery, admission, request, streaming, error, health, and telemetry contracts. Runtime selection follows workload objectives and the selected hardware foundation.

Inference Applications define application demand and service objectives. Optimize and Serve satisfies that demand through model optimization, runtime selection, deployment, scaling, and serving. It does not own application behavior, business outcomes, or enterprise API policy.

Implementation options can include NVIDIA NIM, NVIDIA Dynamo, Triton Inference Server, and TensorRT-LLM. Product inclusion here does not make any one runtime mandatory.

Protect Model IP and Inference Confidentiality

Protect model intellectual property and the confidentiality of inference inputs, retrieved context, and outputs across loading, optimization, serving, caching, and telemetry. Restrict model-artifact access and avoid exposing sensitive request content through logs or traces.

Governance and Assurance defines the protection objectives and acceptance evidence. Platform Services enforces identity, access, and disclosure policy. Select confidential-computing mechanisms only when the threat model and supported deployment require them.

NVIDIA Software Decision Guide

Use this table to distinguish functional roles. It is not a compatibility statement or required bundle. Confirm the target release, GPU, operating system, hypervisor or cloud, Kubernetes distribution, and dependent component versions. Use the NVIDIA AI Enterprise Infrastructure Support Matrix and each product’s current documentation.

ComponentPrimary RoleUse WhenKey Dependencies and Boundary
NVIDIA NeMoModel development, customization, evaluation, and guardrail workflowsTeams need NVIDIA-supported model and data lifecycle capabilitiesRequires governed data, compute, artifacts, evaluation policy, and promotion. It is not the serving control plane.
NVIDIA NIMPackaged model inference endpointsA supported model needs a consistent containerized serving interfaceRequires supported GPU and runtime infrastructure. Service routing, tenancy, and enterprise policy remain adjacent responsibilities.
NVIDIA DynamoDistributed inference execution and scalingServing needs disaggregated or distributed execution across accelerated capacityRequires compatible model runtimes, networking, scheduling, and telemetry. It does not replace enterprise API policy.
Triton Inference ServerMulti-framework model serving protocols and schedulingApplications need a general inference server across supported backendsRequires compatible model repositories and runtimes. Fleet routing and tenant governance remain outside the server.
TensorRT-LLMLarge language model (LLM) runtime optimizationLLM latency, throughput, memory, or scale objectives require NVIDIA GPU optimizationRequires model- and GPU-specific build and runtime compatibility. It is not an API gateway or lifecycle system.
NVIDIA Run:aiGPU scheduling, quotas, and workload orchestrationShared accelerated clusters need governed allocation and utilization controlsRequires a supported platform and resource model. Application service objectives and model lifecycle remain separate.

Orchestrate Resources

Resource orchestration must define:

  • Queues, quotas, priorities, preemption, and fairness.
  • Placement constraints and topology awareness.
  • Whole-GPU, partitioned-GPU, and time-sharing policies where supported.
  • Tenant and project capacity boundaries.
  • Scheduling signals, utilization objectives, and chargeback inputs.
  • Behavior during maintenance, capacity loss, and recovery.

NVIDIA Run:ai is one implementation option for dynamic GPU orchestration.

Scheduling and Accelerator Ownership

Use the following ownership boundaries to separate business policy, Kubernetes scheduling, accelerator enablement, and physical infrastructure:

ResponsibilityOwner in This DesignBoundary
Business priority, project quota, fairness, queue policy, and topology intentResource OrchestrationTranslates workload and organizational policy into placement constraints. The selected implementation can also extend or participate in Pod scheduling and binding.
Default Pod filtering, scoring, binding, reconciliation, and node assignmentEnterprise KubernetesSupplies the default scheduling path against declared constraints and allocatable resources. An integrated scheduler can extend or participate in this path. Kubernetes does not define business priority or physical topology.
GPU drivers, device plugins, container toolkit, node labels, telemetry, and accelerator software lifecycleAccelerator OperatorsReconciles accelerator enablement and exposes resources to Kubernetes. It does not choose workload priority or quota.
Physical systems, fabrics, storage, supported scale points, and topologyEnterprise Hardware Reference ArchitecturesDefines infrastructure capabilities and supported designs. It does not perform software scheduling.

In a Kubernetes-native path, kube-scheduler filters, scores, and binds Pods. With an integrated scheduler such as NVIDIA Run:ai, policy and scheduling responsibilities can extend into that path. Record the selected scheduler, its authority, and its interaction with Kubernetes instead of assuming one universal owner.

Refer to the Kubernetes Scheduling Framework and kube-scheduler descriptions for the default pod-scheduling boundary. Refer to the NVIDIA Run:ai and NVIDIA GPU Operator documentation for implementation-specific orchestration and accelerator-software lifecycle scope.

Interfaces to Adjacent Layers

The AI Platform exchanges the following contracts with adjacent layers:

InterfaceContract
WorkloadsDemand, quality, latency, throughput, tenancy, and availability objectives.
Platform servicesIdentity, gateway, connectors, policy, artifacts, delivery, and telemetry.
Cloud-Native PlatformScheduling primitives, runtimes, networking, storage, secrets, and operators.
Hardware foundationCompute, memory, network, storage, scale, and availability envelope.