Reference Design and Request Flow#

Reference Design Scope#

The reference design defines the roles, boundaries, and interactions required to govern inference for applications and agents while aligning their demand and performance requirements with available capacity. It connects the Portal, Gateway, and inference backend services so platform controls and approved developer access are enforced at runtime, while actual demand and serving performance inform subsequent operating decisions.

NVIDIA IT internal implementation is included as part of the core reference design because it shows how the pattern is realized through deployed components, operational signals, and lessons learned. The reference design covers three connected flows:

  • Platform administration: Platform operators use the Portal and administrative APIs to manage the approved model catalog, organizational access and quota controls, and registered serving backends. These decisions form the trusted control configuration enforced by the Gateway at runtime.

  • Developer access: Developers and application teams use the Portal to discover approved models, request access or additional quota based on application and agent requirements, and provision scoped API keys. Applications and agents use these keys to access permitted models.

  • Runtime operations: Applications and agents submit requests through the Gateway using scoped API keys. The Gateway validates each key, applies the approved controls, and forwards the request to an eligible backend for execution. Runtime demand, serving performance, reliability, and cost signals feed into ongoing model, quota, and capacity decisions.

Together, these flows define the platform, developer, and runtime handoffs shown in the following request flow.

End-to-End Access and Request Flow#

Onboarding and runtime request flow.

Figure 3 Onboarding and runtime request flow.#

Developer Access#

Developers and application teams begin in the Portal by selecting an approved model and provisioning a scoped API key. Each key is associated with a user, providing the identity context used to validate and attribute requests. Users can request access to additional models or higher quotas when needed.

To connect a workload, developers configure an application or agent runtime with the Gateway endpoint, scoped API key, and logical model identifier. The Portal provides the API documentation, authentication guidance, and examples needed for this configuration. Once configured, the application or agent runtime submits inference requests to the Gateway.

Gateway and Intelligent Routing#

The gateway is the network-enforced path to routing. Eligibility precedes optimization. Entitlement and compatibility establish access, while data, geography, and contract rules narrow placement. Policy ranks eligible destinations by health and saturation, balancing latency and cost within service requirements.

During overload, policy may spill or fail over to alternate capacity. Approved degradation or bounded queueing can absorb disruption. Otherwise, the request is rejected. Data eligibility outranks availability and price. The gateway decides where a request may go, while serving controls execution.

NVIDIA’s Model-Serving Approach#

The AI Factory inference hub Reference Design provides a managed inference layer above multiple inference endpoints. The core architecture remains implementation-pattern focused; the NVIDIA components below are standards and components used in practice for self-hosted and managed NVIDIA inference backends.

NVIDIA NIM reduces deployment friction by packaging optimized engines, dependencies, configurations, and API wiring for NVIDIA GPUs. NVIDIA Dynamo improves fleet throughput with available-compute routing and disaggregated prefill and decode, letting each phase scale independently. NVIDIA Run:ai replaces idle static allocation with pooled GPU scheduling, quotas, priority, and fair share. NVIDIA Cloud Functions provides a managed NVIDIA integration path when the enterprise does not operate serving infrastructure. Together, these components address deployment packaging, fleet throughput, GPU scheduling, and managed infrastructure access.