inference hub Reference Implementation#

AI Factory inference hub — LiteLLM Reference Implementation

The reference implementation combines developer self-service with centralized governance, on-premises AI Factory serving, and hybrid model access. Applications use one interface while policy, capacity, lifecycle, and operations remain centrally managed.

The inference hub control plane comprises the management interfaces, APIs, and controllers provided by the components in the Reference Shape below. Together, they manage access, routing, capacity, and service lifecycles.

Reference Shape#

  • Custom Developer Portal: Gives developers self-service model discovery and access through centrally managed catalogs, quotas, and usage controls.

  • LiteLLM Gateway: Anchors the inference interface, centralizing access, budget, and destination policy while shielding applications from backend differences.

  • NVIDIA NeMo Switchyard: Adds centrally managed, intelligent model-selection strategies behind Gateway policy without pushing routing logic into applications.

  • Red Hat OpenShift: Provides the operational foundation for deploying, securing, and managing the lifecycle of the on-premises hub.

  • NVIDIA Run:ai: Translates quotas, priorities, and capacity policy into efficient GPU allocation and workload placement without overriding runtime decisions.

  • NVIDIA Dynamo: Orchestrates and optimizes distributed inference across GPU fleets for higher throughput, lower latency, and efficient scaling.

  • NVIDIA NIM: Packages optimized models, runtimes, and validated hardware profiles into repeatable, supported inference deployments.

  • llm-d: Makes distributed LLM serving declaratively manageable through Kubernetes as inference engines scale across workers and nodes.

  • NVIDIA Cloud Functions: Adds managed deployment, scaling, and lifecycle for elastic GPU-backed execution beyond on-premises capacity.

  • Frontier-model APIs: Expand model choice through provider-hosted capabilities that the hub does not operate locally.

  • Datadog: Correlates Gateway, runtime, and infrastructure signals for shared health, usage, and capacity visibility.

Operating Invariants#

  • All model traffic passes through a governed inference gateway.

  • API normalization must not silently change provider-specific behavior.

  • Access management remains outside the inference request path.

  • Policy, quota, capacity, and lifecycle changes remain outside the inference request path.

  • Routing decisions cannot bypass access policy or destination eligibility.

  • Workload scheduling remains separate from model-request routing.

  • Serving systems retain ownership of inference execution and internal coordination.

Why This Matters#

This stack gives developers one governed interface to hybrid inference services spanning on-premises, managed, and SaaS environments. Centralized control preserves security and operational consistency while models, providers, and serving infrastructure evolve independently of applications.