Overview
Alpamayo 2 Super. For the previous generation, see Alpamayo 1.5 Introduction.
Alpamayo 2 Super NIM serves nvidia/alpamayo-2-super for trajectory prediction and driving-scene understanding. It accepts camera images, ego-motion history, optional navigation text, and task-specific questions.
Alpamayo 2 Super is a vision-language-action model. Its released safetensors checkpoint holds about 35.8 billion parameters in total; summaries that round down or count a subset of components can quote a smaller figure such as 34B. The BF16 checkpoint is about 72 GB (67 GiB) of weights, and every serving profile requires at least 80 GB of GPU memory. Refer to Support Matrix for per-profile memory floors.
The NIM exposes HTTP APIs for trajectory generation, visual question answering, meta-action prediction, auto-labeling, and visual grounding. It also exposes a gRPC API for trajectory generation.
Architecture
At a high level, Alpamayo 2 Super combines vision-language understanding with a trajectory-generation model:
- A vision-language model processes the selected camera context (six cameras by default), ego-motion history, navigation text, and task prompt.
- For trajectory generation, the trajectory expert and diffusion head use that context to sample future ego trajectories.
- For VQA, meta-action, auto-labeling, and grounding, the vision-language path returns natural-language or structured task output.
Precision Selection
The NIM automatically selects a compatible runtime profile for the visible GPU. To request a specific precision, set NIM_PRECISION to bf16 or fp8. DFlash2 profiles are selected explicitly with NIM_MODEL_PROFILE. For the full GPU, profile, and trajectory-count matrix, refer to Support Matrix.
Capabilities
The canonical POST /v1/infer API exposes five explicit tasks:
trajectory: Generate future ego trajectories and chain-of-thought reasoning from the driving cameras (six by default, or a declared subset) and ego-motion history.vqa: Answer a natural-language question about one or more camera streams.meta_action: Explain the scene and return structured longitudinal, lateral, and lane actions.auto_labeling: Analyze critical components and causal relationships using either a future trajectory supplied by the caller or a K=1 trajectory predicted by the NIM.grounding: Locate an object in one camera and return structured bounding boxes tied to that camera.
APIs
The container includes the generated OpenAPI specification at /opt/nim/api_spec.yaml. The primary endpoint is POST /v1/infer for all five tasks. POST /v1/chat/completions provides OpenAI-compatible trajectory generation, POST /v1/vqa provides OpenAI-compatible VQA, and the gRPC Predict API provides trajectory generation.
Input Profiles
Trajectory defaults to the six-camera layout, four context frames per camera, and 16 ego-history poses described in Inference Tasks. It also accepts explicit non-empty subsets of the six driving cameras. Meta-action and auto-labeling retain the complete profile. VQA accepts an explicit non-empty camera subset. Grounding accepts exactly one camera so that every returned box has an unambiguous camera association.
For request contracts and complete examples, refer to Inference Tasks and API Reference. Alpamayo 1.5 clients that still send the legacy mode field continue to work; refer to Release Notes for the differences that affect them.
Serving Backend and Model Results
The serving backend is vLLM-Omni. It executes the vision-language path and trajectory expert behind the NIM APIs, with prefix caching and compiled paths. Requests arrive at the NIM’s HTTP and gRPC front end, which hands them to the model through an in-container PyTriton binding (a Triton Inference Server process started inside the container), and vLLM-Omni executes the model. Startup logs therefore show Triton and PyTriton messages as well as vLLM-Omni messages; they belong to one serving stack, not to alternative backends.
Startup includes model preparation; profile changes require a restart, and each container processes admitted requests serially. A2S in diagnostic material means Alpamayo 2 Super.
For model architecture, parameter accounting and research accuracy results, see the Alpamayo2-Super model card. Model-card results do not establish accuracy for every NIM precision, camera layout or custom checkpoint.