> This page is for version 1.0.0 (Latest) (default). > For other versions, use one of these documentation indexes: > - 1.0.0 (Latest) (default): https://docs.nvidia.com/nim/alpamayo/1.0.0/llms.txt > For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/nim/alpamayo/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nim/alpamayo/_mcp/server. # API Reference ## Endpoints Schema The following HTTP endpoints are available for Alpamayo1.5 NIM: - `/v1/infer` - `/v1/chat/completions` - `/v1/vqa` - `/v1/health/ready` - `/v1/health/live` - `/v1/license` - `/v1/metrics` - `/v1/metadata` - `/v1/manifest` - `/v1/models` - `/v1/version` The NIM also exposes a binary gRPC API on port `50051` for trajectory generation. VQA is served through HTTP. ## API Examples Use the examples in this section to get started with the API. ### Choose an HTTP Request Envelope The three inference endpoints use two different request envelopes: - `POST /v1/infer` uses the native Alpamayo schema. Set `mode` to `trajectory`, `answer`, or `both`. `images`, `question`, `egomotion`, and all trajectory controls are top-level fields. Do not wrap them in `nvext`. `answer` mode does not require ego-motion; `trajectory` and `both` modes require it. - `POST /v1/chat/completions` uses the OpenAI-compatible message schema. Images are `image_url` parts under `messages[].content`. Alpamayo-specific trajectory fields such as `egomotion`, `num_traj_samples`, `nav_text`, `camera_order`, and `num_frames_per_camera` are nested under `nvext`. OpenAI-style controls such as `max_tokens`, `temperature`, and `top_p` remain top-level. `seed` and `top_k` can be top-level or under `nvext`; values under `nvext` take precedence. - `POST /v1/vqa` uses the same OpenAI-compatible message schema. Put the question in a text content part or `nvext.question`. A top-level `question` or `images` field is not accepted by this endpoint. The following abbreviated payloads show where the endpoint-specific fields belong. Replace the placeholders with the full image and ego-motion values shown later on this page. `/v1/infer` trajectory generation: ```json { "mode": "trajectory", "images": ["data:image/jpeg;base64,"], "egomotion": {"ego_history_xyz": [], "ego_history_rot": []}, "num_traj_samples": 1, "seed": 42, "top_k": 1, "top_p": 1 } ``` `/v1/infer` visual question answering: ```json { "mode": "answer", "images": ["data:image/jpeg;base64,"], "question": "What is ahead?", "seed": 42, "top_k": 1, "top_p": 1 } ``` `/v1/infer` combined trajectory generation and visual question answering: ```json { "mode": "both", "images": ["data:image/jpeg;base64,"], "question": "What is ahead?", "egomotion": {"ego_history_xyz": [], "ego_history_rot": []}, "num_traj_samples": 1, "seed": 42, "top_k": 1, "top_p": 1 } ``` Native `/v1/infer` responses contain `answer` for `answer` mode, trajectory fields for `trajectory` mode, and both sets of fields for `both` mode. `/v1/chat/completions`: ```json { "model": "nvidia/alpamayo1.5", "messages": [{ "role": "user", "content": [{ "type": "image_url", "image_url": {"url": "data:image/jpeg;base64,"} }] }], "seed": 42, "top_k": 1, "top_p": 1, "nvext": { "egomotion": {"ego_history_xyz": [], "ego_history_rot": []}, "num_traj_samples": 1, "nav_text": "Drive forward." } } ``` `/v1/vqa`: ```json { "model": "nvidia/alpamayo1.5", "messages": [{ "role": "user", "content": [ {"type": "text", "text": "What is ahead?"}, { "type": "image_url", "image_url": {"url": "data:image/jpeg;base64,"} } ] }], "nvext": {"question": "What is ahead?"} } ``` ### Check Health Use the following command to check server readiness. **cURL Request** ```bash curl -X 'GET' 'http://localhost:8000/v1/health/ready' ``` **Response** ```json { "object": "Triton readiness check", "message": "ready", "status": "ready" } ``` The `object` field reflects the underlying backend readiness check and is not a fixed string. Automation should treat the service as ready based on the HTTP `200` status code and `status` being `"ready"`, rather than matching on `object` or `message`. ### Inspect Service Information The following read-only endpoints help clients identify the running release and selected model profile. The examples use `jq` only to keep large responses compact; omit the filter to retrieve each complete response. - `GET /v1/metadata` Returns deployment metadata, including the NIM version, model resource, and profile selected for the visible GPU. This is the quickest way to confirm which profile an automatically configured container loaded. ```bash curl --fail --silent http://localhost:8000/v1/metadata \ | jq '{version, selectedModelProfileId, modelInfo}' ``` ```json { "version": "1.0.0", "selectedModelProfileId": "", "modelInfo": [ { "modelUrl": "ngc://nim/alpamayo-1-5-10b:", "shortName": "alpamayo-1-5-10b:" } ] } ``` - `GET /v1/manifest` Returns the embedded model manifest as YAML in the `manifest_file` JSON field. The manifest lists available profiles, selection tags, and required model artifacts. To inspect its header: ```bash curl --fail --silent http://localhost:8000/v1/manifest \ | jq -r '.manifest_file' | sed -n '1,12p' ``` ```yaml schema_version: '2.0' profile_selection_criteria: auto model: nvidia/alpamayo1.5 release: 1.0.0 profiles: - id: tags: number_of_gpus: '1' precision: bf16 ``` - `GET /v1/version` Returns the NIM release identifier and the version of the common NIM HTTP API layer. Use it for compatibility checks and support reports. ```bash curl --fail --silent http://localhost:8000/v1/version | jq ``` ```json { "release": "1.0.0", "api": "3.1.0" } ``` ### Generate Trajectory Use trajectory inference to generate sampled future ego trajectories from compressed camera images, ego-motion history, and optional navigation text. The default input layout is 16 images, ordered as four timesteps for each camera: ```text cross_left_t0, cross_left_t1, cross_left_t2, cross_left_t3, front_wide_t0, front_wide_t1, front_wide_t2, front_wide_t3, cross_right_t0, cross_right_t1, cross_right_t2, cross_right_t3, front_tele_t0, front_tele_t1, front_tele_t2, front_tele_t3 ``` `t0` is the oldest frame, approximately `t-0.3s`. `t3` is the current frame. Trajectory requests can use fewer images than the default layout. When the image count is not 16, include `camera_order` and `num_frames_per_camera`. Images must be ordered camera-major according to those fields. For example, a request with `camera_order=["front_wide"]` and `num_frames_per_camera=4` contains four front-wide frames. A request with four camera names and `num_frames_per_camera=1` contains one image per camera. If the image count is not 16 and no explicit layout is supplied, request-model validation returns HTTP 422. Camera names can be full Alpamayo camera names, such as `camera_front_wide_120fov`, or OSS short names, such as `front_wide`. Every trajectory request requires exactly 16 ego-motion history entries, regardless of the number of images. `ego_history_xyz` must have shape `[16, 3]`. `ego_history_rot` must contain 16 quaternions with shape `[16, 4]` (each row `[w, x, y, z]`) or 16 rotation matrices with shape `[16, 3, 3]`. Reduced and single-image layouts do not reduce the ego-motion history length. Requests with another history length are rejected before CUDA inference. The runnable HTTP examples in this section use a payload builder and sample scene shipped in the NIM. Copy both from a running container after you start it: ```bash docker cp alpamayo1.5:/opt/nim/build_http_payload.py ./build_http_payload.py docker cp alpamayo1.5:/workspace/sample_data ./sample_data ``` For trajectory endpoints, the builder uses all 16 real sample images in the default four-camera, four-frame layout and all 16 ego-motion history entries. It emits JSON containing real image data URLs; no base64 substitution is required. The optional `--layout reduced` mode uses only `front_wide_t0` and is intended for quick validation tests, not as the primary usage example. VQA uses the single `front_wide_t0` image because it does not require a trajectory camera layout. For a consolidated description of every helper flag, refer to [Payload Builder Options](/nim/alpamayo/quickstart-guide#payload-builder-options). The gRPC route is recommended for warm trajectory serving because it avoids the HTTP base64 and JSON adapter path. The service is `nvidia.alpamayo.v1.Alpamayo` and the method is `Predict`. The request accepts `num_traj_samples`, `temperature`, `top_p`, `top_k`, `max_tokens`, `seed`, `nav_text`, and `egomotion_json`. Send compressed image frames through `compressed_images`. For non-default trajectory layouts, set `camera_order` and `num_frames_per_camera`. Prefer JPEG payloads for the fastest request path. `local_paths` is also available for controlled deployments where the client and NIM have access to the same container-local files. Its paths are resolved inside the NIM container; they are not client-host paths or network URLs. Use `compressed_images` for normal remote clients. ```protobuf syntax = "proto3"; package nvidia.alpamayo.v1; service Alpamayo { rpc Predict(PredictRequest) returns (PredictResponse); } message ImageFrame { string camera = 1; int32 timestep = 2; string mime_type = 3; bytes data = 4; } message CompressedImageInput { repeated ImageFrame frames = 1; } message LocalPathInput { string scene_dir = 1; repeated string image_paths = 2; } message PredictRequest { reserved 11, 12; string model = 1; int32 num_traj_samples = 2; optional float temperature = 3; optional float top_p = 4; int32 max_tokens = 5; string nav_text = 6; string egomotion_json = 7; optional int64 seed = 8; repeated string camera_order = 9; int32 num_frames_per_camera = 14; optional int32 top_k = 15; oneof input { CompressedImageInput compressed_images = 10; LocalPathInput local_paths = 13; } } message PredictResponse { string model = 1; string backend = 2; int32 num_samples = 3; int32 num_points = 4; float duration = 5; string reasoning_text = 6; float inference_time = 7; repeated float pred_trajectories = 8; repeated int64 pred_trajectories_shape = 9; string raw_json = 10; repeated float pred_rotations = 11; repeated int64 pred_rotations_shape = 12; } ``` In `LocalPathInput`, `scene_dir` is the base directory used for relative paths in `image_paths`. If `image_paths` is empty, the NIM discovers images from `scene_dir` using the requested camera layout. The response includes flattened `pred_trajectories` with `pred_trajectories_shape` and flattened `pred_rotations` with `pred_rotations_shape`, plus `reasoning_text` and `raw_json`. `pred_rotations` contains rotation matrices with shape `[K, num_points, 3, 3]`, matching the OSS `pred_rot` output after squeezing batch and trajectory-set dimensions. `raw_json` contains the same nested Alpamayo payload returned through HTTP, including trajectory metrics. In proto3, an unset non-optional integer is observed as `0`. For the default 16-image layout, leave `camera_order` empty and leave `num_frames_per_camera` unset; the service applies four frames per camera. For every reduced layout, set both fields explicitly and use a positive `num_frames_per_camera` value. Use `POST /v1/chat/completions` for an OpenAI-compatible trajectory request. **cURL Request** ```bash python3 build_http_payload.py \ --endpoint chat --sample-dir sample_data > /tmp/alpamayo-chat.json curl --request POST \ 'http://localhost:8000/v1/chat/completions' \ -H 'Accept: application/json' \ -H 'Content-Type: application/json' \ --data-binary @/tmp/alpamayo-chat.json ``` **Response** ```json { "id": "chatcmpl-alpamayo", "object": "chat.completion", "created": 1778755858, "model": "nvidia/alpamayo1.5", "choices": [ { "index": 0, "message": { "role": "assistant", "content": "{\"pred_trajectories\":[[[1.2,0.0,0.0]]],\"pred_rotations\":[[[[1.0,0.0,0.0],[0.0,1.0,0.0],[0.0,0.0,1.0]]]],\"num_samples\":1,\"num_points\":64,\"time_step\":0.1,\"metrics\":{}}" }, "finish_reason": "stop" } ] } ``` The chat `message.content` value is a JSON string. After parsing that string, trajectory responses also contain `time_step` (currently `0.1` seconds between predicted points) together with the trajectory arrays and metrics. This top-level `time_step` field describes the trajectory horizon. The nested `metrics.time_step_s` field records the finite-difference interval used to compute kinematic metrics. They currently have the same value but are distinct response fields. Use `POST /v1/infer` for the native Alpamayo schema. It accepts the same image layouts and ego-motion payload as the chat-completions path. **cURL Request** ```bash python3 build_http_payload.py \ --endpoint infer --sample-dir sample_data > /tmp/alpamayo-infer.json curl --request POST \ 'http://localhost:8000/v1/infer' \ -H 'Accept: application/json' \ -H 'Content-Type: application/json' \ --data-binary @/tmp/alpamayo-infer.json ``` **Response** ```json { "mode": "trajectory", "pred_trajectories": [[[1.2, 0.0, 0.0]]], "pred_rotations": [[[[1.0, 0.0, 0.0], [0.0, 1.0, 0.0], [0.0, 0.0, 1.0]]]], "reasoning_text": "...", "num_samples": 1, "num_points": 64, "duration": 6.4, "inference_time": 0.8, "metrics": { "comfort_lat_accel": 1.0, "comfort_lon_accel": 1.0, "comfort_lon_jerk": 1.0, "comfort_yaw_accel": 1.0, "comfort_yaw_rate": 1.0 } } ``` ### Trajectory Metrics Trajectory responses include a `metrics` object for trajectory-only kinematic metrics. The same object is returned by `/v1/infer` and embedded in the gRPC `raw_json` payload. Trajectory responses also include `pred_rotations` as rotation matrices with shape `[K, num_points, 3, 3]`. These rotations are the public API equivalent of the OSS `pred_rot` output. The primary comfort metrics are scalar pass rates in `[0, 1]` across the generated trajectory samples: | Metric | Description | | ------------------- | -------------------------------------------- | | `comfort_lat_accel` | Lateral acceleration comfort pass rate. | | `comfort_lon_accel` | Longitudinal acceleration comfort pass rate. | | `comfort_lon_jerk` | Longitudinal jerk comfort pass rate. | | `comfort_yaw_accel` | Yaw acceleration comfort pass rate. | | `comfort_yaw_rate` | Yaw rate comfort pass rate. | Every metrics object also includes: | Field | Description | | ------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------- | | `version` | Metric schema version, currently `kinematic-v2`. | | `time_step_s` | Time interval used for finite-difference dynamics, currently `0.1`. | | `comfort_bounds` | Minimum and maximum thresholds for longitudinal/lateral acceleration, longitudinal jerk, yaw acceleration, and yaw rate. | | `comfort_lat_accel_pass_rate` | Alias for the lateral-acceleration comfort pass rate. | | `path_length_m` and `final_displacement_m` | `min`, `mean`, and `max` summaries across generated samples. | | `max_speed_mps` and `mean_speed_mps` | `min`, `mean`, and `max` summaries across generated samples. | | `max_abs_lon_accel_mps2` and `max_abs_lat_accel_mps2` | `min`, `mean`, and `max` summaries of per-sample acceleration extrema. | | `max_abs_jerk_mps3` and `max_abs_lon_jerk_mps3` | `min`, `mean`, and `max` summaries of per-sample jerk extrema. | | `max_abs_yaw_accel_radps2` and `max_abs_yaw_rate_radps` | `min`, `mean`, and `max` summaries of per-sample yaw dynamics. | | `per_sample` | One object per generated trajectory containing the corresponding scalar kinematic extrema, path values, and five comfort pass values. | All values are derived from generated trajectories and rotations. They are not ground-truth accuracy or map-compliance metrics. When the diagnostic `NIM_ALPAMAYO_INCLUDE_EVALUATION_ONLY_METRICS` setting is enabled, a `debug_evaluation_only_metrics` object explains which evaluation inputs are not available during normal inference. ### Visual Question Answering Use `POST /v1/infer` with `mode="answer"` for the native Alpamayo VQA schema, or use `POST /v1/vqa` for the OpenAI-compatible message schema. Both routes accept one or more compressed images and do not require ego-motion. The images are processed in the order they appear in the request. For native `/v1/infer`, send `images` and `question` as top-level fields. For `/v1/vqa`, send the question as `nvext.question` or as text content in the user message. `/v1/vqa` uses the OpenAI-compatible message envelope; a standalone top-level `question` field is not part of its request schema. The bundled payload builder places the question in both a text content part and `nvext.question`. The literal request shape appears in the following example. Replace `` with the base64-encoded JPEG bytes. Keep the same question in the text content and `nvext.question` so the request is clear to both OpenAI-compatible clients and the Alpamayo adapter. ```json { "model": "nvidia/alpamayo1.5", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "What is in front of the ego vehicle?" }, { "type": "image_url", "image_url": { "url": "data:image/jpeg;base64," } } ] } ], "temperature": 0.6, "top_p": 0.98, "max_tokens": 256, "seed": 42, "nvext": { "question": "What is in front of the ego vehicle?" } } ``` The following builder command substitutes a real image from the bundled sample scene and produces a directly runnable request file. **cURL Request** ```bash python3 build_http_payload.py \ --endpoint vqa --sample-dir sample_data > /tmp/alpamayo-vqa.json curl --request POST \ 'http://localhost:8000/v1/vqa' \ -H 'Accept: application/json' \ -H 'Content-Type: application/json' \ --data-binary @/tmp/alpamayo-vqa.json ``` **Response** ```json { "id": "vqa-alpamayo", "object": "vqa.response", "created": 1778755858, "model": "nvidia/alpamayo1.5", "question": "What is in front of the ego vehicle?", "answer": "...", "inference_time": 0.8 } ``` ### Trajectory Parameters The trajectory APIs accept the following controls: | Parameter | Description | | ----------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `num_traj_samples` | Trajectory count K. Default is `1`. The value must be listed in `NIM_ALPAMAYO_TRAJ_SAMPLES`. | | `temperature` | VLM sampling temperature. Default is `0.6`. | | `top_p` | VLM nucleus sampling top-p. Default is `0.98`. | | `top_k` | Optional VLM top-k sampling limit. Use `1` for top-1 decoding. | | `max_tokens` | Maximum VLM generation tokens. Default is `256`. | | `nav_text` | Optional route context. It is advisory model input, not a deterministic vehicle-control command or a safety control. | | `camera_order` | Camera names for non-default trajectory image layouts. Full Alpamayo camera names and OSS short names are accepted. | | `num_frames_per_camera` | Number of temporal frames supplied for each camera. Omit it for the default 16-image layout, which implies four frames per camera. Set it explicitly, together with `camera_order`, for every reduced layout. | | `seed` | Optional per-request sampling seed. When omitted, the container default seed is used. | For repeatable K=1 trajectory inference in the same deployment, use `num_traj_samples=1` with a fixed `seed`, `top_k=1`, and `top_p=1`. The seed controls diffusion sampling, while top-1 decoding removes VLM sampling variability. This is preferable to relying on `temperature=0` because zero-temperature handling can differ across client protocols and backends. ### Error Handling The HTTP API returns standard status codes to indicate success or failure: - **200 OK**: Request successful - **400 Bad Request**: A request passed schema validation but was rejected by the application, such as an unsupported K value or image bytes that cannot be decoded from the supplied base64/data URL - **422 Unprocessable Entity**: JSON parsing or request-model validation failed. Examples include malformed JSON, a missing required field, an out-of-range value, the wrong ego-motion shape, or a camera layout whose declared camera count and frame count do not match the supplied images - **500 Internal Server Error**: Server-side error The HTTP serving path can buffer or serialize concurrent requests and does not define a queue-full status at a fixed client concurrency. Clients that require an explicit backend backpressure signal should use the direct gRPC API. The gRPC API uses standard gRPC status codes: - `INVALID_ARGUMENT`: Invalid input parameters or malformed payload - `RESOURCE_EXHAUSTED`: Backend request queue is full - `UNAVAILABLE`: Model backend is not loaded - `INTERNAL`: Server-side error ### Reference The HTTP OpenAPI specification is available in the [API Reference](https://docs.nvidia.com/nim/alpamayo/1.0.0/api-reference/infer-v-1-infer-post) tab. The same schema is packaged inside the container at `/opt/nim/api_spec.yaml`.