Is this page helpful?

API Reference#

Endpoints Schema#

The following HTTP endpoints are available for Alpamayo1.5 NIM:

/v1/infer
/v1/chat/completions
/v1/vqa
/v1/health/ready
/v1/health/live
/v1/license
/v1/metrics
/v1/metadata
/v1/manifest
/v1/models
/v1/version

The NIM also exposes a binary gRPC API on port 50051 for trajectory generation. VQA is served through HTTP.

API Examples#

Use the examples in this section to get started with the API.

Choose an HTTP Request Envelope#

The three inference endpoints use two different request envelopes:

POST /v1/infer uses the native Alpamayo schema. Set mode to trajectory, answer, or both. images, question, egomotion, and all trajectory controls are top-level fields. Do not wrap them in nvext. answer mode does not require ego-motion; trajectory and both modes require it.
POST /v1/chat/completions uses the OpenAI-compatible message schema. Images are image_url parts under messages[].content. Alpamayo-specific trajectory fields such as egomotion, num_traj_samples, nav_text, camera_order, and num_frames_per_camera are nested under nvext. OpenAI-style controls such as max_tokens, temperature, and top_p remain top-level. seed and top_k can be top-level or under nvext; values under nvext take precedence.
POST /v1/vqa uses the same OpenAI-compatible message schema. Put the question in a text content part or nvext.question. A top-level question or images field is not accepted by this endpoint.

The following abbreviated payloads show where the endpoint-specific fields belong. Replace the placeholders with the full image and ego-motion values shown later on this page.

/v1/infer trajectory generation:

{
  "mode": "trajectory",
  "images": ["data:image/jpeg;base64,<BASE64_JPEG>"],
  "egomotion": {"ego_history_xyz": [], "ego_history_rot": []},
  "num_traj_samples": 1,
  "seed": 42,
  "top_k": 1,
  "top_p": 1
}

/v1/infer visual question answering:

{
  "mode": "answer",
  "images": ["data:image/jpeg;base64,<BASE64_JPEG>"],
  "question": "What is ahead?",
  "seed": 42,
  "top_k": 1,
  "top_p": 1
}

/v1/infer combined trajectory generation and visual question answering:

{
  "mode": "both",
  "images": ["data:image/jpeg;base64,<BASE64_JPEG>"],
  "question": "What is ahead?",
  "egomotion": {"ego_history_xyz": [], "ego_history_rot": []},
  "num_traj_samples": 1,
  "seed": 42,
  "top_k": 1,
  "top_p": 1
}

Native /v1/infer responses contain answer for answer mode, trajectory fields for trajectory mode, and both sets of fields for both mode.

/v1/chat/completions:

{
  "model": "nvidia/alpamayo1.5",
  "messages": [{
    "role": "user",
    "content": [{
      "type": "image_url",
      "image_url": {"url": "data:image/jpeg;base64,<BASE64_JPEG>"}
    }]
  }],
  "seed": 42,
  "top_k": 1,
  "top_p": 1,
  "nvext": {
    "egomotion": {"ego_history_xyz": [], "ego_history_rot": []},
    "num_traj_samples": 1,
    "nav_text": "Drive forward."
  }
}

/v1/vqa:

{
  "model": "nvidia/alpamayo1.5",
  "messages": [{
    "role": "user",
    "content": [
      {"type": "text", "text": "What is ahead?"},
      {
        "type": "image_url",
        "image_url": {"url": "data:image/jpeg;base64,<BASE64_JPEG>"}
      }
    ]
  }],
  "nvext": {"question": "What is ahead?"}
}

Check Health#

Use the following command to check server readiness.

cURL Request

curl -X 'GET' 'http://localhost:8000/v1/health/ready'

Response

{
   "object": "Triton readiness check",
   "message": "ready",
   "status": "ready"
}

Note

The object field reflects the underlying backend readiness check and is not a fixed string. Automation should treat the service as ready based on the HTTP 200 status code and status being "ready", rather than matching on object or message.

Inspect Service Information#

The following read-only endpoints help clients identify the running release and selected model profile. The examples use jq only to keep large responses compact; omit the filter to retrieve each complete response.

GET /v1/metadata

Returns deployment metadata, including the NIM version, model resource, and profile selected for the visible GPU. This is the quickest way to confirm which profile an automatically configured container loaded.

curl --fail --silent http://localhost:8000/v1/metadata \
   | jq '{version, selectedModelProfileId, modelInfo}'

{
  "version": "1.0.0",
  "selectedModelProfileId": "<profile-id>",
  "modelInfo": [
    {
      "modelUrl": "ngc://nim/alpamayo-1-5-10b:<model-version>",
      "shortName": "alpamayo-1-5-10b:<model-version>"
    }
  ]
}

GET /v1/manifest

Returns the embedded model manifest as YAML in the manifest_file JSON field. The manifest lists available profiles, selection tags, and required model artifacts. To inspect its header:

curl --fail --silent http://localhost:8000/v1/manifest \
   | jq -r '.manifest_file' | sed -n '1,12p'

schema_version: '2.0'
profile_selection_criteria: auto
model: nvidia/alpamayo1.5
release: 1.0.0
profiles:
- id: <profile-id>
  tags:
    number_of_gpus: '1'
    precision: bf16

GET /v1/version

Returns the NIM release identifier and the version of the common NIM HTTP API layer. Use it for compatibility checks and support reports.

curl --fail --silent http://localhost:8000/v1/version | jq

{
  "release": "1.0.0",
  "api": "3.1.0"
}

Generate Trajectory#

Use trajectory inference to generate sampled future ego trajectories from compressed camera images, ego-motion history, and optional navigation text.

The default input layout is 16 images, ordered as four timesteps for each camera:

cross_left_t0,  cross_left_t1,  cross_left_t2,  cross_left_t3,
front_wide_t0,  front_wide_t1,  front_wide_t2,  front_wide_t3,
cross_right_t0, cross_right_t1, cross_right_t2, cross_right_t3,
front_tele_t0,  front_tele_t1,  front_tele_t2,  front_tele_t3

t0 is the oldest frame, approximately t-0.3s. t3 is the current frame.

Trajectory requests can use fewer images than the default layout. When the image count is not 16, include camera_order and num_frames_per_camera. Images must be ordered camera-major according to those fields. For example, a request with camera_order=["front_wide"] and num_frames_per_camera=4 contains four front-wide frames. A request with four camera names and num_frames_per_camera=1 contains one image per camera. If the image count is not 16 and no explicit layout is supplied, request-model validation returns HTTP 422.

Camera names can be full Alpamayo camera names, such as camera_front_wide_120fov, or OSS short names, such as front_wide.

Every trajectory request requires exactly 16 ego-motion history entries, regardless of the number of images. ego_history_xyz must have shape [16, 3]. ego_history_rot must contain 16 quaternions with shape [16, 4] (each row [w, x, y, z]) or 16 rotation matrices with shape [16, 3, 3]. Reduced and single-image layouts do not reduce the ego-motion history length. Requests with another history length are rejected before CUDA inference.

The runnable HTTP examples in this section use a payload builder and sample scene shipped in the NIM. Copy both from a running container after you start it:

docker cp alpamayo1.5:/opt/nim/build_http_payload.py ./build_http_payload.py
docker cp alpamayo1.5:/workspace/sample_data ./sample_data

For trajectory endpoints, the builder uses all 16 real sample images in the default four-camera, four-frame layout and all 16 ego-motion history entries. It emits JSON containing real image data URLs; no base64 substitution is required. The optional --layout reduced mode uses only front_wide_t0 and is intended for quick validation tests, not as the primary usage example. VQA uses the single front_wide_t0 image because it does not require a trajectory camera layout.

For a consolidated description of every helper flag, refer to Payload Builder Options.

gRPC

The gRPC route is recommended for warm trajectory serving because it avoids the HTTP base64 and JSON adapter path. The service is nvidia.alpamayo.v1.Alpamayo and the method is Predict.

The request accepts num_traj_samples, temperature, top_p, top_k, max_tokens, seed, nav_text, and egomotion_json. Send compressed image frames through compressed_images. For non-default trajectory layouts, set camera_order and num_frames_per_camera. Prefer JPEG payloads for the fastest request path.

local_paths is also available for controlled deployments where the client and NIM have access to the same container-local files. Its paths are resolved inside the NIM container; they are not client-host paths or network URLs. Use compressed_images for normal remote clients.

syntax = "proto3";

package nvidia.alpamayo.v1;

service Alpamayo {
  rpc Predict(PredictRequest) returns (PredictResponse);
}

message ImageFrame {
  string camera = 1;
  int32 timestep = 2;
  string mime_type = 3;
  bytes data = 4;
}

message CompressedImageInput {
  repeated ImageFrame frames = 1;
}

message LocalPathInput {
  string scene_dir = 1;
  repeated string image_paths = 2;
}

message PredictRequest {
  reserved 11, 12;

  string model = 1;
  int32 num_traj_samples = 2;
  optional float temperature = 3;
  optional float top_p = 4;
  int32 max_tokens = 5;
  string nav_text = 6;
  string egomotion_json = 7;
  optional int64 seed = 8;
  repeated string camera_order = 9;
  int32 num_frames_per_camera = 14;
  optional int32 top_k = 15;

  oneof input {
    CompressedImageInput compressed_images = 10;
    LocalPathInput local_paths = 13;
  }
}

message PredictResponse {
  string model = 1;
  string backend = 2;
  int32 num_samples = 3;
  int32 num_points = 4;
  float duration = 5;
  string reasoning_text = 6;
  float inference_time = 7;
  repeated float pred_trajectories = 8;
  repeated int64 pred_trajectories_shape = 9;
  string raw_json = 10;
  repeated float pred_rotations = 11;
  repeated int64 pred_rotations_shape = 12;
}

In LocalPathInput, scene_dir is the base directory used for relative paths in image_paths. If image_paths is empty, the NIM discovers images from scene_dir using the requested camera layout.

The response includes flattened pred_trajectories with pred_trajectories_shape and flattened pred_rotations with pred_rotations_shape, plus reasoning_text and raw_json. pred_rotations contains rotation matrices with shape [K, num_points, 3, 3], matching the OSS pred_rot output after squeezing batch and trajectory-set dimensions. raw_json contains the same nested Alpamayo payload returned through HTTP, including trajectory metrics.

In proto3, an unset non-optional integer is observed as 0. For the default 16-image layout, leave camera_order empty and leave num_frames_per_camera unset; the service applies four frames per camera. For every reduced layout, set both fields explicitly and use a positive num_frames_per_camera value.

/v1/chat/completions

Use POST /v1/chat/completions for an OpenAI-compatible trajectory request.

cURL Request

python3 build_http_payload.py \
   --endpoint chat --sample-dir sample_data > /tmp/alpamayo-chat.json

curl --request POST \
   'http://localhost:8000/v1/chat/completions' \
   -H 'Accept: application/json' \
   -H 'Content-Type: application/json' \
   --data-binary @/tmp/alpamayo-chat.json

Response

{
   "id": "chatcmpl-alpamayo",
   "object": "chat.completion",
   "created": 1778755858,
   "model": "nvidia/alpamayo1.5",
   "choices": [
      {
         "index": 0,
         "message": {
            "role": "assistant",
            "content": "{\"pred_trajectories\":[[[1.2,0.0,0.0]]],\"pred_rotations\":[[[[1.0,0.0,0.0],[0.0,1.0,0.0],[0.0,0.0,1.0]]]],\"num_samples\":1,\"num_points\":64,\"time_step\":0.1,\"metrics\":{}}"
         },
         "finish_reason": "stop"
      }
   ]
}

The chat message.content value is a JSON string. After parsing that string, trajectory responses also contain time_step (currently 0.1 seconds between predicted points) together with the trajectory arrays and metrics. This top-level time_step field describes the trajectory horizon. The nested metrics.time_step_s field records the finite-difference interval used to compute kinematic metrics. They currently have the same value but are distinct response fields.

/v1/infer

Use POST /v1/infer for the native Alpamayo schema. It accepts the same image layouts and ego-motion payload as the chat-completions path.

cURL Request

python3 build_http_payload.py \
   --endpoint infer --sample-dir sample_data > /tmp/alpamayo-infer.json

curl --request POST \
   'http://localhost:8000/v1/infer' \
   -H 'Accept: application/json' \
   -H 'Content-Type: application/json' \
   --data-binary @/tmp/alpamayo-infer.json

Response

{
   "mode": "trajectory",
   "pred_trajectories": [[[1.2, 0.0, 0.0]]],
   "pred_rotations": [[[[1.0, 0.0, 0.0], [0.0, 1.0, 0.0], [0.0, 0.0, 1.0]]]],
   "reasoning_text": "...",
   "num_samples": 1,
   "num_points": 64,
   "duration": 6.4,
   "inference_time": 0.8,
   "metrics": {
      "comfort_lat_accel": 1.0,
      "comfort_lon_accel": 1.0,
      "comfort_lon_jerk": 1.0,
      "comfort_yaw_accel": 1.0,
      "comfort_yaw_rate": 1.0
   }
}

Trajectory Metrics#

Trajectory responses include a metrics object for trajectory-only kinematic metrics. The same object is returned by /v1/infer and embedded in the gRPC raw_json payload.

Trajectory responses also include pred_rotations as rotation matrices with shape [K, num_points, 3, 3]. These rotations are the public API equivalent of the OSS pred_rot output.

The primary comfort metrics are scalar pass rates in [0, 1] across the generated trajectory samples:

Metric	Description
`comfort_lat_accel`	Lateral acceleration comfort pass rate.
`comfort_lon_accel`	Longitudinal acceleration comfort pass rate.
`comfort_lon_jerk`	Longitudinal jerk comfort pass rate.
`comfort_yaw_accel`	Yaw acceleration comfort pass rate.
`comfort_yaw_rate`	Yaw rate comfort pass rate.

Every metrics object also includes:

Field	Description
`version`	Metric schema version, currently `kinematic-v2`.
`time_step_s`	Time interval used for finite-difference dynamics, currently `0.1`.
`comfort_bounds`	Minimum and maximum thresholds for longitudinal/lateral acceleration, longitudinal jerk, yaw acceleration, and yaw rate.
`comfort_lat_accel_pass_rate`	Alias for the lateral-acceleration comfort pass rate.
`path_length_m` and `final_displacement_m`	`min`, `mean`, and `max` summaries across generated samples.
`max_speed_mps` and `mean_speed_mps`	`min`, `mean`, and `max` summaries across generated samples.
`max_abs_lon_accel_mps2` and `max_abs_lat_accel_mps2`	`min`, `mean`, and `max` summaries of per-sample acceleration extrema.
`max_abs_jerk_mps3` and `max_abs_lon_jerk_mps3`	`min`, `mean`, and `max` summaries of per-sample jerk extrema.
`max_abs_yaw_accel_radps2` and `max_abs_yaw_rate_radps`	`min`, `mean`, and `max` summaries of per-sample yaw dynamics.
`per_sample`	One object per generated trajectory containing the corresponding scalar kinematic extrema, path values, and five comfort pass values.

All values are derived from generated trajectories and rotations. They are not ground-truth accuracy or map-compliance metrics. When the diagnostic NIM_ALPAMAYO_INCLUDE_EVALUATION_ONLY_METRICS setting is enabled, a debug_evaluation_only_metrics object explains which evaluation inputs are not available during normal inference.

Visual Question Answering#

Use POST /v1/infer with mode="answer" for the native Alpamayo VQA schema, or use POST /v1/vqa for the OpenAI-compatible message schema. Both routes accept one or more compressed images and do not require ego-motion. The images are processed in the order they appear in the request.

For native /v1/infer, send images and question as top-level fields. For /v1/vqa, send the question as nvext.question or as text content in the user message.

/v1/vqa uses the OpenAI-compatible message envelope; a standalone top-level question field is not part of its request schema. The bundled payload builder places the question in both a text content part and nvext.question.

The literal request shape appears in the following example. Replace <BASE64_JPEG> with the base64-encoded JPEG bytes. Keep the same question in the text content and nvext.question so the request is clear to both OpenAI-compatible clients and the Alpamayo adapter.

{
  "model": "nvidia/alpamayo1.5",
  "messages": [
    {
      "role": "user",
      "content": [
        {
          "type": "text",
          "text": "What is in front of the ego vehicle?"
        },
        {
          "type": "image_url",
          "image_url": {
            "url": "data:image/jpeg;base64,<BASE64_JPEG>"
          }
        }
      ]
    }
  ],
  "temperature": 0.6,
  "top_p": 0.98,
  "max_tokens": 256,
  "seed": 42,
  "nvext": {
    "question": "What is in front of the ego vehicle?"
  }
}

The following builder command substitutes a real image from the bundled sample scene and produces a directly runnable request file.

cURL Request

python3 build_http_payload.py \
   --endpoint vqa --sample-dir sample_data > /tmp/alpamayo-vqa.json

curl --request POST \
   'http://localhost:8000/v1/vqa' \
   -H 'Accept: application/json' \
   -H 'Content-Type: application/json' \
   --data-binary @/tmp/alpamayo-vqa.json

Response

{
   "id": "vqa-alpamayo",
   "object": "vqa.response",
   "created": 1778755858,
   "model": "nvidia/alpamayo1.5",
   "question": "What is in front of the ego vehicle?",
   "answer": "...",
   "inference_time": 0.8
}

Trajectory Parameters#

The trajectory APIs accept the following controls:

Parameter	Description
`num_traj_samples`	Trajectory count K. Default is `1`. The value must be listed in `NIM_ALPAMAYO_TRAJ_SAMPLES`.
`temperature`	VLM sampling temperature. Default is `0.6`.
`top_p`	VLM nucleus sampling top-p. Default is `0.98`.
`top_k`	Optional VLM top-k sampling limit. Use `1` for top-1 decoding.
`max_tokens`	Maximum VLM generation tokens. Default is `256`.
`nav_text`	Optional route context. It is advisory model input, not a deterministic vehicle-control command or a safety control.
`camera_order`	Camera names for non-default trajectory image layouts. Full Alpamayo camera names and OSS short names are accepted.
`num_frames_per_camera`	Number of temporal frames supplied for each camera. Omit it for the default 16-image layout, which implies four frames per camera. Set it explicitly, together with `camera_order`, for every reduced layout.
`seed`	Optional per-request sampling seed. When omitted, the container default seed is used.

For repeatable K=1 trajectory inference in the same deployment, use num_traj_samples=1 with a fixed seed, top_k=1, and top_p=1. The seed controls diffusion sampling, while top-1 decoding removes VLM sampling variability. This is preferable to relying on temperature=0 because zero-temperature handling can differ across client protocols and backends.

Error Handling#

The HTTP API returns standard status codes to indicate success or failure:

200 OK: Request successful
400 Bad Request: A request passed schema validation but was rejected by the application, such as an unsupported K value or image bytes that cannot be decoded from the supplied base64/data URL
422 Unprocessable Entity: JSON parsing or request-model validation failed. Examples include malformed JSON, a missing required field, an out-of-range value, the wrong ego-motion shape, or a camera layout whose declared camera count and frame count do not match the supplied images
500 Internal Server Error: Server-side error

The HTTP serving path can buffer or serialize concurrent requests and does not define a queue-full status at a fixed client concurrency. Clients that require an explicit backend backpressure signal should use the direct gRPC API.

The gRPC API uses standard gRPC status codes:

INVALID_ARGUMENT: Invalid input parameters or malformed payload
RESOURCE_EXHAUSTED: Backend request queue is full
UNAVAILABLE: Model backend is not loaded
INTERNAL: Server-side error

Reference#

The HTTP OpenAPI specification is embedded in the following section. The same schema is packaged inside the container at /opt/nim/api_spec.yaml.