Skip to navigation

API Reference

View as Markdown

Endpoints Schema

The following HTTP endpoints are available for Alpamayo1.5 NIM:

  • /v1/infer
  • /v1/chat/completions
  • /v1/vqa
  • /v1/health/ready
  • /v1/health/live
  • /v1/license
  • /v1/metrics
  • /v1/metadata
  • /v1/manifest
  • /v1/models
  • /v1/version

The NIM also exposes a binary gRPC API on port 50051 for trajectory generation. VQA is served through HTTP.

API Examples

Use the examples in this section to get started with the API.

Choose an HTTP Request Envelope

The three inference endpoints use two different request envelopes:

  • POST /v1/infer uses the native Alpamayo schema. Set mode to trajectory, answer, or both. images, question, egomotion, and all trajectory controls are top-level fields. Do not wrap them in nvext. answer mode does not require ego-motion; trajectory and both modes require it.
  • POST /v1/chat/completions uses the OpenAI-compatible message schema. Images are image_url parts under messages[].content. Alpamayo-specific trajectory fields such as egomotion, num_traj_samples, nav_text, camera_order, and num_frames_per_camera are nested under nvext. OpenAI-style controls such as max_tokens, temperature, and top_p remain top-level. seed and top_k can be top-level or under nvext; values under nvext take precedence.
  • POST /v1/vqa uses the same OpenAI-compatible message schema. Put the question in a text content part or nvext.question. A top-level question or images field is not accepted by this endpoint.

The following abbreviated payloads show where the endpoint-specific fields belong. Replace the placeholders with the full image and ego-motion values shown later on this page.

/v1/infer trajectory generation:

{
  "mode": "trajectory",
  "images": ["data:image/jpeg;base64,<BASE64_JPEG>"],
  "egomotion": {"ego_history_xyz": [], "ego_history_rot": []},
  "num_traj_samples": 1,
  "seed": 42,
  "top_k": 1,
  "top_p": 1
}

/v1/infer visual question answering:

{
  "mode": "answer",
  "images": ["data:image/jpeg;base64,<BASE64_JPEG>"],
  "question": "What is ahead?",
  "seed": 42,
  "top_k": 1,
  "top_p": 1
}

/v1/infer combined trajectory generation and visual question answering:

{
  "mode": "both",
  "images": ["data:image/jpeg;base64,<BASE64_JPEG>"],
  "question": "What is ahead?",
  "egomotion": {"ego_history_xyz": [], "ego_history_rot": []},
  "num_traj_samples": 1,
  "seed": 42,
  "top_k": 1,
  "top_p": 1
}

Native /v1/infer responses contain answer for answer mode, trajectory fields for trajectory mode, and both sets of fields for both mode.

/v1/chat/completions:

{
  "model": "nvidia/alpamayo1.5",
  "messages": [{
    "role": "user",
    "content": [{
      "type": "image_url",
      "image_url": {"url": "data:image/jpeg;base64,<BASE64_JPEG>"}
    }]
  }],
  "seed": 42,
  "top_k": 1,
  "top_p": 1,
  "nvext": {
    "egomotion": {"ego_history_xyz": [], "ego_history_rot": []},
    "num_traj_samples": 1,
    "nav_text": "Drive forward."
  }
}

/v1/vqa:

{
  "model": "nvidia/alpamayo1.5",
  "messages": [{
    "role": "user",
    "content": [
      {"type": "text", "text": "What is ahead?"},
      {
        "type": "image_url",
        "image_url": {"url": "data:image/jpeg;base64,<BASE64_JPEG>"}
      }
    ]
  }],
  "nvext": {"question": "What is ahead?"}
}

Check Health

Use the following command to check server readiness.

cURL Request

curl -X 'GET' 'http://localhost:8000/v1/health/ready'

Response

{
   "object": "Triton readiness check",
   "message": "ready",
   "status": "ready"
}

The object field reflects the underlying backend readiness check and is not a fixed string. Automation should treat the service as ready based on the HTTP 200 status code and status being "ready", rather than matching on object or message.

Inspect Service Information

The following read-only endpoints help clients identify the running release and selected model profile. The examples use jq only to keep large responses compact; omit the filter to retrieve each complete response.

  • GET /v1/metadata

    Returns deployment metadata, including the NIM version, model resource, and profile selected for the visible GPU. This is the quickest way to confirm which profile an automatically configured container loaded.

    curl --fail --silent http://localhost:8000/v1/metadata \
       | jq '{version, selectedModelProfileId, modelInfo}'
    {
      "version": "1.0.0",
      "selectedModelProfileId": "<profile-id>",
      "modelInfo": [
        {
          "modelUrl": "ngc://nim/alpamayo-1-5-10b:<model-version>",
          "shortName": "alpamayo-1-5-10b:<model-version>"
        }
      ]
    }

  • GET /v1/manifest

    Returns the embedded model manifest as YAML in the manifest_file JSON field. The manifest lists available profiles, selection tags, and required model artifacts. To inspect its header:

    curl --fail --silent http://localhost:8000/v1/manifest \
       | jq -r '.manifest_file' | sed -n '1,12p'
    schema_version: '2.0'
    profile_selection_criteria: auto
    model: nvidia/alpamayo1.5
    release: 1.0.0
    profiles:
    - id: <profile-id>
      tags:
        number_of_gpus: '1'
        precision: bf16

  • GET /v1/version

    Returns the NIM release identifier and the version of the common NIM HTTP API layer. Use it for compatibility checks and support reports.

    curl --fail --silent http://localhost:8000/v1/version | jq
    {
      "release": "1.0.0",
      "api": "3.1.0"
    }

Generate Trajectory

Use trajectory inference to generate sampled future ego trajectories from compressed camera images, ego-motion history, and optional navigation text.

The default input layout is 16 images, ordered as four timesteps for each camera:

cross_left_t0,  cross_left_t1,  cross_left_t2,  cross_left_t3,
front_wide_t0,  front_wide_t1,  front_wide_t2,  front_wide_t3,
cross_right_t0, cross_right_t1, cross_right_t2, cross_right_t3,
front_tele_t0,  front_tele_t1,  front_tele_t2,  front_tele_t3

t0 is the oldest frame, approximately t-0.3s. t3 is the current frame.

Trajectory requests can use fewer images than the default layout. When the image count is not 16, include camera_order and num_frames_per_camera. Images must be ordered camera-major according to those fields. For example, a request with camera_order=["front_wide"] and num_frames_per_camera=4 contains four front-wide frames. A request with four camera names and num_frames_per_camera=1 contains one image per camera. If the image count is not 16 and no explicit layout is supplied, request-model validation returns HTTP 422.

Camera names can be full Alpamayo camera names, such as camera_front_wide_120fov, or OSS short names, such as front_wide.

Every trajectory request requires exactly 16 ego-motion history entries, regardless of the number of images. ego_history_xyz must have shape [16, 3]. ego_history_rot must contain 16 quaternions with shape [16, 4] (each row [w, x, y, z]) or 16 rotation matrices with shape [16, 3, 3]. Reduced and single-image layouts do not reduce the ego-motion history length. Requests with another history length are rejected before CUDA inference.

The runnable HTTP examples in this section use a payload builder and sample scene shipped in the NIM. Copy both from a running container after you start it:

docker cp alpamayo1.5:/opt/nim/build_http_payload.py ./build_http_payload.py
docker cp alpamayo1.5:/workspace/sample_data ./sample_data

For trajectory endpoints, the builder uses all 16 real sample images in the default four-camera, four-frame layout and all 16 ego-motion history entries. It emits JSON containing real image data URLs; no base64 substitution is required. The optional --layout reduced mode uses only front_wide_t0 and is intended for quick validation tests, not as the primary usage example. VQA uses the single front_wide_t0 image because it does not require a trajectory camera layout.

For a consolidated description of every helper flag, refer to Payload Builder Options.

The gRPC route is recommended for warm trajectory serving because it avoids the HTTP base64 and JSON adapter path. The service is nvidia.alpamayo.v1.Alpamayo and the method is Predict.

The request accepts num_traj_samples, temperature, top_p, top_k, max_tokens, seed, nav_text, and egomotion_json. Send compressed image frames through compressed_images. For non-default trajectory layouts, set camera_order and num_frames_per_camera. Prefer JPEG payloads for the fastest request path.

local_paths is also available for controlled deployments where the client and NIM have access to the same container-local files. Its paths are resolved inside the NIM container; they are not client-host paths or network URLs. Use compressed_images for normal remote clients.

syntax = "proto3";
package nvidia.alpamayo.v1;
service Alpamayo {
  rpc Predict(PredictRequest) returns (PredictResponse);
}
message ImageFrame {
  string camera = 1;
  int32 timestep = 2;
  string mime_type = 3;
  bytes data = 4;
}
message CompressedImageInput {
  repeated ImageFrame frames = 1;
}
message LocalPathInput {
  string scene_dir = 1;
  repeated string image_paths = 2;
}
message PredictRequest {
  reserved 11, 12;
  string model = 1;
  int32 num_traj_samples = 2;
  optional float temperature = 3;
  optional float top_p = 4;
  int32 max_tokens = 5;
  string nav_text = 6;
  string egomotion_json = 7;
  optional int64 seed = 8;
  repeated string camera_order = 9;
  int32 num_frames_per_camera = 14;
  optional int32 top_k = 15;
  oneof input {
    CompressedImageInput compressed_images = 10;
    LocalPathInput local_paths = 13;
  }
}
message PredictResponse {
  string model = 1;
  string backend = 2;
  int32 num_samples = 3;
  int32 num_points = 4;
  float duration = 5;
  string reasoning_text = 6;
  float inference_time = 7;
  repeated float pred_trajectories = 8;
  repeated int64 pred_trajectories_shape = 9;
  string raw_json = 10;
  repeated float pred_rotations = 11;
  repeated int64 pred_rotations_shape = 12;
}

In LocalPathInput, scene_dir is the base directory used for relative paths in image_paths. If image_paths is empty, the NIM discovers images from scene_dir using the requested camera layout.

The response includes flattened pred_trajectories with pred_trajectories_shape and flattened pred_rotations with pred_rotations_shape, plus reasoning_text and raw_json. pred_rotations contains rotation matrices with shape [K, num_points, 3, 3], matching the OSS pred_rot output after squeezing batch and trajectory-set dimensions. raw_json contains the same nested Alpamayo payload returned through HTTP, including trajectory metrics.

In proto3, an unset non-optional integer is observed as 0. For the default 16-image layout, leave camera_order empty and leave num_frames_per_camera unset; the service applies four frames per camera. For every reduced layout, set both fields explicitly and use a positive num_frames_per_camera value.

Trajectory Metrics

Trajectory responses include a metrics object for trajectory-only kinematic metrics. The same object is returned by /v1/infer and embedded in the gRPC raw_json payload.

Trajectory responses also include pred_rotations as rotation matrices with shape [K, num_points, 3, 3]. These rotations are the public API equivalent of the OSS pred_rot output.

The primary comfort metrics are scalar pass rates in [0, 1] across the generated trajectory samples:

MetricDescription
comfort_lat_accelLateral acceleration comfort pass rate.
comfort_lon_accelLongitudinal acceleration comfort pass rate.
comfort_lon_jerkLongitudinal jerk comfort pass rate.
comfort_yaw_accelYaw acceleration comfort pass rate.
comfort_yaw_rateYaw rate comfort pass rate.

Every metrics object also includes:

FieldDescription
versionMetric schema version, currently kinematic-v2.
time_step_sTime interval used for finite-difference dynamics, currently 0.1.
comfort_boundsMinimum and maximum thresholds for longitudinal/lateral acceleration, longitudinal jerk, yaw acceleration, and yaw rate.
comfort_lat_accel_pass_rateAlias for the lateral-acceleration comfort pass rate.
path_length_m and final_displacement_mmin, mean, and max summaries across generated samples.
max_speed_mps and mean_speed_mpsmin, mean, and max summaries across generated samples.
max_abs_lon_accel_mps2 and max_abs_lat_accel_mps2min, mean, and max summaries of per-sample acceleration extrema.
max_abs_jerk_mps3 and max_abs_lon_jerk_mps3min, mean, and max summaries of per-sample jerk extrema.
max_abs_yaw_accel_radps2 and max_abs_yaw_rate_radpsmin, mean, and max summaries of per-sample yaw dynamics.
per_sampleOne object per generated trajectory containing the corresponding scalar kinematic extrema, path values, and five comfort pass values.

All values are derived from generated trajectories and rotations. They are not ground-truth accuracy or map-compliance metrics. When the diagnostic NIM_ALPAMAYO_INCLUDE_EVALUATION_ONLY_METRICS setting is enabled, a debug_evaluation_only_metrics object explains which evaluation inputs are not available during normal inference.

Visual Question Answering

Use POST /v1/infer with mode="answer" for the native Alpamayo VQA schema, or use POST /v1/vqa for the OpenAI-compatible message schema. Both routes accept one or more compressed images and do not require ego-motion. The images are processed in the order they appear in the request.

For native /v1/infer, send images and question as top-level fields. For /v1/vqa, send the question as nvext.question or as text content in the user message.

/v1/vqa uses the OpenAI-compatible message envelope; a standalone top-level question field is not part of its request schema. The bundled payload builder places the question in both a text content part and nvext.question.

The literal request shape appears in the following example. Replace <BASE64_JPEG> with the base64-encoded JPEG bytes. Keep the same question in the text content and nvext.question so the request is clear to both OpenAI-compatible clients and the Alpamayo adapter.

{
  "model": "nvidia/alpamayo1.5",
  "messages": [
    {
      "role": "user",
      "content": [
        {
          "type": "text",
          "text": "What is in front of the ego vehicle?"
        },
        {
          "type": "image_url",
          "image_url": {
            "url": "data:image/jpeg;base64,<BASE64_JPEG>"
          }
        }
      ]
    }
  ],
  "temperature": 0.6,
  "top_p": 0.98,
  "max_tokens": 256,
  "seed": 42,
  "nvext": {
    "question": "What is in front of the ego vehicle?"
  }
}

The following builder command substitutes a real image from the bundled sample scene and produces a directly runnable request file.

cURL Request

python3 build_http_payload.py \
   --endpoint vqa --sample-dir sample_data > /tmp/alpamayo-vqa.json
curl --request POST \
   'http://localhost:8000/v1/vqa' \
   -H 'Accept: application/json' \
   -H 'Content-Type: application/json' \
   --data-binary @/tmp/alpamayo-vqa.json

Response

{
   "id": "vqa-alpamayo",
   "object": "vqa.response",
   "created": 1778755858,
   "model": "nvidia/alpamayo1.5",
   "question": "What is in front of the ego vehicle?",
   "answer": "...",
   "inference_time": 0.8
}

Trajectory Parameters

The trajectory APIs accept the following controls:

ParameterDescription
num_traj_samplesTrajectory count K. Default is 1. The value must be listed in NIM_ALPAMAYO_TRAJ_SAMPLES.
temperatureVLM sampling temperature. Default is 0.6.
top_pVLM nucleus sampling top-p. Default is 0.98.
top_kOptional VLM top-k sampling limit. Use 1 for top-1 decoding.
max_tokensMaximum VLM generation tokens. Default is 256.
nav_textOptional route context. It is advisory model input, not a deterministic vehicle-control command or a safety control.
camera_orderCamera names for non-default trajectory image layouts. Full Alpamayo camera names and OSS short names are accepted.
num_frames_per_cameraNumber of temporal frames supplied for each camera. Omit it for the default 16-image layout, which implies four frames per camera. Set it explicitly, together with camera_order, for every reduced layout.
seedOptional per-request sampling seed. When omitted, the container default seed is used.

For repeatable K=1 trajectory inference in the same deployment, use num_traj_samples=1 with a fixed seed, top_k=1, and top_p=1. The seed controls diffusion sampling, while top-1 decoding removes VLM sampling variability. This is preferable to relying on temperature=0 because zero-temperature handling can differ across client protocols and backends.

Error Handling

The HTTP API returns standard status codes to indicate success or failure:

  • 200 OK: Request successful
  • 400 Bad Request: A request passed schema validation but was rejected by the application, such as an unsupported K value or image bytes that cannot be decoded from the supplied base64/data URL
  • 422 Unprocessable Entity: JSON parsing or request-model validation failed. Examples include malformed JSON, a missing required field, an out-of-range value, the wrong ego-motion shape, or a camera layout whose declared camera count and frame count do not match the supplied images
  • 500 Internal Server Error: Server-side error

The HTTP serving path can buffer or serialize concurrent requests and does not define a queue-full status at a fixed client concurrency. Clients that require an explicit backend backpressure signal should use the direct gRPC API.

The gRPC API uses standard gRPC status codes:

  • INVALID_ARGUMENT: Invalid input parameters or malformed payload
  • RESOURCE_EXHAUSTED: Backend request queue is full
  • UNAVAILABLE: Model backend is not loaded
  • INTERNAL: Server-side error

Reference

The HTTP OpenAPI specification is available in the API Reference tab. The same schema is packaged inside the container at /opt/nim/api_spec.yaml.