Inference Tasks

Use POST /v1/infer for new Alpamayo 2 Super integrations. Set task explicitly so that request validation, required inputs, and response fields are clear. If both task and the legacy mode field are omitted, the service runs task="trajectory". For the previous generation, see Alpamayo 1.5 Documentation.

Example Target

Set these for your deployment before running the shell examples. Bash is assumed; pipefail prevents a failed builder or HTTP request being hidden by the last command in a pipeline. Use the container name you own.

export NIM_URL="${NIM_URL:-http://localhost:8000}"
export NIM_CONTAINER="${NIM_CONTAINER:-alpamayo2-super}"
export GRPC_TARGET="${GRPC_TARGET:-localhost:50051}"
export ALPAMAYO_OUT="${ALPAMAYO_OUT:-$(mktemp -d "${TMPDIR:-/tmp}/alpamayo.XXXXXX")}"
set -o pipefail

NIM_URL is the HTTP base URL and GRPC_TARGET the gRPC host:port as published on the host; use your remapped host ports if you changed them. The packaged host clients read both variables. ALPAMAYO_OUT is a private directory for the request and response files written below.

Task Summary

Trajectory

Set task to trajectory. The default uses all six cameras in the driving camera layout, with four frames per camera in camera-major order. An explicit camera_order can select any non-empty subset of those cameras. Every request still needs 16 ego-history translations and rotations. Meta-action and auto-labeling retain the complete six-camera profile.

For four-camera evaluation, use cross_left, front_wide, cross_right, and front_tele, with four frames each (16 images). The builder selects those frames and declares the layout:

docker exec "$NIM_CONTAINER" python3 /opt/nim/build_http_payload.py \
  --endpoint infer --task trajectory --sample-dir /workspace/sample_data \
  --camera-order cross_left,front_wide,cross_right,front_tele \
  > "$ALPAMAYO_OUT/alpamayo-four-camera.json"
curl --fail-with-body -sS "${NIM_URL}/v1/infer" \
  -H 'Content-Type: application/json' \
  --data-binary "@$ALPAMAYO_OUT/alpamayo-four-camera.json"

Trajectory also accepts 1, 2 or 3 frames per camera instead of four, with all six cameras or a declared subset such as the four cameras above. Declare num_frames_per_camera, send the most recent frames, and keep the 16-pose ego history. The model was trained with four frames, so fewer frames reduce temporal context and can change the result. With the payload builder, add --num-frames-per-camera 1, 2 or 3; it keeps the newest frames and declares camera_order and num_frames_per_camera. See Preparing Inputs for the supported layouts, frame timing, image sizes and ego-motion axes.

The response contains pred_trajectories, pred_rotations, reasoning_text, and trajectory metadata.

Trajectory response fields

The model plans from whatever cameras you declare and does not know when forward context is missing: a rear-camera-only request still returns a forward plan. Include the front cameras for trajectory. The service validates camera names and image counts, but cannot detect mislabeled cameras or timesteps from pixels. Supply camera-major order; a timestep-major array can pass validation while changing the result.

The commands on this page assume Bash on the Docker host and a ready container named alpamayo2-super, started using Quickstart Guide. Run the packaged builder inside the container; the JSON file is written on the host:

docker exec "$NIM_CONTAINER" python3 /opt/nim/build_http_payload.py \
    --endpoint infer --sample-dir /workspace/sample_data \
    --task trajectory > "$ALPAMAYO_OUT/alpamayo-trajectory.json"
curl --fail-with-body -sS "${NIM_URL}/v1/infer" \
    -H 'Content-Type: application/json' \
    --data-binary "@$ALPAMAYO_OUT/alpamayo-trajectory.json"

Navigation-Conditioned Trajectory

nav_text adds a navigation instruction to the prompt. nav_guidance_weight (0 through 10, default 3.0) controls classifier-free guidance: the action expert evaluates guided and unguided prompts for each sample. Set 1.0 for prompt-only conditioning. Omit nav_text when no instruction is available; the weight is then ignored. Navigation text is advisory model input, not a vehicle-control or safety command.

Use a maneuver and distance, such as "Turn left in 15m" or "Stop in 15m". The model responds more strongly to this form than to general prose, but the effect depends on the scene. At the default weight of 3.0, guidance steers the plan in only part of the samples on some scenes, and the outcome varies with the seed; the reference model behaves the same. A higher weight, around 6, or several trajectory samples (K>1) increase compliance. Higher weights can also degrade trajectory quality and cause excessive braking, so evaluate the weight on your scenes and check each returned path against the instruction. Guidance latency varies with the profile, scene, and execution path. It can be faster or slower than an otherwise identical unconditioned request; measure the actual path and settings used by your clients.

The builder command below writes the complete request. The following is only the navigation-field fragment to add to an existing trajectory request:

{
  "nav_text": "Turn right in 15m",
  "nav_guidance_weight": 3.0
}

Use the packaged navigation scene to try a junction with left and right branches. Its source.json identifies the PhysicalAI-AV clip and frame times; the recorded future takes the right branch. Compare instructions on this same scene and inspect both the resulting path and the guidance metrics. The metrics confirm execution of guidance, not compliance with an instruction.

The packaged builder creates a complete request from this scene:

docker exec "$NIM_CONTAINER" python3 /opt/nim/build_http_payload.py \
    --endpoint infer --sample-dir /workspace/sample_data/navigation --task trajectory \
    --nav-text 'Turn right in 15m' \
    --nav-guidance-weight 3.0 > "$ALPAMAYO_OUT/alpamayo-nav.json"
curl --fail-with-body -sS "${NIM_URL}/v1/infer" \
    -H 'Content-Type: application/json' \
    --data-binary "@$ALPAMAYO_OUT/alpamayo-nav.json"

Startup warmup exercises ordinary and guided trajectories for each configured K. If disabled with NIM_ALPAMAYO_STARTUP_WARMUP=0, the first navigation request prepares the guidance path. Before benchmarking, send an untimed request with the exact navigation settings; ordinary trajectory warmup does not cover this path.

Navigation requests add three keys to metrics:

Navigation metrics

Requests without nav_text do not include these keys. nav_guidance_weight is only valid for task="trajectory" and legacy mode="both"; on other tasks an explicit non-default value, or any value together with nav_text, returns HTTP 422. Meta-action, auto-labeling, VQA and grounding reject nav_text with HTTP 422. For repeatability, follow Sampling Control and also keep nav_text and nav_guidance_weight fixed.

VQA

Using the ready quickstart container:

docker exec "$NIM_CONTAINER" python3 /opt/nim/build_http_payload.py \
    --endpoint infer --sample-dir /workspace/sample_data \
    --task vqa --question 'What is the vehicle ahead doing?' > "$ALPAMAYO_OUT/alpamayo-vqa.json"
curl --fail-with-body -sS "${NIM_URL}/v1/infer" \
    -H 'Content-Type: application/json' \
    --data-binary "@$ALPAMAYO_OUT/alpamayo-vqa.json"

Set task to vqa and provide a non-empty question. VQA does not require ego-motion and accepts one or more images. For a single-image request, set camera_order=["front_wide"] and num_frames_per_camera=1 explicitly. Otherwise the default expects 24 images. For multi-frame or multi-camera requests, declare the layout and make the image count equal to len(camera_order) * num_frames_per_camera.

Without camera_order, 24 VQA images are read in the driving layout used by trajectory and the bundled samples: cross_left, front_wide, cross_right, rear_left, rear_right, front_tele, four frames each. For any other set of cameras, including rear_tele, declare camera_order explicitly.

Ask questions in English. Questions in other languages can return unrelated or nonsensical text, and the response does not flag it.

Presence and location questions can make the model answer with grounding output (a JSON list of bbox_2d boxes) instead of prose. The answer is returned unchanged, and answer_reason says so. Use task='grounding' to localize objects, or rephrase the question.

Ask a focused factual question, such as "What is the vehicle directly ahead doing?" or "Is the traffic light green?". Broad description requests can produce fragments of the model’s structured analysis template instead of a clean prose answer. Treat answer as model-generated text and inspect it before using it in an application; it is not a guaranteed structured schema. VQA can decline or answer incorrectly on some scene questions, and its temporal reasoning is weak: answers can be identical with one and four frames. Single images are reliable for front_wide 16:9 frames; for other views, ask inside the six-camera layout. These limits are shared with the reference model; see Known Issues and Limitations.

Meta-Action

Using the ready quickstart container:

docker exec "$NIM_CONTAINER" python3 /opt/nim/build_http_payload.py \
    --endpoint infer --sample-dir /workspace/sample_data \
    --task meta_action > "$ALPAMAYO_OUT/alpamayo-meta_action.json"
curl --fail-with-body -sS "${NIM_URL}/v1/infer" \
    -H 'Content-Type: application/json' \
    --data-binary "@$ALPAMAYO_OUT/alpamayo-meta_action.json"

Set task to meta_action and use the trajectory input profile. The response includes the unmodified generated text, chain-of-thought reasoning, and normalized action axes:

{
  "task": "meta_action",
  "raw_text": "...",
  "reasoning_text": "...",
  "meta_action": {
    "longitudinal": "decelerate",
    "lateral": "keep",
    "lane": "current"
  }
}

The normalized values are constrained to these enums:

• longitudinal: stop, decelerate, accelerate, or maintain;
• lateral: left, right, or keep; and
• lane: left, right, or current.

raw_text preserves the model’s wording, such as Strong Deceleration, and has no guaranteed stable format. The meta_action fields hold the normalized values. Integrations should consume the normalized meta_action object. It always contains all three axes; an axis that cannot be parsed is null and meta_action_reason explains the incomplete result. finish_reason is length when the text reached max_tokens.

Send four frames per camera. Other counts are served with a warnings entry because they were not evaluated. On scenes where the action is borderline, an axis can change with the decoding settings: for example, greedy decoding and different seeds at the default temperature can return right or keep for lateral on the same input. Use temperature=0 to reduce sampling variation, but check repeatability after other tasks and across restarts; cache history can still affect greedy results. Treat a single label as one observation.

Auto-Labeling

Auto-labeling requires future_source; omitting it returns HTTP 422. The payload builder defaults --future-source to predicted and writes the field into every auto-labeling request.

• provided

  Set future_source="provided" and supply future_trajectory.xyz with shape [64, 3] and future_trajectory.rotations with shape [64, 3, 3].

  The packaged auto_labeling scene includes authentic positions and rotations from PhysicalAI-AV. Build a complete request inside the container:

  docker exec "$NIM_CONTAINER" python3 /opt/nim/build_http_payload.py \
      --endpoint infer --task auto_labeling --future-source provided \
      --sample-dir /workspace/sample_data/auto_labeling \
      --future-trajectory /workspace/sample_data/auto_labeling/gt_trajectory.json \
      > "$ALPAMAYO_OUT/alpamayo-auto-labeling-provided.json"
  curl --fail-with-body -sS "${NIM_URL}/v1/infer" \
      -H 'Content-Type: application/json' \
      --data-binary "@$ALPAMAYO_OUT/alpamayo-auto-labeling-provided.json"

  The builder also accepts caller JSON containing xyz and rotations directly or under future_trajectory. It also accepts the sample-file keys ego_future_xyz and ego_future_rot and normalizes them to the API fields; those aliases are builder inputs, not API field names.

• predicted

  Set future_source="predicted" and omit future_trajectory. The NIM first predicts one trajectory and then applies auto-labeling to that exact future. The response includes future_trajectory (positions and rotations) and trajectory_cot used by the labeling request. Build and send a predicted-future request from the packaged six-camera scene:

  docker exec "$NIM_CONTAINER" python3 /opt/nim/build_http_payload.py \
      --endpoint infer --task auto_labeling --future-source predicted \
      --sample-dir /workspace/sample_data \
      > "$ALPAMAYO_OUT/alpamayo-auto-labeling-predicted.json"
  curl --fail-with-body -sS "${NIM_URL}/v1/infer" \
      -H 'Content-Type: application/json' \
      --data-binary "@$ALPAMAYO_OUT/alpamayo-auto-labeling-predicted.json"

The normalized auto_labeling object always contains the four string fields critical_components_analysis, ego_vehicle_motion_analysis, trajectory_analysis, and chain_of_causation. Expect some of them to be empty. Alpamayo 2 Super was trained on labels in which sections that do not apply to a scene are null, and the model reproduces this: a simple scene often populates only chain_of_causation, and trajectory_analysis is frequently empty. The NIM returns such sections as empty strings, sets auto_labeling_reason, and preserves the unparsed model output in raw_text. This is expected model behavior, not a failure. On the packaged auto_labeling scene the straight, constant-speed gt_trajectory.json (provided) populates only chain_of_causation, while the predicted future for the same scene populates three of the four fields. Which sections are filled can change with small differences in the supplied future.

With provided, the model weighs the camera images more heavily than the supplied future. When the future contradicts what the images show, for example a stop, a reversal or a lane change while the scene suggests following the lead vehicle, ego_vehicle_motion_analysis and chain_of_causation can describe the motion the scene implies rather than the supplied future. Check the labels against the trajectory before using them as ground truth. A provided future that moves more than 1 m behind the current pose returns a warnings entry.

auto_labeling_reason distinguishes two other outcomes that do indicate a problem: the model output did not contain a structured JSON object, or the object was cut off at the generation budget. In both cases the four fields are empty; inspect raw_text rather than treating the empty fields as a label. The NIM always decodes auto-labeling with at least the released reference budget, so a cut-off response should be reported.

A partial response looks like this:

{
  "task": "auto_labeling",
  "future_source": "provided",
  "auto_labeling": {
    "critical_components_analysis": "",
    "ego_vehicle_motion_analysis": "",
    "trajectory_analysis": "",
    "chain_of_causation": "Maintain a safe distance from the lead vehicle since it is directly ahead in the same lane."
  },
  "auto_labeling_reason": "One or more auto-labeling fields were missing, null, or empty in the model response: critical_components_analysis, ego_vehicle_motion_analysis, trajectory_analysis. ...",
  "raw_text": "{\"critical_components_analysis\": null, ...}"
}

future_source="predicted" runs a trajectory prediction before labeling and is therefore slower than provided; use provided when the future is known.

Grounding

Using the ready quickstart container:

docker exec "$NIM_CONTAINER" python3 /opt/nim/build_http_payload.py \
    --endpoint infer --sample-dir /workspace/sample_data \
    --task grounding --question 'Locate the vehicles.' > "$ALPAMAYO_OUT/alpamayo-grounding.json"
curl --fail-with-body -sS "${NIM_URL}/v1/infer" \
    -H 'Content-Type: application/json' \
    --data-binary "@$ALPAMAYO_OUT/alpamayo-grounding.json"

Set task to grounding and provide exactly one camera in the camera_order request field so each returned bounding box identifies its source camera. For one image, also set num_frames_per_camera=1. The builder’s --camera-order selects cameras for every task. Grounding requires exactly one camera; without the flag, the builder selects front-wide and emits camera_order for you. The optional question selects the object to locate. When omitted, the released default asks for bounding boxes for vehicles and pedestrians. Each returned entry contains bbox, label, and camera:

{
  "task": "grounding",
  "raw_text": "[{\"bbox_2d\":[474,417,546,562],\"label\":\"Car\"}]",
  "grounding_boxes": [
    {
      "bbox": [474.0, 417.0, 546.0, 562.0],
      "label": "Car",
      "camera": "camera_front_wide_120fov"
    }
  ],
  "bbox_space": "grid_0_1000"
}

bbox contains the model’s 0–1000 coordinate grid in [x_min, y_min, x_max, y_max] order. The NIM passes these coordinates through without converting them to source-image pixels. To draw a retained box on an image of width W and height H, use [x_min * W / 1000, y_min * H / 1000, x_max * W / 1000, y_max * H / 1000]. For example, the illustrative box above maps to [910.08, 450.36, 1048.32, 606.96] on a 1920×1080 image.

Every grounding response also carries "bbox_space": "grid_0_1000", so a client can tell from the response alone that bbox is [x_min, y_min, x_max, y_max] on the model’s 0–1000 grid, not pixels. Multiply x by width/1000 and y by height/1000 of the source image to get pixels. Other tasks do not set bbox_space.

The NIM validates boxes after scaling the grid coordinates to the source-image size and returns valid boxes in grid coordinates. An empty list does not prove that the requested object is absent; inspect grounding_reason and raw_text.

Labels are whitespace-trimmed model text, and raw_text is returned verbatim. Label case and wording vary with the question phrasing and frame: the same vehicle can be labeled car, Car or Truck. Match labels case-insensitively and allow for synonyms. Absent-object requests can return an empty list with a grounding_reason, or a hallucinated box, often around the most salient visible object and sometimes labeled with the requested word. Do not treat a non-empty grounding_boxes list as proof that the object exists. The NIM sets grounding_reason when the returned labels do not name any object asked for in the question, when boxes were dropped for lying outside the model grid, or when no box could be parsed; treat any response with grounding_reason as unverified and filter on label. No per-box confidence is returned. Ask only for classes the model grounds in driving scenes (vehicles, pedestrians, traffic lights, signs, cones or barriers). Grounding is phrasing-sensitive, can miss visible objects and can return spurious labels. It is reliable for front_wide 16:9 frames; with side, rear or telephoto labels or other aspect ratios, boxes may not land on the objects. These limits are shared with the reference model.

Validation Errors

Request-schema and task-contract errors return HTTP 422 with an actionable message. Inputs that pass schema validation but fail image decoding or the backend context budget return HTTP 400. Common errors include an incorrect image count, missing ego-motion, a missing VQA question, an unspecified auto-labeling future source, a future trajectory with the wrong shape, and multiple cameras in a grounding request.

Single-Image Canonical Request Templates

These bodies go to POST /v1/infer. Replace the image placeholder with actual base64 data; the builder commands above produce runnable requests. The camera label must describe the supplied image. For your own data, send the newest frame of that camera, for example front_wide_t3.jpg in the sample layout; the builder uses t0 only as a fixed demonstration input. See Known Issues and Limitations for camera- and question-dependent quality limits.

{
  "task": "vqa",
  "images": ["data:image/jpeg;base64,<BASE64_IMAGE>"],
  "camera_order": ["front_wide"],
  "num_frames_per_camera": 1,
  "question": "What is the vehicle ahead doing?"
}

{
  "task": "grounding",
  "images": ["data:image/jpeg;base64,<BASE64_IMAGE>"],
  "camera_order": ["front_wide"],
  "num_frames_per_camera": 1,
  "question": "Locate the vehicles."
}

Expect answer for VQA and grounding_boxes for grounding. The grounding response above is a fragment from the bundled front-wide scene, not a complete annotation or a guaranteed detection. Questions such as Locate the vehicles. are useful starting points; wording changes can change the detections.

Payload Builder Controls

The payload builder creates complete requests from the sample files. Its sampling defaults are --seed 42, --temperature 0.6 and --num-traj-samples 1. These are builder defaults; an API request that omits its seed uses the container default. Seeds must be integers from 0 through 2147483647. Temperature accepts 0 through 1.5 for driving tasks and 0 through 2 for VQA and grounding.

--num-traj-samples accepts 1 through 32. Values above 1 require trajectory or chat, a positive temperature, and a sample count enabled in the running service. --model applies only to the chat and VQA compatibility endpoints.

--camera-order accepts short aliases or full camera names. Trajectory/chat allow supported driving-camera subsets. Meta-action and auto-labeling require the canonical six driving cameras in their canonical order. VQA permits selected cameras; grounding requires exactly one. With a reduced sample layout, VQA uses the t0 image of each selected camera and defaults to front-wide t0. The builder declares the actual six driving cameras for full sample VQA requests, rather than relying on the API’s default camera layout.

For the /v1/vqa compatibility endpoint (--endpoint vqa), the builder puts the question only in the message text; for --endpoint infer it sets the question field. If constructing requests yourself, prefer one question source to avoid conflicting values.