> This page is for version 2.0.0 (Latest) (default).
> For other versions, use one of these documentation indexes:
> - 2.0.0 (Latest) (default): https://docs.nvidia.com/nim/alpamayo/2.0.0/llms.txt
> - 1.0.0: https://docs.nvidia.com/nim/alpamayo/1.0.0/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nim/alpamayo/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nim/alpamayo/_mcp/server.

# Inference Tasks

Use `POST /v1/infer` for new Alpamayo 2 Super integrations. Set `task`
explicitly so that request validation, required inputs, and response fields are
clear. If both `task` and the legacy `mode` field are omitted, the service
runs `task="trajectory"`. For the previous generation, see
[Alpamayo 1.5 Documentation](/nim/alpamayo/1.0.0/overview).

## Example Target

Set these for your deployment before running the shell examples. Bash is
assumed; `pipefail` prevents a failed builder or HTTP request being hidden
by the last command in a pipeline. Use the container name you own.

```bash
export NIM_URL="${NIM_URL:-http://localhost:8000}"
export NIM_CONTAINER="${NIM_CONTAINER:-alpamayo2-super}"
export GRPC_TARGET="${GRPC_TARGET:-localhost:50051}"
export ALPAMAYO_OUT="${ALPAMAYO_OUT:-$(mktemp -d "${TMPDIR:-/tmp}/alpamayo.XXXXXX")}"
set -o pipefail
```

`NIM_URL` is the HTTP base URL and `GRPC_TARGET` the gRPC `host:port` as
published on the host; use your remapped host ports if you changed them. The
packaged host clients read both variables. `ALPAMAYO_OUT` is a private
directory for the request and response files written below.

## Task Summary

| Task | Required inputs | Primary output |
| ---- | --------------- | -------------- |
| `trajectory` | Declared driving-camera subset (default six), temporal frames, and 16 ego-history poses | Trajectories, rotations, and reasoning |
| `vqa` | One or more cameras and `question` | `answer` |
| `meta_action` | Six cameras x four frames and 16 ego-history poses | Reasoning and structured action axes |
| `auto_labeling` | Driving inputs plus an explicit future source | Four-field causal analysis JSON |
| `grounding` | Exactly one camera and an optional question | Camera-associated bounding boxes |

## Trajectory

Set `task` to `trajectory`. The default uses all six cameras in the
[driving camera layout](/nim/alpamayo/api-reference#camera-order-and-sample-files), with four frames per
camera in camera-major order. An explicit `camera_order` can select any
non-empty subset of those cameras. Every request still needs 16 ego-history
translations and rotations. Meta-action and auto-labeling retain the complete
six-camera profile.

For four-camera evaluation, use `cross_left`, `front_wide`, `cross_right`,
and `front_tele`, with four frames each (16 images). The builder selects those
frames and declares the layout:

```bash
docker exec "$NIM_CONTAINER" python3 /opt/nim/build_http_payload.py \
  --endpoint infer --task trajectory --sample-dir /workspace/sample_data \
  --camera-order cross_left,front_wide,cross_right,front_tele \
  > "$ALPAMAYO_OUT/alpamayo-four-camera.json"
curl --fail-with-body -sS "${NIM_URL}/v1/infer" \
  -H 'Content-Type: application/json' \
  --data-binary "@$ALPAMAYO_OUT/alpamayo-four-camera.json"
```

Trajectory also accepts 1, 2 or 3 frames per camera instead of four, with all
six cameras or a declared subset such as the four cameras above. Declare
`num_frames_per_camera`, send the most recent frames, and keep the 16-pose
ego history. The model was trained with four frames, so fewer frames reduce
temporal context and can change the result. With the payload builder, add
`--num-frames-per-camera 1`, `2` or `3`; it keeps the newest frames and
declares `camera_order` and `num_frames_per_camera`. See
[Preparing Inputs](/nim/alpamayo/preparing-inputs) for the supported layouts, frame timing, image sizes
and ego-motion axes.

The response contains `pred_trajectories`, `pred_rotations`,
`reasoning_text`, and trajectory metadata.

**Trajectory response fields**

| Field | Meaning |
| ----- | ------- |
| `pred_trajectories` | XYZ translations with shape `[num_samples, 64, 3]`. |
| `pred_rotations` | Rotation matrices with shape `[num_samples, 64, 3, 3]`. |
| `num_samples` / `num_points` | Returned trajectory count and points per trajectory. |
| `duration` | Predicted horizon in seconds: 6.4 seconds for the 64-point trajectory. |
| `inference_time` | Backend inference time in seconds. |
| `reasoning_text` | Model-generated trajectory reasoning. |
| `metrics` | Kinematic, comfort, and optional runtime diagnostics described in [Sampling Control](/nim/alpamayo/sampling-params). |

The model plans from whatever cameras you declare and does not know when
forward context is missing: a rear-camera-only request still returns a forward
plan. Include the front cameras for trajectory. The service validates camera names and image counts, but cannot detect mislabeled
cameras or timesteps from pixels. Supply camera-major order; a timestep-major
array can pass validation while changing the result.

The commands on this page assume Bash on the Docker host and a ready container
named `alpamayo2-super`, started using [Quickstart Guide](/nim/alpamayo/quickstart-guide). Run the
packaged builder inside the container; the JSON file is written on the host:

```bash
docker exec "$NIM_CONTAINER" python3 /opt/nim/build_http_payload.py \
    --endpoint infer --sample-dir /workspace/sample_data \
    --task trajectory > "$ALPAMAYO_OUT/alpamayo-trajectory.json"
curl --fail-with-body -sS "${NIM_URL}/v1/infer" \
    -H 'Content-Type: application/json' \
    --data-binary "@$ALPAMAYO_OUT/alpamayo-trajectory.json"
```

### Navigation-Conditioned Trajectory

`nav_text` adds a navigation instruction to the prompt.
`nav_guidance_weight` (0 through 10, default `3.0`) controls classifier-free
guidance: the action expert evaluates guided and unguided prompts for each
sample. Set `1.0` for prompt-only conditioning. Omit `nav_text` when no
instruction is available; the weight is then ignored. Navigation text is
advisory model input, not a vehicle-control or safety command.

Use a maneuver and distance, such as `"Turn left in 15m"` or `"Stop in 15m"`.
The model responds more strongly to this form than to general prose, but the
effect depends on the scene. At the default weight of 3.0, guidance steers the
plan in only part of the samples on some scenes, and the outcome varies with
the seed; the reference model behaves the same. A higher weight, around 6, or
several trajectory samples (K>1) increase compliance. Higher weights can also
degrade trajectory quality and cause excessive braking, so evaluate the weight
on your scenes and check each returned path against the instruction.
Guidance latency varies with the profile, scene, and execution path.
It can be faster or slower than an otherwise identical unconditioned request;
measure the actual path and settings used by your clients.

The builder command below writes the complete request. The following is only
the navigation-field fragment to add to an existing trajectory request:

```json
{
  "nav_text": "Turn right in 15m",
  "nav_guidance_weight": 3.0
}
```

Use the packaged `navigation` scene to try a junction with left and right
branches. Its `source.json` identifies the PhysicalAI-AV clip and frame times;
the recorded future takes the right branch. Compare instructions on this same
scene and inspect both the resulting path and the guidance metrics. The
metrics confirm execution of guidance, not compliance with an instruction.

The packaged builder creates a complete request from this scene:

```bash
docker exec "$NIM_CONTAINER" python3 /opt/nim/build_http_payload.py \
    --endpoint infer --sample-dir /workspace/sample_data/navigation --task trajectory \
    --nav-text 'Turn right in 15m' \
    --nav-guidance-weight 3.0 > "$ALPAMAYO_OUT/alpamayo-nav.json"
curl --fail-with-body -sS "${NIM_URL}/v1/infer" \
    -H 'Content-Type: application/json' \
    --data-binary "@$ALPAMAYO_OUT/alpamayo-nav.json"
```

Startup warmup exercises ordinary and guided trajectories for each configured K.
If disabled with `NIM_ALPAMAYO_STARTUP_WARMUP=0`, the first navigation request
prepares the guidance path. Before benchmarking, send an untimed request with
the exact navigation settings; ordinary trajectory warmup does not cover this path.

Navigation requests add three keys to `metrics`:

**Navigation metrics**

| Key | Meaning |
| --- | ------- |
| `nav_text_rendered` | `true`; the instruction was rendered into the prompt. |
| `nav_guidance_weight` | The guidance weight that was applied. |
| `nav_cfg_applied` | `true` when classifier-free guidance was applied: both the guided and the unguided branch were evaluated with an effective weight other than 0 or 1. `false` when the effective weight is 0 or 1, or when the engine fell back to the guided prompt alone; treat such a result as prompt-only conditioning. |

Requests without `nav_text` do not include these keys. `nav_guidance_weight`
is only valid for `task="trajectory"` and legacy `mode="both"`; on other
tasks an explicit non-default value, or any value together with `nav_text`,
returns HTTP 422. Meta-action, auto-labeling, VQA and grounding reject `nav_text` with HTTP 422.
For repeatability, follow [Sampling Control](/nim/alpamayo/sampling-params) and also keep
`nav_text` and `nav_guidance_weight` fixed.

## VQA

Using the ready quickstart container:

```bash
docker exec "$NIM_CONTAINER" python3 /opt/nim/build_http_payload.py \
    --endpoint infer --sample-dir /workspace/sample_data \
    --task vqa --question 'What is the vehicle ahead doing?' > "$ALPAMAYO_OUT/alpamayo-vqa.json"
curl --fail-with-body -sS "${NIM_URL}/v1/infer" \
    -H 'Content-Type: application/json' \
    --data-binary "@$ALPAMAYO_OUT/alpamayo-vqa.json"
```

Set `task` to `vqa` and provide a non-empty `question`. VQA does not
require ego-motion and accepts one or more images. For a single-image request,
set `camera_order=["front_wide"]` and `num_frames_per_camera=1` explicitly.
Otherwise the default expects 24 images. For multi-frame or multi-camera
requests, declare the layout and make the image count equal to
`len(camera_order) * num_frames_per_camera`.

Without `camera_order`, 24 VQA images are read in the driving layout used by
trajectory and the bundled samples: `cross_left`, `front_wide`,
`cross_right`, `rear_left`, `rear_right`, `front_tele`, four frames
each. For any other set of cameras, including `rear_tele`, declare
`camera_order` explicitly.

Ask questions in English. Questions in other languages can return unrelated
or nonsensical text, and the response does not flag it.

Presence and location questions can make the model answer with grounding
output (a JSON list of `bbox_2d` boxes) instead of prose. The answer is
returned unchanged, and `answer_reason` says so. Use `task='grounding'` to
localize objects, or rephrase the question.

Ask a focused factual question, such as `"What is the vehicle directly ahead
doing?"` or `"Is the traffic light green?"`. Broad description requests can
produce fragments of the model's structured analysis template instead of a
clean prose answer. Treat `answer` as model-generated text and inspect it
before using it in an application; it is not a guaranteed structured schema.
VQA can decline or answer incorrectly on some scene questions, and its temporal
reasoning is weak: answers can be identical with one and four frames. Single
images are reliable for `front_wide` 16:9 frames; for other views, ask inside
the six-camera layout. These limits are shared with the reference model; see
[Known Issues and Limitations](/nim/alpamayo/known-issues).

## Meta-Action

Using the ready quickstart container:

```bash
docker exec "$NIM_CONTAINER" python3 /opt/nim/build_http_payload.py \
    --endpoint infer --sample-dir /workspace/sample_data \
    --task meta_action > "$ALPAMAYO_OUT/alpamayo-meta_action.json"
curl --fail-with-body -sS "${NIM_URL}/v1/infer" \
    -H 'Content-Type: application/json' \
    --data-binary "@$ALPAMAYO_OUT/alpamayo-meta_action.json"
```

Set `task` to `meta_action` and use the trajectory input profile. The
response includes the unmodified generated text, chain-of-thought reasoning,
and normalized action axes:

```json
{
  "task": "meta_action",
  "raw_text": "...",
  "reasoning_text": "...",
  "meta_action": {
    "longitudinal": "decelerate",
    "lateral": "keep",
    "lane": "current"
  }
}
```

The normalized values are constrained to these enums:

- `longitudinal`: `stop`, `decelerate`, `accelerate`, or `maintain`;
- `lateral`: `left`, `right`, or `keep`; and
- `lane`: `left`, `right`, or `current`.

`raw_text` preserves the model's wording, such as `Strong Deceleration`,
and has no guaranteed stable format. The `meta_action` fields hold the
normalized values. Integrations should
consume the normalized `meta_action` object. It always contains all three
axes; an axis that cannot be parsed is `null` and `meta_action_reason`
explains the incomplete result. `finish_reason` is `length` when the text
reached `max_tokens`.

Send four frames per camera. Other counts are served with a `warnings`
entry because they were not evaluated. On scenes where the action is
borderline, an axis can
change with the decoding settings: for example, greedy decoding and different
seeds at the default temperature can return `right` or `keep` for
`lateral` on the same input. Use `temperature=0` to reduce sampling
variation, but check repeatability after other tasks and across restarts;
cache history can still affect greedy results. Treat a single label as one
observation.

## Auto-Labeling

Auto-labeling requires `future_source`; omitting it returns HTTP 422.
The payload builder defaults `--future-source` to `predicted` and writes
the field into every auto-labeling request.

- `provided`

  Set `future_source="provided"` and supply `future_trajectory.xyz` with
  shape `[64, 3]` and `future_trajectory.rotations` with shape
  `[64, 3, 3]`.

  The packaged `auto_labeling` scene includes authentic positions and
  rotations from PhysicalAI-AV. Build a complete request inside the container:

  ```bash
  docker exec "$NIM_CONTAINER" python3 /opt/nim/build_http_payload.py \
      --endpoint infer --task auto_labeling --future-source provided \
      --sample-dir /workspace/sample_data/auto_labeling \
      --future-trajectory /workspace/sample_data/auto_labeling/gt_trajectory.json \
      > "$ALPAMAYO_OUT/alpamayo-auto-labeling-provided.json"
  curl --fail-with-body -sS "${NIM_URL}/v1/infer" \
      -H 'Content-Type: application/json' \
      --data-binary "@$ALPAMAYO_OUT/alpamayo-auto-labeling-provided.json"
  ```

  The builder also accepts caller JSON containing `xyz` and `rotations`
  directly or under `future_trajectory`. It also accepts the sample-file
  keys `ego_future_xyz` and `ego_future_rot` and normalizes them to
  the API fields; those aliases are builder inputs, not API field names.

- `predicted`

  Set `future_source="predicted"` and omit `future_trajectory`. The NIM
  first predicts one trajectory and then applies auto-labeling to that exact
  future. The response includes `future_trajectory` (positions and rotations)
  and `trajectory_cot` used by
  the labeling request. Build and send a predicted-future request from the
  packaged six-camera scene:

  ```bash
  docker exec "$NIM_CONTAINER" python3 /opt/nim/build_http_payload.py \
      --endpoint infer --task auto_labeling --future-source predicted \
      --sample-dir /workspace/sample_data \
      > "$ALPAMAYO_OUT/alpamayo-auto-labeling-predicted.json"
  curl --fail-with-body -sS "${NIM_URL}/v1/infer" \
      -H 'Content-Type: application/json' \
      --data-binary "@$ALPAMAYO_OUT/alpamayo-auto-labeling-predicted.json"
  ```

The normalized `auto_labeling` object always contains the four string fields
`critical_components_analysis`, `ego_vehicle_motion_analysis`,
`trajectory_analysis`, and `chain_of_causation`. Expect some of them to be
empty. Alpamayo 2 Super was trained on labels in which sections that do not
apply to a scene are `null`, and the model reproduces this: a simple scene
often populates only `chain_of_causation`, and `trajectory_analysis` is
frequently empty. The NIM returns such sections as empty strings, sets
`auto_labeling_reason`, and preserves the unparsed model output in
`raw_text`. This is expected model behavior, not a failure. On the packaged
`auto_labeling` scene the straight, constant-speed `gt_trajectory.json`
(`provided`) populates only `chain_of_causation`, while the `predicted`
future for the same scene populates three of the four fields. Which sections
are filled can change with small differences in the supplied future.

With `provided`, the model weighs the camera images more heavily than the
supplied future. When the future contradicts what the images show, for example
a stop, a reversal or a lane change while the scene suggests following the
lead vehicle, `ego_vehicle_motion_analysis` and `chain_of_causation` can
describe the motion the scene implies rather than the supplied future. Check
the labels against the trajectory before using them as ground truth. A
provided future that moves more than 1 m behind the current pose returns a
`warnings` entry.

`auto_labeling_reason` distinguishes two other outcomes that do indicate a
problem: the model output did not contain a structured JSON object, or the
object was cut off at the generation budget. In both cases the four fields are
empty; inspect `raw_text` rather than treating the empty fields as a label.
The NIM always decodes auto-labeling with at least the released reference
budget, so a cut-off response should be reported.

A partial response looks like this:

```json
{
  "task": "auto_labeling",
  "future_source": "provided",
  "auto_labeling": {
    "critical_components_analysis": "",
    "ego_vehicle_motion_analysis": "",
    "trajectory_analysis": "",
    "chain_of_causation": "Maintain a safe distance from the lead vehicle since it is directly ahead in the same lane."
  },
  "auto_labeling_reason": "One or more auto-labeling fields were missing, null, or empty in the model response: critical_components_analysis, ego_vehicle_motion_analysis, trajectory_analysis. ...",
  "raw_text": "{\"critical_components_analysis\": null, ...}"
}
```

`future_source="predicted"` runs a trajectory prediction before labeling and
is therefore slower than `provided`; use `provided` when the future is known.

## Grounding

Using the ready quickstart container:

```bash
docker exec "$NIM_CONTAINER" python3 /opt/nim/build_http_payload.py \
    --endpoint infer --sample-dir /workspace/sample_data \
    --task grounding --question 'Locate the vehicles.' > "$ALPAMAYO_OUT/alpamayo-grounding.json"
curl --fail-with-body -sS "${NIM_URL}/v1/infer" \
    -H 'Content-Type: application/json' \
    --data-binary "@$ALPAMAYO_OUT/alpamayo-grounding.json"
```

Set `task` to `grounding` and provide exactly one camera in the
`camera_order` request field so each returned bounding box identifies its
source camera. For one image, also set `num_frames_per_camera=1`.
The builder's `--camera-order` selects cameras for every task. Grounding
requires exactly one camera; without the flag, the builder selects front-wide
and emits `camera_order` for you. The
optional `question` selects the object to locate. When omitted, the released
default asks for bounding boxes for vehicles and pedestrians.
Each returned entry contains `bbox`, `label`, and `camera`:

```json
{
  "task": "grounding",
  "raw_text": "[{\"bbox_2d\":[474,417,546,562],\"label\":\"Car\"}]",
  "grounding_boxes": [
    {
      "bbox": [474.0, 417.0, 546.0, 562.0],
      "label": "Car",
      "camera": "camera_front_wide_120fov"
    }
  ],
  "bbox_space": "grid_0_1000"
}
```

`bbox` contains the model's **0–1000 coordinate grid** in
`[x_min, y_min, x_max, y_max]` order. The NIM passes these coordinates
through without converting them to source-image pixels. To draw a retained
box on an image of width `W` and height `H`, use
`[x_min * W / 1000, y_min * H / 1000, x_max * W / 1000, y_max * H / 1000]`.
For example, the illustrative box above maps to
`[910.08, 450.36, 1048.32, 606.96]` on a 1920×1080 image.

Every grounding response also carries `"bbox_space": "grid_0_1000"`, so a
client can tell from the response alone that `bbox` is
`[x_min, y_min, x_max, y_max]` on the model's 0–1000 grid, not pixels.
Multiply x by `width/1000` and y by `height/1000` of the source image to
get pixels. Other tasks do not set `bbox_space`.

The NIM validates boxes after scaling the grid coordinates to the source-image
size and returns valid boxes in grid coordinates. An empty list does not prove
that the requested object is absent; inspect `grounding_reason` and
`raw_text`.

Labels are whitespace-trimmed model text, and `raw_text` is returned
verbatim. Label case and wording vary with the question phrasing and frame:
the same vehicle can be labeled `car`, `Car` or `Truck`. Match labels
case-insensitively and allow for synonyms. Absent-object requests can return an empty list with a
`grounding_reason`, or a hallucinated box, often
around the most salient visible object and sometimes labeled with the requested
word. Do not treat a non-empty `grounding_boxes` list as proof that the object
exists. The NIM sets `grounding_reason` when the returned labels do not name
any object asked for in the `question`, when boxes were dropped for lying
outside the model grid, or when no box could be parsed; treat any response with
`grounding_reason` as unverified and filter on `label`. No per-box
confidence is returned. Ask only for classes the model grounds in driving
scenes (vehicles, pedestrians, traffic lights, signs, cones or barriers).
Grounding is phrasing-sensitive, can miss visible objects and can return
spurious labels. It is reliable for `front_wide` 16:9 frames; with side, rear
or telephoto labels or other aspect ratios, boxes may not land on the objects.
These limits are shared with the reference model.

## Validation Errors

Request-schema and task-contract errors return HTTP 422 with an actionable
message. Inputs that pass schema validation but fail image decoding or the
backend context budget return HTTP 400.
Common errors include an incorrect image count, missing ego-motion, a missing
VQA question, an unspecified auto-labeling future source, a future trajectory
with the wrong shape, and multiple cameras in a grounding request.

## Single-Image Canonical Request Templates

These bodies go to `POST /v1/infer`. Replace the image placeholder with
actual base64 data; the builder commands above produce runnable requests.
The camera label must describe the supplied image. For your own data, send
the newest frame of that camera, for example `front_wide_t3.jpg` in the
sample layout; the builder uses `t0` only as a fixed demonstration input. See
[Known Issues and Limitations](/nim/alpamayo/known-issues) for camera- and question-dependent quality limits.

```json
{
  "task": "vqa",
  "images": ["data:image/jpeg;base64,<BASE64_IMAGE>"],
  "camera_order": ["front_wide"],
  "num_frames_per_camera": 1,
  "question": "What is the vehicle ahead doing?"
}
```

```json
{
  "task": "grounding",
  "images": ["data:image/jpeg;base64,<BASE64_IMAGE>"],
  "camera_order": ["front_wide"],
  "num_frames_per_camera": 1,
  "question": "Locate the vehicles."
}
```

Expect `answer` for VQA and `grounding_boxes` for grounding. The grounding
response above is a fragment from the bundled front-wide scene, not a complete
annotation or a guaranteed detection. Questions such as `Locate the vehicles.`
are useful starting points; wording changes can change the detections.

## Payload Builder Controls

The [payload builder](./examples/build_http_payload.py "download") creates complete
requests from the sample files. Its sampling defaults are `--seed 42`,
`--temperature 0.6` and `--num-traj-samples 1`. These are builder defaults;
an API request that omits its seed uses the container default. Seeds must be
integers from 0 through 2147483647. Temperature accepts 0 through 1.5 for
driving tasks and 0 through 2 for VQA and grounding.

`--num-traj-samples` accepts 1 through 32. Values above 1 require trajectory
or chat, a positive temperature, and a sample count enabled in the running
service. `--model` applies only to the chat and VQA compatibility endpoints.

`--camera-order` accepts short aliases or full camera names. Trajectory/chat
allow supported driving-camera subsets. Meta-action and auto-labeling require
the canonical six driving cameras in their canonical order. VQA permits
selected cameras; grounding requires exactly one. With a reduced sample layout,
VQA uses the t0 image of each selected camera and defaults to front-wide t0.
The builder declares the actual six driving cameras for full sample VQA
requests, rather than relying on the API's default camera layout.

For the `/v1/vqa` compatibility endpoint (`--endpoint vqa`), the builder puts
the question only in the message text; for `--endpoint infer` it sets the
`question` field. If constructing requests yourself, prefer one question source to avoid conflicting values.