Sampling Control

Alpamayo1.5 NIM exposes request parameters for controlling VLM rollout, trajectory sampling, and optional navigation text.

Trajectory Inputs

Trajectory requests require compressed camera images and ego-motion history. The default layout is 16 images: four timesteps for each of four cameras. When camera_order is omitted, the NIM assumes this default camera-major order:

cross_left_t0,  cross_left_t1,  cross_left_t2,  cross_left_t3,
front_wide_t0,  front_wide_t1,  front_wide_t2,  front_wide_t3,
cross_right_t0, cross_right_t1, cross_right_t2, cross_right_t3,
front_tele_t0,  front_tele_t1,  front_tele_t2,  front_tele_t3

t0 is the oldest frame, approximately t-0.3s. t3 is the current frame.

Reduced image layouts are supported. For any trajectory request that does not use the default 16-image layout, include both camera_order and num_frames_per_camera. Images must still be ordered camera-major: all timesteps for the first camera, then all timesteps for the next camera. For example:

{
   "camera_order": ["front_wide"],
   "num_frames_per_camera": 4
}

This layout expects front_wide_t0 through front_wide_t3. A layout such as camera_order=["cross_left", "front_wide", "cross_right", "front_tele"] and num_frames_per_camera=1 expects one current-frame image for each camera. If a trajectory request sends fewer or more than 16 images without camera_order, request-model validation returns HTTP 422 because the NIM cannot infer the image grouping.

Camera names can be full Alpamayo camera names, such as camera_front_wide_120fov, or OSS short names, such as front_wide.

The ego-motion history length is independent of the image layout. Every trajectory request, including reduced and single-image requests, requires all 16 history entries. Do not shorten the history to match the number of images.

Use JPEG input images when possible. JPEG payloads are smaller than PNG and use the fastest image decode path in the NIM.

VQA Inputs

Visual question answering accepts one or more compressed images and preserves the image order sent by the client. VQA requests do not require ego-motion. Use JPEG images when possible. Common benchmark-style layouts such as one image, three images, five frames, and six camera views are accepted.

Sampling Parameters

Name	Type	Default	Required	Description
num_traj_samples	integer	1	No	Number of trajectory samples to generate. The value must be listed in NIM_ALPAMAYO_TRAJ_SAMPLES.
nav_text	string	unset	No	Optional route context for trajectory generation. It is advisory model input, not a deterministic vehicle-control command or a safety control.
temperature	float	0.6	No	VLM sampling temperature.
top_p	float	0.98	No	VLM nucleus sampling top-p.
top_k	integer	unset	No	Optional VLM top-k sampling limit. Use 1 for top-1 decoding.
max_tokens	integer	256	No	Maximum VLM generation tokens.
egomotion	object	unset	Yes (trajectory); No (VQA)	Ego-motion payload containing ego_history_xyz with shape [16, 3] and ego_history_rot with shape [16, 4] (each quaternion [w, x, y, z]) or [16, 3, 3] for rotation matrices. Other history lengths are rejected before inference.
images	array	unset	Yes	Compressed camera images as base64 strings, data URLs, or compressed image bytes depending on the API used. The default trajectory layout is 16 images. Non-default trajectory layouts require camera_order and num_frames_per_camera.
camera_order	array	unset	No	Camera names matching the camera-major image order. Full Alpamayo camera names and OSS short names are accepted. Required for trajectory requests when image count is not 16.
num_frames_per_camera	integer	unset; the default layout implies 4	Required for non-default layouts	Number of temporal frames supplied for each camera. Omit this field and camera_order for the default 16-image layout. Set both fields explicitly for every reduced layout.
seed	integer	unset	No	Optional per-request sampling seed. When omitted, the container default seed is used.

Reproducibility

HTTP and gRPC requests can include seed to initialize sampling state. When seed is omitted, the container-level NIM_ALPAMAYO_SEED value is used. For repeatable K=1 trajectory inference in the same deployment, use the same input with num_traj_samples=1, a fixed seed, top_k=1, and top_p=1:

{
  "num_traj_samples": 1,
  "seed": 42,
  "top_k": 1,
  "top_p": 1
}

The seed controls diffusion sampling. top_k=1 requests top-1 VLM decoding, and top_p=1 leaves nucleus sampling unrestricted so it does not filter the top-1 token.

K Values

The active K values are controlled by NIM_ALPAMAYO_TRAJ_SAMPLES. Configure only the K values your deployment needs. Use an explicit comma-separated list such as 1,2,3 or 1,10. Each configured value is warmed during startup and retained for CUDA graph performance, so broad K lists can increase startup time and GPU memory usage. Requests outside the configured list return a client error. For example, num_traj_samples=2 when NIM_ALPAMAYO_TRAJ_SAMPLES=1,10 returns an error that the profile supports only those values.

For the full profile matrix, refer to Support Matrix.

Trajectory Metrics

Trajectory responses include a metrics object with trajectory-only kinematic metrics. The primary comfort metrics are scalar pass rates in [0, 1] across the generated trajectory samples:

• comfort_lat_accel

• comfort_lon_accel

• comfort_lon_jerk

• comfort_yaw_accel

• comfort_yaw_rate

These metrics are computed from the generated trajectory and predicted ego rotations. Evaluation-only metrics that require ground truth or map geometry are not part of the normal inference response.

The object also reports its schema version, time_step_s, comfort thresholds, aggregate min/mean/max kinematic summaries, and a per_sample breakdown. Refer to API Reference for the full field list and units.

The response includes pred_rotations as rotation matrices with shape [K, num_points, 3, 3]. This matches the OSS pred_rot output after squeezing the batch and trajectory-set dimensions.