Preparing Inputs

Alpamayo 2 Super. Alpamayo 1.5 uses a different camera layout; see Alpamayo 1.5 Documentation.

The bundled sample_data scene and the payload builder produce valid requests for you. Use this page when you replace the sample with your own camera frames and ego-motion. The service validates names, counts, shapes and image decoding. Final-pose checks reject some ego-motion convention errors, but mislabeled cameras, reversed image frames, and some incorrectly converted histories can pass validation and silently change the output.

Checklist

1. Choose the cameras and frame count, and list the images camera-major with each camera’s frames oldest first.
2. Use frames spaced 0.1 seconds apart, ending with the frame at the prediction time.
3. Give all images in one request the same width and height; prefer 1920×1080 JPEG frames from cameras matching the driving layout.
4. Supply exactly 16 ego-history poses sampled at 0.1 seconds, expressed in the ego frame at the prediction time (x forward, y left, z up, meters).
5. Keep the request below the 134217728-byte HTTP body or gRPC message limit.
6. Compare the result with the bundled sample before trusting a new pipeline.

Camera Layout and Frame Order

The driving layout contains six cameras in this order: cross_left, front_wide, cross_right, rear_left, rear_right and front_tele. The full camera names are listed in Camera order and sample files. Supply images camera-major: every frame for the first camera, then every frame for the next camera. Within each camera, the first frame is the oldest (t0) and the last is the newest.

Supported camera and frame layouts

Whenever you send fewer than six cameras or other than four frames, declare camera_order and num_frames_per_camera (inside nvext on the compatibility routes). The image count must equal the number of cameras multiplied by the frames per camera. Requests with more frames can exceed the model’s context limit and return HTTP 400. Missing cameras are never padded or synthesized.

When you use fewer frames, send the most recent ones: the last frame for every camera must still be the frame at the prediction time. For single-image VQA or grounding on your own data, send the newest frame. Start single-image requests with front_wide 16:9 frames and evaluate the results on your data. For VQA about other views, include them in the six-camera layout; grounding still requires exactly one camera. For trajectory, always include the front cameras: the model cannot tell that forward context is missing and still returns a forward plan. A trajectory request without front_wide or front_tele is served with a warnings entry. The payload builder uses the sample’s oldest front-wide frame, front_wide_t0, only as a fixed demonstration input.

Frame Timing

The bundled scenes use frames 0.1 seconds apart: t0 through t3 were captured 0.3, 0.2, 0.1 and 0 seconds before the prediction time, and the last frame coincides with the last ego-history pose. The navigation scene’s source.json records the requested and retrieved camera timestamps. Use the same spacing and alignment. Frames from different cameras at the same index should be captured as close together as your rig allows.

Image Format and Resolution

HTTP accepts base64 strings or data:image/...;base64,... URLs; gRPC accepts raw encoded bytes. JPEG (image/jpeg or image/jpg), PNG, WebP and BMP are supported. Data URLs must carry the ;base64 marker, and remote http(s) image URLs are not fetched. Baseline JPEG decodes on the GPU; other formats decode on the CPU with the same validation. All images in one request must have identical dimensions. Each side must be at least 2 pixels, and the longer side at most 200 times the shorter side. An image outside these bounds returns HTTP 400, for example Image width and height must each be at least 2 pixels.

The bundled scenes, like the recorded data the model was trained on, use 1920×1080 (16:9) frames. The model’s processor rescales every image to a fixed pixel budget while preserving its aspect ratio, so sending more than 1920×1080 adds upload and decoding work without adding model detail. Keep the 16:9 aspect ratio and the camera field of view named by each camera (for example 120° for front_wide). Other aspect ratios, crops and camera models are accepted but can produce weaker answers and trajectories; see Known Issues and Limitations.

Request Size and Uncompressed Images

Both transports enforce a 134217728-byte (128 MiB) limit. HTTP counts the whole JSON body, including the base64 expansion of every image (about 4/3 of its encoded size); gRPC counts the binary message, and LocalPathInput applies the same limit to the sum of the referenced files.

The 24 JPEG frames of the bundled scene total about 8 MB (about 11 MB as base64 JSON). Uncompressed BMP does not. A 1920×1080 24-bit BMP frame is about 6.2 MB, so the default 24-frame layout is about 149 MB of raw images (about 199 MB as base64 JSON) and exceeds both limits. For a 24-image request, budget at most about 4 MB of encoded image data per frame over HTTP and about 5.5 MB over gRPC. For BMP this means frames of about 1280×720 or smaller. Prefer JPEG or WebP for full-resolution frames.

Oversized HTTP requests return HTTP 413. Send them with the packaged http_client.py or another client that implements Expect: 100-continue to receive the 413 before uploading. A client that uploads immediately can see a broken pipe or connection reset instead, because the server stops reading an oversized upload after a bounded amount; see API Reference.

Ego-Motion History

egomotion.ego_history_xyz has exactly 16 rows of [x, y, z] in meters. egomotion.ego_history_rot has exactly 16 rotations, each either a 3×3 rotation matrix or a scalar-first unit quaternion [w, x, y, z]. History is chronological, sampled every 0.1 seconds, and ends at the prediction time.

Coordinates use the ego vehicle’s frame at the prediction time: a right-handed frame with x forward, y to the left and z up. The last history row is therefore [0, 0, 0] with the identity rotation, and earlier rows of a forward-moving vehicle have negative x. Quaternions are scalar-first [w, x, y, z]. The NIM rejects a history whose last row is not the origin (within 0.01 m) with the identity rotation (within 0.001 per matrix element) with HTTP 422 or gRPC INVALID_ARGUMENT; the message points to this page. A history implying a speed above 70 m/s is served but adds an entry to the response warnings, because it usually means centimeters instead of meters or the wrong row spacing. The last-row check rejects a history whose quaternions are all scalar-last or whose rows are simply reversed, and the speed check flags centimeter positions of a moving vehicle. Mixed or re-anchored convention errors and centimeter positions of a slow or stationary vehicle are not detected, and a history that implies reversing faster than about 2 m/s (for example a 180-degree heading error) is served with a warnings entry; the model, like the reference model, returns confident but wrong plans for them. In the bundled sample the vehicle is driving forward, so its history runs from about [-18.1, -0.47, -0.02] to [0, 0, 0].

To convert global (world) poses with positions p_i and orientations R_i, where R_i maps the vehicle frame at pose i into the world frame, use the latest pose p_t, R_t. The helper uses numpy, which the client virtual environment in Set up a client outside the NIM installs:

import numpy as np
def to_ego_frame(p_world, R_world):
    """p_world: [16, 3] positions; R_world: [16, 3, 3], oldest first.
    Accepts numpy arrays or nested Python lists.
    """
    p_world = np.asarray(p_world, dtype=float)
    R_world = np.asarray(R_world, dtype=float)
    R_t_inv = R_world[-1].T
    xyz = (p_world - p_world[-1]) @ R_t_inv.T
    rot = R_t_inv @ R_world
    return xyz.tolist(), rot.tolist()

Predicted trajectories use the same frame and continue from the prediction time: 64 future poses at 0.1-second intervals. A caller-provided auto-labeling future must use the same convention. An all-zero history describes a vehicle that has been standing still; submitted with images of a moving scene, it can produce a plausible-looking path that reverses or stops. Use real history.

gRPC Ordering, Local Paths and Defaults

For compressed_images, label every frame with camera and timestep. When any frame carries a camera label, every frame must, and the server orders frames by those labels: by camera_order when it is set, otherwise by the canonical camera order, and by timestep from 0 (oldest) upward. timestep is the frame’s position within its camera, not an absolute frame number: with num_frames_per_camera set to 2, label each camera’s frames 0 and 1. Array order is then ignored. Duplicate, missing or extra camera/timestep pairs return INVALID_ARGUMENT. Unlabeled frames are used in array order and must already be camera-major; prefer labels. Set mime_type on every frame. A declared type must match the bytes, or the request returns INVALID_ARGUMENT; an empty value lets the server detect the format from the bytes.

local_paths (LocalPathInput) reads files that already exist inside the server container. It is disabled unless the operator sets NIM_ALPAMAYO_GRPC_LOCAL_PATH_ROOT, and every path, including symlink targets, must resolve under that root. Two layouts are supported:

• scene_dir alone: the directory must contain egomotion.json and images/<short-name>_t<N>.<ext> for each selected camera and frame, for example images/front_wide_t0.jpg. Extensions .jpg, .jpeg, .png, .webp and .bmp are recognized. The bundled sample_data directory uses this layout. With fewer than four frames per camera, the newest frames are read (t2 and t3 for two frames), so they end at the same time as the ego-motion history.
• image_paths: an explicit camera-major list. Relative entries are resolved under scene_dir when it is set. Without a scene directory, also send egomotion_json and declare the layout.

If a request sets both local_paths.scene_dir and egomotion_json, egomotion_json is used and the scene directory’s egomotion.json is ignored.

Proto3 does not transmit unset scalar fields. The server treats an unset or zero num_traj_samples as 1 and an unset or zero num_frames_per_camera as 4. temperature, top_p, top_k, seed and nav_guidance_weight are optional fields, so an explicit zero is honored: temperature=0 requests greedy decoding and seed=0 is a valid seed. Leave them unset to use the NIM defaults. An empty model selects the served model.

Verify a New Input Pipeline

1. Rebuild the bundled scene with your own conversion code, starting from the raw files in sample_data, and compare the request with the one produced by build_http_payload.py.
2. Send both with temperature=0 and num_traj_samples=1 and compare the trajectories. The same inputs on the same deployment should give closely matching results.
3. For your own scenes, check that predicted paths continue in the direction of travel shown by the history and images. A path that reverses on a moving scene usually indicates a sign, frame or ordering error. A history that implies reversing faster than about 2 m/s returns a warnings entry; it usually means a frame-convention mistake (180-degree heading or a transposed world-to-ego rotation).