> This page is for version 2.0.0 (Latest) (default). > For other versions, use one of these documentation indexes: > - 2.0.0 (Latest) (default): https://docs.nvidia.com/nim/alpamayo/2.0.0/llms.txt > - 1.0.0: https://docs.nvidia.com/nim/alpamayo/1.0.0/llms.txt > For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/nim/alpamayo/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nim/alpamayo/_mcp/server. # Preparing Inputs **Alpamayo 2 Super.** Alpamayo 1.5 uses a different camera layout; see [Alpamayo 1.5 Documentation](/nim/alpamayo/1.0.0/overview). The bundled `sample_data` scene and the payload builder produce valid requests for you. Use this page when you replace the sample with your own camera frames and ego-motion. The service validates names, counts, shapes and image decoding. Final-pose checks reject some ego-motion convention errors, but mislabeled cameras, reversed image frames, and some incorrectly converted histories can pass validation and silently change the output. ## Checklist 1. Choose the cameras and frame count, and list the images camera-major with each camera's frames oldest first. 2. Use frames spaced 0.1 seconds apart, ending with the frame at the prediction time. 3. Give all images in one request the same width and height; prefer 1920×1080 JPEG frames from cameras matching the driving layout. 4. Supply exactly 16 ego-history poses sampled at 0.1 seconds, expressed in the ego frame at the prediction time (x forward, y left, z up, meters). 5. Keep the request below the 134217728-byte HTTP body or gRPC message limit. 6. Compare the result with the bundled sample before trusting a new pipeline. ## Camera Layout and Frame Order The driving layout contains six cameras in this order: `cross_left`, `front_wide`, `cross_right`, `rear_left`, `rear_right` and `front_tele`. The full camera names are listed in [Camera order and sample files](/nim/alpamayo/api-reference#camera-order-and-sample-files). Supply images **camera-major**: every frame for the first camera, then every frame for the next camera. Within each camera, the first frame is the oldest (`t0`) and the last is the newest. **Supported camera and frame layouts** | Task | Cameras | Frames per camera | | ---- | ------- | ----------------- | | `trajectory` | All six (default), or any declared non-empty subset of the six, such as `cross_left`, `front_wide`, `cross_right`, `front_tele`. | 4 (default). 1, 2 or 3 are also accepted with six cameras or with a subset such as four cameras. The model was trained with four frames; fewer frames reduce temporal context and can change the result. Larger counts are served up to the context limit with a `warnings` entry; they are not evaluated. | | `meta_action`, `auto_labeling` | All six, in the order above. | 4 (default). These tasks are evaluated with 4 frames per camera; other counts are served with a `warnings` entry. | | `vqa` | Any declared non-empty subset; `rear_tele` is also accepted. Without `camera_order`, 24 images are read in the driving order above. | 1 or more. | | `grounding` | Exactly one. | 1 is recommended. | Whenever you send fewer than six cameras or other than four frames, declare `camera_order` and `num_frames_per_camera` (inside `nvext` on the compatibility routes). The image count must equal the number of cameras multiplied by the frames per camera. Requests with more frames can exceed the model's context limit and return HTTP 400. Missing cameras are never padded or synthesized. When you use fewer frames, send the most recent ones: the last frame for every camera must still be the frame at the prediction time. For single-image VQA or grounding on your own data, send the newest frame. Start single-image requests with `front_wide` 16:9 frames and evaluate the results on your data. For VQA about other views, include them in the six-camera layout; grounding still requires exactly one camera. For trajectory, always include the front cameras: the model cannot tell that forward context is missing and still returns a forward plan. A trajectory request without `front_wide` or `front_tele` is served with a `warnings` entry. The payload builder uses the sample's oldest front-wide frame, `front_wide_t0`, only as a fixed demonstration input. ## Frame Timing The bundled scenes use frames 0.1 seconds apart: `t0` through `t3` were captured 0.3, 0.2, 0.1 and 0 seconds before the prediction time, and the last frame coincides with the last ego-history pose. The `navigation` scene's `source.json` records the requested and retrieved camera timestamps. Use the same spacing and alignment. Frames from different cameras at the same index should be captured as close together as your rig allows. ## Image Format and Resolution HTTP accepts base64 strings or `data:image/...;base64,...` URLs; gRPC accepts raw encoded bytes. JPEG (`image/jpeg` or `image/jpg`), PNG, WebP and BMP are supported. Data URLs must carry the `;base64` marker, and remote `http(s)` image URLs are not fetched. Baseline JPEG decodes on the GPU; other formats decode on the CPU with the same validation. All images in one request must have identical dimensions. Each side must be at least 2 pixels, and the longer side at most 200 times the shorter side. An image outside these bounds returns HTTP 400, for example `Image width and height must each be at least 2 pixels.` The bundled scenes, like the recorded data the model was trained on, use 1920×1080 (16:9) frames. The model's processor rescales every image to a fixed pixel budget while preserving its aspect ratio, so sending more than 1920×1080 adds upload and decoding work without adding model detail. Keep the 16:9 aspect ratio and the camera field of view named by each camera (for example 120° for `front_wide`). Other aspect ratios, crops and camera models are accepted but can produce weaker answers and trajectories; see [Known Issues and Limitations](/nim/alpamayo/known-issues). ## Request Size and Uncompressed Images Both transports enforce a 134217728-byte (128 MiB) limit. HTTP counts the whole JSON body, including the base64 expansion of every image (about 4/3 of its encoded size); gRPC counts the binary message, and `LocalPathInput` applies the same limit to the sum of the referenced files. The 24 JPEG frames of the bundled scene total about 8 MB (about 11 MB as base64 JSON). Uncompressed BMP does not. A 1920×1080 24-bit BMP frame is about 6.2 MB, so the default 24-frame layout is about 149 MB of raw images (about 199 MB as base64 JSON) and exceeds both limits. For a 24-image request, budget at most about 4 MB of encoded image data per frame over HTTP and about 5.5 MB over gRPC. For BMP this means frames of about 1280×720 or smaller. Prefer JPEG or WebP for full-resolution frames. Oversized HTTP requests return HTTP 413. Send them with the packaged `http_client.py` or another client that implements `Expect: 100-continue` to receive the 413 before uploading. A client that uploads immediately can see a broken pipe or connection reset instead, because the server stops reading an oversized upload after a bounded amount; see [API Reference](/nim/alpamayo/api-reference). ## Ego-Motion History `egomotion.ego_history_xyz` has exactly 16 rows of `[x, y, z]` in meters. `egomotion.ego_history_rot` has exactly 16 rotations, each either a 3×3 rotation matrix or a scalar-first unit quaternion `[w, x, y, z]`. History is chronological, sampled every 0.1 seconds, and ends at the prediction time. Coordinates use the ego vehicle's frame **at the prediction time**: a right-handed frame with x forward, y to the left and z up. The last history row is therefore `[0, 0, 0]` with the identity rotation, and earlier rows of a forward-moving vehicle have negative x. Quaternions are scalar-first `[w, x, y, z]`. The NIM rejects a history whose last row is not the origin (within 0.01 m) with the identity rotation (within 0.001 per matrix element) with HTTP 422 or gRPC `INVALID_ARGUMENT`; the message points to this page. A history implying a speed above 70 m/s is served but adds an entry to the response `warnings`, because it usually means centimeters instead of meters or the wrong row spacing. The last-row check rejects a history whose quaternions are all scalar-last or whose rows are simply reversed, and the speed check flags centimeter positions of a moving vehicle. Mixed or re-anchored convention errors and centimeter positions of a slow or stationary vehicle are not detected, and a history that implies reversing faster than about 2 m/s (for example a 180-degree heading error) is served with a `warnings` entry; the model, like the reference model, returns confident but wrong plans for them. In the bundled sample the vehicle is driving forward, so its history runs from about `[-18.1, -0.47, -0.02]` to `[0, 0, 0]`. To convert global (world) poses with positions `p_i` and orientations `R_i`, where `R_i` maps the vehicle frame at pose `i` into the world frame, use the latest pose `p_t`, `R_t`. The helper uses `numpy`, which the client virtual environment in [Set up a client outside the NIM](/nim/alpamayo/quickstart-guide#set-up-a-client-outside-the-nim) installs: ```python import numpy as np def to_ego_frame(p_world, R_world): """p_world: [16, 3] positions; R_world: [16, 3, 3], oldest first. Accepts numpy arrays or nested Python lists. """ p_world = np.asarray(p_world, dtype=float) R_world = np.asarray(R_world, dtype=float) R_t_inv = R_world[-1].T xyz = (p_world - p_world[-1]) @ R_t_inv.T rot = R_t_inv @ R_world return xyz.tolist(), rot.tolist() ``` Predicted trajectories use the same frame and continue from the prediction time: 64 future poses at 0.1-second intervals. A caller-provided auto-labeling future must use the same convention. An all-zero history describes a vehicle that has been standing still; submitted with images of a moving scene, it can produce a plausible-looking path that reverses or stops. Use real history. ## gRPC Ordering, Local Paths and Defaults For `compressed_images`, label every frame with `camera` and `timestep`. When any frame carries a camera label, every frame must, and the server orders frames by those labels: by `camera_order` when it is set, otherwise by the canonical camera order, and by `timestep` from 0 (oldest) upward. `timestep` is the frame's position within its camera, not an absolute frame number: with `num_frames_per_camera` set to 2, label each camera's frames 0 and 1. Array order is then ignored. Duplicate, missing or extra camera/timestep pairs return `INVALID_ARGUMENT`. Unlabeled frames are used in array order and must already be camera-major; prefer labels. Set `mime_type` on every frame. A declared type must match the bytes, or the request returns `INVALID_ARGUMENT`; an empty value lets the server detect the format from the bytes. `local_paths` (`LocalPathInput`) reads files that already exist inside the server container. It is disabled unless the operator sets `NIM_ALPAMAYO_GRPC_LOCAL_PATH_ROOT`, and every path, including symlink targets, must resolve under that root. Two layouts are supported: - `scene_dir` alone: the directory must contain `egomotion.json` and `images/_t.` for each selected camera and frame, for example `images/front_wide_t0.jpg`. Extensions `.jpg`, `.jpeg`, `.png`, `.webp` and `.bmp` are recognized. The bundled `sample_data` directory uses this layout. With fewer than four frames per camera, the newest frames are read (`t2` and `t3` for two frames), so they end at the same time as the ego-motion history. - `image_paths`: an explicit camera-major list. Relative entries are resolved under `scene_dir` when it is set. Without a scene directory, also send `egomotion_json` and declare the layout. If a request sets both `local_paths.scene_dir` and `egomotion_json`, `egomotion_json` is used and the scene directory's `egomotion.json` is ignored. Proto3 does not transmit unset scalar fields. The server treats an unset or zero `num_traj_samples` as 1 and an unset or zero `num_frames_per_camera` as 4. `temperature`, `top_p`, `top_k`, `seed` and `nav_guidance_weight` are `optional` fields, so an explicit zero is honored: `temperature=0` requests greedy decoding and `seed=0` is a valid seed. Leave them unset to use the NIM defaults. An empty `model` selects the served model. ## Verify a New Input Pipeline 1. Rebuild the bundled scene with your own conversion code, starting from the raw files in `sample_data`, and compare the request with the one produced by `build_http_payload.py`. 2. Send both with `temperature=0` and `num_traj_samples=1` and compare the trajectories. The same inputs on the same deployment should give closely matching results. 3. For your own scenes, check that predicted paths continue in the direction of travel shown by the history and images. A path that reverses on a moving scene usually indicates a sign, frame or ordering error. A history that implies reversing faster than about 2 m/s returns a `warnings` entry; it usually means a frame-convention mistake (180-degree heading or a transposed world-to-ego rotation).