API Reference

Use this compact reference for runtime routing, the shared Generator request envelope, media inputs, and the common Generator response. Task-specific rules and examples live with each task workflow.

Check /openapi.json on the running Generator or Reasoner for routes supplied by the active runtime and selected image.

Runtime and Primary Endpoints

One selected profile starts one backend. A Generator profile does not serve Reasoner completion APIs, and a Reasoner profile does not serve /v1/infer.

Runtime	API	Guide
Generator	POST /v1/infer for T2I, T2V, I2V, and V2V	Generation
Generator	POST /v1/infer with action_params	Action
Generator	POST /v1/infer with transfer	Transfer
Reasoner	POST /v1/chat/completions	Reasoning
Reasoner	POST /v1/responses and optional state routes	Reasoning
Reasoner (streaming enabled)	WebSocket /v1/streaming/ws and REST /v1/streaming/*	Streaming Endpoints

Reasoner features are in Reasoning. Check the active model_type and inference_endpoint in /v1/metadata. Other framework endpoints are in Inspect the Running Service.

Generator Endpoint

The Generator serves POST /v1/infer. It accepts one synchronous JSON object and rejects unknown fields. Every request must set model_mode to identify the task.

NIM_MODEL_VARIANT chooses the checkpoint at container start, and model_mode chooses the task per request.

model_mode	Task	input_reference
text2image	T2I	Forbidden
text2video	T2V	Forbidden
image2video	I2V	Required image
video2video	V2V	Required video
video2video with transfer	Transfer	Optional for precomputed controls; required for derived edge/blur
forward_dynamics	Forward dynamics	Required image
policy	Policy	Required image
inverse_dynamics	Inverse dynamics	Required video

The task guides define the remaining required inputs and invalid combinations.

Common Generator Request Fields

Defaults are for video / text-to-image (T2I). Transfer and Action defaults are on their pages.

Field	Type	Video / T2I Default	Applies To
model_mode	required enum	None	Selects one of the seven modes above
prompt	string or null; maximum 20,000 characters	None	Required and non-empty for T2I/T2V; task-dependent otherwise
negative_prompt	string or null; maximum 20,000 characters	Built-in / empty	Tasks and models that permit negative prompting
input_reference	encoded image/video string or null	None	I2V, V2V, Action, and derived Transfer; media type follows model_mode
seed	non-negative integer or null	Service-generated	Reproducibility
guidance_scale	finite number in [1.0,7.0]	6.0 / 4.0	Sampling when not owned by a specialist model
num_inference_steps	integer in [1,100]	35 / 50	Sampling when not owned by a specialist model
flow_shift	finite number greater than 0	10.0 / 3.0	Sampling when not owned by a specialist model
resolution	enum	720 / 720_1_1	Generation and Transfer; refer to Resolution Keys
num_frames	integer	189 / 1	Generation and Transfer; Action derives it and rejects the field
fps	finite number in [1.0,60.0]	24.0	Video tasks; retained but not encoded for T2I
condition_frame_indexes_vision	integer array or null	[0, 1]	V2V only
condition_video_keep	first, last, or null	first	V2V only
action_params	object	None	Action modes only; Action
transfer	object	None	video2video only; Transfer

Empty or whitespace-only media strings are treated as absent.

Strict JSON Types

Integer and finite-number fields use strict JSON types. For example, "35", 35.0, and true are not accepted spellings of integer num_inference_steps=35. Unknown nested fields are rejected too.

Media Inputs

Generator image and video inputs (input_reference and Transfer control videos) can be supplied in the following ways:

• Raw base64

• A MIME-aware data URL such as data:image/jpeg;base64,... or data:video/mp4;base64,...

• An HTTP(S) URL when NIM_ALLOW_URL_INPUT is enabled (the default)

A data URL must use ;base64 and an image/* or video/* MIME type matching the field. An HTTP(S) URL is fetched with a 30-second timeout and must respond with a Content-Type of image/* or video/*. Generic application/octet-stream is accepted for video only when the URL or Content-Disposition names an .mp4 file. Prefer data URLs. Remote inputs need container network access and can time out or change.

Input videos must use an MP4/MOV, Matroska/WebM, FLV, or AVI container with H.264, H.265/HEVC, MPEG-4 Part 2, VP8, VP9, or AV1. Other codecs return HTTP 422. Decode paths and image formats are in the Support Matrix.

Generation inputs have these size caps:

• Image-to-video (I2V) images: 20,000,000 encoded characters

• Video-to-video (V2V) decoded video: 75 MB

Transfer limits are in Transfer.

Generator Response

A successful Generator response contains an image, a video, or—on a compatible specialist policy profile—an action without visual media. T2I returns:

{
  "b64_image": "<RAW_BASE64_JPEG>"
}

Video modes return:

{
  "b64_video": "<RAW_BASE64_MP4>",
  "action": null
}

Media fields are raw base64 (JPEG for T2I, VP9-in-MP4 for video), not data or file URLs. Inactive fields are omitted or null. Only general Policy and inverse dynamics return video plus an action object.

A specialist action-only policy can return the following response. The data array is shortened to one row here. A complete response has 32 rows of eight values, matching shape: [32, 8]:

{
  "action": {
    "data": [[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]],
    "shape": [32, 8],
    "dtype": "float32",
    "raw_action_dim": 8,
    "action_mode": "policy",
    "domain_id": 8
  }
}

In that case both b64_image and b64_video are absent or null. Clients must branch on the fields actually present rather than assuming every non-T2I request has b64_video. Refer to Edge-/Nano-DROID Policy.

Errors and Live Schema

A request for the other runtime’s primary route returns HTTP 404 with the common NIM error envelope naming the active runtime and its endpoint. Refer to Errors and Troubleshooting. Do not automate on exact error-message text.

Save the active runtime’s OpenAPI document:

curl -fsS http://localhost:8000/openapi.json -o openapi.json
python3 -m json.tool openapi.json >/dev/null

Treat the running NIM’s schema as authoritative for available routes and constraints in the deployed image.