> This page is for version 1.0.0 (Latest) (default). > For other versions, use one of these documentation indexes: > - 1.0.0 (Latest) (default): https://docs.nvidia.com/nim/harmonizer/1.0.0/llms.txt > For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/nim/harmonizer/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nim/harmonizer/_mcp/server. # API Reference > Look up Harmonizer HTTP and gRPC endpoints, request and response schemas, examples, and error behavior. Use this reference to look up the HTTP and gRPC inference contracts. For a complete deployment walkthrough, start with the [Quickstart](/nim/harmonizer/quickstart-guide). ## HTTP Endpoints The release HTTP server exposes the following routes: | Category | Method and Route | Purpose | | ---------- | ---------------------- | ------------------------------------------------------- | | Inference | `POST /v1/infer` | Native Harmonizer request and response schema. | | Health | `GET /v1/health/live` | Process or server liveness. | | Health | `GET /v1/health/ready` | Service readiness for inference. | | Management | `GET /v1/metrics` | Prometheus metrics in text exposition format. | | Management | `GET /v1/license` | Packaged NIM license information and content. | | Management | `GET /v1/metadata` | Version, license, selected profile, and model metadata. | | Management | `GET /v1/version` | NIM release and API versions. | | Management | `GET /v1/manifest` | Active model manifest and repository override. | | Management | `GET /v1/models` | Served model in the standard NIM model-list shape. | The NIM also exposes a binary gRPC API on port `50051` that mirrors the `/v1/infer` payload as `harmonizer.v1.Harmonizer/Process`. ## API Examples Use the examples in this section to get started with the API. The examples use `localhost` because Docker publishes the port on the client host. For a remote NIM, replace it with the server host name or IP address. From inside Kubernetes, use the Service DNS name. `0.0.0.0` is a server listen address and should not be used as a client destination. ### Check Health Use the following command to check server readiness. **cURL Request** ```bash curl -X 'GET' 'http://localhost:8000/v1/health/ready' ``` **Response** ```json { "object": "health.response", "message": "ready", "status": "ready" } ``` For the HTTP entrypoint, model loading and startup warmup complete before the server begins accepting connections. During that phase a client normally sees a connection failure or a pending deployment startup probe, not an HTTP `503`. After the server is reachable, `200` with `"status": "ready"` means the backend is ready. A running HTTP server can return `503` if its internal backend readiness state later becomes false. For the gRPC entrypoint, the model also loads and warms before the gRPC server starts. The standard `grpc.health.v1.Health/Check` service changes to `SERVING` after server startup and to `NOT_SERVING` during shutdown or during the short transition between detecting an unrecoverable CUDA error and terminating the inference process. Its companion HTTP `live` and `ready` routes query that gRPC health service. Deployment probes should use `/v1/health/ready`. ### Process Images Use the process endpoint to enhance one or more images. The `images` list is ordered: with `context_length=4` the list is treated as a frame sequence and the temporal lane builds autoregressive context from previously generated outputs. With `context_length=0` each image is processed independently. Each returned image or tensor has the corresponding input's original dimensions. The runtime stretches the input to `1024x576` for inference and stretches the result back to the original dimensions. Both processing modes are available through either inference transport. A container activates one inference transport at startup, so the HTTP and gRPC tabs below are alternatives rather than simultaneous inference endpoints. See the [Quickstart](/nim/harmonizer/quickstart-guide) for the corresponding launch commands. #### /v1/infer Use `POST /v1/infer` for the native Harmonizer schema. HTTP image payloads are base64-encoded PNG or JPEG bytes. This example uses [an input frame from the NVIDIA Harmonizer model-publication repository](https://github.com/NVIDIA/harmonizer/blob/d1234ff7afb00b6084967dc31b18b526c9b9e7e0/examples/0.png). The API accepts encoded image data rather than an image URL, so the first command downloads the frame as `input.png`. The remaining commands insert its complete base64 payload into the request. **cURL Request** ```bash curl -L \ https://raw.githubusercontent.com/NVIDIA/harmonizer/d1234ff7afb00b6084967dc31b18b526c9b9e7e0/examples/0.png \ -o input.png IMAGE_B64="$(base64 < input.png | tr -d '\n')" curl -X 'POST' \ 'http://localhost:8000/v1/infer' \ -H 'Accept: application/json' \ -H 'Content-Type: application/json' \ --data-binary @- <", "mime_type": "image/png", "width": 1024, "height": 576 }, "tensor": null } ] } ``` #### gRPC Use gRPC for latency-sensitive inference. It sends image payloads as `bytes` and uses protobuf enums for parameter choices. The service is `harmonizer.v1.Harmonizer` and the method is `Process`. Follow the **gRPC Request** instructions in the [Quickstart](/nim/harmonizer/quickstart-guide) to generate the client modules and run a request against this service. Download the canonical schema as harmonizer.proto . Server reflection is disabled, so clients such as `grpcurl` require this schema explicitly. #### Complete gRPC Schema ```protobuf syntax = "proto3"; package harmonizer.v1; service Harmonizer { rpc Process(ProcessRequest) returns (ProcessResponse); } message ProcessRequest { optional string request_id = 1; repeated InputImage images = 2; ProcessParameters parameters = 3; } message InputImage { reserved 3, 4; reserved "mime_type", "metadata"; optional string id = 1; oneof payload { bytes data = 2; RawTensor tensor = 5; } } message ProcessParameters { reserved 3, 5, 6, 7; reserved "context_policy", "target_size", "include_metadata", "include_timings"; OutputFormat output_format = 1; uint32 context_length = 2; ReturnPolicy return_policy = 4; } enum OutputFormat { OUTPUT_FORMAT_UNSPECIFIED = 0; OUTPUT_FORMAT_PNG = 1; OUTPUT_FORMAT_JPEG = 2; OUTPUT_FORMAT_RAW_TENSOR = 3; } enum ReturnPolicy { RETURN_POLICY_UNSPECIFIED = 0; RETURN_POLICY_ALL = 1; RETURN_POLICY_EXCLUDE_CONTEXT = 2; RETURN_POLICY_LAST = 3; } message ProcessResponse { reserved 1, 2, 5, 6, 7, 8; reserved "id", "object", "model", "timings", "summary", "warnings"; optional string request_id = 3; repeated ProcessOutput outputs = 4; } message ProcessOutput { reserved 4; reserved "metadata"; optional string id = 1; oneof payload { EncodedImage image = 2; RawTensor tensor = 3; } } message EncodedImage { bytes data = 1; string mime_type = 2; uint32 width = 3; uint32 height = 4; } message RawTensor { reserved 2, 5, 6; reserved "dtype", "color_space", "encoding"; bytes data = 1; repeated int64 shape = 3; string layout = 4; } ``` The gRPC server accepts unary messages up to 256 MiB (268,435,456 bytes) so a full temporal clip can be sent in a single request. Set the client `grpc.max_receive_message_length` and `grpc.max_send_message_length` channel options accordingly. ## Transport Parameter Values HTTP represents parameter choices as strings. gRPC represents the same choices as protobuf enums. | Parameter | HTTP Value | gRPC Value | | --------------- | ----------------- | ------------------------------- | | `output_format` | `png` | `OUTPUT_FORMAT_PNG` | | `output_format` | `jpeg` | `OUTPUT_FORMAT_JPEG` | | `output_format` | `raw_tensor` | `OUTPUT_FORMAT_RAW_TENSOR` | | `return_policy` | `all` | `RETURN_POLICY_ALL` | | `return_policy` | `exclude_context` | `RETURN_POLICY_EXCLUDE_CONTEXT` | | `return_policy` | `last` | `RETURN_POLICY_LAST` | ## Request Fields The following table describes the request fields: | Field | Description | | --------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `request_id` | Optional client-supplied request identifier echoed back in the response. | | `images` | Ordered list of input images. Each image carries exactly one of `data` (base64 PNG/JPEG over HTTP, `bytes` over gRPC) or `tensor` (raw `uint8` RGB tensor). Encoded formats are detected from the payload. | | `parameters` | Optional. If omitted, the NIM uses `output_format=png`, `context_length=0`, and `return_policy=all`. | | `parameters.output_format` | Optional. `png` (default), `jpeg`, or `raw_tensor`. | | `parameters.context_length` | Optional. `0` (default) for the independent image lane or `4` for the rolling temporal lane. This NIM release supports only these two values. | | `parameters.return_policy` | Optional. `all` (default), `exclude_context`, or `last`. | ## Raw Tensor Response For HTTP responses, both `image` and `tensor` keys are serialized and exactly one is non-null. gRPC represents the alternatives as a protobuf `oneof`. With `output_format=raw_tensor`, HTTP base64-encodes `tensor.data` and gRPC returns it as `bytes`. The decoded payload contains tightly packed RGB `uint8` values in the `[0, 255]` range. Output layout is always `HWC` and output shape is `[height, width, 3]`. The following HTTP response is for a `1024x576` input: ```json { "request_id": "demo-0001", "outputs": [ { "id": "frame-000000", "image": null, "tensor": { "data": "", "shape": [576, 1024, 3], "layout": "HWC" } } ] } ``` ## Response Fields The following table describes the response fields: | Field | Description | | ------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `request_id` | Client-supplied request identifier, when provided. | | `outputs` | Processed outputs in input order, subject to `return_policy`. Each output carries its optional input `id` and exactly one non-null `image` or `tensor` payload. Its dimensions match the corresponding input's original dimensions. | ## Model and NIM Information Use the standard NIM management endpoints rather than inference responses for service and model information. `/v1/version` reports the release and API versions; `/v1/metadata` combines version, license, model, and selected-profile information; `/v1/models` identifies the served model; and `/v1/manifest` returns the active model manifest. `/v1/license` returns the packaged license, and `/v1/metrics` returns Prometheus text rather than JSON. ## Error Handling The HTTP API returns standard status codes to indicate success or failure: * **200 OK**: Request successful * **400 Bad Request**: nimlib request-body validation or invalid input parameters (for example an unsupported `context_length` or an invalid `return_policy` for the request shape) * **503 Service Unavailable**: A running backend becomes unavailable * **500 Internal Server Error**: Server-side error Every HTTP error uses the `{"error": ...}` envelope shown below. The nested `error` object always contains the required `message`, `type`, and `code` fields; the OpenAPI schema defines their types. A missing model artifact normally fails model loading before the HTTP or gRPC server starts, so clients see startup failure rather than an inference response. The current nimlib configuration includes field-specific validation details. For example, setting both image payload fields returns: ```json { "error": { "message": "images -> 0: Value error, exactly one of data or tensor must be set", "type": "BadRequestError", "code": 400 } } ``` The gRPC API uses standard gRPC status codes: * `OK` (code `0`): The RPC completed successfully * `INVALID_ARGUMENT`: Invalid input parameters or malformed payload * `UNAVAILABLE`: Model backend is not loaded * `UNKNOWN`: Unhandled server-side error ## OpenAPI Specification The HTTP OpenAPI specification is generated from the release nimlib interface and packaged at `/opt/nim/api_spec.yaml` inside the container. The interactive endpoint reference is available in the **API Reference** tab.