Is this page helpful?

API Usage Examples (gRPC) for NVIDIA NIM for Nemotron Parse#

Use the examples in this documentation to help you get started using the API for NVIDIA NIM for Nemotron Parse. For the full API reference, refer to API Reference (gRPC) for NVIDIA NIM for Nemotron Parse.

gRPC Support#

The Nemotron Parse supports the Open Inference Protocol (KServe V2 Protocol). You can make gRPC inference requests by using the KServe V2 protocol buffers with a gRPC client.

Launch the Nemotron Parse NIM#

Follow the Get Started guide. Add -e NIM_SERVER_GRPC_BIND_ADDR=0.0.0.0:8001 to enable the gRPC listener and -p 8001:8001 to publish the port. The listener is disabled when its bind address is unset.

Make Inference Calls#

Install Python dependencies and generate the KServe V2 gRPC Python stubs.

python3 -m pip install grpcio grpcio-tools numpy
curl -LO https://raw.githubusercontent.com/kserve/open-inference-protocol/main/specification/protocol/open_inference_grpc.proto
python3 -m grpc_tools.protoc -I. --python_out=. --grpc_python_out=. open_inference_grpc.proto

Send one page image and decode the response tensors:

from pathlib import Path
import struct

import grpc
import open_inference_grpc_pb2 as oip
import open_inference_grpc_pb2_grpc as oip_grpc


def bytes_input(name, value):
    return oip.ModelInferRequest.InferInputTensor(
        name=name,
        datatype="BYTES",
        shape=[1],
        contents=oip.InferTensorContents(bytes_contents=[value]),
    )


with grpc.insecure_channel("localhost:8001") as channel:
    client = oip_grpc.GRPCInferenceServiceStub(channel)
    response = client.ModelInfer(oip.ModelInferRequest(
        model_name="nvidia_nemotron_3_5_parse",
        inputs=[
            bytes_input("document", Path("page.png").read_bytes()),
            bytes_input("media_type", b"image/png"),
        ],
    ))

for output, raw in zip(response.outputs, response.raw_output_contents):
    if output.datatype == "BYTES":
        length = struct.unpack_from("<I", raw)[0]
        value = raw[4:4 + length].decode("utf-8")
    else:
        value = struct.unpack("<I", raw)[0]
    print(output.name, value)