Is this page helpful?

Function Creation

This page describes the steps to create a function within Cloud Functions.

Attention

Please ensure before function creation, you’ve installed and configured the NGC CLI for working with the NGC Private Registry.

Functions can be created in one of three ways, listed below, and also visible in the Cloud Functions UI.

1. Custom Container

• Enables any container-based workload as long as the container exposes an inference endpoint and a health check.

• Option to leverage any server, ex. PyTriton, FastAPI, Triton.

• See Container-Based Function Creation.

2. Helm Chart

• Enables orchestration across multiple containers. For complex use cases where a single container isn’t flexible enough.

• Requires one “mini-service” container defined as the inference entry point for the function.

• Does not support partial response reporting, gRPC or HTTP streaming-based invocation.

• See Helm-Based Function Creation.

Working with NGC Private Registry

Function creation requires your model, container, helm chart and/or static resources to be hosted within NGC Private Registry as a prerequisite. Follow the steps below to optimally configure the NGC CLI to work with NGC Private Registry and Cloud Functions.

Warning

NGC Private Registry has size constraints on layers, images, models and resources.

Ensure that your uploaded resources conform to these constraints.

Generate an NGC Personal API Key

Do this by navigating to the Personal Keys Page. For more details see Generate an NGC Personal API Key.

Note

It’s recommended that the API Key that you generate includes both Cloud Functions and Private Registry scopes to enable ideal Cloud Functions workflows.

Download & Configure the NGC CLI

1. Navigate to the NGC CLI Installer Page to download the CLI and follow the installation instructions for your platform.

2. Find your NGC organization name within the NGC Organization Profile Page. This is not the Display Name. For example: qdrlnbkss123.

3. Run ngc config set and input the Personal API Key generated in the previous step, along with your organization name. If prompted, default to no-team and no-ace.

> ngc config set
Enter API key [****bi9Z]. Choices: [<VALID_APIKEY>, 'no-apikey']: <api key>
Enter CLI output format type [json]. Choices: ['ascii', 'csv', 'json']: json
Enter org [ax3ysqem02xw]. Choices: ['$ORG_NAME']: <org name>
Enter team [no-team]. Choices: ['no-team']:
Enter ace [no-ace]. Choices: ['no-ace']:

Authenticate with NGC Docker Registry

1. Run docker login nvcr.io and input the following, note $oauthtoken is the actual string to input, and <api key> is the Personal API key generated in the first step.

> docker login nvcr.io
Username: $oauthtoken
Password: $API_KEY

(Optional) Push a Container to the NGC Private Registry

You should now be able to push a container to the NGC Private Registry. Optionally, validate this by pushing an example container from the samples repository:

1. First clone and build the docker image.

> git clone https://github.com/NVIDIA/nv-cloud-function-helpers.git
> cd nv-cloud-function-helpers/examples/function_samples/fastapi_echo_sample
> docker build . -t fastapi_echo_sample

2. Now tag and push the docker image to the NGC Private Registry.

> docker tag fastapi_echo_sample:latest nvcr.io/$ORG_NAME/fastapi_echo_sample:latest
> docker push nvcr.io/$ORG_NAME/fastapi_echo_sample:latest

Warning

Note that any additional slashes in the path when tagging and pushing to nvcr.io will be detected by Private Registry as specifying a team. This is most likely not what you want.

3. Once this finishes, you’ll now be able to see the new container in the NGC Private Registry Containers Page and it will be available for use in function creation.

Best Practices with NGC Docker Registry and Cloud Functions

Container Versioning

• Ensure that any resources that you tag for deployment into production environments are not simply using “latest” and are following a standard version control convention.

  • During autoscaling, a function scaling any additional instances will pull the same specificed container image and version. If version is set to “latest”, and the “latest” container image is updated between instance scaling, this can lead to undefined behavior.

• Function versions created are immutable, this means that the container image and version cannot be updated for a function without creating a new version of the function.

Usage of NGC Teams

• For easier handling of authorization and accessibility, we recommend pushing your containers, helm charts, models and resources to the root of your NGC organization (i.e. “No Team”), not to a team within the organization.

• Note that any additional slashes in the path when tagging and pushing to nvcr.io will be detected as an NGC team.

Security

• Do not run containers as root user: Running containers as root is not supported in Cloud Functions. Always specify a non-root user in your Dockerfile using the USER instruction.

• Use Kubernetes Secrets: For sensitive information like API keys, credentials, or tokens, use Secrets instead of environment variables. This provides better security and follows Kubernetes best practices for secret management.

Container-Based Function Creation

Container-based functions require building and pushing a Cloud Functions compatible Docker container image to the NGC Private Registry.

Attention

Before proceeding, ensure that you have the NGC CLI installed and configured with an API Key that has the required scopes for Cloud Functions and Private Registry.

See Working with NGC Private Registry for instructions.

Resources

• Example containers can be found in the examples repository.

• The repository also contains helper functions that are useful when authoring your container, including:

  • Helpers that parse Cloud Functions-specific parameters on invocation

  • Helpers that can be used to instrument your container with Cloud Functions compatible logs

• After container creation, but before proceeding to deployment, it is strongly recommended to validate your container’s configuration locally, see Deployment Validation.

• It’s always a best practice to emit logs from your inference container. See Observability Guide for how to add logs to your container. Cloud Functions also supports third-party logging and metrics emission from your container.

Attention

Please note that container functions should not run as root user, running as root is not formally supported on any Cloud Functions backend.

Container Endpoints

Any server can be implemented within the container, as long as it implements the following:

• For HTTP-based functions, a health check endpoint that returns a 200 HTTP Status Code on success.

• For gRPC-based functions, a standard gRPC health check. See these docs for more info also gRPC Health Checking.

• An inference endpoint (this endpoint will be called during function invocation)

These endpoints are expected to be served on the same port, defined as the inferencePort.

Warning

Cloud Functions reserves the following ports on your container for internal monitoring and metrics:

• Port 8080

• Port 8010

Cloud Functions also expects the following directories in the container to remain read-only for caching purposes:

• /config/ directory

• Nested directories created inside /config/

Composing a FastAPI Container

It’s possible to use any container with Cloud Functions as long as it implements a server with the above endpoints. The below is an example of a FastAPI-based container compatible with Cloud Functions. Clone the FastAPI echo example.

Create the “requirements.txt” File

requirements.txt

fastapi==0.110.0
uvicorn==0.29.0

Implement the Server

http_echo_server.py

import os
import time
import uvicorn
from pydantic import BaseModel
from fastapi import FastAPI, status
from fastapi.responses import StreamingResponse


app = FastAPI()

class HealthCheck(BaseModel):
    status: str = "OK"

# Implement the health check endpoint
@app.get("/health", tags=["healthcheck"], summary="Perform a Health Check", response_description="Return HTTP Status Code 200 (OK)", status_code=status.HTTP_200_OK, response_model=HealthCheck)
def get_health() -> HealthCheck:
    return HealthCheck(status="OK")

class Echo(BaseModel):
    message: str
    delay: float = 0.000001
    repeats: int = 1
    stream: bool = False


# Implement the inference endpoint
@app.post("/echo")
async def echo(echo: Echo):
    if echo.stream:
        def stream_text():
            for _ in range(echo.repeats):
                time.sleep(echo.delay)
                yield f"data: {echo.message}\n\n"
        return StreamingResponse(stream_text(), media_type="text/event-stream")
    else:
        time.sleep(echo.delay)
        return echo.message*echo.repeats

# Serve the endpoints on a port
if __name__ == "__main__":
    uvicorn.run(app, host="0.0.0.0", port=8000, workers=int(os.getenv('WORKER_COUNT', 500)))

Note in the example above, the function’s configuration during creation will be:

• Inference Protocol: HTTP

• Inference Endpoint: /echo

• Health Endpoint: /health

• Inference Port (also used for health check): 8000

Create the Dockerfile

Dockerfile

FROM python:3.10.13-bookworm

ENV WORKER_COUNT=10

WORKDIR /app

COPY requirements.txt ./

RUN python -m pip install --no-cache-dir -U pip && \
    python -m pip install --no-cache-dir -r requirements.txt

COPY http_echo_server.py /app/

CMD uvicorn http_echo_server:app --host=0.0.0.0 --workers=$WORKER_COUNT

Build the Container & Create the Function

See the Functions Quickstart for the remaining steps.

Composing a PyTriton Container

NVIDIA’s PyTriton is a Python native solution of Triton inference server. A minimum version of 0.3.0 is required.

Create the “requirements.txt” File

• This file should list the Python dependencies required for your model.

• Add nvidia-pytriton to your requirements.txt file.

Here is an example of a requirements.txt file:

requirements.txt

--extra-index-url https://pypi.ngc.nvidia.com
opencv-python-headless
pycocotools
matplotlib
torch==2.1.0
nvidia-pytriton==0.3.0
numpy

Create the “run.py” File

1. Your run.py file (or similar Python file) needs to define a PyTriton model.

2. This involves importing your model dependencies, creating a PyTritonServer class with an __init__ function, an _infer_fn function and a run function that serves the inference_function, defining the model name, the inputs and the outputs along with optional configuration.

Here is an example of a run.py file:

run.py

import numpy as np
from pytriton.model_config import ModelConfig, Tensor
from pytriton.triton import Triton, TritonConfig
import time
....
class PyTritonServer:
    """triton server for timed_sleeper"""

    def __init__(self):
        # basically need to accept image, mask(PIL Images), prompt, negative_prompt(str), seed(int)
        self.model_name = "timed_sleeper"

    def _infer_fn(self, requests):
        responses = []
        for req in requests:
            req_data = req.data
            sleep_duration = numpy_array_to_variable(req_data.get("sleep_duration"))
            # deal with header dict keys being lowerscale
            request_parameters_dict = uppercase_keys(req.parameters)
            time.sleep(sleep_duration)
            responses.append({"sleep_duration": np.array([sleep_duration])})

        return responses

    def run(self):
        """run triton server"""
        with Triton(
            config=TritonConfig(
                http_header_forward_pattern="NVCF-*",  # this is required
                http_port=8000,
                grpc_port=8001,
                metrics_port=8002,
            )
        ) as triton:
            triton.bind(
                model_name="timed_sleeper",
                infer_func=self._infer_fn,
                inputs=[
                    Tensor(name="sleep_duration", dtype=np.uint32, shape=(1,)),
                ],
                outputs=[Tensor(name="sleep_duration", dtype=np.uint32, shape=(1,))],
                config=ModelConfig(batching=False),
            )
            triton.serve()
if __name__ == "__main__":
    server = PyTritonServer()
    server.run()

Create the “Dockerfile”

1. Create a file named Dockerfile in your model directory.

2. It’s strongly recommended to use NVIDIA-optimized containers like CUDA, Pytorch or TensorRT as your base container. They can be downloaded from the NGC Catalog.

3. Make sure to install your Python requirements in your Dockerfile.

4. Copy in your model source code, and model weights unless you plan to host them in NGC Private Registry.

Here is an example of a Dockerfile:

Dockerfile

FROM nvcr.io/nvidia/cuda:12.1.1-devel-ubuntu22.04
RUN apt-get update && apt-get install -y \
    git \
    python3 \
    python3-pip \
    python-is-python3 \
    libsm6 \
    libxext6 \
    libxrender-dev \
    curl \
    && rm -rf /var/lib/apt/lists/*
WORKDIR /workspace/

# Install requirements file
COPY requirements.txt requirements.txt
RUN pip install --no-cache-dir --upgrade pip
RUN pip install --no-cache-dir -r requirements.txt
ENV DEBIAN_FRONTEND=noninteractive

# Copy model source code and weights
COPY model_weights /models
COPY model_source .
COPY run.py .

# Set run command to start PyTriton to serve the model
CMD python3 run.py

Build the Docker Image

1. Open a terminal or command prompt.

2. Navigate to the my_model directory.

3. Run the following command to build the docker image:

docker build -t my_model_image .

Replace my_model_image with the desired name for your docker image.

Push the Docker Image

Before beginning, ensure that you have authenticated with the NGC Docker Registry.

1. Tag and push the docker image to the NGC Private Registry.

> docker tag my_model_image:latest nvcr.io/$ORG_NAME/my_model_image:latest
> docker push nvcr.io/$ORG_NAME/my_model_image:latest

Create the Function

1. Create the function via API by running the following curl with an $API_KEY and your $ORG_NAME. In this example, we defined the inference endpoint as 8000 and are using the default inference and health endpoint paths.

API_KEY=<your api key>
ORG_NAME=<your organization name>

 curl --location 'https://api.ngc.nvidia.com/v2/nvcf/functions' \
 --header 'Content-Type: application/json' \
 --header 'Accept: application/json' \
 --header "Authorization: Bearer $API_KEY" \
 --data '{
     "name": "my-model-function",
     "inferenceUrl": "/v2/models/my_model_image/infer",
     "inferencePort": 8000,
     "containerImage": "nvcr.io/'$ORG_NAME'/my_model_image:latest",
     "health": {
                 "protocol": "HTTP",
                 "uri": "/v2/health/ready",
                 "port": 8000,
                 "timeout": "PT10S",
                 "expectedStatusCode": 200
             }
 }'

Additional Examples

See more examples of containers that are Cloud Functions compatible in the function samples directory.

Creating Functions with NGC Models & Resources

When creating a function, models and resources can be mounted to the function instance. The models will be available under /config/models/{modelName} and /config/resources/{resourceName} where modelName and resourceName are specified as part of the API request.

Here is an example where a model and resource are added to a function creation API call, for an echo sample function:

curl -X 'POST' \
  'https://api.ngc.nvidia.com/v2/nvcf/functions' \
  -H "Authorization: Bearer $API_KEY" \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
     "name": "echo_function",
     "inferenceUrl": "/echo",
     "containerImage": "nvcr.io/$ORG_NAME/echo:latest",
     "apiBodyFormat": "CUSTOM",
     "models": [
     {
         "name": "simple_int8",
         "version": "1",
         "uri": "v2/org/'$ORG_NAME'/models/simple_int8/1/files"
     }
     ],
     "resources": [
     {
         "name": "simple_resource",
         "version": "1",
         "uri": "v2/org/'$ORG_NAME'/resources/simple_resource/1/files"
     }
     ]
 }'

Within the container, once the function instance is deployed, the model would be mounted at /config/models/simple_int8 and resource mounted at /config/resources/simple_int8

Creating gRPC-based Functions

Cloud Functions supports function invocation via gRPC. During function creation, specify that the function is a gRPC function by setting the “Inference Protocol”, or inferenceUrl field to /grpc.

Prerequisites

• The function container must implement a gRPC port, endpoint and health check. The health check is expected to be served by the gRPC inference port, there is no need to define a separate health endpoint path.

  • See gRPC health checking.

  • See an example container with a gRPC server that is Cloud Functions compatible.

gRPC Function Creation via UI

In the Function Creation Page, set the “Inference Protocol” to gRPC and port to whatever your gRPC server has implemented.

gRPC Function Creation via CLI

When creating the gRPC function, set the --inference-url argument to /grpc:

ngc cf function create --inference-port 8001 --container-image nvcr.io/$ORG_NAME/grpc_echo_sample:latest --name my-grpc-function --inference-url /grpc

gRPC Function Creation via API

When creating the gRPC function, set the inferenceURl field to /grpc:

curl --location 'https://api.ngc.nvidia.com/v2/nvcf/functions' \
--header 'Content-Type: application/json' \
--header 'Accept: application/json' \
--header "Authorization: Bearer $API_KEY" \
--data '{
    "name": "my-grpc-function",
    "inferenceUrl": "/grpc",
    "inferencePort": 8001,
    "containerImage": "nvcr.io/'$ORG_NAME'/grpc_echo_sample:latest"
}'

gRPC Function Invocation

See gRPC Invocation for details on how to authenticate and invoke your gRPC function.

Creating Low Latency Streaming (LLS AKA GameStreamSDK/WebRTC) Functions

Cloud Functions supports the ability to stream video, audio, and other data using WebRTC.

Here is a detailed diagram of the overall setup.

For complete examples of LLS streaming functions, see NVCF LLS Function Samples.

Currently LLS Streaming is only supported with GFN based instances. Either a single container or a helm chart can be used.

Building the Streaming Server Application

The streaming application needs to be packaged inside a container and should be leveraging the StreamSDK. The streaming application needs to follow the below guidelines: 1. Expose an HTTP server at port CONTROL_SERVER_PORT with following 2 endpoints:

1. Health endpoint: This endpoint should return 200 HTTP status code only when the streaming application container is ready to start streaming a session. If the streaming application container doesn’t want to serve any more streaming sessions of current container deployment this endpoint should return HTTP status code 500. .. code-block:: yaml

Request:

Endpoint:

GET /v1/streaming/ready

Responses:

Status code:

200 OK 500 Internal Server Error

2. STUN creds endpoint: This endpoint should accept the access details and credentials for STUN server and keep it cached in the memory of the streaming application. When the streaming requests comes, the streaming application can use these access details and credentials to communicate with STUN server and request for opening of ports for streaming. .. code-block:: yaml

Request:

Endpoint:

POST /v1/streaming/creds

Headers:

Content-Type: application/json

Request Body:

{

“stunIp”: “<string>”, “stunPort”: <int>, “username”: “<string>”, “password”: “<string>”

}

Responses:

Status code:

200 OK

2. Expose a server at port STREAMING_SERVER_PORT to accepting WebSocket connection

   1. An endpoint STREAMING_START_ENDPOINT should be exposed by this server

3. Post websocket connection establishment guidelines:

   1. When the browser client requests for opening port for specific protocols (e.g. WebRTC), the streaming application needs to request STUN server to open port. This port should be in the range of 47998 and 48020 which would be referred as STREAMING_PORT_BINDING_RANGE in this doc.

4. Containerization guidelines:

   1. The container should make sure that the CONTROL_SERVER_PORT, STREAMING_SERVER_PORT and STREAMING_PORT_BINDING_RANGE are exposed by the container and accessible from outside the container.

   2. If multiple sessions one after another needs to be supported with a fresh start of container, then exit the container after a streaming session ends.

Creating the LLS Streaming Function

When creating the function, we need to ensure functionType is set to STREAMING:

curl -X 'POST' \
    'https://api.ngc.nvidia.com/v2/nvcf/functions' \
    -H "Authorization: Bearer $API_KEY" \
    -H 'accept: application/json' \
    -H 'Content-Type: application/json' \
    -d '{
        "name": "'$STREAMING_FUNCTION_NAME'",
        "inferenceUrl": "/sign_in",
        "inferencePort": '$STREAMING_SERVER_PORT',
        "health": {
            "protocol": "HTTP",
            "uri": "/v1/streaming/ready",
            "port": '$CONTROL_SERVER_PORT',
            "timeout": "PT10S",
            "expectedStatusCode": 200
        },
        "containerImage": "'$STREAMING_CONTAINER_IMAGE'",
        "apiBodyFormat": "CUSTOM",
        "description": "'$STREAMING_FUNCTION_NAME'",
        "functionType": "STREAMING"
        }
    }'

Connecting to a streaming function with a client

Intermediary Proxy

An intermediary proxy service needs to be deployed in order to facilitate the connection to the streaming function.

The intermediate proxy serves to handle authentication and the headers that are required for NVCF, and also to align the connection behavior with NVCF that the browser can’t handle on its own, or the browser behavior is unpredictable.

Proxy Responsibilities

The intermediary proxy performs the following functionalities:

1. Authenticate the user token coming from the browser to the intermediary proxy

2. Authorize the user to have access to specific streaming function

3. Once the user is authenticated and authorized, modify the websocket connection coming in to append the required NVCF headers (NVCF_API_KEY and STREAMING_FUNCTION_ID)

4. Forward the websocket connection request to NVCF

Technical Implementation Guidance

nvcf-function-id Header

NVCF requires this header to be present to identify the function that needs to be reached. Browser does not have the mechanism to set any kind of headers in case of WebSocket connections other than Sec-Websocket-Protocol, so the intermediate proxy can serve to either add the nvcf-function-id header on its own, or to parse Sec-Websocket-Protocol if the browser added it there and get the function id from there.

See http-request add-header documentation in HAProxy.

Authentication

Originally, NVCF did not support client authentication tokens, so the role of intermediate proxy is to add the required server authentication here (e.g. http-request set-header Authorization "Bearer NVCF_BEARER_TOKEN").

Connection Keepalive

NVCF controls the session lifetime based on the TCP connection lifetime to the function and the type of disconnection that happens. The intermediate proxy helps to keep the connection with the browser alive.

Resume Support

NVCF returns the cookie with nvcf-request-id, but given the browser may reject the cookie since it is not from the same domain, the intermediate proxy helps to align this.

CORS Headers

For browsers to allow traffic with NVCF, the intermediate proxy needs to add the relevant CORS headers to responses from NVCF:

• access-control-expose-headers: *

• access-control-allow-headers: *

• access-control-allow-origin: *

For guidance on implementing this in HAProxy, see http-response set-header documentation.

Example HAProxy Dockerfile

Below is an all-in-one Dockerfile sample for setting up an HAProxy intermediary proxy with optional TLS/SSL support:

Note

This example focuses on NVCF integration. In production, you should also implement user authentication and authorization to control access to your streaming function.

For certain applications, TLS/SSL support is required. The proxy can be configured to use self-signed certificates for development and testing purposes by setting PROXY_SSL_INSECURE=true.

FROM haproxy:3.2

# Switch to root user for package installation
USER root

# Install necessary tools
RUN apt-get update && apt-get install -y \
    bash \
    gettext-base \
    lua5.3 \
    openssl \
    && rm -rf /var/lib/apt/lists/*

# Create directory for configuration and certificates
RUN mkdir -p /usr/local/etc/haproxy/lua \
    && mkdir -p /usr/local/etc/haproxy/certs \
    && chown -R haproxy:haproxy /usr/local/etc/haproxy

# Create certificate generation script
COPY <<EOF /usr/local/bin/generate-cert.sh
#!/bin/bash
cd /usr/local/etc/haproxy/certs
openssl req -x509 -newkey rsa:2048 -keyout server.key -out server.crt -days 365 -nodes -subj "/CN=localhost" -quiet
# Combine certificate and key into a single file for HAProxy
cat server.crt server.key > server.pem
chown haproxy:haproxy server.key server.crt server.pem
chmod 600 server.key server.pem
chmod 644 server.crt
EOF

RUN chmod +x /usr/local/bin/generate-cert.sh

# Create the HAProxy configuration template file
COPY --chown=haproxy:haproxy <<EOF /usr/local/etc/haproxy/haproxy.cfg.template
global
        log stdout    local0 info
        stats timeout 30s
        user haproxy

        # Default SSL material locations
        ca-base /etc/ssl/certs
        crt-base /etc/ssl/private

        # SSL server verification enabled for security
        ssl-server-verify required

        # See: https://ssl-config.mozilla.org/#server=haproxy&server-version=3.2&config=intermediate
        ssl-default-bind-ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384
        ssl-default-bind-ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256
        ssl-default-bind-options ssl-min-ver TLSv1.2 no-tls-tickets

defaults
        log     global
        option  httplog
        option  dontlognull
        option  logasap
        timeout connect 5000
        timeout client  50000
        timeout server  50000

frontend test_frontend
        log  global
        bind *:\${PROXY_PORT} \${PROXY_SSL_BIND_OPTIONS}
        mode http
        timeout client       7s
        timeout http-request 30m
        use_backend webrtc_backend

backend webrtc_backend
        log  global
        mode http
        timeout connect 4s
        timeout server  7s
        http-request set-header Host \${NVCF_SERVER}
        http-request set-header Authorization "Bearer \${NGC_CLI_API_KEY}"
        http-request set-header Function-ID \${STREAMING_FUNCTION_ID}
        server s1 \${NVCF_SERVER}:443 ssl ca-file /etc/ssl/certs/ca-certificates.crt verify required
EOF

# Create the entrypoint script
COPY <<EOF /entrypoint.sh
#!/bin/bash

# Check required environment variables
if [ -z "\${NGC_CLI_API_KEY:+x}" ]; then
    echo "NGC_CLI_API_KEY must be set"
    exit 1
fi

if [ -z "\${STREAMING_FUNCTION_ID:+x}" ]; then
    echo "STREAMING_FUNCTION_ID must be set"
    exit 1
fi

# Use default NVCF_SERVER if not set
if [ -z "\${NVCF_SERVER:+x}" ]; then
    export NVCF_SERVER=grpc.nvcf.nvidia.com
    echo "NVCF_SERVER not set, using default: \${NVCF_SERVER}"
fi

# Use default PROXY_PORT if not set
if [ -z "\${PROXY_PORT:+x}" ]; then
    export PROXY_PORT=49100
    echo "PROXY_PORT not set, using default: \${PROXY_PORT}"
fi

# Use default PROXY_SSL_INSECURE if not set
if [ -z "\${PROXY_SSL_INSECURE:+x}" ]; then
    export PROXY_SSL_INSECURE=false
    echo "PROXY_SSL_INSECURE not set, using default: \${PROXY_SSL_INSECURE}"
fi

echo "Launching intermediate proxy:"
echo "  API Key: \${NGC_CLI_API_KEY:0:6}**********\${NGC_CLI_API_KEY: -3}"
echo "  Function ID: \${STREAMING_FUNCTION_ID}"
echo "  Version ID: \${STREAMING_FUNCTION_VERSION_ID}"
echo "  NVCF Server: \${NVCF_SERVER}"
echo "  Proxy Port: \${PROXY_PORT}"
echo "  Proxy SSL (Insecure): \${PROXY_SSL_INSECURE}"

# Generate self-signed certificate if SSL is enabled
if [ "\${PROXY_SSL_INSECURE}" = "true" ]; then
    /usr/local/bin/generate-cert.sh
    export PROXY_SSL_BIND_OPTIONS="ssl crt /usr/local/etc/haproxy/certs/server.pem"
    echo "SSL enabled - self-signed certificate generated"
else
    export PROXY_SSL_BIND_OPTIONS=""
    echo "SSL disabled - running in HTTP mode"
fi

# Process the template and create the final config
envsubst < /usr/local/etc/haproxy/haproxy.cfg.template > /usr/local/etc/haproxy/haproxy.cfg

# Function to handle signals and forward them to HAProxy
handle_signal() {
    echo "Received signal, shutting down HAProxy..."
    if [ -n "\$HAPROXY_PID" ]; then
        kill -TERM "\$HAPROXY_PID" 2>/dev/null
        wait "\$HAPROXY_PID"
    fi
    exit 0
}

# Set up signal handlers
trap handle_signal SIGTERM SIGINT

# Start HAProxy in background and capture PID
echo "Starting HAProxy..."
haproxy -f /usr/local/etc/haproxy/haproxy.cfg &
HAPROXY_PID=\$!

# Wait for HAProxy process
wait "\$HAPROXY_PID"
EOF

RUN chmod +x /entrypoint.sh

# Switch back to haproxy user
USER haproxy

# Set the entrypoint
ENTRYPOINT ["/entrypoint.sh"]

Environment Variables

The following environment variables control proxy behavior:

Variable	Required	Default	Description
NGC_CLI_API_KEY	Yes		Your NGC Personal API Key
STREAMING_FUNCTION_ID	Yes		Your NVCF streaming function ID
STREAMING_FUNCTION_ VERSION_ID	No		Specific version of your function (optional)
NVCF_SERVER	No	grpc.nvcf. nvidia.com	NVCF server endpoint
PROXY_PORT	No	49100	Port for the proxy to listen on
PROXY_SSL_INSECURE	No	false	Enable SSL with self-signed certificate (set to “true” to enable)

Usage Examples

1. HTTP Mode (Default)

Standard configuration without SSL:

export STREAMING_FUNCTION_ID=your-function-id
export NGC_CLI_API_KEY=your-ngc-cli-api-key
export PROXY_PORT=49100

docker build -t nvcf-haproxy-proxy .

docker run --rm -it \
    -p 127.0.0.1:${PROXY_PORT}:${PROXY_PORT}/tcp \
    -e PROXY_PORT="$PROXY_PORT" \
    -e NGC_CLI_API_KEY="$NGC_CLI_API_KEY" \
    -e STREAMING_FUNCTION_ID="$STREAMING_FUNCTION_ID" \
    nvcf-haproxy-proxy

2. HTTPS Mode with Self-Signed Certificate

Configuration with SSL enabled using a self-signed certificate:

export STREAMING_FUNCTION_ID=your-function-id
export NGC_CLI_API_KEY=your-ngc-cli-api-key
export PROXY_SSL_INSECURE=true
export PROXY_PORT=48322

docker build -t nvcf-haproxy-proxy .

docker run --rm -it \
    -p 127.0.0.1:${PROXY_PORT}:${PROXY_PORT}/tcp \
    -e PROXY_PORT=${PROXY_PORT} \
    -e PROXY_SSL_INSECURE=${PROXY_SSL_INSECURE} \
    -e NGC_CLI_API_KEY="$NGC_CLI_API_KEY" \
    -e STREAMING_FUNCTION_ID="$STREAMING_FUNCTION_ID" \
    nvcf-haproxy-proxy

Note

Since this configuration uses self-signed certificates for development and testing, you will need to configure your client to accept untrusted certificates. In production environments, you should use proper CA-signed certificates.

Web Browser Client

Using the proxy, a browser client can be used to connect to the stream. The browser client needs to be developed by the customer leveraging the raganrok dev branch 0.0.1503 version. Please ensure that the flags are set: .. code-block:: javascript

const configData: RagnarokConfigData = {

overrideData: “disableworkerws=true”

}

ConfigureRagnarokSettings(configData);

Available Container Variables

The following is a reference of available variables via the headers of the invocation message (auto-populated by Cloud Functions), accessible within the container.

For examples of how to extract and use some of these variables, see NVCF Container Helper Functions.

Name	Description
NVCF-REQID	Request ID for this request.
NVCF-SUB	Message subject.
NVCF-NCAID	Function’s organization’s NCA ID.
NVCF-FUNCTION-NAME	Function name.
NVCF-FUNCTION-ID	Function ID.
NVCF-FUNCTION-VERSION-ID	Function version ID.
NVCF-LARGE-OUTPUT-DIR	Large output directory path.
NVCF-MAX-RESPONSE-SIZE-BYTES	Max response size in bytes for the function.
NVCF-NSPECTID	NVIDIA reserved variable.
NVCF-BACKEND	Backend or “Cluster Group” the function is deployed on.
NVCF-INSTANCETYPE	Instance type the function is deployed on.
NVCF-REGION	Region or zone the function is deployed in.
NVCF-ENV	Spot environment if deployed on spot instances.

Environment Variables

The following environment variables are automatically injected into your function containers when they are deployed and can be accessed using standard environment variable access methods in your application code:

Name	Description
NVCF_BACKEND	Backend or “Cluster Group” the function is deployed on.
NVCF_ENV	Spot environment if deployed on spot instances.
NVCF_FUNCTION_ID	Function ID.
NVCF_FUNCTION_NAME	Function name.
NVCF_FUNCTION_VERSION_ID	Function version ID.
NVCF_INSTANCETYPE	Instance type the function is deployed on.
NVCF_NCA_ID	Function’s organization’s NCA ID.
NVCF_REGION	Region or zone the function is deployed in.

Note

All environment variables with the NVCF_* prefix are reserved and should not be overridden in your application code or function configuration.

Helm-Based Function Creation

Cloud functions support helm-based functions for orchestration across multiple containers.

Prerequisites

Warning

Ensure that your helm charts version does not contain - For example v1 is ok but v1-test will cause issues.

1. The helm chart must have a “mini-service” container defined, which will be used as the inference entry point.

2. The name of this service in your helm chart should be supplied by setting helmChartServiceName during the function definition. This allows Cloud Functions to communicate and make inference requests to the “mini-service” endpoint.

Attention

The servicePort defined within the helm chart should be used as the inferencePort supplied during function creation. Otherwise, Cloud Functions will not be able to reach the “mini-service”.

3. Ensure you have the NGC CLI configured and have pushed your helm chart to NGC Private Registry. Refer to Managing Helm Charts Using the NGC CLI.

Secret Management

For pulling containers defined as part of the helm chart from NGC Private Registry, define ngcImagePullSecretName in values.yaml. This value will be used in the deployment spec as spec.imagePullSecrets.name for the pods.

For nested Helm charts, define global.ngcImagePullSecretName in values.yaml, which will be referenced in the deployment spec under spec.imagePullSecrets.name for the pods.

Warning

Containers defined in the helm chart should be in the same NGC Organization and Team that the helm chart itself is being pulled from.

Create a Helm-based Function

1. Ensure your helm chart is uploaded to NGC Private Registry and adheres to the Prerequisites listed above.

2. Create the function:

   • Include the following additional parameters in the function definition

     • helmChart

     • helmChartServiceName

   • The helmChart property should be set to the URL hosted by the NGC Model Registry pointing to the helm chart that will deploy the “mini-service”. Please note, that this helm chart URL should be accessible to the NGC org in which the function will eventually be deployed. The helm chart URL should follow the format: https://helm.ngc.nvidia.com/$ORG_ID/$TEAM_NAME/charts/$NAME-X.Y.Z.tgz for example, https://helm.ngc.nvidia.com/abc123/teamA/charts/nginx-0.1.5.tgz would be a valid chart URL but https://helm.ngc.nvidia.com/abc123/teamA/charts/nginx-0.1.5-hello.tgz would not.

   • The helmChartServiceName is used for checking if the “mini-service” is ready for inference and is also scraped for function metrics. At this time, templatized service names are not supported. This must match the service name of your “mini-service” with the exposed entry point port.

   • Important: The Helm chart name should not contain underscores or other special symbols, as that may cause issues during deployment.

Example Creation via API

Please see our sample helm chart used in this example for reference.

Below is an example function creation API call creating a helm-based function:

curl -X 'POST' \
    'https://api.ngc.nvidia.com/v2/nvcf/functions' \
    -H "Authorization: Bearer $API_KEY" \
    -H 'accept: application/json' \
    -H 'Content-Type: application/json' \
    -d '{
    "name": "function_name",
    "inferenceUrl": "v2/models/model_name/versions/model_version/infer",
    "inferencePort": 8001,
    "helmChart": "https://helm.ngc.nvidia.com/'$ORG_ID'/'$TEAM_NAME'/charts/inference-test-1.0.tgz",
    "helmChartServiceName": "service_name",
    "apiBodyFormat": "CUSTOM"
}'

Note

For gRPC-based functions, set "inferenceURL" : "/gRPC". This signals to Cloud Functions that the function is using gRPC protocol and is not expected to have a /gRPC endpoint exposed for inferencing requests.

3. Proceed with function deployment and invocation normally.

Multi-node helm deployment To create a multi-node helm deployment, you need to use the following format for the instanceType: <CSP>.GPU.<GPU_NAME>_<number of gpus per node>x[.x<number of nodes>]. For example, DGXC.GPU.L40S_1x is a single L40S instance while ON-PREM.GPU.B200_8x.x2 is two full nodes of 8-way B200.

A sample helm chart for a multi-node deployment can be found in the multi-node helm example.

Limitations

When using Helm Charts to deploy a function, the following limitations need to be taken into consideration.

1. Asset caching

NGC model and resource caching - Automatic mounting of NGC Models and Resources for your container is not supported (coming soon). - For any downloads (such as of assets or models) occurring within your function’s containers, download size is limited by the disk space on the VM - for GFN this is 100GB approximately, and for other clusters this limit will vary.

2. Inference

Progress/partial response reporting is not supported, including any additional artifacts generated during inferencing. Consider opting for HTTP streaming or gRPC bidirectional support.

3. Security Constraints

Helm charts must conform to certain security standards to be deployable as a function. This means that certain helm and Kubernetes features are restricted in NVCF backends. NVCF will process your helm chart on function creation, then later on deployment with your Helm values and other deployment metadata, to ensure standards are enforced.

NVCF may automatically modify certain objects in your chart so they conform to these standards; it will only do so if modification will not break your chart when it is installed in the targeted backend. Possible areas amenable to modification will be noted in the restrictions section below. Any standard that cannot be enforced by modification will result in error(s) during function creation.

Restrictions

• Supported k8s artifacts under Helm Chart Namespace are listed below; others will be rejected:

  • ConfigMaps

  • Secrets

  • Services - Only type: ClusterIP or none

  • Deployments

  • ReplicaSets

  • StatefulSets

  • Jobs

  • CronJobs

  • Pods

  • ServiceAccounts (GFN backend only)

  • Roles (GFN backend only)

  • Rolebindings (GFN backend only)

  • PersistentVolumeClaims (GFN backend only)

• The only allowed Pod or Pod template volume types are:

  • configMap

  • secret

  • projected.sources.* of any of the above

  • persistentVolumeClaim (GFN backend only)

  • emptyDir

• No chart hooks are allowed; if specified in the chart, they will not be executed.

Note

CustomResourceDefinitions in helm charts will be skipped on installation. There is no need to modify your chart to remove them from helm template output for NVCF.

Helm charts _should_ conform to these additional security standards. While not enforced now, they will be at a later date.

• All containers have resource limits for at least cpu and memory (and nvidia.com/gpu, ephemeral-storage if required for certain containers).

• All Pod’s and resources that define a Pod template conform to the Kubernetes Pod Security Standards Baseline and Restricted policies.

• Pod and container securityContext’s conform to these parameters:

• automountServiceAccountToken must be unset or set to false

• runAsNonRoot must be explicitly set to true

• hostIPC, hostPID, and hostNetwork must be unset or set to false

• No privilege escalation, root capabilities, or non-defalt Seccomp, AppArmor, or SELinux profiles are allowed. See the Baseline and Restricted Pod security standards for fields that cannot be explicitly set.

Helm Chart Overrides

To override keys in your helm chart values.yml, you can provide the configuration parameter and supply corresponding key-value pairs in JSON format which you would like to be overridden when the function is deployed.

Example helm chart override

curl -X 'POST' \
 'https://api.ngc.nvidia.com/v2/nvcf/deployments/functions/fe6e6589-12bb-423a-9bf6-8b9d028b8bf4/versions/fe6e6589-12bb-423a-9bf6-8b9d028b8bf4' \
 -H "Authorization: Bearer $API_KEY" \
 -H 'accept: application/json' \
 -H 'Content-Type: application/json' \
 -d '{
     "deploymentSpecifications": [{
         "gpu": "L40",
         "backend": "OCI",
         "maxInstances": 2,
         "minInstances": 1,
         "configuration": {
         "key_one": "<value>",
         "key_two": { "key_two_subkey_one": "<value>", "key_two_subkey_two": "<value>" }
     ...
     },
     {
         "gpu": "T10",
         "backend": "GFN",
         "maxInstances": 2,
         "minInstances": 1
     }]
 }'