Deploy Behind a TLS Proxy for NVIDIA NIM for LLMs#

When NVIDIA NIM for LLMs is deployed behind a TLS proxy, additional setup is required to download model files from Hugging Face or NGC.

Prerequisites#

Before configuring NVIDIA NIM for LLMs to work behind a TLS proxy, ensure you have the following information and understanding:

Required Information#

  • Proxy server address and port: The full URL of your TLS proxy, such as https://proxy.company.com:3128.

  • Authentication credentials (if required): The username and password required to authenticate and access the proxy.

  • Certificate information: Whether your proxy uses a CA-signed certificate or a self-signed certificate.

  • Certificate file (for self-signed certificates): The CA certificate file that validates your proxy’s TLS connection.

Understand Your Proxy Setup#

  • TLS Proxy: A proxy server that terminates TLS connections and forwards requests to external services.

  • CA Certificate: A certificate signed by a trusted Certificate Authority (CA) that browsers and applications automatically trust.

  • Self-signed Certificate: A certificate created by your organization that is not automatically trusted and must be manually configured as trusted on your system.

Network Requirements#

Your proxy must allow connections to the following external services:

  • authn.nvidia.com - NVIDIA authentication service

  • api.ngc.nvidia.com - NVIDIA NGC API

  • xfiles.ngc.nvidia.com - NVIDIA NGC file storage

  • huggingface.co - Hugging Face model repository

  • cas-bridge.xethub.hf.co - Hugging Face content delivery network

System Requirements#

  • File descriptor limits: You may need to increase your system’s file descriptor limits when using a proxy.

  • Container runtime: Ensure you are running in a Docker or Kubernetes environment with the necessary permissions.

  • Network access: Make sure you can set environment variables and mount certificate files as needed.

Proxy Server Address#

Provide NIM with the proxy address using the https_proxy environment variable. The https_proxy value must include the scheme and port, for example: https://example.proxy:3128. If the proxy requires authentication, the format must be https://$user:$pass@proxy_host:$port.

Docker#

When you use a proxy with a CA certificate, run the Docker container as follows:

docker run -e https_proxy=<address_of_the_proxy> ...

If you use a self-signed certificate, in addition to the -e https_proxy=<address_of_the_proxy> argument, mount the certificate and set the path to the certificate in the SSL_CERT_FILE environment variable:

docker run \
  -v /path/to/certificate/on/host/machine:/path/to/certificate/inside/container \
  -e SSL_CERT_FILE=/path/to/certificate/inside/container \
  ...

Helm#

If the proxy has a CA certificate, provide the https_proxy environment variable:

env:
  - name: https_proxy
    value: <proxy-address>

If you use a self-signed certificate, in addition to setting https_proxy, provide the certificate to the deployment as a secret:

  1. Create a secret with the certificate:

kubectl create secret generic <proxy_certificate_secret> \
    --from-file=<proxy_certificate_key>=/home/apeganov/mitmproxy-deployment/mitmproxy-ca-cert.pem \
    -n <nim_namespace_name>

  1. Add a proxyCA section to the values file:

proxyCA:
  enabled: true
  secretName: <proxy_certificate_secret>
  keyName: <proxy_certificate_key>

Common Issues#

1. Unsuccessful Tunnel#

To troubleshoot an unsuccessful tunnel:

  • Check Proxy Access: Ensure your proxy allows outbound connections to all required endpoints:

    • authn.nvidia.com

    • api.ngc.nvidia.com

    • xfiles.ngc.nvidia.com

    • huggingface.co

    • cas-bridge.xethub.hf.co

  • Confirm Protocol Support: Verify that the proxy supports HTTP 1.1 connections.

Verifying Connectivity to the Proxy#

Run the following command inside the container to test the connection to the proxy:

Run the following command inside the container:

proxy_name_port="${https_proxy#https://}"
proxy_name_port="${proxy_name_port#*@}"
openssl s_client -connect ${proxy_name_port} -verify_return_error -brief < /dev/null

If the proxy requires password authentication, provide -proxy_user ... -proxy_pass ....

Look for CONNECTION ESTABLISHED and Verification: OK in the output. Example output:

CONNECTION ESTABLISHED
Protocol version: TLSv1.3
Ciphersuite: TLS_AES_256_GCM_SHA384
Peer certificate: CN = example.proxy
Hash used: SHA256
Signature type: RSA-PSS
Verification: OK
Server Temp Key: X25519, 253 bits
DONE

Verifying Connectivity to the Target Endpoint#

Run the following commands inside the container to test the connection to the target:

target_name="huggingface.co"
proxy_name_port="${https_proxy\#https://}"
proxy_name_port="${proxy_name_port\#*@}"
printf "CONNECT ${target_name}:443 HTTP/1.1\r\nHost: ${target_name}:443\r\n\r\n" | \
  openssl s_client \
    -quiet \
    -connect ${proxy_name_port} \
    -verify_return_error

If the proxy requires password authentication, provide -proxy_user ... -proxy_pass ....

You can set target_name to any of the following: authn.nvidia.com, api.ngc.nvidia.com, xfiles.ngc.nvidia.com, huggingface.co, or cas-bridge.xethub.hf.co.

Look for verify return:1 and HTTP/1.1 200 Connection Established in the output. The openssl command is expected to hang and wait for further TLS handshake with the target. You can interrupt the openssl command after the output is printed. Example output:

depth=0 CN = example.proxy
verify return:1
HTTP/1.1 200 Connection Established
Proxy-agent: nginx

2. Authentication Failed#

If you use a self-signed certificate, verify that the SSL_CERT_FILE environment variable points to the correct certificate.

If the proxy requires password authentication, verify that the credentials in the https_proxy URL are correct.

3. Too Many Open Files Error#

File Descriptor Limits: When downloading through a proxy, NIM creates many file descriptors that can exceed system limits (often 1024 per process). This may cause download failures with errors like “Too many open files.” The solution is to increase the limit, for example to 1,048,576 file descriptors.

Docker#

Increase the file descriptor limit by adding --ulimit nofile=1048576 to your docker run command. This sets both soft and hard limits to 1,048,576 file descriptors.

Example:

docker run \
  --ulimit nofile=1048576 \
  -e https_proxy=<address_of_the_proxy> \
  ...

Helm#

Solutions:

  1. Host-level fix (recommended): Increase the soft limit on the number of per-process opened file descriptors to 1048576 on host nodes prior to deployment:

    ulimit -S -n 1048576

  2. Container-level workaround: Use a custom command that sets the limit before launching NIM:

    customCommand: "ulimit -S -n 1048576; python -m nim_llm_sdk.entrypoints.launch"

Note

The host-level approach is preferred as it ensures the limit is properly set for the entire container lifecycle.