Skip to navigation
User Guide

Security and Authentication

View as Markdown

As a developer, you are responsible for securing access to any application using the NeMo ecosystem, including an authentication layer between users and your application and securing communication between services in your application.

Rate Limiting

NIMs do not impose rate limits. If you want to restrict access to your application, it is your responsibility to implement a strategy. One strategy is to use the NIM_TRITION_MAX_QUEUE_SIZE option to set Triton’s dynamic batching maximum queue size.

Ports

The NIM uses multiple ports, but only the HTTP API Port needs to be accessible outside of the cluster. The service port is set at start up based on the NIM_HTTP_API_PORT environment variable (default: 8000).

Additional Security Reminders

As a Developer, you must secure your own API endpoints. We suggest using a proxy as well as HTTPS/TLS 1.2.

Incident Response

Secrets

If you deploy a NIM for BGR component using Helm charts, you will need at least two secrets to set up in that namespace:

  • An image pull secret for NGC
  • An NGC API key secret

If your cluster requires additional image pull secrets for custom init containers, you’ll need those also.

Create the secrets according to your organization’s requirements and Kubernetes secrets best practices, but for POCs and to set up secrets quickly you can use the following, where NAMESPACE is the name of your namespace:

kubectl \
    --create secret -n NAMESPACE docker-registry ngc-secret  \
    --docker-server=nvcr.io --docker-username='$oauthtoken' \
    --docker-password=$NGC_API_KEY
kubectl \
    --create secret -n NAMESPACE generic ngc-api-secret  \
    --from-literal=NGC_API_KEY=$NGC_API_KEY

NeMo Retriever uses connection strings that may contain credentials. We recommend that you store these connection string in a secret management solution. For more information, refer to Manage Secrets.

Logging

To view the service logs for your running container, first identify the container ID and then stream the logs.

  1. Find the container ID:

    docker ps

  2. View the logs:

    docker logs <container-id> -f

For information about securing your Docker container and log files, refer to Docker security.