Weights & Biases Keys#

When running a customization job, individual users manage their own Weight & Biases keys and pass them as headers (--header 'wandb-api-key: <WANDB_API_KEY>' \) in the Customizer REST API call.

How It Works#

When a key is provided, NeMo Customizer service sends telemetry data to W&B including:

  • Job ID

  • Training loss

  • Validation loss

  • Other relevant metrics

The service uses PyTorch’s W&B integration to authenticate and report metrics. All metrics are stored under the user’s nvidia-nemo-customizer project in their W&B account.

Security Considerations#

  • The W&B API key is encrypted and not logged internally

  • The key is only referenced during the duration of a training job

  • Gateway layer logging should be configured to exclude this value

Configure Gateway Logging for Sensitive Headers#

This guide explains how to configure your API gateway to exclude sensitive headers, such as the wandb-api-key, from logs.

Nginx Ingress Controller#

If you’re using Nginx Ingress Controller, you can mask sensitive headers using the following configurations:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/configuration-snippet: |
      # Mask sensitive headers in logs
      map $http_wandb_api_key $masked_wandb_key {
          default       'REDACTED';
          ''           '';
      }
    nginx.ingress.kubernetes.io/log-format-upstream: |
      $remote_addr - $remote_user [$time_local] "$request" 
      $status $body_bytes_sent "$http_referer" 
      "$http_user_agent" $request_length $request_time 
      [$proxy_upstream_name] [$proxy_alternative_upstream_name] $upstream_addr 
      $upstream_response_length $upstream_response_time $upstream_status 
      $req_id $masked_wandb_key

Kong API Gateway#

For Kong Gateway, use the request-transformer plugin:

apiVersion: configuration.konghq.com/v1
kind: KongPlugin
metadata:
  name: mask-sensitive-headers
config:
  remove:
    headers: ["wandb-api-key"]
  add:
    headers:
      - "X-Sensitive-Headers-Masked: true"

Apply this to your Kong route or service:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    konghq.com/plugins: mask-sensitive-headers

Envoy Proxy#

For Envoy, modify the access logging configuration:

apiVersion: v1
kind: ConfigMap
metadata:
  name: envoy-config
data:
  envoy.yaml: |
    static_resources:
      listeners:
        - access_log:
          - name: envoy.access_loggers.file
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog
              path: "/dev/stdout"
              log_format:
                json_format:
                  time: "%START_TIME%"
                  protocol: "%PROTOCOL%"
                  request_method: "%REQ(:METHOD)%"
                  request_path: "%REQ(X-ENVOY-ORIGINAL-PATH?:PATH)%"
                  response_code: "%RESPONSE_CODE%"
                  response_flags: "%RESPONSE_FLAGS%"
                  bytes_received: "%BYTES_RECEIVED%"
                  bytes_sent: "%BYTES_SENT%"
                  duration: "%DURATION%"
                  upstream_service_time: "%RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)%"
                  forwarded_for: "%REQ(X-FORWARDED-FOR)%"
                  user_agent: "%REQ(USER-AGENT)%"
                  request_id: "%REQ(X-REQUEST-ID)%"
                  # Explicitly exclude wandb-api-key header

Verification#

To verify your configuration:

  1. Send a test request with the sensitive header:

    curl -H "wandb-api-key: test-key" https://your-api-endpoint

  2. Check your logs to ensure the header is masked:

    kubectl logs -n ingress-nginx -l app.kubernetes.io/name=ingress-nginx

Troubleshooting#

If sensitive headers are still visible in logs:

  1. Verify the configuration is applied:

    kubectl get ingress your-ingress -o yaml

  2. Check if the gateway configuration was reloaded:

    kubectl rollout restart deployment/your-gateway-controller

  3. Validate log format settings:

    kubectl describe configmap your-gateway-config