Is this page helpful?

UCC: Configure#

This guide provides detailed information on the USD Content Cache (UCC) component and its use in a self-hosted NVCF cluster.

UCC provides a content store running as an intermediary between object storage and Cloud Function clients. It caches USD (Universal Scene Description) content to accelerate scene loading and reduce bandwidth usage. UCC acts as a reverse proxy cache, intercepting requests for USD content from object storage sources like S3, Azure Blob Storage, or NVIDIA Omniverse Nucleus, and serving cached content when available.

Cached content reduces egress costs from cloud storage providers and improves scene load times. When content is requested, UCC first checks its local cache. On a cache hit, content is served directly from persistent storage. On a cache miss, UCC fetches the content from the upstream source, caches it locally, and serves it to the requesting client.

Base Configuration#

UCC requires some configuration to be properly installed. Create a file on your local machine called values.yaml. A base configuration is provided in the following dropdown.

values.yaml#
 1replicaCount: 3
 2
 3image:
 4  pullSecrets:
 5    - name: ngc-container-pull
 6
 7persistence:
 8  storageClassName: "gp3"
 9  volumes:
10    - name: az
11      path: /proxy_cache_az
12      sizeGi: 256
13      minFreeSizePercentage: 7
14    - name: s3
15      path: /proxy_cache_s3
16      sizeGi: 256
17      minFreeSizePercentage: 7
18    - name: nucleus
19      path: /proxy_cache_nucleus
20      sizeGi: 256
21      minFreeSizePercentage: 7
22
23nginx:
24  proxyCache:
25    validity:
26      "200": "1d"
27      "206": "1d"
28  backends:
29    azure:
30      include: true
31      cacheTtl: 30
32    s3:
33      include: true
34      cacheTtl: 30
35
36metrics:
37  prometheus:
38    enabled: true
39    serviceMonitor:
40      enabled: false
41
42tls:
43  enabled: false
Complete Configuration Reference

The base configuration above covers the essential settings for most deployments. For advanced configuration options or to explore all available settings, refer to the complete values file below. This reference includes all configuration options available in the UCC Helm chart, including advanced settings for TLS, OpenTelemetry, and resource management.

Complete UCC values.yaml reference#
  1replicaCount: 3
  2
  3image:
  4  registry: nvcr.io
  5  repository: nvidia/omniverse/usd-content-cache
  6  pullPolicy: IfNotPresent
  7  # Default tag is latest, often overridden by CI
  8  tag: "0.1.0"
  9  pullSecrets:
 10    # Name of the Kubernetes secret needed to pull from the GitLab registry.
 11    - name: gitlab-registry-secret
 12
 13# Service configuration for accessing the cache.
 14# Port 14128 avoids issues with privileged port 443 (HTTPS), which non-root containers
 15# can't bind to, preventing permission errors.
 16service:
 17  ucc:
 18    type: ClusterIP
 19    port: 14128
 20    containerPort: 14128
 21    annotations: {}
 22    # Optionally specify a specific IP for the LoadBalancer.
 23    loadBalancerIP: null
 24  nucleus:
 25    # Nucleus service configuration for Large File Transfer (LFT) operations.
 26    type: ClusterIP
 27    port: 14129
 28    containerPort: 14129
 29    annotations: {}
 30    # Optionally specify a specific IP for the LoadBalancer.
 31    loadBalancerIP: null
 32
 33# TLS configuration for HTTPS.
 34tls:
 35  # Enables HTTPS for the service.
 36  enabled: true
 37  # Name of the Kubernetes Secret containing tls.crt and tls.key
 38  # For local Minikube testing, you'll need to create this secret:
 39  #   - openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout /tmp/tls.key -out /tmp/tls.crt -subj "/CN=ucc.local.test/O=MyOrg"
 40  #   - kubectl create secret tls usd-content-cache-tls --key /tmp/tls.key --cert /tmp/tls.crt
 41  #   - rm /tmp/tls.key /tmp/tls.crt
 42  secretName: "usd-content-cache-tls"
 43
 44# Persistence Volume Claim (PVC) configuration.
 45persistence:
 46  storageClassName: "managed-csi-premium"
 47  # Defines claims for the volumes declared in the "/docker/openresty.Dockerfile" file.
 48  # `name` is used as the logical key to link with the corresponding "nginx.proxyCache.paths" entry.
 49  volumes:
 50    - name: az
 51      path: /proxy_cache_az
 52      sizeGi: 50
 53      minFreeSizePercentage: 7
 54      # Override storage class for this volume.
 55      # storageClassName: null
 56    - name: s3
 57      path: /proxy_cache_s3
 58      sizeGi: 50
 59      minFreeSizePercentage: 7
 60    - name: nucleus
 61      path: /proxy_cache_nucleus
 62      sizeGi: 50
 63      minFreeSizePercentage: 7
 64
 65# NGINX/OpenResty cache behavior settings.
 66nginx:
 67  workerConnections: 1024
 68  proxyCache:
 69    # List the HTTP codes and how long they should be cached.
 70    # https://nginx.org/en/docs/syntax.html
 71    validity:
 72      "200": "1d"
 73      "206": "1d"
 74    # List defining NGINX "proxy_cache_path" directives.
 75    # `name` is NGINX keys_zone name and must match a `name` in "persistence.volumes".
 76    # `path` specifies the NGINX cache directory on disk and must reside within a "persistence.volumes".
 77    paths:
 78      - name: az
 79        path: /proxy_cache_az/ucc_data
 80        # maxSizeGi: 90
 81        maxIdleTime: 1d
 82        metadataMemorySize: 10m
 83      - name: s3
 84        path: /proxy_cache_s3/ucc_data
 85        maxIdleTime: 1d
 86        metadataMemorySize: 10m
 87      - name: nucleus
 88        path: /proxy_cache_nucleus/ucc_data
 89        maxIdleTime: 1d
 90        metadataMemorySize: 10m
 91  logging:
 92    buffer: 1m
 93    flushInterval: 10s
 94  sharedMemory:
 95    limits:
 96      # TODO DOC
 97      presignedUrlCache: "1024m"
 98      # TODO DOC
 99      lockByRequestUriTable: "1024m"
100  resolver:
101    config: "local=on ipv6=off"
102    timeout: 8s
103  # List of backends supported.
104  # The `proxyCacheName` must match a `name` in "nginx.proxyCache.paths.name".
105  backends:
106    azure:
107      include: true
108      allowCacheReset: false
109      serverName: ~^(?<container_name>[^\.]+)\.blob\.core\.windows\.net$
110      proxyCacheName: az
111      proxyPass: $scheme://$host
112      proxyAuthPass: $scheme://$host
113      cacheTtl: 30
114    s3:
115      include: true
116      allowCacheReset: false
117      serverName: ~^[^.]+\.s3\.[^.]+\.amazonaws\.com$
118      proxyCacheName: s3
119      proxyPass: $scheme://$host
120      proxyAuthPass: $scheme://$host
121      cacheTtl: 30
122    nucleus:
123      allowCacheReset: true
124      # Default server - catches any hostname not matched by other servers.
125      serverName: _
126      proxyCacheName: nucleus
127      proxyPass: $scheme://$host
128      proxyAuthPass: null
129      # DNS resolver for Nucleus Bridge connectivity.
130      # Each Nucleus Connectivity configuration is setup into the DNS, reuse it to access the bridge.
131      # Uses global resolver if not specified.
132      bridgeResolver: null
133  lua:
134    # When true, Lua scripts can be injected into the running containers using a configmap.
135    #
136    # To simplify the creation of the configmap see the script at `scripts/debug/lua_configmap.sh`
137    # See for more information: `./scripts/debug/lua_configmap.sh -h`
138    # Example: `./scripts/debug/lua_configmap.sh -x -n namespace lua-access.lua`
139    #    Note: include '-x' option to execute the statement
140    debug: false
141    # When true Lua scripts are reloaded for each execution. This setting will only take
142    # affect when 'debug' mode is also enabled.
143    noCache: false
144    # The name of the configmap to look for. Will be prefixed with the full name of the deployment.
145    configMap: debug-lua
146    # Maps keys in the configmap to file names that will be placed in `/etc/nginx/conf.d/lua/`
147    files: [lua-access.lua, lua-metrics.lua, lua-generation.lua]
148
149# Resource requests and limits (define later for specific environments)
150resources: {}
151  # limits:
152  #   cpu: 2000m
153  #   memory: 4Gi
154  # requests:
155  #   cpu: 1000m
156  #   memory: 2Gi
157
158# Node selection, tolerations, affinity (define later if needed)
159nodeSelector: {}
160tolerations: []
161affinity: {}
162
163# Configuration options related to how Container Cache Metrics are configured.
164metrics:
165  # Creates a configmap entry that will contain the json for the UCC grafana dashboard.
166  # Useful in cases where `kube-prometheus-stack` is used for monitoring.
167  dashboard:
168    # When true the configmap value will be created.
169    include: false
170    # Allows the configmap to be placed in a different namespace.
171    namespace:
172      override: false
173      value: "default"
174    # Addtional labels for the configmap for the operator to pick up.
175    labels:
176      label: false
177  # Size for storing cache metrics in memory.
178  cacheMetricsStorageSize: 16m
179  # Configure the histogram buckets used for metrics.
180  # Must be a comma-separated list of positive integers.
181  buckets:
182    byte: 1000,2000,4000,8000,16000,32000,64000,128000,256000,1048576, 10485760, 104857600, 1000000000, 10000000000, 100000000000
183    time: 0.001, 0.002, 0.003, 0.005, 0.01, 0.02, 0.03, 0.05, 0.075, 0.1, 0.2, 0.3, 0.4, 0.5, 0.75, 1, 1.5, 2, 3, 4, 5, 10
184
185  prometheus:
186    # Enables a ServiceMonitor and Service to export prometheus metrics
187    enabled: true
188    port: 9145
189    serviceMonitor:
190      enabled: false
191      interval: 5s
192      path: /cache_metrics
193      port: cache-metrics
194      scheme: http
195      scrapeTimeout: 5s
196
197otel:
198  # Enable or disable import of otel module and collection off otel events.
199  enabled: false
200  # Additional headers to include in otel export.
201  headers:
202    #- header: X-API-Token
203    #  value: "my-token-value"
204  # interval (in seconds) in which to report buffered batches of OTEL events.
205  interval: 5
206  # The number of items in a batch.
207  batchSize: 512
208  # The number of batches to collect. If the buffer runs out of space, new events
209  # are discarded
210  batchCount: 4
211  # The collector to send the otel events to.
212  endpoint: localhost:4317
213  # See `otel_trace_context` in https://nginx.org/en/docs/ngx_otel_module.html
214  context: propagate
215  # Settings for collection of a ratio of all events.
216  ratio:
217    # When true a portion of all events are collected.
218    enabled: false
219    # The index on which to ratio.
220    source: otel_trace_id
221    # The percent of total requests to report.
222    percent: 10
223  # Controls an OpenTelemetry collector/operator deployment that can be used to
224  # deploy sidcars in each pod. This sidecar will collect the metrics from the running service
225  # and sends them to another OTEL collector.
226  #
227  # If enabled, this option will disable the ServiceMonitor.
228  collector:
229    # When enabled `endpoint` must be `localhost:4317`.
230    enabled: false
231    batch:
232    export:
233      otlp:
234        # Set this to the correct location of the collector in your cluster.
235        endpoint: otel-collector.svc.svc.cluster.local:4317
236        tls:
237          insecure: true

1. Provision and Scale#

UCC replicas operate independently of each other and do not share cached data. Each pod maintains its own cache and distributes requests from clients. Multiple replicas provide high availability and can handle higher request volumes.

Important

UCC is not designed to scale up and down dynamically. Set the replica count based on your expected workload and maintain it consistently.

The number of UCC pods is controlled through the replicaCount value:

values.yaml#
1replicaCount: 3

Recommended Replica Counts:

  • Small deployments: 1-2 replicas

  • Typical deployments: 3 replicas (recommended baseline)

  • Large deployments: 3-5 replicas

2. Storage Configuration#

UCC uses persistent volumes to cache USD content from upstream sources. Each configured backend (Azure Blob, S3, Nucleus) has its own persistent volume. Storage performance directly impacts cache hit performance and the speed at which cold assets can be served.

values.yaml#
 7persistence:
 8  storageClassName: "gp3"
 9  volumes:
10    - name: az
11      path: /proxy_cache_az
12      sizeGi: 256
13      minFreeSizePercentage: 7
14    - name: s3
15      path: /proxy_cache_s3
16      sizeGi: 256
17      minFreeSizePercentage: 7
18    - name: nucleus
19      path: /proxy_cache_nucleus
20      sizeGi: 256
21      minFreeSizePercentage: 7

storageClassName determines the performance characteristics of persistent volumes. Select a storage class that provides high IOPS and throughput suitable for cache workloads.

Storage Class Recommendations:

  • AWS (EKS): gp3 (recommended) or io1/io2 for higher performance

  • Azure (AKS): managed-csi-premium

  • On-Premises: Select a storage class backed by high-performance hardware

sizeGi determines the volume size for each cache backend. Plan cache size based on your workload:

  • Small workloads: 50-100GB per provider

  • Medium workloads: 100-500GB per provider

  • Large workloads: 500GB+ per provider

minFreeSizePercentage restricts cached content size by maintaining a minimum number of free bytes. The default value of 7 means the service will restrict cached content to 93% of the available capacity, leaving 7% free space for filesystem operations and preventing disk exhaustion.

Example Kubernetes StorageClass for AWS

If a StorageClass named gp3 does not already exist in your cluster, one can be created using the following configuration:

storageclass-gp3.yaml#
 1apiVersion: storage.k8s.io/v1
 2kind: StorageClass
 3metadata:
 4  name: gp3
 5provisioner: kubernetes.io/aws-ebs
 6volumeBindingMode: WaitForFirstConsumer
 7parameters:
 8  type: gp3
 9  iops: "5000"
10  throughput: "1000"

Apply this StorageClass to your cluster:

kubectl apply -f storageclass-gp3.yaml

3. Cache Configuration#

UCC cache behavior is controlled through NGINX proxy cache settings. These settings determine how long content is cached, how cache metadata is stored, and when cached content expires.

values.yaml#
23nginx:
24  proxyCache:
25    validity:
26      "200": "1d"
27      "206": "1d"
28  backends:
29    azure:
30      include: true
31      cacheTtl: 30
32    s3:
33      include: true
34      cacheTtl: 30
Cache Validity (validity):

Controls how long content is cached based on HTTP status codes. When the upstream source returns the specified HTTP status code, the result is kept for the given time period. Default values cache successful responses (200, 206) for 1 day.

Backend Configuration:

UCC supports multiple upstream backends. Each backend can be enabled or disabled, and has specific routing rules:

  • Azure Blob (azure): Matches Azure Blob Storage URLs using regex pattern

  • S3 (s3): Matches Amazon S3 URLs using regex pattern

  • Nucleus (nucleus): Default backend that catches any hostname not matched by other servers

Each backend configuration includes: * include: Enable or disable the backend * cacheTtl: Cache time-to-live in seconds

5. Telemetry#

UCC exports Prometheus metrics for monitoring cache performance, hit rates, and storage utilization. Collection of these metrics is important for diagnosing potential problems with UCC performance and optimizing cache configuration.

values.yaml#
36metrics:
37  prometheus:
38    enabled: true
39    serviceMonitor:
40      enabled: false
41
42tls:
43  enabled: false

Metrics Configuration:

  • prometheus.enabled: When true, creates a Kubernetes service that exports metrics in Prometheus format at /cache_metrics

  • prometheus.serviceMonitor.enabled: When true, creates a ServiceMonitor resource for Prometheus Operator integration

Important

The ServiceMonitor CRD must be installed in the cluster for ServiceMonitor resources to work.

Summary#

This guide covered the configuration options for UCC, including replica scaling, storage sizing, memory allocation, cache behavior, and monitoring setup. Proper configuration of these settings is essential for optimal UCC performance in your self-hosted NVCF cluster.

Once you have prepared your values.yaml file with the appropriate configuration, proceed to the UCC: Deployment guide to deploy UCC using Helm.

For TLS configuration instructions, refer to the UCC: TLS Configuration guide.