Is this page helpful?

DDCS: Configure#

This guide provides detailed information on the Derived Data Cache Service (DDCS) component and its use in a self-hosted NVCF cluster.

DDCS reduces scene load time and improves performance when properly configured and sized for the workload. Derived data generation is computationally expensive and time-consuming. DDCS trades network bandwidth for compute time by caching derived data, allowing multiple GPUs to share pre-generated content.

Derived data is often many times the size of the source content. During initial scene loads, GPUs generate the most derived content as they encounter assets for the first time. While DDCS may add some time to cold scene loads (as content must be generated and written to DDCS synchronously), the generated data becomes immediately available to other GPUs. During “warm” scene loads, render workers read all derived content from DDCS instead of regenerating it, significantly reducing load times.

Base Configuration#

DDCS requires some configuration to be properly installed. Create a file on your local machine called values.yaml. A base configuration is provided in the following dropdown.

values.yaml#
 1image:
 2  pullSecrets:
 3    - name: ngc-container-pull
 4
 5cluster:
 6  replicas: 1
 7  selfAntiAffinity: false
 8  affinity:
 9    nodeAffinity:
10      preferredDuringSchedulingIgnoredDuringExecution:
11        - weight: 100
12          preference:
13            matchExpressions:
14              - key: node-type
15                operator: In
16                values:
17                  - compute
18    podAntiAffinity:
19      preferredDuringSchedulingIgnoredDuringExecution:
20        - weight: 5
21          podAffinityTerm:
22            labelSelector:
23              matchExpressions:
24                - key: ddcs
25                  operator: In
26                  values:
27                    - "kvnode"
28            topologyKey: "kubernetes.io/hostname"
29
30  container:
31    resources:
32      #limits:
33      #  memory: 32Gi
34      requests:
35        memory: 32Gi
36
37    storage:
38      volume:
39        size: 330Gi
40        storageClassName: "gp3"
41
42    settings:
43      storageLimit: 300G
44      engine:
45        sys.cache_size: "10G"
46        sys.block_cache_size: "18G"
47        cf.max_write_buffer_number: 128
48        sys.increase_parallelism: 8
49        db.max_background_jobs: 8
50
51monitoring:
52  enabled: false
Complete Configuration Reference

The base configuration above covers the essential settings for most deployments. For advanced configuration options or to explore all available settings, refer to the complete values file below. This reference includes all configuration options available in the DDCS Helm chart, including advanced settings for TLS, OpenTelemetry, and resource management.

Complete DDCS values.yaml reference#
  1# Default values for ddcs.
  2
  3name: ddcs
  4
  5global:
  6  additionalLabels:
  7
  8image:
  9  registry: nvcr.io
 10  repository: nvidia/omniverse/ddcs-dist-kv
 11  pullPolicy: IfNotPresent
 12  pullSecrets:
 13    - name: regcred
 14  tag: "latest"
 15  overrideTag: false
 16
 17imagePullSecrets: []
 18nameOverride: ""
 19fullnameOverride: ""
 20
 21# Whole cluster configuration
 22cluster:
 23  # The number of pods to deploy
 24  #
 25  # By default the pods have a scheduler preference to be placed on different nodes. If an OSSKV pod
 26  # is already deployed to a node, another will still be scheduled if possible to do so.
 27  # This allows the load to spread between nodes without making it impossible to scale up on a single node.
 28  #
 29  # When using networked storage classes like 'managed-csi-premium' on AKS or NVMesh classes, it may
 30  # benefit the deployment to have several replicas scheduled on the same node.
 31  # For example, in AKS *EACH* volume can likely burst to ~150MB/s.
 32  replicas: 1
 33
 34  nodeSelector: {}
 35
 36  tolerations: []
 37
 38  # When true, the scheduler will prefer to place KV pods on nodes that do not already
 39  # have a KV store.
 40  selfAntiAffinity: true
 41
 42  affinity: {}
 43
 44  podAnnotations: {}
 45
 46  podSecurityContext: {}
 47    # fsGroup: 2000
 48
 49  # Controls an OpenTelemetry collector/operator deployment that can be used to
 50  # deploy sidcars in each pod. This sidecar will collect the metrics from the running service
 51  # and sends them to another OTEL collector.
 52  #
 53  # If enabled, this option will disable the ServiceMonitor.
 54  otelCollector:
 55    # When enabled `otelHost` and `otelPort` should be `localhost` and `4317`.
 56    enabled: false
 57    # The sidecar itself produces metrics, when true collect those metrics.
 58    includeCollectorMetrics: true
 59    batch:
 60    export:
 61      otlp:
 62        # Set this to the correct location of the collector in your cluster.
 63        endpoint: otel-collector.svc.svc.cluster.local:4317
 64        tls:
 65          insecure: true
 66
 67  monitoring:
 68    labels:
 69    spec:
 70
 71  # Configuration applied to each node.
 72  container:
 73
 74    rustLog: "info"
 75    rustBacktrace: "full"
 76
 77    securityContext: {}
 78      # capabilities:
 79      #   drop:
 80      #   - ALL
 81      # readOnlyRootFilesystem: true
 82      # runAsNonRoot: true
 83      # runAsUser: 1000
 84
 85    resources:
 86      #limits:
 87      #  memory: 64G
 88
 89    storage:
 90      volume:
 91        # When enabled the DB environment will reside on a k8s mounted volume.
 92        enabled: true
 93        # This number should be greater than what is provided in settings.size
 94        size: 300Gi
 95        # This is azure specific and should be changed to match a class available in your cluster.
 96        storageClassName: managed-csi
 97
 98    settings:
 99      # Format to use when logging.
100      # Allowed values are: human, human_no_color, json
101      logFormat: "json"
102      # The maximum amount of storage space the DB will target for use on the persistent volume.
103      #
104      # This includes all space necessary to maintain a WAL, SST files etc.
105      storageLimit: 275G
106      # Configuration for garbage collection.
107      garbageCollection:
108        # The minimum free capacity target for the target map.
109        #
110        # If the percentage of free map space falls below the given amount, GC
111        # will begin.
112        minFreeCapacity: 40
113        # Once GC begins, the system will attempt to remove the given quantile of the entire
114        # keyspace.
115        #
116        # If this number is 60, then 60% of keys will be deleted.
117        deleteKeyspaceQuantile: 60
118        # The interval on which to check the database capacity.
119        checkDbCapacityMs: 1000
120      # Settings for telemetry services.
121      telemetry:
122        # Enables or disables prometheus metric export.
123        #
124        # An http service is started on the specified port.
125        prometheusMetricsExposition: true
126        # The port to serve metrics on.
127        prometheusMetricsPort: 3051
128        # Enables or disables OTEL exposition.
129        otelExposition: false
130        # The OTEL service collection port.
131        otelPort: 4317
132        # The OTEL service host address.
133        #
134        # If otelCollector is enabled, this option should remain as `localhost`.
135        otelHost: localhost
136        # When true the IP env var must be specified.
137        #
138        # Useful for situations in which the host has the or pod runs an OTEL collector.
139        otelPodIpAsHost: false
140      # Engine specific configuration.
141      engine:
142        # Think of the row cache like a hashmap. When a key-pair is read it is placed in the map. The next lookup
143        # of that key will be served by the row cache. The engine does not do a row scan for the data and no disk
144        # io is necessary.
145        # This is primarily where HOT data is served from.
146        sys.cache_size: "32G"
147        # Key/value pairs can also be served from the block cache. When the engine searches rows on disk it can keep
148        # some amount of them cached in memory. This allows the next lookup to be fast if the data resides in the file
149        # blocks that have been cached.
150        # There is more that goes in the block cache, in general a large block cache can help reduce scan time.
151        sys.block_cache_size: "8G"
152        sys.block_cache_num_shard_bits: 8
153        sys.increase_parallelism: 8
154        sys.use_write_buffer_manager: 1
155
156        # Should be equal to the number of threads.
157        # Rocks db uses background jobs to flush data to disk
158        # These two values should remain equal
159        db.max_background_jobs: 8
160
161        db.table_cache_numshardbits: 6
162
163        db.allow_concurrent_memtable_write: true
164
165        # This number limits the files that rocks will keep open. This is required because K8s
166        # will consider the page files associated with open files as used memory by the POD. IE it will
167        # the pod can be OOM killed for having too many open files.
168        db.max_open_files: 128
169
170        # Write buffers have a default size of 64mb
171        # 128 * 64 MB = ~8GB of write capacity before writes are slowed to disk speed
172        # When 1 buffer is full the engine switches to the next. When all buffers are full, writes are stalled until another
173        # becomes available.
174        cf.min_write_buffer_number_to_merge: 2
175        cf.max_write_buffer_number: 128
176
177        # Enables the usage of blob db
178        cf.enable_blob_files: true
179        cf.enable_blob_garbage_collection: true
180
181        # Values of the given size are placed in new blob files instead of going through compaction.
182        cf.min_blob_size: 1MB
183
184        # Ensure that stale files are removed more often so that GC is not triggered when it does not need to be.
185        # This is 3 minutes.
186        #cf.delete_obsolete_files_period_micros: 180000000
187
188        # Ensure that SST files are flowed through the compaction filter every 4 hours.
189        cf.periodic_compaction_seconds: 14400
190      # Configure the gRPC HTTP service.
191      grpc:
192        # Set the initial HTTP2 stream window size.
193        # Must not be 0.
194        initialStreamWindowSize: 512K
195        # Set the initial HTTP2 connection window size.
196        # Must not be 0.
197        initialConnectionWindowSize: 32M
198        # Max number of requests that can be operated on concurrently per connection.
199        # Must not be 0.
200        connectionConcurrencyLimit: 32
201        # Max number of HTTP2 streams per connection.
202        # 0 for unlimited.
203        maxConcurrentStreams: 0
204        # Max frame size for each HTTP2 data frame.
205        # 0 for HTTP2 default.
206        maxFrameSize: 0
207        # Max time that a request can take.
208        # Must not be 0.
209        timeoutSeconds: 30
210        # The amount of time to wait before timing out a proto buf write.
211        writeTimeoutSeconds: 30
212        # The amount of time in seconds before keepalive probes are sent.
213        tcpKeepaliveAfterIdleSeconds: 5
214        # The amount of time between each keepalive probe.
215        tcpKeepaliveIntervalSeconds: 3
216        # The number of probes to send before the socket is considered reset.
217        tcpKeepaliveRetries: 15
218        # The amount of time to wait between http2 keepalive probes.
219        http2KeepaliveIntervalSeconds: 5
220        # The amount of time to wait before connection is considered reset.
221        http2KeepaliveTimeoutSeconds: 45
222        # The max size that grpc service will allow.
223        maxDecodingMessageSize: 5M
224        # If true a tenant ID can be set through a string set in the metadata.
225        tenantFromMetadata: false
226        # TLS configuration options.
227        tls:
228          # If true, requires TLS/https encrypted transport
229          enabled: false
230          # Use existing kubernetes.io/tls secret
231          secretName: ddcs-tls
232          # The path to the certificate to use. Must be same directory as key.
233          cert: "/tls/tls.crt"
234          # The key for the certificate. Must be same directory as cert.
235          key: "/tls/tls.key"
236          # If true, the given root ca is used.
237          includeCaRoot: false
238          # The path to the ca root.
239          caRoot: "/cert/path/ca.pem"
240        # Controls verification of JWT
241        jwt:
242          # When enabled, any jwt provided in the authorization header will be validated
243          enabled: false
244          # When true, a valid JWT must be provided.
245          require: false
246          # The url to get the public JWK set from.
247          jwkPublicKeysetUrl:
248            - "https://example.com/jwk.json"
249            - "test"
250          # The interval on which to get the JWK set.
251          jwkUpdateIntervalSecs: 1500
252          # The size of the cross-connection cache for JWTs.
253          cacheSizeMb: 128
254          # When true the aud claim of the token is verified.
255          verifyAud: false
256          # Tokens will be required to have this value as their audience claim.
257          audClaim: ""
258          # When true verifies the expiration time of provided JWT.
259          verifyExp: false
260      # Settings that control how DDCS stores items in the storage engine.
261      store:
262        # The size after which values are treated as blobs.
263        blobCutoff: 4M
264        # The size of chunks to create when storing blobs.
265        blobChunkSize: 4M
266        mode: "legacy"
267
268service:
269  # Use loadbalancer for the service - allows external IP
270  annotations: {}
271  loadBalancer: false
272  loadBalancerSourceRanges: []
273  grpcPort: 3010
274  metricsPort: 3051
275
276monitoring:
277  prometheusAlerts: true
278  enabled: true
279  interval: 5s
280  path: /metrics
281  port: http-metrics
282  scheme: http
283  scrapeTimeout: 5s

1. Provision and Scale#

Proper provisioning and scaling are critical for DDCS performance. When undersized or misconfigured for the workload, simulations will slow down or even fail. Ensure adequate network bandwidth and storage capacity based on your GPU count and scene complexity.

This guide assumes a “CPU/GPU” node split. That is, workloads that do not require a GPU are scheduled on Kubernetes nodes without GPUs (CPU nodes). Workloads that do require a GPU are scheduled on Kubernetes nodes with GPUs (GPU nodes).

As a baseline plan for ~3.3 Gbps for each GPU in the cluster.

AWS EKS Example

Assuming the SKUs for CPU/GPU nodes are m5.8xlarge and g6e.4xlarge respectively.

SKU

vCPU

RAM

NIC

GPU

m5.8xlarge (compute)

32

128 G

10 Gbps

g6e.4xlarge (gpu)

16

128 G

20 Gbps

1 L40S

If a cluster is provisioned with 25 GPUs a matching 83 Gbps (3.3 Gbps x 25 GPU) of bandwidth is necessary to facilitate caching.

Provision 9 compute nodes with 10Gbps and create 9 DDCS replicas.

A single DDCS pod should be placed on each compute node. The goal is to expand network capability with each pod, distributing cache load across multiple nodes. If 9 compute nodes were provisioned for caching, schedule one DDCS pod on each node.

values.yaml#
5cluster:
6  replicas: 1

2. Memory Configuration#

DDCS memory configuration includes Kubernetes resource limits and requests, along with engine-level cache settings. The base configuration allocates memory for the Point Cache, Block Cache, and Write Buffers.

Here are some common configuration modes that provide for various CPU and RAM allocations.

values.yaml#
container:
  resources:
    #limits:
    #  memory: 16Gi
    requests:
      memory: 16Gi

  settings:
    engine:
      sys.cache_size: "4G"
      sys.block_cache_size: "8G"
      cf.max_write_buffer_number: 64
      sys.increase_parallelism: 4
      db.max_background_jobs: 4
values.yaml#
container:
  resources:
    #limits:
    #  memory: 32Gi
    requests:
      memory: 32Gi

  settings:
    engine:
      sys.cache_size: "10G"
      sys.block_cache_size: "18G"
      cf.max_write_buffer_number: 128
      sys.increase_parallelism: 8
      db.max_background_jobs: 8
values.yaml#
container:
  resources:
    #limits:
    #  memory: 32Gi
    requests:
      memory: 32Gi

  settings:
    engine:
      sys.cache_size: "20G"
      sys.block_cache_size: "36G"
      cf.max_write_buffer_number: 128
      sys.increase_parallelism: 12
      db.max_background_jobs: 12

Advanced Memory Configuration

The provided guide for sizing DDCS should be followed where possible. If the configuration needs more advanced tuning, or the environment restricts memory this section may be helpful in understanding what the options mean.

Skip this if using the provided configuration.

See more…

Memory Components:

  • Point Cache (sys.cache_size): The first cache level for serving cached content. Set to 10G in the base configuration.

  • Block Cache (sys.block_cache_size): The second cache level used by RocksDB for metadata, filters, and file blocks. Set to 18G in the base configuration.

  • Write Buffers (cf.max_write_buffer_number): In-memory buffers that absorb write bursts before flushing to disk. Each buffer consumes 64MB. The base configuration sets this to 128 buffers (approximately 8GB total capacity).

  1. Volume Configuration

Reading from persistent storage, even a network-attached volume, is often faster than regenerating derived data. Therefore, DDCS persists content to a Kubernetes volume.

values.yaml#
37    storage:
38      volume:
39        size: 330Gi
40        storageClassName: "gp3"
41
42    settings:
43      storageLimit: 300G
size determines the volume size that will be created and attached to each pod.

Depending on the cloud environment, volume size may also control the IOPS and throughput of the volume. Refer to your cloud provider’s documentation for specific performance characteristics.

storageClassName determines the performance characteristics of persistent volumes.

Select a storage class that provides high IOPS and throughput suitable for database workloads. Ideal conditions are 4000 IOPS and greater than 800 Mb/s sustained throughput.

storageLimit controls the maximum disk content before garbage collection is triggered.

This value should be 30-60GB smaller than the volume size to provide headroom for filesystem operations and prevent disk exhaustion. Filling a volume will result in IO errors and instability. When disk usage reaches 60% of this limit, garbage collection begins automatically. Smaller volumes will trigger garbage collection more frequently, which can reduce performance.

Example Kubernetes StorageClass for AWS

If a StorageClass named gp3 does not already exist in your cluster, one can be created using the following configuration:

storageclass-gp3.yaml#
 1apiVersion: storage.k8s.io/v1
 2kind: StorageClass
 3metadata:
 4  name: gp3
 5provisioner: kubernetes.io/aws-ebs
 6volumeBindingMode: WaitForFirstConsumer
 7parameters:
 8  type: gp3
 9  iops: "5000"
10  throughput: "1000"

Apply this StorageClass to your cluster:

kubectl apply -f storageclass-gp3.yaml

4. Telemetry#

DDCS exports Prometheus metrics for monitoring cache performance, hit rates, and storage utilization. Collection of these metrics is important for diagnosing potential problems with DDCS performance and optimizing cache configuration.

values.yaml#
51monitoring:
52  enabled: false

Metrics Configuration:

  • monitoring.enabled: When true, enables Prometheus metrics collection. Metrics are exposed via a Kubernetes service in Prometheus format.

Important

The ServiceMonitor CRD must be installed in the cluster for ServiceMonitor resources to work.

Configuration Recommendations#

The following configurations provide complete values files for different cluster sizes. Each configuration includes all base settings optimized for the specified GPU count and bandwidth requirements.

values.yaml#
image:
  pullSecrets:
    - name: ngc-container-pull

cluster:
  replicas: 2  # 2 nodes x 12.5 Gbps = 25 Gbps capacity
  selfAntiAffinity: false
  affinity:
    nodeAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
        - weight: 100
          preference:
            matchExpressions:
              - key: node-type
                operator: In
                values:
                  - compute
    podAntiAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
        - weight: 5
          podAffinityTerm:
            labelSelector:
              matchExpressions:
                - key: ddcs
                  operator: In
                  values:
                    - "kvnode"
            topologyKey: "kubernetes.io/hostname"

  container:
    resources:
      #limits:
      #  memory: 32Gi
      requests:
        memory: 32Gi

    storage:
      volume:
        size: 330Gi
        storageClassName: "gp3"

    settings:
      storageLimit: 300G
      engine:
        sys.cache_size: "10G"
        sys.block_cache_size: "18G"
        cf.max_write_buffer_number: 128
        sys.increase_parallelism: 8
        db.max_background_jobs: 8

monitoring:
  enabled: false
values.yaml#
image:
  pullSecrets:
    - name: ngc-container-pull

cluster:
  replicas: 3  # 3 nodes x 12.5 Gbps = 37.5 Gbps capacity
  selfAntiAffinity: false
  affinity:
    nodeAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
        - weight: 100
          preference:
            matchExpressions:
              - key: node-type
                operator: In
                values:
                  - compute
    podAntiAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
        - weight: 5
          podAffinityTerm:
            labelSelector:
              matchExpressions:
                - key: ddcs
                  operator: In
                  values:
                    - "kvnode"
            topologyKey: "kubernetes.io/hostname"

  container:
    resources:
      #limits:
      #  memory: 32Gi
      requests:
        memory: 32Gi

    storage:
      volume:
        size: 330Gi
        storageClassName: "gp3"

    settings:
      storageLimit: 300G
      engine:
        sys.cache_size: "10G"
        sys.block_cache_size: "18G"
        cf.max_write_buffer_number: 128
        sys.increase_parallelism: 8
        db.max_background_jobs: 8

monitoring:
  enabled: false
values.yaml#
image:
  pullSecrets:
    - name: ngc-container-pull

cluster:
  replicas: 6  # 6 nodes x 12.5 Gbps = 75 Gbps capacity
  selfAntiAffinity: false
  affinity:
    nodeAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
        - weight: 100
          preference:
            matchExpressions:
              - key: node-type
                operator: In
                values:
                  - compute
    podAntiAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
        - weight: 5
          podAffinityTerm:
            labelSelector:
              matchExpressions:
                - key: ddcs
                  operator: In
                  values:
                    - "kvnode"
            topologyKey: "kubernetes.io/hostname"

  container:
    resources:
      #limits:
      #  memory: 32Gi
      requests:
        memory: 32Gi

    storage:
      volume:
        size: 330Gi
        storageClassName: "gp3"

    settings:
      storageLimit: 300G
      engine:
        sys.cache_size: "10G"
        sys.block_cache_size: "18G"
        cf.max_write_buffer_number: 128
        sys.increase_parallelism: 8
        db.max_background_jobs: 8

monitoring:
  enabled: false

Summary#

This guide covered the configuration options for DDCS, including scaling considerations, memory allocation, storage sizing, and monitoring setup. Proper configuration of these settings is essential for optimal DDCS performance in your self-hosted NVCF cluster.

Once you have prepared your values.yaml file with the appropriate configuration, proceed to the DDCS: Deployment guide to deploy DDCS using Helm.