Is this page helpful?

Post-Installation Configuration#

To avoid errors when you copy configuration files from the documentation, download the archive from the Holoscan for Media Platform Setup resource in the NGC catalog. You can find the resource in the Holoscan for Media collection.

Node Labeling#

In this section, label each worker node with the workloads it supports and the node pool it belongs to.

Before labeling the worker nodes, organize them into node pools. A node pool is a logical group of worker nodes that share the same workload type and hardware or network configuration. A pool can contain one node or several nodes. Examples include a container-only pool, a virtual machine (VM)-only pool, or a pool that supports both container and VM workloads.

Label every worker node with the following labels:

  • Both workload labels, which indicate whether the node supports container workloads, VM workloads, or both.

  • One node-pool label, which identifies the pool the node belongs to.

For example, a five-node cluster has three control-plane nodes and two worker nodes. If worker-1 is a container-only node and worker-2 is a VM-only node, you can define the following pools:

Node Pool

Nodes

Workload

container-pool

worker-1

container

vm-pool

worker-2

vm

The following examples show other common node-pool layouts. Create a separate pool whenever nodes need a different workload type or virtual function (VF) configuration.

Example: Different VF layout on container-only nodes

Both worker nodes run container workloads only, but they use different VF layouts. Because the configuration differs, place them in separate pools.

For example, worker-1 uses 32 VFs on each interface, and worker-2 uses 64 VFs on each interface:

NIC

Worker 1

Worker 2

NIC1

ens15f0np0 (32 VFs)

ens15f0np0 (64 VFs)

ens15f1np1 (32 VFs)

ens15f1np1 (64 VFs)

NIC2

ens16f0np0 (32 VFs)

ens16f0np0 (64 VFs)

ens16f1np1 (32 VFs)

ens16f1np1 (64 VFs)

NIC3

ens20f0np0 (32 VFs)

ens20f0np0 (64 VFs)

ens20f1np1 (32 VFs)

ens20f1np1 (64 VFs)

NIC4

ens21f0np0 (32 VFs)

ens21f0np0 (64 VFs)

ens21f1np1 (32 VFs)

ens21f1np1 (64 VFs)

Node Pool

Nodes

Workload

VF layout

pool-1

worker-1

container

32 VFs per interface

pool-2

worker-2

container

64 VFs per interface
Example: Only some NICs are virtualized

On a worker node that supports VMs, only some NICs carry VM workloads. The remaining NICs stay container-only. Because the nodes have different workloads and different VF layouts, place them in separate pools.

For example, on worker-1 only NIC1 and NIC2 are virtualized for VMs, and worker-2 is container-only:

NIC

Worker 1 (VM)

Worker 2

NIC1

ens15f0np0 (VM workload)

ens15f0np0

ens15f1np1 (VM workload)

ens15f1np1

NIC2

ens16f0np0 (VM workload)

ens16f0np0

ens16f1np1 (VM workload)

ens16f1np1

NIC3

ens20f0np0

ens20f0np0

ens20f1np1

ens20f1np1

NIC4

ens21f0np0

ens21f0np0

ens21f1np1

ens21f1np1

Node Pool

Nodes

Workload

NICs for VMs

pool-1

worker-1

container and vm

NIC1, NIC2

pool-2

worker-2

container

None
Example: VF partitioning

On the same interface, split VFs between container and VM workloads. Because the nodes have different workloads and different VF layouts, place them in separate pools.

For example, on worker-1 each interface assigns VFs 0-15 to containers and VFs 16-63 to VMs, and worker-2 is container-only:

NIC

Worker 1 (VM)

Worker 2

NIC1

ens15f0np0 (container 0-15, VM 16-63)

ens15f0np0

ens15f1np1 (container 0-15, VM 16-63)

ens15f1np1

NIC2

ens16f0np0 (container 0-15, VM 16-63)

ens16f0np0

ens16f1np1 (container 0-15, VM 16-63)

ens16f1np1

NIC3

ens20f0np0 (container 0-15, VM 16-63)

ens20f0np0

ens20f1np1 (container 0-15, VM 16-63)

ens20f1np1

NIC4

ens21f0np0 (container 0-15, VM 16-63)

ens21f0np0

ens21f1np1 (container 0-15, VM 16-63)

ens21f1np1

Node Pool

Nodes

Workload

pool-1

worker-1

container and vm

pool-2

worker-2

container

Workload Labels#

Apply the following workload labels:

Node Workload

Container Workload Label

VM Workload Label

container

nvidia.com/holoscan-for-media.container-workloads=true

nvidia.com/holoscan-for-media.vm-workloads=false

vm

nvidia.com/holoscan-for-media.container-workloads=false

nvidia.com/holoscan-for-media.vm-workloads=true

container and vm

nvidia.com/holoscan-for-media.container-workloads=true

nvidia.com/holoscan-for-media.vm-workloads=true

For a container-only node:

oc label node <node-name> nvidia.com/holoscan-for-media.container-workloads=true --overwrite
oc label node <node-name> nvidia.com/holoscan-for-media.vm-workloads=false --overwrite

For a VM-only node:

oc label node <node-name> nvidia.com/holoscan-for-media.container-workloads=false --overwrite
oc label node <node-name> nvidia.com/holoscan-for-media.vm-workloads=true --overwrite

For a node that supports both container and VM workloads:

oc label node <node-name> nvidia.com/holoscan-for-media.container-workloads=true --overwrite
oc label node <node-name> nvidia.com/holoscan-for-media.vm-workloads=true --overwrite

Node-Pool Labels#

Apply the node-pool label to each node:

oc label node <node-name> nvidia.com/holoscan-for-media.node-pools=<pool-name> --overwrite

Verify that the expected nodes match each label:

oc get nodes -l nvidia.com/holoscan-for-media.container-workloads=true
oc get nodes -l nvidia.com/holoscan-for-media.vm-workloads=true
oc get nodes -l nvidia.com/holoscan-for-media.node-pools=<pool-name>

Worker Node Tuning#

Create MachineConfigPool#

Skip this section if you are on a compact (three-node) cluster or single-node OpenShift (SNO). This step is specific to five-node clusters because it creates a new MachineConfigPool (MCP) that inherits from the worker MCP. In a three-node cluster or SNO, all configuration applies to the master MCP itself.

  1. Label all worker nodes with a custom label. For example, node-role.kubernetes.io/holoscanmedia=

    oc label node <worker-node> node-role.kubernetes.io/holoscanmedia=

  2. Create holoscanmedia_mcp.yaml with the following content:

    apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfigPool
metadata:
    labels:
        holoscanmedia: ''
        machineconfiguration.openshift.io/role: holoscanmedia
    name: holoscanmedia
spec:
    machineConfigSelector:
        matchExpressions:
        -   key: machineconfiguration.openshift.io/role
            operator: In
            values:
            - worker
            - holoscanmedia
    nodeSelector:
        matchLabels:
            node-role.kubernetes.io/holoscanmedia: ''

  3. Apply the machine config pool:

    oc create -f holoscanmedia_mcp.yaml

Create Performance Profile#

A performance profile tunes the cluster nodes for high-performance, low-latency workloads.

  1. Create holoscanmedia_profile.yaml based on the following template:

    apiVersion: performance.openshift.io/v2
kind: PerformanceProfile
metadata:
    name: irq-performanceprofile
spec:
    globallyDisableIrqLoadBalancing: true
    additionalKernelArgs:
    - nmi_watchdog=0
    - audit=0
    - mce=ignore_ce
    - processor.max_cstate=0
    - idle=poll
    - intel_idle.max_cstate=0
    - intel_pstate=disable
    - nosmt
    - nosoftlockup
    - cpufreq.default_governor=performance
    - module_blacklist=irdma
    - nokaslr
    cpu:
        isolated: 8-63
        reserved: 0-7
    hugepages:
        defaultHugepagesSize: 2M
        pages:
        - count: 32768
          size: 2M
    numa:
        topologyPolicy: single-numa-node
    nodeSelector:
        node-role.kubernetes.io/<machine-config-pool>: ''
    realTimeKernel:
        enabled: false

    Note the following about this template:

    • The additional kernel arguments are derived from the DOCA Rivermax Documentation or find the latest version of the Rivermax Getting Started.

    • Replace <machine-config-pool> with holoscanmedia for a five-node cluster and master for a three-node cluster or SNO.

    • Reserve four to eight CPU cores for the system and isolate the rest of the CPUs for application pods. Set the isolated and reserved fields in the template accordingly. These CPU cores are physical cores on the nodes (not hyperthreaded cores) because the additionalKernelArgs specify nosmt.

    • Maximum performance requires Non-Uniform Memory Access (NUMA)-aware scheduling. However, for servers with fewer network adapters or GPUs, or an unbalanced topology, add numa=off to the additionalKernelArgs and remove the numa section to use the available resources.

    Important

    After you configure the Topology Manager single-numa-node policy, pods that request NUMA-aligned resources and use the default-scheduler can cause runaway pod creation errors (ContainerStatusUnknown). To avoid these errors, install and use the NUMA-aware scheduler.

  2. Apply the performance profile:

    oc create -f holoscanmedia_profile.yaml

    This command reboots the worker nodes one at a time.

  3. You can monitor the status using the following command. Wait for the UPDATED column to become True. This step can take 30–45 minutes to complete.

    oc get mcp
NAME            CONFIG                                                    UPDATED   UPDATING   DEGRADED   MACHINECOUNT   READYMACHINECOUNT   UPDATEDMACHINECOUNT   DEGRADEDMACHINECOUNT   AGE
holoscanmedia   rendered-holoscanmedia-7e2a6af0d84770766c9246dffdf55a40   False     True       False      2              0                   0                     0                      100m
master          rendered-master-fd62216b1239cd90a6a3cd7c3b9e213d          True      False      False      3              3                   3                     0                      4h
worker          rendered-worker-7e2a6af0d84770766c9246dffdf55a40          True      False      False      0              0                   0                     0                      4h

  4. After the servers come back online, check the hugepages allocation from the jump node using the following command:

    oc get node <worker-node> -o jsonpath="{.status.allocatable.hugepages-2Mi}" 64Gi

Hardware Tuning#

  1. Create holoscanmedia_tuned.yaml based on the following template:

    apiVersion: tuned.openshift.io/v1
kind: Tuned
metadata:
    name: holoscanmedia-tuning-profile
    namespace: openshift-cluster-node-tuning-operator
spec:
    profile:
    - data: |
        [main]
        summary=Boot time configuration for Holoscan for Media
        include=openshift-node-performance-irq-performanceprofile
        [sysctl]
        vm.swappiness=0
        vm.zone_reclaim_mode=0
        kernel.numa_balancing=0
        kernel.sched_rt_runtime_us=-1
        [service]
        service.irqbalance=stop,disable
        [vm]
        transparent_hugepages=never
      name: openshift-node-holoscanmedia
    recommend:
    - machineConfigLabels:
        machineconfiguration.openshift.io/role: "<machine-config-pool>"
      priority: 10
      profile: openshift-node-holoscanmedia

    Note the following about this template:

    • The additional settings in this profile come from the DOCA Rivermax Documentation.

    • Replace <machine-config-pool> with holoscanmedia for a five-node cluster and master for a three-node cluster or SNO.

  2. Apply the TuneD profile:

    oc create -f holoscanmedia_tuned.yaml

Disable Chronyd Service#

  1. Create disable_chronyd.yaml based on the following template. Replace <role> with worker for a five-node cluster and master for a three-node cluster or SNO:

    apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
    labels:
        machineconfiguration.openshift.io/role: <role>
    name: disable-chronyd
spec:
    config:
        ignition:
            version: 3.2.0
        systemd:
            units:
            - contents: |
                [Unit]
                Description=NTP client/server
                Documentation=man:chronyd(8) man:chrony.conf(5)
                After=ntpdate.service sntp.service ntpd.service
                Conflicts=ntpd.service systemd-timesyncd.service
                ConditionCapability=CAP_SYS_TIME
                [Service]
                Type=forking
                PIDFile=/run/chrony/chronyd.pid
                EnvironmentFile=-/etc/sysconfig/chronyd
                ExecStart=/usr/sbin/chronyd $OPTIONS
                ExecStartPost=/usr/libexec/chrony-helper update-daemon
                PrivateTmp=yes
                ProtectHome=yes
                ProtectSystem=full
                [Install]
                WantedBy=multi-user.target
              enabled: false
              name: "chronyd.service"

  2. Apply the machine config:

    oc create -f disable_chronyd.yaml

    This command reboots the worker nodes. Wait 15 to 20 minutes for the nodes to come back up before you proceed.

  3. Ensure that READYMACHINECOUNT shows the total number of worker nodes:

    oc get mcp
NAME            CONFIG                                                    UPDATED   UPDATING   DEGRADED   MACHINECOUNT   READYMACHINECOUNT   UPDATEDMACHINECOUNT   DEGRADEDMACHINECOUNT   AGE
holoscanmedia   rendered-holoscanmedia-7e2a6af0d84770766c9246dffdf55a40   False     True       False      2              0                   0                     0                      100m
master          rendered-master-fd62216b1239cd90a6a3cd7c3b9e213d          True      False      False      3              3                   3                     0                      4h
worker          rendered-worker-7e2a6af0d84770766c9246dffdf55a40          True      False      False      0              0                   0                     0                      4h

PCIe ACS Configuration#

Configure PCIe Access Control Services (ACS) based on the workload type. Virtualization workloads require the virtualization-specific ACS configuration that follows. For non-virtualization workloads, disable ACS to maximize bandwidth.

ACS configuration uses the VM workload label applied during node labeling.

  1. Create configure_acs.yaml based on the following template. Replace <role> with worker for a five-node cluster and master for a three-node cluster or SNO:

    apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
    labels:
        machineconfiguration.openshift.io/role: <role>
    name: configure-acs
spec:
    config:
        ignition:
            version: 3.4.0
        storage:
            files:
            - path: /usr/local/bin/configure-acs.sh
              contents:
                source: data:;charset=utf-8;base64,IyEvYmluL2Jhc2gKIyBtdXN0IGJlIHJvb3QgdG8gYWNjZXNzIGV4dGVuZGVkIFBDSSBjb25maWcgc3BhY2UKCmlmIFsgIiRFVUlEIiAtbmUgMCBdOyB0aGVuCiAgICBlY2hvICJFUlJPUjogJDAgbXVzdCBiZSBydW4gYXMgcm9vdCIKICAgIGV4aXQgMQpmaQoKIyBHZXQgbm9kZSBuYW1lCk5PREVfTkFNRT0kKGhvc3RuYW1lKQoKIyBDaGVjayBpZiBrdWJlY29uZmlnIGV4aXN0cwpLVUJFQ09ORklHPSIvdmFyL2xpYi9rdWJlbGV0L2t1YmVjb25maWciCmlmIFsgISAtZiAiJEtVQkVDT05GSUciIF07IHRoZW4KICAgIGVjaG8gIkt1YmVjb25maWcgbm90IGZvdW5kLCBza2lwcGluZyBBQ1MgY29uZmlndXJhdGlvbiIKICAgIGV4aXQgMQpmaQoKIyBXYWl0IHVudGlsIEFQSSBzZXJ2ZXIgaXMgcmVhZHkKZWNobyAiV2FpdGluZyBmb3IgQVBJIHNlcnZlci4uLiIKd2hpbGUgISBvYyAtLWt1YmVjb25maWc9IiRLVUJFQ09ORklHIiBnZXQgbm9kZSAiJE5PREVfTkFNRSIgLS1zaG93LWxhYmVscyAmPi9kZXYvbnVsbDsgZG8KICAgIHNsZWVwIDUKZG9uZQplY2hvICJBUEkgc2VydmVyIHJlYWR5IgoKIyBDaGVjayBpZiBub2RlIGhhcyB0aGUgaG9sb3NjYW4tZm9yLW1lZGlhIHZtLXdvcmtsb2FkcyBsYWJlbCBzZXQgdG8gInRydWUiClZNX1dPUktMT0FEU19MQUJFTD0kKG9jIC0ta3ViZWNvbmZpZz0iJEtVQkVDT05GSUciIGdldCBub2RlICIkTk9ERV9OQU1FIiAtbyBqc29ucGF0aD0ney5tZXRhZGF0YS5sYWJlbHMubnZpZGlhXC5jb20vaG9sb3NjYW4tZm9yLW1lZGlhXC52bS13b3JrbG9hZHN9JyAyPi9kZXYvbnVsbCkKaWYgWyAiJFZNX1dPUktMT0FEU19MQUJFTCIgPSAidHJ1ZSIgXTsgdGhlbgogICAgIyBDb25maWd1cmUgQUNTIGZvciB2aXJ0dWFsaXphdGlvbgogICAgZWNobyAiQ29uZmlndXJpbmcgQUNTIGZvciB2aXJ0dWFsaXphdGlvbiIKICAgIGZvciBCREYgaW4gJChsc3BjaSAtZCAiKjoqOioiIHwgYXdrICd7cHJpbnQgJDF9Jyk7IGRvCiAgICAgICAgIyBza2lwIGlmIGl0IGRvZXNuJ3Qgc3VwcG9ydCBBQ1MKICAgICAgICBzZXRwY2kgLXYgLXMgJHtCREZ9IEVDQVBfQUNTKzB4Ni53ID4gL2Rldi9udWxsIDI+JjEKICAgICAgICBpZiBbICQ/IC1uZSAwIF07IHRoZW4KICAgICAgICAgICAgIyBlY2hvICIke0JERn0gZG9lcyBub3Qgc3VwcG9ydCBBQ1MsIHNraXBwaW5nIgogICAgICAgICAgICBjb250aW51ZQogICAgICAgIGZpCiAgICAgICAgbG9nZ2VyICJFbmFibGluZyBBQ1Mgb24gJChsc3BjaSAtcyAke0JERn0pIgogICAgICAgIHNldHBjaSAtdiAtcyAke0JERn0gRUNBUF9BQ1MrMHg2LncKICAgICAgICBzZXRwY2kgLXYgLXMgJHtCREZ9IEVDQVBfQUNTKzB4Ni53PTB4NUQKICAgICAgICBzZXRwY2kgLXYgLXMgJHtCREZ9IEVDQVBfQUNTKzB4Ni53CiAgICAgICAgaWYgWyAkPyAtbmUgMCBdOyB0aGVuCiAgICAgICAgICAgIGxvZ2dlciAiRXJyb3IgZW5hYmxpbmcgQUNTIG9uICR7QkRGfSIKICAgICAgICAgICAgY29udGludWUKICAgICAgICBmaQogICAgICAgIE5FV19WQUw9JChzZXRwY2kgLXYgLXMgJHtCREZ9IEVDQVBfQUNTKzB4Ni53IHwgYXdrICd7cHJpbnQgJE5GfScpCiAgICAgICAgaWYgWyAiJHtORVdfVkFMfSIgIT0gIjB4NUQiIF07IHRoZW4KICAgICAgICAgICAgbG9nZ2VyICJGYWlsZWQgdG8gRW5hYmxlIEFDUyBvbiAke0JERn0iCiAgICAgICAgICAgIGNvbnRpbnVlCiAgICAgICAgZmkKICAgIGRvbmUKICAgIGVjaG8gIkFDUyBjb25maWd1cmVkIGZvciB2aXJ0dWFsaXphdGlvbiIKZWxzZQogICAgIyBEaXNhYmxlIEFDUwogICAgZWNobyAiRGlzYWJsaW5nIEFDUyIKICAgIGZvciBCREYgaW4gJChsc3BjaSAtZCAiKjoqOioiIHwgYXdrICd7cHJpbnQgJDF9Jyk7IGRvCiAgICAgICAgIyBza2lwIGlmIGl0IGRvZXNuJ3Qgc3VwcG9ydCBBQ1MKICAgICAgICBzZXRwY2kgLXYgLXMgJHtCREZ9IEVDQVBfQUNTKzB4Ni53ID4gL2Rldi9udWxsIDI+JjEKICAgICAgICBpZiBbICQ/IC1uZSAwIF07IHRoZW4KICAgICAgICAgICAgIyBlY2hvICIke0JERn0gZG9lcyBub3Qgc3VwcG9ydCBBQ1MsIHNraXBwaW5nIgogICAgICAgICAgICBjb250aW51ZQogICAgICAgIGZpCiAgICAgICAgbG9nZ2VyICJEaXNhYmxpbmcgQUNTIG9uICQobHNwY2kgLXMgJHtCREZ9KSIKICAgICAgICBzZXRwY2kgLXYgLXMgJHtCREZ9IEVDQVBfQUNTKzB4Ni53PTAwMDAKICAgICAgICBpZiBbICQ/IC1uZSAwIF07IHRoZW4KICAgICAgICAgICAgbG9nZ2VyICJFcnJvciBkaXNhYmxpbmcgQUNTIG9uICR7QkRGfSIKICAgICAgICAgICAgY29udGludWUKICAgICAgICBmaQogICAgICAgIE5FV19WQUw9JChzZXRwY2kgLXYgLXMgJHtCREZ9IEVDQVBfQUNTKzB4Ni53IHwgYXdrICd7cHJpbnQgJE5GfScpCiAgICAgICAgaWYgWyAiJHtORVdfVkFMfSIgIT0gIjAwMDAiIF07IHRoZW4KICAgICAgICAgICAgbG9nZ2VyICJGYWlsZWQgdG8gZGlzYWJsZSBBQ1Mgb24gJHtCREZ9IgogICAgICAgICAgICBjb250aW51ZQogICAgICAgIGZpCiAgICBkb25lCiAgICBlY2hvICJBQ1MgZGlzYWJsZWQiCmZpCgpleGl0IDA=
              filesystem: root
              mode: 0755
        systemd:
            units:
            - name: configure-acs.service
              enabled: true
              contents: |
                [Unit]
                Description=Configure PCIe ACS
                After=network-online.target kubelet.service
                Wants=network-online.target
                [Service]
                Type=oneshot
                ExecStart=/usr/local/bin/configure-acs.sh
                RemainAfterExit=yes
                [Install]
                WantedBy=multi-user.target

    The Base64-encoded data in the YAML above corresponds to the following plain text:

    #!/bin/bash
# must be root to access extended PCI config space

if [ "$EUID" -ne 0 ]; then
    echo "ERROR: $0 must be run as root"
    exit 1
fi

# Get node name
NODE_NAME=$(hostname)

# Check if kubeconfig exists
KUBECONFIG="/var/lib/kubelet/kubeconfig"
if [ ! -f "$KUBECONFIG" ]; then
    echo "Kubeconfig not found, skipping ACS configuration"
    exit 1
fi

# Wait until API server is ready
echo "Waiting for API server..."
while ! oc --kubeconfig="$KUBECONFIG" get node "$NODE_NAME" --show-labels &>/dev/null; do
    sleep 5
done
echo "API server ready"

# Check if node has the holoscan-for-media vm-workloads label set to "true"
VM_WORKLOADS_LABEL=$(oc --kubeconfig="$KUBECONFIG" get node "$NODE_NAME" -o jsonpath='{.metadata.labels.nvidia\.com/holoscan-for-media\.vm-workloads}' 2>/dev/null)
if [ "$VM_WORKLOADS_LABEL" = "true" ]; then
    # Configure ACS for virtualization
    echo "Configuring ACS for virtualization"
    for BDF in $(lspci -d "*:*:*" | awk '{print $1}'); do
        # skip if it doesn't support ACS
        setpci -v -s ${BDF} ECAP_ACS+0x6.w > /dev/null 2>&1
        if [ $? -ne 0 ]; then
            # echo "${BDF} does not support ACS, skipping"
            continue
        fi
        logger "Enabling ACS on $(lspci -s ${BDF})"
        setpci -v -s ${BDF} ECAP_ACS+0x6.w
        setpci -v -s ${BDF} ECAP_ACS+0x6.w=0x5D
        setpci -v -s ${BDF} ECAP_ACS+0x6.w
        if [ $? -ne 0 ]; then
            logger "Error enabling ACS on ${BDF}"
            continue
        fi
        NEW_VAL=$(setpci -v -s ${BDF} ECAP_ACS+0x6.w | awk '{print $NF}')
        if [ "${NEW_VAL}" != "0x5D" ]; then
            logger "Failed to Enable ACS on ${BDF}"
            continue
        fi
    done
    echo "ACS configured for virtualization"
else
    # Disable ACS
    echo "Disabling ACS"
    for BDF in $(lspci -d "*:*:*" | awk '{print $1}'); do
        # skip if it doesn't support ACS
        setpci -v -s ${BDF} ECAP_ACS+0x6.w > /dev/null 2>&1
        if [ $? -ne 0 ]; then
            # echo "${BDF} does not support ACS, skipping"
            continue
        fi
        logger "Disabling ACS on $(lspci -s ${BDF})"
        setpci -v -s ${BDF} ECAP_ACS+0x6.w=0000
        if [ $? -ne 0 ]; then
            logger "Error disabling ACS on ${BDF}"
            continue
        fi
        NEW_VAL=$(setpci -v -s ${BDF} ECAP_ACS+0x6.w | awk '{print $NF}')
        if [ "${NEW_VAL}" != "0000" ]; then
            logger "Failed to disable ACS on ${BDF}"
            continue
        fi
    done
    echo "ACS disabled"
fi

exit 0

    The ACS configuration depends on the node label:

    • Nodes with nvidia.com/holoscan-for-media.vm-workloads=true use the following ACS configuration for virtualization:

      • SrcValid (Source Validation) - enabled

      • TransBlk (Translation Blocking) - disabled

      • ReqRedir (P2P Request Redirection) - enabled

      • CmpltRedir (P2P Completion Redirection) - enabled

      • UpstreamFwd (Upstream Forwarding) - enabled

      • EgressCtrl (Egress Control) - disabled

      • DirectTrans (Direct Translated P2P) - enabled

    • Nodes with nvidia.com/holoscan-for-media.vm-workloads=false have ACS disabled.

  2. Apply the machine config:

    oc create -f configure_acs.yaml

    This command reboots the worker nodes. Wait 15 to 20 minutes for the nodes to come back up before you proceed.

  3. Ensure that READYMACHINECOUNT shows the total number of worker nodes:

    oc get mcp
NAME            CONFIG                                                    UPDATED   UPDATING   DEGRADED   MACHINECOUNT   READYMACHINECOUNT   UPDATEDMACHINECOUNT   DEGRADEDMACHINECOUNT   AGE
holoscanmedia   rendered-holoscanmedia-7e2a6af0d84770766c9246dffdf55a40   False     True       False      2              0                   0                     0                      100m
master          rendered-master-fd62216b1239cd90a6a3cd7c3b9e213d          True      False      False      3              3                   3                     0                      4h
worker          rendered-worker-7e2a6af0d84770766c9246dffdf55a40          True      False      False      0              0                   0                     0                      4h