PDR Deterministic Plugin

Overview

The PDR deterministic plugin, overseen by the UFM, is a docker container that isolates malfunctioning ports, and then reinstates the repaired links to their previous condition by lifting the isolation. The PDR plugin uses a specific algorithm to isolate ports, which is based on telemetry data from the UFM Telemetry. This data includes packet drop rate, BER counter values, link down counter, and port temperature. Any decisions made by the plugin will trigger an event in the UFM for tracking purposes.

The PDR plugin performs the following tasks:

1. Collects telemetry data using UFM Dynamic Telemetry

2. Identifies potential failures based on telemetry calculations and isolates them to avert any interruption to traffic flow

3. Maintains a record of maintenance procedures that can be executed to restore an isolated link

4. After performing the required maintenance, the system verifies if the ports can be de-isolated and restored to operational status (brought back online).

The plugin can simulate port isolation without actually executing it for the purpose of analyzing the algorithm's performance and decision-making process in order to make future adjustments. This behavior is achieved through the implementation of a "dry_run" flag that changes the plugin's behavior to solely record its port "isolation" decisions in the log, rather than invoking the port isolation API. All decisions will be recorded in the plugin's log.

To deploy the plugin, follow these steps:

1. Download the ufm-plugin-pdr_deterministic-image from the NVIDIA License Portal (NLP).

2. Load the downloaded image onto the UFM server. This can be done either by using the UFM GUI by navigating to the Settings -> Plugins Management tab or by loading the image via the following instructions:

   1. Log in to the UFM server terminal.

   2. Run:

      Copy

      Copied!

                  
      
                  
      docker load -I <path_to_image>
              

   3. After successfully loading the plugin image, the plugin should become visible in the plugin management table within the UFM GUI. To initiate the plugin’s execution, simply right-click on the respective in the table.

      

      
      

NDR Link Validation Procedure

Verify ports that are in INIT, ARMED or ACTIVE states only. Track the SymbolErrorsExt of every such link for at least 120m. If polling period is Pm, need to keep N=(125+Pm+1)/Pm samples. Also, two delta samples are computed: number of samples covering 12 minutes S12m = (12 + Pm + 1)/Pm and S125m = (125 + Pm + 1)/Pm. 12m_thd = LinkBW_Gbps*1e9*12*60*1e-14 (2.88 for NDR) and
125m_thd = LinkBW_Gbps*1e9*125*60*1e-15 (3 for NDR).

Check the following conditions for every port in the given set:

1. If the Delta(LinkDownedCounterExt) port is > 0 and the Delta(LinkDownedCounterExt) remote port is > 0, add it to the list of bad_ports. This condition should be ignored if the --no_down_count flag is provided.

2. If the symbol_errors[now_idx] – symbol_errors[now_idx – S12m] is > 12m_thd, add the link to the list of bad_ports, and continue with next link.

3. If the symbol_errors[now_idx] – symbol_errors[now_idx – S125m] is > 125m_thd, add the link to the list of bad_ports, continue with next linkPacket drop rate criteria

When packet drops due to the link health are detected, isolate the problematic link. To achieve this, a target packet_drop/packet_delivered ratio can be employed to include TX ports with a receiver exceeding this threshold in the list of bad_ports. However, the drawback of this method is that such links may fluctuate between bad/good state since their BER may be normal. Therefore, it is advisable to track their statistics over time and refrain from reintegrating them after their second or third de-isolation.

Return to Service

Continuously monitoring the collection of bad_ports, the plugin persistently assess their Bit Error Rate (BER) and determines their reintegration when they successfully pass the 126m test without errors.

Configuration

The following parameters are configurable via the plugin’s configuration file. (pdr_deterministic.conf)

Calculating BER Counters

For calculating BER counters, the plugin extracts the maximum window it needs to wait for calculating the BER value, using the following formula:

Example:

BER counters are calculated with the following formula:

1. Install UFM with the latest software version.

2. Run:

   Copy

   Copied!

               
   
               
   /etc/init.d/ufmd start
           

3. To get PDR plugin image, please contact the NVIDIA Support team. After that, load the plugin using this command:

   When working with UFM in HA mode, load the plugin on the standby node.

   Copy

   Copied!

               
   
               
   ufmapl [ mgmt-sa ] (config) # docker load ufm-plugin-pdr-determinitic.tar
           

4. Run the following command. Add -p pdr-determinitic to enable the plugin:

   Copy

   Copied!

               
   
               
   /opt/ufm/scripts/manage_ufm_plugins.sh add -p pdr-determinitic
           

5. Ensure that the plugin is up and running. Run: /opt/ufm/scripts/manage_ufm_plugins.sh show