ClusterMinder Plugin

Changes and New Features

Bug Fixes

The ClusterMinder plugin collects telemetry data from multiple data sources and aggregates, streams and visualizes the backend data. The plugin can cluster/group aggregated data from multiple machines that allows operational anomaly and misconfiguration detection. The plugin provides Cluster-wide histograms of hardware telemetry which details compute node configuration and inventory, PCIe bus, hardware information (SN and FW version) and health alerts of all relevant devices.

The plugin can be deployed as a container and supports multiple data sources

Data Source Supported Types and Devices

SATEC PM 130,135,175

ABB vendor

The plugin can be deployed using the following methods:

1. On the UFM Appliance

2. On the UFM Software

To deploy the plugin, follow these steps:

• The plugin is included in the default plugin bundle available at NVIDIA's Licensing Portal .

• Load the downloaded image onto the UFM server. This can be done either by using the UFM GUI by navigating to the Settings -> Plugins Management tab or by loading the image via the following instructions:

  • Log in to the UFM server terminal.

  • Run:

    Copy

    Copied!

                
    
                
    docker load < <path_to_image> 
            

  • After successfully loading the plugin image, the plugin should become visible within the plugins management table within the UFM GUI. To initiate the plugin’s execution, simply right-click on the respective in the table.

After the successful deployment of the plugin, a new item is shown in the UFM side menu for the ClusterMinder plugin

Managing data sources with ClusterMinder is intuitive and efficient. Users can easily add new data sources, update existing ones, or remove those that are no longer needed. This feature ensures your data is always current and relevant.

Example of Adding Data Source

Adding hosts is done through the "Data Sources" section. To add a data source, you start by selecting the appropriate tab, clicking , and filling out the information on the endpoint. Then you must first test the endpoint and if the endpoint status is "up," the add button will become clickable, allowing you to add the data source.

Note: we support adding hosts in hostlist format for exmaple: agx[01-10].

for example:

Example of Adding a Redfish Host

After pressing the button you will be presented with a form for inputting the following fields: " BMC IP", "Protocol","Username" and "Password". after inputting the needed information, pressing the button tests the connection to the host. You will then be presented with a window notifying if the connection was successful. if successful you can click the button to add the data source.

For example:

Note: when adding multiple redfish hosts, they should all be of the same model and vendor. If this is not done it can cause issues with data collection and presentation.

Example of Adding a Switch Host

After pressing the button you will be presented with a form for inputting the following fields: "Switch IP", "Username", "Password". after inputting the needed information, pressing the button tests the connection to the host. You will then be presented with a window notifying if the connection was successful. if successful you can click the button to add the data source.

Note: when adding multiple switch hosts, they should all be of the same OS type. If this is not done it can cause issues with data collection and presentation.

Example of Adding a User Generated Data Source Type

Pressing the button in the row of data types allows the user to create custom user generated data source, you will be presented with a form for inputting the following fields: "Type", "Name". After inputting the needed information, you can click the button to add the data source.

Example of Removing Data Source Endpoint

Removing hosts is done through the "Data Sources" section. here you can right click any available host and click the remove option.

You can also use the button

to open the Remove Data Sources form that has the IP field. After choosing the appropriate IP you can click the button to remove those endpoints

Removing User Generated Data Source Type

Removing user generated data source type is done through the "Data Sources" section. here you can right click any user generated data source type and click the remove option.

Example of Updating Data Source Endpoint

Updating hosts is done through the "Data Sources" section. here you can right click any available host and click the update option.

You can also use the button to open the Update Data Sources form that thar has the IP field. After filling the appropriate fields, pressing the button tests the connection to the host. You will then be presented with a window notifying if the connection was successful. if successful you can click the button to update the endpoint.

Data Tab

The Data tab provides a comprehensive view of all your data in one place, organized in an easy-to-navigate tree or table view. Additionally, there is a group view tab that allows users to see differences in host groups, helping to identify and understand misconfigurations within their data. This feature allows users to quickly access and navigate through their data, making it easier to perform analyses and derive insights. Users can right click tree APIs to copy API path or content or add to filter preset

Redfish Data Example

Switch Data Example

Additional Features

Filter Bar

The filter allows to choose specific API based on frequency level. The search box allows to search for substring in the API or a specific data field. You can also choose to only show APIs with a minimum or maximum number of groups .

Group View

The group view inner tab allows to view endpoints divided to groups based on differences in data.

Group Differences Tab

The “Group differences” tab helps users identify and understand misconfigurations within their data. By comparing different data groups, users can easily spot discrepancies and take corrective actions. At the start of the report, there is a summary table that lists the most problematic hosts in descending order based on the number of appearances. This table provides a quick overview of the hosts that require the most attention, allowing users to prioritize their troubleshooting efforts effectively. Additionally, each table in the report (besides the summary table) includes the API that provided the data, a column for the number of hosts per group, and the fields where their values differed, ensuring transparency and traceability of the information presented.

Redfish Group Differences Example

Switch Group Differences Example

Suspected Errors Tab

The “Suspected Errors” tab provides a comprehensive report on APIs that have returned values flagged as potential issues. This report is crucial for maintaining the reliability and integrity of your data analysis. At the start of the report, a summary table lists the number of hosts with errors compared to the total number of hosts, giving users a quick snapshot of the overall health of their data environment. This summary helps prioritize troubleshooting efforts and allocate resources effectively. Each entry in the report details the host ID, the specific fields where values were problematic, and the problematic values themselves. These values are highlighted in red or orange to indicate the severity of the issue, with red denoting more critical problems and orange indicating less severe ones. This color-coding allows users to quickly assess the urgency of each issue and take appropriate corrective actions.

Redfish Suspected Errors Example

Switch Suspected Errors Example

Alerts and Alerts Rules Tab

The Alert tab displays all fields whose values meet or violate specific thresholds defined by the user. The Alert Rules tab provides an overview of all the rules the user has created. The Alert Rules tab provides an overview of all the rules the user has created. Alert condition "contains" and "not contains" support regex format. By default, an alert monitors and logs endpoint status changes (between 'Up' and 'Failed to connect' states)

Updating alerts

Right clicking any alert rule, selecting update allows to modify the following parameters:

• Field name and source path to monitor

• Threshold conditions (equals, not equals, contains, etc.)

• Alert actions (immediate, queued)

• Email notifications and frequency settings

• Minimum time between alerts

clicking "Update" will save the changes made.

Removing alerts

To remove an alert rule from the Alerts Rules tab: Right-click on the alert rule you want to delete. Select "Remove" from the context menu. The alert rule will be deleted and the table will automatically refresh

Additional Features

Historical Change log Tab

Displays fields that have had their values changed, sorted by SRC ID and API.

API Response Duration Tab

Displays the duration of each APIs data collection.

Log Tab

The log page in Cluster Minder provides a comprehensive view of all product logging activities. This feature is essential for tracking changes, monitoring system performance, and diagnosing issues.

Visualizing data is made simple with ClusterMinder’s histogram feature. Histograms are premade depending on the type of data source as well as the make and model. The histograms range from rpm speeds, voltages, wattage, amperage, temperatures (Celsius) and more. By clicking the histograms, users can filter specific hosts or values, allowing for a more targeted analysis. Additionally, using the dropdown menu, users can further filter the histogram bars to hone in on specific data subsets and attributes, enhancing their ability to derive meaningful insights.

Redfish Histogram Example

Switch Histogram Example

ClusterMinder’s telemetry page allows users to create custom graphs based on their data. Additionally, there are premade graphs available depending on the telemetry data the data source provides. This feature provides flexibility in how data is displayed and analyzed, enabling users to tailor their graphs to meet specific needs.

Redfish Telemetry Example

Switch Telemetry Example

Additional Features

Graphs Tab

Allows user to create a custom graph. To do so users should navigate to the “Graphs” tab. Here, they can press the button to initiate new graph creation. Next, users select a provider, device, and counter to add to the graph. Finally, they name the graph and complete the process by pressing the finish button. Graphs offer various options such as time frame and resolution, as well as built-in filtering by pressing the specific counter, allowing for more precise and customizable data visualization. After creating a graph and accessing it, selecting a host will highlight that specific host and display only it.

Dashboards

Dashboards in this project enable users to collect and organize specific graphs they create, grouping related visualizations within a single dashboard for easier access and comparison. This allows users to save sets of graphs that serve a common purpose and effortlessly switch between multiple dashboards to manage different groups of visualizations tailored to various needs.

Dashboard example

Annotations

The Annotation tab allows users to mark and document specific time periods in telemetry data for devices and metrics. Annotations provide context for unusual patterns, incidents, maintenance windows, or other events of interest in the monitoring data.

Users can upload a JSON file containing an array of annotation objects, where each annotation requires src_id, device, provider, metric, annotation_start_time, annotation_end_time, and notes fields, with optional start_time, end_time, and query_resolution fields for customizing the displayed data range.

Icons

- allows (for left to right) to view current dashboard, delete dashboard, rename dashboard, add dashboard, download dashboard, upload dashboard, refresh data

- allows to select specific time range (or custom) and resolution for telemetry graph

ClusterMinder’s analytics page allows users to view suspected errors and group differences reports in one aggregated location as well as viewing common suspected errors and common group differences. Analytics page also provides alerts (currently only CDU supported) for various user determined scenarios. This page provides ease of use viewing of all the text reports in a singular location as well as important user tailored alerts.