Configure System Event Notifications
This section describes how to configure NetQ to send event notifications through a third-party-application, such as syslog, PagerDuty, Slack, email, or a generic webhook channel. NetQ can generate notifications for system and threshold-crossing events or according to a predefined set of rule keys.
You can implement a proxy server (that sits between the NetQ appliance or VM and the integration channels) that receives, processes, and distributes the notifications rather than having them sent directly to the integration channel. If you use such a proxy, you must configure NetQ with the proxy information.
Event Message Format
Messages have the following structure:
<message-type><timestamp><opid><hostname><severity><message>
Element | Description |
---|---|
message type | Category of event |
timestamp | Date and time event occurred |
opid | Identifier of the service or process that generated the event |
hostname | Hostname of network device where event occurred |
severity | Severity classification: error or info |
message | Text description of event |
For example:
To set up the integrations, you must configure NetQ with at least one channel, one rule, and one filter. To refine what messages you want to view and where to send them, you can add additional rules and filters and set thresholds on supported event types. You can also configure a proxy server to receive, process, and forward the messages. This is accomplished in the following order:
Configure Basic NetQ Event Notifications
The simplest configuration you can create is one that sends all events generated by all interfaces to a single notification application. A notification configuration must contain one channel, one rule, and one filter. Creation of the configuration follows this same path:
- Create a channel.
- Create a rule that accepts a selected set of events.
- Create a filter that associates this rule with the newly created channel.
Create a Channel
The first step is to create a Slack, PagerDuty, syslog, email, or generic channel to receive the notifications.
Expand the Menu and select Notification channels.
The Slack tab is displayed by default.
Add a channel.
- When no channels have been specified, click Add Slack channel.
- When at least one channel has been specified, click Add above the table.
Provide a unique name for the channel. Note that spaces are not allowed—use dashes or camelCase instead.
Create an incoming webhook as described in the Slack documentation. Then copy and paste it in the Webhook URL field.
(Optional) Select the toggle to send all notifications to this channel.
Click Add.
(Optional) To verify the channel configuration, click Test.
To create and verify a Slack channel, run:
netq add notification channel slack <text-channel-name> webhook <text-webhook-url> [severity info|severity error] [tag <text-slack-tag>] [default]
netq show notification channel [json]
Option | Description |
---|---|
<text-channel-name> | User-specified Slack channel name |
webhook <text-webhook-url> | WebHook URL for the desired channel. For example: https://hooks.slack.com/services/text/moretext/evenmoretext |
severity <level> | The log level, either info or error. The severity defaults to info if unspecified. |
tag <text-slack-tag> | Optional tag appended to the Slack notification to highlight particular channels or people. An @ sign must precede the tag value. For example, @netq-info. |
default | Set the channel as default and send all notifications to this channel |
The following example shows the creation of a slk-netq-events channel and verifies the configuration.
Create an incoming webhook as described in the documentation for your version of Slack.
Create the channel.
cumulus@switch:~$ netq add notification channel slack slk-netq-events webhook https://hooks.slack.com/services/text/moretext/evenmoretext Successfully added/updated channel slk-netq-events
Verify the configuration.
cumulus@switch:~$ netq show notification channel Matching config_notify records: Name Type Severity Channel Info Is Default --------------- ---------------- -------- -------------------------- ------------ slk-netq-events slack info webhook:https://hooks.s False lack.com/services/text/ moretext/evenmoretext.
Expand the Menu and select Notification channels.
Click PagerDuty.
Add a channel.
- When no channels have been specified, click Add PagerDuty channel.
- When at least one channel has been specified, click Add above the table.
Provide a unique name for the channel. Note that spaces are not allowed. Use dashes or camelCase instead.
Obtain and enter an integration key (also called a service key or routing key).
(Optional) Select the toggle to send all notifications to this channel.
Click Add.
(Optional) To verify the channel configuration, click Test.
To create and verify a PagerDuty channel, run:
netq add notification channel pagerduty <text-channel-name> integration-key <text-integration-key> [severity info|severity error] [default]
netq show notification channel [json]
Option | Description |
---|---|
<text-channel-name> | User-specified PagerDuty channel name |
integration-key <text-integration-key> | The integration key is also called the service_key or routing_key. The default is an empty string (""). |
severity <level> | The log level, either info or error. The severity defaults to info if unspecified. |
default | Set the channel as default and send all notifications to this channel |
Obtain an integration key as described in this PagerDuty support page.
Create the channel.
cumulus@switch:~$ netq add notification channel pagerduty pd-netq-events integration-key c6d666e210a8425298ef7abde0d1998 Successfully added/updated channel pd-netq-events
Verify the configuration.
cumulus@switch:~$ netq show notification channel Matching config_notify records: Name Type Severity Channel Info Is Default --------------- ---------------- ---------------- ------------------------ ------------ pd-netq-events pagerduty info integration-key: c6d666e False 210a8425298ef7abde0d1998
Expand the Menu and select Notification channels.
Click Syslog.
Add a channel.
- When no channels have been specified, click Add syslog channel.
- When at least one channel has been specified, click Add above the table.
Provide a unique name for the channel. Note that spaces are not allowed. Use dashes or camelCase instead.
Enter the IP address and port of the syslog server.
(Optional) Select the toggle to send all notifications to this channel.
Click Add.
(Optional) To verify the channel configuration, click Test.
To create and verify a syslog channel, run:
netq add notification channel syslog <text-channel-name> hostname <text-syslog-hostname> port <text-syslog-port> [severity info | severity error] [default]
netq show notification channel [json]
Option | Description |
---|---|
<text-channel-name> | User-specified syslog channel name |
hostname <text-syslog-hostname> | Hostname or IP address of the syslog server to receive notifications |
port <text-syslog-port> | Port on the syslog server to receive notifications |
severity <level> | The log level, either info or error. The severity defaults to info if unspecified. |
default | Set the channel as default and send all notifications to this channel |
The following example shows the creation of a syslog-netq-events channel and verifies the configuration.
Obtain the syslog server hostname (or IP address) and port.
Create the channel.
cumulus@switch:~$ netq add notification channel syslog syslog-netq-events hostname syslog-server port 514 Successfully added/updated channel syslog-netq-events
Verify the configuration.
cumulus@switch:~$ netq show notification channel Matching config_notify records: Name Type Severity Channel Info Is Default --------------- ---------------- -------- ---------------------- ------------ syslog-netq-eve syslog info host:syslog-server False nts port: 514
Expand the Menu and select Notification channels.
Click Email.
Add a channel.
- When no channels have been specified, click Add email channel.
- When at least one channel has been specified, click Add above the table.
Provide a unique name for the channel. Note that spaces are not allowed. Use dashes or camelCase instead.
Enter a list of recipient email addresses, separated by commas, and no spaces. For example:
user1@domain.com,user2@domain.com,user3@domain.com
(Optional) Select the toggle to send all notifications to this channel.
The first time you configure an email channel, you must also specify the SMTP server information:
- Host: hostname or IP address of the SMTP server
- Port: port of the SMTP server (typically 587)
- User ID/Password: your administrative credentials
- From: email address that indicates who sent the notifications
After the initial setup, any additional email channels you create can use this configuration, by clicking Existing.
Click Add.
(Optional) To verify the channel configuration, click Test.
To create and verify the specification of an email channel, run:
netq add notification channel email <text-channel-name> to <text-email-toids> [smtpserver <text-email-hostname>] [smtpport <text-email-port>] [login <text-email-id>] [password <text-email-password>] [severity info | severity error] [default]
netq add notification channel email <text-channel-name> to <text-email-toids> [default]
netq show notification channel [json]
The configuration is different depending on whether you are using the on-premises or cloud version of NetQ. Do not configure SMTP for cloud deployments as the NetQ cloud service uses the NetQ SMTP server to push email notifications.
For an on-premises deployment:
Set up an SMTP server. The server can be internal or public.
Create a user account (login and password) on the SMTP server. NetQ sends notifications to this address.
Create the notification channel using the following command:
netq add notification channel email <text-channel-name> to <text-email-toids> [smtpserver <text-email-hostname>] [smtpport <text-email-port>] [login <text-email-id>] [password <text-email-password>] [severity info | severity error] [default]
cumulus@switch:~$ netq add notification channel email onprem-email to netq-notifications@domain.com smtpserver smtp.domain.com smtpport 587 login smtphostlogin@domain.com password MyPassword123
Successfully added/updated channel onprem-email
Verify the configuration.
cumulus@switch:~$ netq show notification channel Matching config_notify records: Name Type Severity Channel Info Is Default --------------- ---------------- ---------------- ------------------------ --------------- onprem-email email info password: MyPassword123, False port: 587, isEncrypted: True, host: smtp.domain.com, from: smtphostlogin@doma in.com, id: smtphostlogin@domain .com, to: netq-notifications@d omain.com
For a cloud deployment:
Create the notification channel using the following command:
netq add notification channel email <text-channel-name> to <text-email-toids>
cumulus@switch:~$ netq add notification channel email cloud-email to netq-cloud-notifications@domain.com
Successfully added/updated channel cloud-email
Verify the configuration.
cumulus@switch:~$ netq show notification channel Matching config_notify records: Name Type Severity Channel Info Is Default --------------- ---------------- ---------------- ------------------------ -------------- cloud-email email info password: TEiO98BOwlekUP False TrFev2/Q==, port: 587, isEncrypted: True, host: netqsmtp.domain.com, from: netqsmtphostlogin@doma in.com, id: smtphostlogin@domain .com, to: netq-notifications@d omain.com
Click Menu, then click Notification channels.
Click Generic.
Add a channel.
- When no channels have been specified, click Add generic channel.
- When at least one channel has been specified, click Add above the table.
Provide a unique name for the channel. Note that spaces are not allowed. Use dashes or camelCase instead.
Specify a webhook URL.
Set the desired notification severity, SSL, and authentication parameters for this channel.
(Optional) Select the toggle to send all notifications to this channel.
Click Add.
(Optional) To verify the channel configuration, click Test.
To create and verify a generic channel, run:
netq add notification channel generic <text-channel-name> webhook <text-webhook-url> [severity info | severity error ] [use-ssl True | use-ssl False] [auth-type basic-auth generic-username <text-generic-username> generic-password <text-generic-password> | auth-type api-key key-name <text-api-key-name> key-value <text-api-key-value>] [default]
netq show notification channel [json]
Option | Description |
---|---|
<text-channel-name> | User-specified generic channel name |
webhook <text-webhook-url> | URL of the remote application to receive notifications |
severity <level> | The log level, either info or error. The severity defaults to info if unspecified. |
use-ssl [True | False] | Enable or disable SSL |
auth-type [basic-auth | api-key] | Set authentication parameters. Either basic-auth with generic-username and generic-password or api-key with a key-name and key-value |
default | Set the channel as default and send all notifications to this channel |
Create a Rule
The second step is to create and verify a rule that accepts a set of events. You create rules for system events using the NetQ CLI.
To create and verify a rule, run:
netq add notification rule <text-rule-name> key <text-rule-key> value <text-rule-value>
netq show notification rule [json]
Refer to the Rule Keys and Values Reference for a list of available keys and values.
To remove notification rules, run:
netq del notification rule <text-rule-name-anchor>
Create a Filter
The final step is to create a filter to tie the rule to the channel. You create filters for system events using the NetQ CLI.
To create and verify a filter, run:
netq add notification filter <text-filter-name> rule <text-rule-name-anchor> channel <text-channel-name-anchor>
netq show notification filter [json]
These examples use the channels and rules created in the previous sections. After creating this filter, NetQ will send all interface events to your designated channel.
cumulus@switch:~$ netq add notification filter notify-all-ifs rule all-interfaces channel pd-netq-events
Successfully added/updated filter notify-all-ifs
cumulus@switch:~$ netq show notification filter
Matching config_notify records:
Name Order Severity Channels Rules
--------------- ---------- ---------------- ---------------- ----------
notify-all-ifs 1 info pd-netq-events all-interfaces
cumulus@switch:~$ netq add notification filter notify-all-ifs rule all-interfaces channel slk-netq-events
Successfully added/updated filter notify-all-ifs
cumulus@switch:~$ netq show notification filter
Matching config_notify records:
Name Order Severity Channels Rules
--------------- ---------- ---------------- ---------------- ----------
notify-all-ifs 1 info slk-netq-events all-interfaces
cumulus@switch:~$ netq add notification filter notify-all-ifs rule all-interfaces channel syslog-netq-events
Successfully added/updated filter notify-all-ifs
cumulus@switch:~$ netq show notification filter
Matching config_notify records:
Name Order Severity Channels Rules
--------------- ---------- ---------------- ---------------- ----------
notify-all-ifs 1 info syslog-netq-events all-ifs
cumulus@switch:~$ netq add notification filter notify-all-ifs rule all-interfaces channel onprem-email
Successfully added/updated filter notify-all-ifs
cumulus@switch:~$ netq show notification filter
Matching config_notify records:
Name Order Severity Channels Rules
--------------- ---------- ---------------- ---------------- ----------
notify-all-ifs 1 info onprem-email all-ifs
Filter names can contain spaces, but must be enclosed with single quotes in commands. It is easier to use dashes in place of spaces or mixed case for better readability. For example, use bgpSessionChanges or BGP-session-changes or BGPsessions, instead of 'BGP Session Changes'. Filter names are also case sensitive.
As you create filters, they are added to the bottom of a list of filters. By default, NetQ processes event messages against filters starting at the top of the filter list and works its way down until it finds a match. NetQ applies the first filter that matches an event message, ignoring the other filters. Then it moves to the next event message and reruns the process, starting at the top of the list of filters. NetQ ignores events that do not match any filter.
You might have to change the order of filters in the list to ensure you capture the events you want and drop the events you do not want. This is possible using the before
or after
keywords to ensure one rule is processed before or after another.
To delete notification filters, run:
netq del notification filter <text-filter-name-anchor>
Delete a Channel
You can remove channels if they are not part of an existing notification configuration.
To remove notification channels:
Expand the Menu and select Notification channels.
Select the tab for the type of channel you want to remove.
Select one or more channels.
Click Delete.
To remove notification channels, run:
netq del notification channel <text-channel-name-anchor>
This example removes a Slack integration and verifies it is no longer in the configuration:
cumulus@switch:~$ netq del notification channel slk-netq-events
cumulus@switch:~$ netq show notification channel
Matching config_notify records:
Name Type Severity Channel Info
--------------- ---------------- ---------------- ------------------------
pd-netq-events pagerduty info integration-key: 1234567
890
Configure a Proxy Server
To send notification messages through a proxy server instead of directly to a notification channel, you configure NetQ with the hostname and optionally a port of a proxy server. If you do not specify a port, NetQ defaults to port 80. NetQ supports one proxy server. To simplify deployment, configure your proxy server before configuring channels, rules, or filters.
To configure and verify the proxy server, run:
netq add notification proxy <text-proxy-hostname> [port <text-proxy-port>]
netq show notification proxy
This example configures and verifies the proxy4 server on port 80 to act as a proxy for event notifications.
cumulus@switch:~$ netq add notification proxy proxy4
Successfully configured notifier proxy proxy4:80
cumulus@switch:~$ netq show notification proxy
Matching config_notify records:
Proxy URL Slack Enabled PagerDuty Enabled
------------------ -------------------------- ----------------------------------
proxy4:80 yes yes
You can remove the proxy server with netq del notification proxy
. This changes the NetQ behavior to send events directly to the notification channels.
Rule Keys and Values Reference
A single key-value pair comprises each rule. The key-value pair indicates what messages to include or drop from event information sent to a notification channel. You can create more than one rule for a single filter. Creating multiple rules for a given filter can provide a very defined filter. For example, you can specify rules around hostnames or interface names, enabling you to filter messages specific to those hosts or interfaces. You can only create rules after you have set up your notification channels.
NetQ includes a predefined fixed set of valid rule keys. You enter values as regular expressions, which vary according to your deployment.
Service | Rule Key | Description | Example Rule Values |
---|---|---|---|
BGP | message_type | Network protocol or service identifier | bgp |
hostname | User-defined, text-based name for a switch or host | server02, leaf11, exit01, spine-4 | |
peer | User-defined, text-based name for a peer switch or host | server4, leaf-3, exit02, spine06 | |
desc | Text description | ||
vrf | Name of VRF interface | mgmt, default | |
old_state | Previous state of the BGP service | Established, Failed | |
new_state | Current state of the BGP service | Established, Failed | |
old_last_reset_time | Previous time that BGP service was reset | Apr3, 2019, 4:17 PM | |
new_last_reset_time | Most recent time that BGP service was reset | Apr8, 2019, 11:38 AM | |
ConfigDiff | message_type | Network protocol or service identifier | configdiff |
hostname | User-defined, text-based name for a switch or host | server02, leaf11, exit01, spine-4 | |
vni | Virtual Network Instance identifier | 12, 23 | |
old_state | Previous state of the configuration file | created, modified | |
new_state | Current state of the configuration file | created, modified | |
EVPN | message_type | Network protocol or service identifier | evpn |
hostname | User-defined, text-based name for a switch or host | server02, leaf-9, exit01, spine04 | |
vni | Virtual Network Instance identifier | 12, 23 | |
old_in_kernel_state | Previous VNI state, in kernel or not | true, false | |
new_in_kernel_state | Current VNI state, in kernel or not | true, false | |
old_adv_all_vni_state | Previous VNI advertising state, advertising all or not | true, false | |
new_adv_all_vni_state | Current VNI advertising state, advertising all or not | true, false | LCM | message_type | Network protocol or service identifier | clag |
hostname | User-defined, text-based name for a switch or host | server02, leaf-9, exit01, spine04 | |
old_conflicted_bonds | Previous pair of interfaces in a conflicted bond | swp7 swp8, swp3 swp4 | |
new_conflicted_bonds | Current pair of interfaces in a conflicted bond | swp11 swp12, swp23 swp24 | |
old_state_protodownbond | Previous state of the bond | protodown, up | |
new_state_protodownbond | Current state of the bond | protodown, up | |
Link | message_type | Network protocol or service identifier | link |
hostname | User-defined, text-based name for a switch or host | server02, leaf-6, exit01, spine7 | |
ifname | Software interface name | eth0, swp53 | |
LLDP | message_type | Network protocol or service identifier | lldp |
hostname | User-defined, text-based name for a switch or host | server02, leaf41, exit01, spine-5, tor-36 | |
ifname | Software interface name | eth1, swp12 | |
old_peer_ifname | Previous software interface name | eth1, swp12, swp27 | |
new_peer_ifname | Current software interface name | eth1, swp12, swp27 | |
old_peer_hostname | Previous user-defined, text-based name for a peer switch or host | server02, leaf41, exit01, spine-5, tor-36 | |
new_peer_hostname | Current user-defined, text-based name for a peer switch or host | server02, leaf41, exit01, spine-5, tor-36 | MLAG (CLAG) | message_type | Network protocol or service identifier | clag |
hostname | User-defined, text-based name for a switch or host | server02, leaf-9, exit01, spine04 | |
old_conflicted_bonds | Previous pair of interfaces in a conflicted bond | swp7 swp8, swp3 swp4 | |
new_conflicted_bonds | Current pair of interfaces in a conflicted bond | swp11 swp12, swp23 swp24 | |
old_state_protodownbond | Previous state of the bond | protodown, up | |
new_state_protodownbond | Current state of the bond | protodown, up | |
Node | message_type | Network protocol or service identifier | node |
hostname | User-defined, text-based name for a switch or host | server02, leaf41, exit01, spine-5, tor-36 | |
ntp_state | Current state of NTP service | in sync, not sync | |
db_state | Current state of DB | Add, Update, Del, Dead | |
NTP | message_type | Network protocol or service identifier | ntp |
hostname | User-defined, text-based name for a switch or host | server02, leaf-9, exit01, spine04 | |
old_state | Previous state of service | in sync, not sync | |
new_state | Current state of service | in sync, not sync | |
Port | message_type | Network protocol or service identifier | port |
hostname | User-defined, text-based name for a switch or host | server02, leaf13, exit01, spine-8, tor-36 | |
ifname | Interface name | eth0, swp14 | |
old_speed | Previous speed rating of port | 10 G, 25 G, 40 G, unknown | |
old_transreceiver | Previous transceiver | 40G Base-CR4, 25G Base-CR | |
old_vendor_name | Previous vendor name of installed port module | Amphenol, OEM, NVIDIA, Fiberstore, Finisar | |
old_serial_number | Previous serial number of installed port module | MT1507VS05177, AVE1823402U, PTN1VH2 | |
old_supported_fec | Previous forward error correction (FEC) support status | none, Base R, RS | |
old_advertised_fec | Previous FEC advertising state | true, false, not reported | |
old_fec | Previous FEC capability | none | |
old_autoneg | Previous activation state of auto-negotiation | on, off | |
new_speed | Current speed rating of port | 10 G, 25 G, 40 G | |
new_transreceiver | Current transceiver | 40G Base-CR4, 25G Base-CR | |
new_vendor_name | Current vendor name of installed port module | Amphenol, OEM, NVIDIA, Fiberstore, Finisar | |
new_part_number | Current part number of installed port module | SFP-H10GB-CU1M, MC3309130-001, 603020003 | |
new_serial_number | Current serial number of installed port module | MT1507VS05177, AVE1823402U, PTN1VH2 | |
new_supported_fec | Current FEC support status | none, Base R, RS | |
new_advertised_fec | Current FEC advertising state | true, false | |
new_fec | Current FEC capability | none | |
new_autoneg | Current activation state of auto-negotiation | on, off | |
Sensors | sensor | Network protocol or service identifier | Fan: fan1, fan-2 Power Supply Unit: psu1, psu2 Temperature: psu1temp1, temp2 |
hostname | User-defined, text-based name for a switch or host | server02, leaf-26, exit01, spine2-4 | |
old_state | Previous state of a fan, power supply unit, or thermal sensor | Fan: ok, absent, bad PSU: ok, absent, bad Temp: ok, busted, bad, critical | |
new_state | Current state of a fan, power supply unit, or thermal sensor | Fan: ok, absent, bad PSU: ok, absent, bad Temp: ok, busted, bad, critical | |
old_s_state | Previous state of a fan or power supply unit. | Fan: up, down PSU: up, down | |
new_s_state | Current state of a fan or power supply unit. | Fan: up, down PSU: up, down | |
new_s_max | Current maximum temperature threshold value | Temp: 110 | |
new_s_crit | Current critical high temperature threshold value | Temp: 85 | |
new_s_lcrit | Current critical low temperature threshold value | Temp: -25 | |
new_s_min | Current minimum temperature threshold value | Temp: -50 | |
Services | message_type | Network protocol or service identifier | services |
hostname | User-defined, text-based name for a switch or host | server02, leaf03, exit01, spine-8 | |
name | Name of service | clagd, lldpd, ssh, ntp, netqd, netq-agent | |
old_pid | Previous process or service identifier | 12323, 52941 | |
new_pid | Current process or service identifier | 12323, 52941 | |
old_status | Previous status of service | up, down | |
new_status | Current status of service | up, down |
Examples of Advanced Notification Configurations
The following section lists examples of advanced notification configurations.
Create a Notification for BGP Events from a Selected Switch
This example creates a notification integration with a PagerDuty channel called pd-netq-events. It then creates a rule bgpHostname and a filter called 4bgpSpine for any notifications from spine-01. The result is that any info severity event messages from Spine-01 is filtered to the pd-netq-events channel.
Create a Notification for Errors on a Given EVPN VNI
This example creates a notification integration with a PagerDuty channel called pd-netq-events. It then creates a rule evpnVni and a filter called 3vni42 for any error messages from VNI 42 on the EVPN overlay network. The result is that any event messages from VNI 42 with a severity level of ‘error’ are filtered to the pd-netq-events channel.
Create a Notification for Configuration File Changes
This example creates a notification integration with a Slack channel called slk-netq-events. It then creates a rule sysconf and a filter called configChange for any configuration file update messages. The result is that any configuration update messages are filtered to the slk-netq-events channel.
Create a Notification for When a Service Goes Down
This example creates a notification integration with a Slack channel called slk-netq-events. It then creates a rule svcStatus and a filter called svcDown for any services state messages indicating a service is no longer operational. The result is that any service down messages are filtered to the slk-netq-events channel.
Create a Filter to Drop Notifications from a Given Interface
This example creates a notification integration with a Slack channel called slk-netq-events. It then creates a rule swp52 and a filter called swp52Drop that drops all notifications for events from interface swp52.
Create a Notification for a Given Device that Has a Tendency to Overheat (Using Multiple Rules)
This example creates a notification when switch leaf04 has passed over the high temperature threshold. Two rules were necessary to create this notification, one to identify the specific device and one to identify the temperature trigger. NetQ then sends the message to the pd-netq-events channel.