Use netconsole with syslog on Cumulus Linux Switches
netconsole
is a Linux feature that allows you to redirect kernel messages output from dmesg
to a location across the network using UDP. You can capture and store these messages on a syslog
server to investigate issues on the Cumulus Linux switch that is generating the dmesg
output. This is useful where a physical console is not connected and you need to debug kernel events, such as system crashes and unexpected reboots.
netconsole
is not a replacement for a physical console. It does not provide an interactive console to the switch; it is a remote logging service only. netconsole
is also not available until the network has initialized on boot. Log data from early in the boot cycle is not captured. Use netconsole
whenever a physical console is not available to log data.
Configure the netconsole Module
You must reboot the switch at the end of this process to apply the changes.
To configure the netconsole
module on your Cumulus Linux switch:
Set up the
netconsole
kernel module to load on boot:cumulus@switch:~$ echo netconsole | sudo tee /etc/modules-load.d/netconsole.conf
Configure the
netconsole
kernel module options to point to yoursyslog
server.cumulus@switch:~$ echo 'options netconsole netconsole=[...]' | sudo tee /etc/modprobe.d/netconsole.conf
In the above command, replace
[...]
with the desired configuration. The format for the options is as follows (with default values in parentheses):netconsole=[+][src-port]@[src-ip]/[<dev>],[tgt-port]@<tgt-ip>/[tgt-macaddr] where + if present, enable extended console support src-port source UDP port (6665) src-ip source IP address (<dev> interface address) dev network interface (eth0) tgt-port UDP port of the syslog server (6666) tgt-ip IP address of the syslog server tgt-macaddr Ethernet MAC address of the next hop to the syslog server (broadcast)
- The
netconsole
module requires only thetgt-ip
parameter; other parameters use their default value if unspecified. - Because the
netconsole
module might get loaded before you configuredev
, you must specifysrc-ip
. - If the
syslog
server is not on the same Ethernet segment as the source device, you must specifytgt-macaddr
. - It is more efficient to specify
tgt-macaddr
than to use the default, which is an Ethernet broadcast.
To determine the values
dev
andtgt-macaddr
, use the following procedure. When running the commands, replace the values between angle brackets (< >) to match your configuration.Use
ip route get
to determine the interface and IP address of the next hop to thesyslog
server.If the
syslog
server is reachable through a front port, run:cumulus@switch:~$ ip route get <tgt-ip>
If the
syslog
server is reachable through the management port (mgmt VRF), run:cumulus@switch:~$ ip route get <tgt-ip> vrf mgmt
Look at the output of the
ip route get
command. If it is in the following format (without avia
keyword), thesyslog
server is on the same Ethernet segment. The value to use for thedev
parameter is thedev
value reported in the output,eth0
in this example. The nexthop ip is the first field.10.230.130.20 dev eth0 table mgmt src 10.230.130.211 uid 1000
If the output of the
ip route get
command is in the following format (with avia
keyword), you reached thesyslog
server through a gateway. The value to use for thedev
parameter is thedev
value reported in the output,eth0
in this example. The nexthop ip is thevia
value, 10.230.130.1 in this example.10.230.15.31 via 10.230.130.1 dev eth0 table mgmt src 10.230.130.211 uid 1000
Use the
arping
command to determine the next hop MAC address:cumulus@switch:~$ sudo arping -i <dev> -c1 -r <nexthop ip>
The value to use for the
tgt-macaddr
parameter is the output of the previous command.For example, to configure a switch with
netconsole
logging to asyslog
server reachable at IP address 10.230.15.31 and port 514, run the following commands:cumulus@switch:~$ ip route get 10.230.15.31 vrf mgmt 10.230.15.31 via 10.230.130.1 dev eth0 table mgmt src 10.230.130.211 uid 0 cache cumulus@switch:~$ sudo arping -i eth0 -c1 -r 10.230.130.1 d8:c4:97:b5:be:b7 cumulus@switch:~$ ip -4 addr show dev eth0 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master mgmt state UP group default qlen 1000 inet 10.230.130.211/24 brd 10.230.130.255 scope global dynamic eth0 valid_lft 6855sec preferred_lft 6855sec cumulus@switch:~$ echo netconsole > /etc/modules-load.d/netconsole.conf cumulus@switch:~$ echo 'options netconsole netconsole=@10.230.130.211/eth0,514@10.230.15.31/d8:c4:97:b5:be:b7' > /etc/modprobe.d/netconsole.conf cumulus@switch:~$ sudo reboot
- The
You can increase or decrease the amount of data you want to log.
- To increase the amount of data the kernel logs (see Introduction to the Linux kernel log levels at linuxconfig.org), adjust the log level. By default, a Cumulus Linux switch logs kernel data at level 3 (KERN_ERR). It might be useful to log all the data when trying to debug an issue. To do this, increase the kernel
printk
value to 7 in the/etc/systctl.d/99-sysctl.conf
file:
$ echo 'kernel.printk = 7 4 1 7' | sudo tee -a /etc/sysctl.d/99-sysctl.conf
To limit the data to just kernel panic logs, set the kernel module option
oops_only
to 1 by appendingoops_only=1
to theoptions netconsole
line you used in/etc/modprobe.d/netconsole.conf
.cumulus@switch:~$ echo 'options netconsole netconsole=@10.230.130.211/eth0,514@10.230.15.31/d8:c4:97:b5:be:b7 oops_only=1' | sudo tee /etc/modprobe.d/netconsole.conf
- To increase the amount of data the kernel logs (see Introduction to the Linux kernel log levels at linuxconfig.org), adjust the log level. By default, a Cumulus Linux switch logs kernel data at level 3 (KERN_ERR). It might be useful to log all the data when trying to debug an issue. To do this, increase the kernel
Reboot the switch. The boot sequence applies the settings.
cumulus@switch:~$ sudo reboot
Create a Running Configuration
The following procedure only impacts the running kernel (otherwise known as a non-persistent configuration) on the switch. After the switch reboots, you lose these settings.
To create a running configuration on a Cumulus Linux switch:
Increase the kernel logging level (optional).
cumulus@switch:~$ sudo dmesg -n 7
Load the
netconsole
kernel module with the appropriate options.If the
syslog
server is reachable through the management VRF, when loading thenetconsole
module at runtime, <dev> must be the name of the management VRF interface, mgmt.cumulus@switch:~$ sudo modprobe netconsole netconsole=@10.20.30.40/eth0,6666@10.20.30.255/00:22:33:aa:bb:cc
Using the same configuration as above:
cumulus@switch:~$ sudo modprobe netconsole netconsole=@10.230.130.211/mgmt,514@10.230.15.31/d8:c4:97:b5:be:b7
To use the
oops_only
setting, append this option to themodprobe
command:cumulus@switch:~$ sudo modprobe netconsole netconsole=@10.230.130.211/mgmt,514@10.230.15.31/d8:c4:97:b5:be:b7 oops_only=1
Configure an rsyslog Server to Receive Console Log Data
The following steps show how to configure an rsyslog
server to receive UDP traffic on port 6666 from two devices and create separate log files for each. You can add this to your existing rsyslog
configuration.
You must be the root (super) user on your server to perform these steps.
Create a specific configuration file with your favorite editor:
cumulus@switch:~$ sudo vi /etc/rsyslog.d/remote-netconsole.conf
Add the following content to the file. Change the IP addresses to match the IP addresses of your switches and the appropriate destination log files.
$ModLoad imudp $RuleSet remote # Modify the following template according to the devices on which you want to # store logs. Change the IP address and subdirectory name on each # line. Add or remove "else if" lines according to the number of your # devices. if $fromhost-ip=='10.20.30.40' then /var/log/remote/spineswitch1/console.log else if $fromhost-ip=='10.20.30.41' then /var/log/remote/leafswitch1/console.log else if $fromhost-ip=='10.20.30.42' then /var/log/remote/leafswitch2/console.log else /var/log/remote/other/console.log & stop $InputUDPServerBindRuleset remote $UDPServerRun 6666 $RuleSet RSYSLOG_DefaultRuleset
Create a directory to store the log files. The following example creates a directory called
/var/log/remote
.# mkdir /var/log/remote
Restart
rsyslog
.# systemctl restart rsyslog.service
Test the Setup
You can test this setup in one of two ways:
- Append data to the kernel log
- Intentionally crash the switch (which causes a catastrophic failure of the switch)
Append Data to the Kernel Log
To create a new kernel log message and verify that the syslog
server recorded it, run the following command on the switch configured with netconsole
:
cumulus@switch:~$ echo "<0>test message $(date +%s)" | sudo tee /dev/kmsg
Confirm that the same message output by this command is also recorded on the syslog
server.
Crash a Switch
This causes a catastrophic failure of the switch and results in an immediate reboot. Ensure your network is ready for this to occur and you understand the consequences.
To invoke a kernel panic to test the process, log in to the switch you want to crash and run the following command:
cumulus@switch:~$ echo c | sudo tee /proc/sysrq-trigger
If the process is working correctly, you see log data sent to the rsyslog
server.
Log File Sample Output
Here is some sample output from the rsyslog
server:
May 12 17:13:59 leafswitch1.network.com [17593.272492] sysrq: SysRq :
May 12 17:13:59 Trigger a crash
May 12 17:13:59 leafswitch1.network.com [17593.277181] BUG: unable to handle kernel
May 12 17:13:59 NULL pointer dereference
May 12 17:13:59 leafswitch1.network.com at (null)
May 12 17:13:59 leafswitch1.network.com [17593.285951] IP:
May 12 17:13:59 leafswitch1.network.com [<ffffffff81496256>] sysrq_handle_crash+0x16/0x20
May 12 17:13:59 leafswitch1.network.com [17593.292773] PGD 4cb06067
May 12 17:13:59 PUD 4ca44067
May 12 17:13:59 PMD 0
May 12 17:13:59 leafswitch1.network.com
May 12 17:13:59 leafswitch1.network.com [17593.297566] Oops: 0002 [#1]
May 12 17:13:59 SMP
...