[Linux] How to configure Kdump

Hi all,

in this tutorial we are going to configure Kdump on a Red Hat Linux 7 server. Kdump is a useful utility that dumps all the memory when the server crashes.

1 – Kernel configuration

First of all, Kdump requires the following package to be installed:

yum install kexec-tools

Next, allocate the memory for the crash kernel that is started after a failure/crash and that has to dump the memory. To to this, modify in the grub file setting the crashkernel memory size in the GRUB_CMDLINE_LINUX entry and then regenerate the configuration:

vi grub

...
GRUB_CMDLINE_LINUX="rd.lvm.lv=rhel/root crashkernel=512M@64M rd.lvm.lv=rhel/swap vconsole.font=latarcyrheb-sun16 vconsole.keymap=us rhgb quiet nomodeset"
...

#for legacy BIOS systems
grub2-mkconfig -o /boot/grub2/grub.cfg

#for modern UEFI systems
grub2-mkconfig -o /boot/efi/EFI/redhat/grub.cfg

Note that for systems with more than 128 GB of memory, the recommended setting is crashkernel=512M@64M.

Then, issue a reboot for changes to take effect.

2 – Kdump configuration

Kdump could potentially dump the entire content of memory, so it might require a very large filesystem. Let’s start from the following situation:

#about 750GB of memory
free -h
              total        used        free      shared  buff/cache   available
Mem:           755G        598G        136G        5.5G         19G        136G
Swap:           23G          0B         23G

#about 1TB of free space in the volume group
vgs
  VG      #PV #LV #SN Attr   VSize VFree
  VGExaDb   2   6   0 wz--n- 1.63t <995.08g

So, in the worst case a 850GB filesystem is required to dump all the memory. Let’s proceed to create it:

#creation of the logical volume
lvcreate -L 850G -n LVKDUMP1 VGExaDb

#create an ext4 filesystem on the logical volume
mkfs -t ext4 /dev/VGExaDb/LVKDUMP1

#create a label for the logical volume
e2label /dev/mapper/VGExaDb-LVKDUMP1 KDUMP

#check that the label has been successfully created
ls -latr /dev/disk/by-label/
total 0
drwxr-xr-x 8 root root 160 Jun  6 17:17 ..
...
lrwxrwxrwx 1 root root  10 Jun  6 17:30 KDUMP -> ../../dm-5

NOTE: It is important to create the label, because it will be used in kdump configuration file. Logical volume names are not accepted.

NOTE: Creating a dedicated logical volume is useful because if there is the need to take back the allocated space, it can easily be done with LVM.

Next, create the directory and mount the logical volume on it. Edit the /etc/fstab file to make configuration persistent after a reboot.

mkdir /kdump
mount /dev/VGExaDb/LVKDUMP1 /kdump

mkdir /kdump/crashfiles #directory where dumps will be actually written

vi /etc/fstab
...
/dev/VGExaDb/LVKDUMP1             /kdump                       ext4    defaults        1 1

Next, modify the kdump configuration file (the default is /etc/kdump.conf):

#add or edit the follwing lines
ext4 LABEL=KDUMP
path crashfiles

Finally, start the kdump daemon:

systemctl enable kdump
systemctl start kdump
systemctl status kdump

3 – Test – optional

To test if it is working, reboot the system and check if kdump automatically starts up.

systemctl is-active kdump

Then, issue the following commands to force a kernel crash:

echo 1 > /proc/sys/kernel/sysrq
echo c > /proc/sysrq-trigger

Once the system is up again, vmcore files should be located under /kdump/crashfiles directory.

That’s all, see you in the next tutorial!