Kernel Panic!!!

I’ve been using Fedora for a couple years now and only had 2 kernel panics, both after updating the kernel, this however is not the only reason why that would happen and it might happen to you as well.

What is a Kernel Panic

From Wikipedia (the ultimate source of truth):

A kernel panic “is a safety measure taken by an operating system’s kernel upon detecting an internal fatal error in which either it is unable to safely recover or continuing to run the system would have a higher risk of major data loss. The term is largely specific to Unix and Unix-like systems. The equivalent on Microsoft Windows operating systems is a stop error, often called a “blue screen of death”.

Common Causes

• Installation of unstable or corrupt kernels.
• Faulty hardware, such as RAM or storage devices.
• Corrupted or missing initramfs.
• Incompatible or faulty kernel modules (drivers) - often NVIDIA.
• Memory overflow issues, causing access violations.

Symptoms

A kernel panic occurs when the Linux kernel encounters a fatal error it cannot recover from and can’t load properly, causing:

• Complete system crash: Your keyboard, mouse, everything becomes frozen.
• Error message: A “kernel panic” message appears on the screen, often with some technical details or explanations about the issue.
• Fail to boot: The system fails to boot, either triggering an automatic reboot or remains frozen indefinitely.

You might see something like this at boot:

Common error message:

error: ../../grub-core/fs/fshelp.c:257:file
/initramfs-<kernel_version>.img
not found
Press any key to continue...

KERNEL PANIC!
Please reboot your computer.
VFS: Unable to mount root fs on unknown-block(0,0)

Troubleshooting

Boot with a Previous Stable Kernel

Background: Kernel Retention (Optional)

Most Linux distributions (such as Fedora/RHEL), maintain a history of installed kernels to enable recovery from failed kernel updates.

Fedora Kernel Retention History

• Standard Package Behavior: When updating most packages with dnf or yum, the package manager replaces the current version of the package with the new one, removing the old version entirely.
• Kernel Update Behavior: Kernel updates are different. The dnf update kernel command installs the new kernel alongside existing kernel versions rather than replacing them.
• Default Retention Policy: By default, dnf and yum retain the last 3 installed kernels in the /boot/ directory.

Modify Kernel Retention

• Edit /etc/dnf/dnf.conf.
• Adjust the number of retained kernels in installonly_limit=<number>.

# Edit /etc/dnf/dnf.conf with your favorite text editor
# Example /etc/dnf/dnf.conf
[main]
...
# Keep the last 5 kernels installed with dnf/yum
installonly_limit=5

Boot Procedure

1. Restart your system.
2. Access the GRUB boot menu (typically by holding Shift or pressing Esc during boot.
3. Use the up and down arrow keys to select a previous working stable kernel from the list.
4. Boot into the selected stable kernel.

Check Missing initramfs (Optional)

After successfully booting into a previous stable kernel, you can confirm if the faulty kernel is missing its initramfs file, which is a common cause of kernel panics.

Check Current Kernel Version

uname -r

# Example output (stable kernel): 6.12.0-55.12.1.el10_0.x86_64

Inspect Boot Directory

• Check the contents of the /boot/ directory to identify any missing initramfs files.
• Each installed kernel file (identified by vmlinuz-<version>) should have a corresponding initramfs-<kernel_version>.img file. If the faulty kernel lacks its initramfs file, this confirms the root cause of the boot failure.

cd /boot
ls -l

Regenerate initramfs (Fix)

If the initramfs file for a kernel is missing or corrupted, you must regenerate it to fix the kernel panic. After logging into the previous stable kernel, use the dracut command to rebuild the initramfs.

Regenerate a Specific Kernel Initramfs

• --force (-f): Overwrites any existing initramfs file for the specified kernel version, ensures a fresh regeneration even if a corrupted file is present.
• --kver <kernel_version>: Specifies the kernel version for which to generate the initramfs. The version must match the kernel filename in /boot/ (without the vmlinuz- prefix).

# Syntax
sudo dracut --force --kver <kernel_version>

# Example:
sudo dracut --force --kver 6.12.0-55.27.1.el10_0.x86_64

# Note: There is no need to regenerate `initramfs` for all kernels, but this single command is easy to execute and will do the job
sudo dracut --force --regenerate-all

Finally, reboot and you should be able to login using the faulty kernel.

reboot

Workarounds

If the kernel panic keeps occurring no matter what you do, you might want to implement a temporary workaround while a fix is implemented.

Set a Previous Kernel as Default

You can configure an older, stable kernel as the default boot option. This approach is useful when you need immediate system access while waiting for a kernel update that resolves the issue or while finding other solutions.

Show The Current Default Kernel

• First, identify which kernel is currently configured to load by default at boot time.

# Option 1: Show the default boot kernel (likely the faulty kernel causing the panic)
sudo grubby --info=DEFAULT

# Example output (panic): /boot/vmlinuz-6.27.0-55.12.1.el10_0.x86_64

Identify Target Stable Kernel Version

• Option 1: Show all available kernel entries in the system to identify which version you want to set as boot time default.
• Option 2: Show the kernel currently in use (most likely a stable kernel).

# Option 1: Show GRUB configuration for all installed kernel entries
sudo grubby --info=ALL

# Option 2: Show the currently running stable kernel
uname -r

Set a Stable Older Kernel as Default

• Modify GRUB to boot to the specified stable kernel by default.

# Syntax
sudo grubby --set-default /boot/vmlinuz-<kernel_version>

# Example
sudo grubby --set-default /boot/vmlinuz-6.12.0-55.12.1.el10_0.x86_64

Verify the Change

• The command should return the path of the stable kernel you just set.

sudo grubby --default-kernel
# Output after the change: /boot/vmlinuz-6.12.0-55.12.1.el10_0.x86_64

Temporarily Exclude Kernel Updates from DNF/YUM

After selecting a stable kernel as the default, you can prevent dnf or yum from automatically updating kernel packages. This ensures the system continues using the working kernel until the issue is resolved.

Procedure

• Edit /etc/dnf/dnf.conf.
• Edit DNF configuration to exclude kernel packages from updates.

# Edit /etc/dnf/dnf.conf with your favorite text editor
# Example /etc/dnf/dnf.conf
[main]
...
# Excludes all kernel packages from dnf update
exclude=kernel*

Note: This is a temporary measure only. Kernel updates often contain critical security patches and bug fixes. Excluding kernel updates for extended periods can leave your system vulnerable.

Screenshots are from a Rocky Linux VM I forced a kernel panic on, however the content is applicable to Rocky Linux, Fedora, RHEL, CentOS distributions.