When rebuilding the kernel remotely, a kernel panic can be disastrous. GRUB provides the fallback and panic options to automatically reboot the machine if a panic occurs and to select an alternate boot entry (e.g., the previous stable kernel).
Example GRUB configuration:
default saved timeout 5 fallback 1 title Gentoo Linux root (hd0,4) kernel /boot/vmlinuz root=/dev/sda7 panic=5 nouveau.modeset=1 video=uvesafb:1280@800-32,mtrr:3,ywrap vga=0x318 savedefault fallback title Gentoo Linux Old root (hd0,4) kernel /boot/vmlinuz.old root=/dev/sda7 nouveau.modeset=1 video=uvesafb:1280@800-32,mtrr:3,ywrap vga=0x318 savedefault 0
By default, default is 0. If you select entry 0 (new kernel), the fallback will be set to 1 (the old, verified kernel) for that boot. If a kernel panic occurs (triggered by panic=5 after 5 seconds), GRUB will boot entry 1 instead.
After booting the old kernel, default will remain 0, so the next boot will attempt the new kernel again.
Note: In GRUB2, the fallback option does not exist and has been replaced with other mechanisms. Refer to the GRUB2 documentation for similar functionality.