Can't boot UEFI VM after migration - grub configuration problem - #160
Can't boot UEFI VM after migration - grub configuration problem#160sparimi-rh wants to merge 1 commit into
Conversation
UEFI boot mode based Linux VM, which reference /boot and /boot/efi partitions by device files (e.g. /dev/sdaX) in /etc/fstab, fail to boot post migration as virt-v2v re-writes the grub boot loader if it finds references to device files instead of disk UUIDs. The libguestfs appliance is configured to use BIOS boot mode. This causes the target VM's grub to be configured in BIOS boot mode. The target VM fails to boot in Qemu/KVM environment. The solution is to replace the BIOS boot mode specific grub commands which loads Linux kernel and init RAM disk, with UEFI boot mode specific commands. This is done only if source VM is in UEFI mode. Fixes: https://redhat.atlassian.net/browse/RHEL-173538 Signed-off-by: Srihari Parimi <sparimi@redhat.com>
| if g#exists grub_cfg then ( | ||
| try | ||
| ignore (g#sh (sprintf "sed -i 's/\\blinux16\\b/linuxefi/g; | ||
| s/\\binitrd16\\b/initrdefi/g' %s" |
There was a problem hiding this comment.
We wouldn't normally want to run commands out of the guest like this. Augeas lets us programatically edit files in the guest which is considerably safer.
A bigger question that isn't answered is why do we need to change these linux16 commands? What is that doing exactly?
Also the commit message doesn't really explain anything, at least not in terms that I can understand. An example would go a long way.
There was a problem hiding this comment.
Hi @rwmjones, Here is an explanation of what is happening and significance of linux16/initrd16 in failure case
- Guest VM is configured with EFI boot mode - Guest VM would have /sys/firmware/efi via sys FS mount
- Supermin appliance is configured in Legacy BIOS boot mode (NO /sys/firmware/efi)
- The /etc/fstab modified to use /dev/sdaX for mount points {/boot, /boot/efi}, instead of the normal mkfs created UUID reference. This was done in the source VM in the VMware
- supermin appliance conversion reads the /etc/fstab,
- if it finds UUIDs, it does NOT invoke grub configuration change
- if it finds /dev/sdaX instead, it decides that there is a need to re-write the grub configuration. These device files need to be replaced with virtio block storage files.
- For regenerating the Grub configuration,
- Guest VM's (storage mounted) grub configuration scripts (RHEL 7.9 /etc/grub.d/10_linux) are used
- Supermin's Legacy boot mode becomes and input - this ends up generating linux16, initrd16 - even though the Guest VM is EFI based.
- When the system boots, UEFI firmware fails to recognize the {linux16, initrd16}
There was a problem hiding this comment.
- The /etc/fstab modified to use /dev/sdaX for mount points {/boot, /boot/efi}, instead of the normal mkfs created UUID reference. This was done in the source VM in the VMware
Was this change made after the VM was created (and if so, why?)
As an aside we should really be rewriting any /dev/sdX references in /etc/fstab as UUIDs, at least for all modern Linuxes that support it. We've had other issues with leaving device references there in the past that would be solved if they were UUIDs.
I don't understand points 5-8 TBH.
Probably best to really go right back to basics and simplify everything. What does linux16 etc mean? What do the bootloader files look like before conversion? What commands are run now and what do they do? Why is what they are doing now wrong? What should we be doing instead?
There was a problem hiding this comment.
Was this change made after the VM was created (and if so, why?)
Yes this change was made after the VM was created (source VM in VMware) to reproduce the problem described in the Jira issue.
As an aside we should really be rewriting any /dev/sdX references in /etc/fstab as UUIDs, at least for all modern Linuxes that support it. We've had other issues with leaving device references there in the past that would be solved if they were UUIDs.
The first idea that occurred to me was replace the /dev/sdX references with UUID. But this idea of replacing GRUB strings (linux16, initrd16) with (linuxefi, initrdefi) came from problem description in the Jira. Now as you say - it is better t o use UUIDs for future proofing. I will work upon that
Probably best to really go right back to basics and simplify everything. What does linux16 etc mean? What do the bootloader files look like before conversion? What commands are run now and what do they do? Why is what they are doing now wrong? What should we be doing instead?
Let me get back on this
There was a problem hiding this comment.
I really think using UUIDs is going to be better, assuming that this would fix the problem as well.
Using /dev/ devices in /etc/fstab isn't stable since this change was made to Linux: https://bugzilla.redhat.com/show_bug.cgi?id=1804207#c0 This change to Linux (probably) doesn't affect RHEL 7 specifically, but it does affect later versions, and using UUIDs has been suggested as the solution for other problems in the past.
There was a problem hiding this comment.
@rwmjones, Here are the answers to the questions - for sake of readability, I split your questions into multiple lines
What does linux16 etc mean?
- linux16, initrd16 seems to be specific to RHEL7 and may be older versions. Generated by RHEL7 /etc/grub/10_linux configuration script if it DOES NOT detect the system to be configured in UEFI boot mode (Absence of /sys/firmware/efi)
- It is supported on Legacy BIOS systems and these strings are GRUB commands triggering a specific HW boot protocol
What do the bootloader files look like before conversion?
- Relevant Snippet of bootloader file /boot/efi/EFI/redhat/grub.cfg in the Source VM. It shows use linuxefi, initrdefi which get generated for systems which are in UEFI bootmode. Point to the kernel and initramfs to load
### BEGIN /etc/grub.d/10_linux ###
menuentry 'Red Hat Enterprise Linux Server (3.10.0-1160.el7.x86_64) 7.9 (Maipo)' --class red --class gnu-linux --class gnu --class os --unrestricted $menuentry_id_option 'gnulinux-3.10.0-1160.el7.x86_64-advanced-6a608d01-5254-46e5-85ff-d71679eac669' {
load_video
...
...
linuxefi /vmlinuz-3.10.0-1160.el7.x86_64 root=/dev/mapper/rhel-root ro crashkernel=auto rd.lvm.lv=rhel/root rd.lvm.lv=rhel/swap rhgb quiet
initrdefi /initramfs-3.10.0-1160.el7.x86_64.img
}
- Post conversion too, the GRUB configuration file MUST have the same. However,in RHEL 7 case, linux16, initrd16 are generated because the Guest VM's (RHEL 7) GRUB config script /etc/grub.d/10_linux is running inside libguestfs appliance that is based on Legacy BIOS.
What commands are run now and what do they do?
- virt-v2v runs grub2-mkconfig under the following conditions (source file convert/convert_linux.ml)
- if contents of /etc/fstab entries which had a change from /dev/sd* to corresponding Virtio storage devices /dev/vd*.
- if GRUB_CMDLINE_LINUX has "resume=" (device or part used for hibernation) was modified.
- /etc/crypttab - replaces occurrence of /dev/sd* with corresponding UUIDs. This was implemented for sles12sp5
Why is what they are doing now wrong?
- Change the /dev/sd* to /dev/vd* have potential of causing disk access problems, invalid boot devices etc.
- Replacement with UUIDs NOT being done in /etc/fstab. This can cause problems when the migrated VM boots.
- GRUB_CMDLINE_LINUX may have root= having /dev/sd* device. So this can cause boot crash.
What should we be doing instead?
- Replacement with UUIDs should also be carried out in /etc/fstab entries too
- In /etc/default/grub, for GRUB_CMDLINE_LINUX, replace any occurrence of root=/dev/sd* with UUID
- Finally if GRUB config was indeed executed, replace linux16, initrd16 with linuxefi, initrdefi
|
I'm going to speak clearly now so you understand my position here. I can only review what's in the upstream pull request, and currently it's the same as it always was. We can't run commands like This change will be run on 1000s of VMs per day, across hundreds of customers, on critical workloads of all kinds (some life safety critical) and it absolutely cannot mess up. It must do the minimum thing possible that fixes the problem. As an example, if the file doesn't need to be changed (eg there are no We don't have good QE still, for reasons outside our control, so everyone needs to clearly understand what the change does and why. And I still do not understand this change because it hasn't been explained clearly to me. I can read code myself, I don't need someone to explain what code does. I need to know exactly why this is being done. So, no. |
2 participants
UEFI boot mode based Linux VM, which reference /boot and /boot/efi partitions by device files (e.g. /dev/sdaX) in /etc/fstab, fail to boot post migration as virt-v2v re-writes the grub boot loader if it finds references to device files instead of disk UUIDs. The libguestfs appliance is configured to use BIOS boot mode. This causes the target VM's grub to be configured in BIOS boot mode. The target VM fails to boot in Qemu/KVM environment. The solution is to replace the BIOS boot mode specific grub commands which loads Linux kernel and init RAM disk, with UEFI boot mode specific commands. This is done only if source VM is in UEFI mode.
Fixes: https://redhat.atlassian.net/browse/RHEL-173538