Bsrgvty

I have mdadm raid1 for my boot/root drive.
The disks were partitioned as MBR and I can boot Ubuntu 18.04

One of the drives failed and I replaced it with a 4TB drive and would like to switch to GPT so I can grow the array later.

The new drive is partitioned GPT:

Number Start End Size File system Name Flags
 1 1049kB 2097kB 1049kB BIOS boot partition bios_grub
 2 2097kB 4001GB 4001GB Linux RAID raid

The old drive is still MBR:

Number Start End Size Type File system Flags
 1 1049kB 1000GB 1000GB primary boot, raid

I ran grub-install on the new GPT drive and it finished without errors but I could not boot from the drive.

Next I ran boot-repair and was able to bios boot the new GPT drive, but only when the mirror is present.

When I remove the MBR disk and boot from the GPT disk, I get the same grub rescue mode prompt and it can't find my boot/root lvms. But running ls from rescue lists other raid lvms on differnt mbr disks.

My grub.cfg has:
set root='lvmid/xxxx'

lvmid/xxx is the one grub can't find. The lvmid is correct for /dev/mapper/boot

blkid output:

/dev/mapper/bootdisk-boot: UUID="f13dfe22-a31e-413e-a587-af3f68b82913" TYPE="ext4"
/dev/mapper/bootdisk-data: UUID="702554d1-1113-4aba-bdc0-5db182287343" TYPE="ext4"
/dev/mapper/bootdisk-root: UUID="5b4bc8f5-dc1c-4850-9404-22be00669e23" TYPE="ext4"
/dev/mapper/datavg-datalv: UUID="b6c2eeb5-e868-4b93-89f8-d056f4a43202" TYPE="ext4"
/dev/sda1: PARTLABEL="BIOS boot partition" PARTUUID="a513db1e-893a-4e58-b31f-b848b7adec09"
/dev/sda2: UUID="96535c2c-f839-8d5d-f055-7396b19a6914" UUID_SUB="9a164f93-31ab-e8fc-091b-f95e91e2dfa1" LABEL="bootdisknew:0" TYPE="linux_raid_member" PARTLABEL="Linux RAID" PARTUUID="387427c6-8e47-4c39-8d98-fc1a096164f1"
/dev/sdc1: UUID="96535c2c-f839-8d5d-f055-7396b19a6914" UUID_SUB="2fddbf50-6d71-8724-c8e7-afa45ff686ea" LABEL="bootdisknew:0" TYPE="linux_raid_member" PARTUUID="a5d01761-01"

FIXED. Good grief what a pain.

Short answer: fixed with boot-repair and selecting ATA disk support

Long answer: replacing/upgrading software RAID1 to larger drives with GPT is doable but tricky.

First step: make a bootable USB for your system. Not a rescue/livecd - you may want to have one anyway but make sure your own system is bootable from USB. SystemrescueCD booted my system and somehow degraded a different mirror so be careful.

Optional: I created a virtualbox mock up of my 18.0.4 system with the same MBR RAID1/LVM to test everything. I confirmed some problems/issues with grub-install so it was worth it.

Bootable USB: I used YUMI/Multiboot and copied my kernel and initrd to the USB /boot and edited /multiboot/syslinux.cfg to add:

label Mysystem
menu label Mysystem
MENU INDENT 1
KERNEL /boot/vmlinuz-4.15.0-47-generic
APPEND ro noapic root=/dev/mapper/system-root
INITRD /boot/initrd.img-4.15.0-47-generic

Boot-repair

• You can try the default repair.


• Note that purging grub will delete /etc/default/grub. With Ubuntu defaults my system would have a kernel panic. Also default Ubuntu grub.cfg hides important boot messages. After trying boot-repair a few times I gave up on purging GRUB and used Advanced options


• My system ended up needing the ATA disk support

Converting MBR to GPT

This assumes you have backups or you're sure you won't have a drive failure while converting. Breaking the mirror and repartitioning is destructive so you will be relying on one drive for a while. Rebuilding mirrors may stress an old drive so you've been warned.

I had another drive/mirror as a backup in case I needed to recover.

Fail the drive you want to convert:

mdadm /dev/md0 --fail /dev/sdb1
mdadm /dev/md0 -r /dev/sdb1

Create new GPT partition with gdisk. You need a 1 MB bios-boot for grub. I added an EFI partition in case I move the drives to a new system. Rest of the drive is allocated to RAID and the added back to /dev/md0. It's OK to make a larger RAID partition on the new drive. The original /dev/md0 will stay the same until you grow it.

after gdisk:
Number Start (sector) End (sector) Size Code Name
 1 2048 4095 1024.0 KiB EF02 BIOS boot partition
 2 4096 208895 100.0 MiB EF00 EFI System
 3 208896 7814037134 3.6 TiB FD00 Linux RAID

I had a problem with grub-install failing, and boot-repair not fixing or complaining either (even after purging grub).

Wiping the bios-boot partition seemed to fix grub-install. Make sure it's the correct drive/partition!

dd if=/dev/zero of=/dev/sdb1 

Then: (note the change in raid partition!)

mdadm /dev/md0 -a /dev/sdb3

Best to wait for the raid to finish rebuilding. Grub-install should work on the new drive.

At this point you should have raid1 with your old MBR drive and a new GPT drive. If you're not replacing the MBR drive, you could repeat the process of failing, GPT partitioning and re-adding to /dev/md0.

But I wanted to replace the MBR drive with a larger one so I had to shut down. This old drive was going to be my backup and I didn't want to fail it in mdadm, just in case there were problems. So I pulled the drive giving me an offline backup.

If you fail the old drive, replace it and reboot, hopefully Ubuntu boots with a degraded array and you can repartition with GPT and re-add to the array.

My mdadm radi1 system seems to boot with a degraded mirror, so failing the drive then shutting down might have worked for me.

However, if your drive actually fails, or you replace it without failing it, Linux may not boot properly and drop you into initramfs.

The problem is in /scripts/local-block/mdadm:

mdadm -q --assemble --scan --no-degraded || true

This prevents starting /dev/md0 properly so it can't mount the root filesystem.

You can get the system up from the initramfs shell:

# /dev/md0 is "up" but not mountable so stop it
mdadm --stop /dev/md0

# start it degraded
mdadm -q --assemble --scan

# if /dev/md0 is extfs it should be mountable now
# I use lvm and had to start manually:
lvm vgchange -ay

You may want to fix the iniramfs scripts. Ubuntu docs may be incorrect about the BOOT_DEGRADED option. The script is /usr/share/initramfs-tools/scripts/local-block/mdadm and you would need to rebuild your initrd etc.

After booting, you can finish partitioning the replacement drive with GPT and re-installing grub

Resizing

The whole point of all this was to use GPT and make use of a larger drive.
After /dev/md0 was fully synced and the system was working/bootable I extended the RAID, physical volume, logical volume and ext4 filesystems.

On new larger drives you should have made new, larger RAID partitions. (If you didn't, you should be able increase the partition without data loss using gparted).

Make sure the array is up and synced [UU] and state clean

# cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]

 md0 : active raid1 sdb3[3] sda3[2]
 3906783047 blocks super 1.2 [2/2] [UU]
 bitmap: 2/8 pages [8KB], 262144KB chunk

# mdadm -D /dev/md0
/dev/md0:
 Version : 1.2
 Creation Time : Thu May 3 22:06:08 2018
 Raid Level : raid1
 Array Size : 3906783047 (3725.80 GiB 4000.55 GB)
 Used Dev Size : 3906783047 (3725.80 GiB 4000.55 GB)
 Raid Devices : 2
 Total Devices : 2
 Persistence : Superblock is persistent

 Intent Bitmap : Internal

 Update Time : Sun Jun 2 13:45:17 2019
 State : clean
 Active Devices : 2
 Working Devices : 2
 Failed Devices : 0
 Spare Devices : 0

Consistency Policy : bitmap

 Name : mysystem:0 (local to host mysystem)
 UUID : xxxxx
 Events : 1995897

 Number Major Minor RaidDevice State
 3 8 35 0 active sync /dev/sdb3
 2 8 3 1 active sync /dev/sda3

Now you can extend the system for more disk space

# Grow the RAID. Docs suggest using a backup file on another drive
# mdadm didn't seem to use the backup file but doesn't hurt to include

mdadm --grow --backup-file=/mnt/otherdisk/grow_md0.bak /dev/md0 --size=max

# Have to grow the physical volume. By default uses all available space
pvresize /dev/md0

# You can extend logical volumes or create new ones
# I extended an existing one to use it all 
lvextend -l +100%FREE /dev/mysystem/data

# 
resize2fs /dev/mapper/mysystem-data # should be same as /dev/mysystem/data

At this point, I have a bootable mdadm raid1 and LVM, as well as a backup USB just in case