Multi-device and RAID1 with btrfs

I have been using btrfs, a modern modern copy on write filesystem for Linux, since many years now. But only recently I realized how amateurish my usage has been. Over the last day I switched to multiple devices and threw in a RAID1 level at the same time.

For the last years, I have been using btrfs in a completely naive way, simply creating new filesystems, mounting them, moving data over, linking the directories into my home dir, etc etc. It all became a huge mess over time. I have heard of “multi-device support“, but always thought that this is for the big players in the data centers – not realizing that it works trivially on your home system, too. Thanks to an article by Mark McBride I learned how to better use it!

Btrfs has an impressive list of features, and is often compared to (Open)ZFS (btw, they domain openzfs.org has a botched SSL certificate .. umpf, my trust disappears even more) due to the high level of data security. I have been playing around with the idea to use ZFS for quite some time, but first of all it requires compiling extra modules all the time, because ZFS cannot be included in the kernel source. And much more, I realized that ZFS is simply too inflexible with respect to disks of different sizes in a storage pool.

Btrfs on the other hand allows adding and removing devices to the “filesystem” on a running system. I just added a 2TB disk to my rig, and called:

btrfs device add /dev/sdh1 /

and with that alone, my root filesystem grew immediately. At the end I have consolidated data from 4 extra SSDs into this new “filesystem” spanning multiple disks, and got rid of all the links and loops.

For good measure, and since I had enough space left, I also switched to RAID1 for this filesystem. This again, surprisingly, works on a running system!

btrfs balance start -dconvert=raid1 -mconvert=raid1 /

Here, both data and metadata are mirrored on the devices. With 6TB of total disk space, the balancing operation took quite some time, about 6h in my case, but finished without a hiccup.

After all that, the filesystem now looks like this:

$ sudo btrfs fi show /
Label: none  uuid: XXXXXX
	Total devices 5 FS bytes used 2.19TiB
	devid    1 size 899.01GiB used 490.03GiB path /dev/sdb3
	devid    2 size 489.05GiB used 207.00GiB path /dev/sdd1
	devid    3 size 1.82TiB used 1.54TiB path /dev/sde1
	devid    4 size 931.51GiB used 649.00GiB path /dev/sdf1
	devid    5 size 1.82TiB used 1.54TiB path /dev/sdc1

and using btrfs fi usage / I can get detailed information about the device usage and status.

Stumbling blocks

You wouldn’t expect such a deep rebuilding of the intestines of a system to go without a few bumps, and indeed, there are a few:

First of all, update-grub is broken when device names are used. If you have GRUB_DISABLE_LINUX_UUID=true, so that actual device nodes are used in grub.cfg, the generated entries are broken because they list all the devices. This comes from the fact that grub-mkconfig uses grub-probe --target=device / to determine the root device, and this returns in our case:

# grub-probe --target=device /
/dev/sdb3
/dev/sdd1
/dev/sde1
/dev/sdf1
/dev/sdc1

and thus the grub config file contains entries like:

...
menuentry 'Debian GNU/Linux ...' ... {
  ...
  linux	/boot/vmlinuz-5.7.0-rc7 root=/dev/sdb3
/dev/sdd1
/dev/sde1
/dev/sdf1
/dev/sdc1 ro 
}

This is of course an invalid entry, but fortunately grub still boots, but ignores the rest of the command line options.

So I decided to turn back to using UUID for the root entry, which should be better supported. But alas, what happened, I couldn’t even boot anymore. Grub gave me very cryptic messages like cannot find UUID device and dropping you into the grub rescue shell, then having the grub rescue shell being unable to read any filesystem at all (not even FAT or ext2!). The most cryptic one was grub header bytenr is not equal node addr, where even Google gave up on it.

At the end I booted into a rescue image (you always have something like SystemRescueCD on an USB stick next to you during these operations, right?), mounted the filesystem manually, and reinstalled grub, which fixed the problem, and now the grub config file contains only the UUID for root.

I don’t blame btrfs for that, this is more like we are, after sooo many years, we still don’t have a good boot system 🙁

All in all, a very smooth transition, and at least for some time I don’t have to worry about which partition has still some space left.

Thanks btrfs and Open Source!