btrfs: No space left on device
I’ve been using btrfs on a SSD for around six months now. I have a slightly unusual setup: btrfs on LVM, even though btrfs lets you create subvolumes and resize them at will. My reasoning was that I might not necessarily stick with btrfs, but I’d appreciate the flexibility to change my partition layouts and use other filesystems.
Anywho, around a week ago, I started seeing messages like the following:
No space left on device
This was odd.
df -h doesn’t report any full volumes. Doing
btrfs check reports no errors, and playing around with my LVM settings didn’t change anything. What could be going wrong?
Filesystem Size Used Avail Use% Mounted on /dev/dm-0 15G 368M 13G 3% / udev 10M 0 10M 0% /dev tmpfs 1.6G 2.7M 1.6G 1% /run tmpfs 3.8G 136K 3.8G 1% /dev/shm tmpfs 3.8G 0 3.8G 0% /sys/fs/cgroup tmpfs 5.0M 0 5.0M 0% /run/lock tmpfs 16G 768K 16G 1% /tmp tmpfs 100M 8.0K 100M 1% /run/user /dev/sda1 1.9G 42M 1.7G 3% /boot /dev/mapper/linux-opt 15G 19M 13G 1% /opt /dev/mapper/linux-var 7.5G 2.4G 4.4G 36% /var /dev/mapper/linux-usr 15G 11G 3.5G 75% /usr /dev/mapper/linux-home 299G 159G 138G 54% /home
I panicked for a bit, thinking my SSD was failing, before deleting a few gigabytes of downloaded files and forgetting about the problem like it never existed.
But they came back.
cp: cannot create directory ‘linux-kernel/kernel/arch/um’: No space left on device cp: cannot create directory ‘linux-kernel/kernel/arch/ia64’: No space left on device cp: cannot create directory ‘linux-kernel/kernel/arch/parisc’: No space left on device cp: cannot create directory ‘linux-kernel/kernel/arch/unicore32’: No space left on device cp: cannot create directory ‘linux-kernel/kernel/arch/x86’: No space left on device cp: cannot create directory ‘linux-kernel/kernel/arch/powerpc’: No space left on device
I looked around everywhere.
tmpfs wasn’t full, but I tried mounting it explicitly in
/etc/fstab with 16GiB of space just to be sure.
Eventually my system became unusable.
By some magic, I happened upon this btrfs wiki page. They warn about metadata. First, a verification that
home isn’t full:
root@dionysus:/# btrfs filesystem show Label: 'home' uuid: 4216df47-7fb8-442f-b432-5e732ab24166 Total devices 1 FS bytes used 147.94GiB devid 1 size 298.02GiB used 298.02GiB path /dev/mapper/linux-home Btrfs v3.14.1
And surely enough:
root@dionysus:/# btrfs fi df /home Data, single: total=292.50GiB, used=145.70GiB System, DUP: total=8.00MiB, used=40.00KiB System, single: total=4.00MiB, used=0.00 Metadata, DUP: total=2.75GiB, used=2.25GiB Metadata, single: total=8.00MiB, used=0.00 unknown, single: total=512.00MiB, used=0.00
Metadata, DUP - that’s 81% usage. The btrfs wiki says that 75% or greater metadata space usage can cause
No space left on device errors, even when the filesystem isn’t full.
For now, a workaround is to run a partial balance:
$ sudo btrfs filesystem balance start -dusage=5 /mount/point
-dusage=5 bit means only chunks that are at most 5% used will be relocated.
btrfs filesystem balance spreads usage across the disk and reclaims allocated but unused metadata chunks.
root@dionysus:/# btrfs fi balance start -dusage=5 -v /home Dumping filters: flags 0x1, state 0x0, force is off DATA (flags 0x2): balancing, usage=5 Done, had to relocate 0 out of 303 chunks
And now, the
Metadata, DUP field shows 72% usage. Better, I suppose.
root@dionysus:/# btrfs fi df /home Data, single: total=285.50GiB, used=151.25GiB System, DUP: total=8.00MiB, used=40.00KiB System, single: total=4.00MiB, used=0.00 Metadata, DUP: total=3.25GiB, used=2.34GiB Metadata, single: total=8.00MiB, used=0.00 unknown, single: total=512.00MiB, used=0.00
I’ll be exploring this more if I see any more issues, and update this post with information. For reference, I run a custom Linux 3.15 release, and have built and booted the 3.16-rc1, rc2, and some in-between RCs.