Category Archives: btrfs

btrfs, Linux

Recovering from unmountable btrfs filesystem issues

1 Comment

Here are some notes of how I recovered most of the data after my btrfs disk got horribly corrupted by bad memory. Fortunately I had upgraded the disk 6 months ago so I was able to start from that image left behind on the old disk, copied over using the excellent btrfs-clone tool.

After that I could restore most of my files to the last backup (a month or two back) and git repositories from the main server. But I still had a number of documents and other bits that I needed to recover.

The first thing prior to formatting the disk (I don’t have another spare fast SSD lying around) was to take a backup of the entire btrfs disk. However it was quite a bit larger than I easily had spare on another disk. So, I stored it in squashfs which reduced size by 50%.

mkdir empty-dir
mksquashfs empty-dir squash.img -p 'sdb3_backup.img f 444 root root dd if=/dev/sdb3 bs=4M'

After that I tested that it was mountable:

mount squash.img /mnt/tmp
btrfs restore -l /mnt/tmp/sdb3_backup.img

And erased and cloned the old btrfs disk to it.

I then started using the btrfs restore tool to try to recover the data. First you need to list the roots, usually the highest number will be the latest snapshot and it may have consistent data:

btrfs restore -l /mnt/tmp/sdb3_backup.img

Then you can get a listing of the files under that root and whether they may be recoverable using the -v -D flags (-v means list files, -D means don’t actually try to restore any data. For example:

btrfs restore -r 290 -v -D sdb3_backup.img /laptop/restore/

If that looks good then you can run the command with a few extra flags to try to get the files back as much as possible:

btrfs restore -r 290 -x -m -i sdb3_backup.img /laptop/restore/

This can take a while but it seems to work well on smaller files. Unfortunately some virtual machine images (60gb or so each) didn’t recover because they had got corrupted in the middle.

If you want to recover only a particular point under the tree you can use the --path-regex parameter to specify this, however writing the regexps is very difficult. Here is a short bit of code which will generate the path regex correctly:

perl -E 'for(@ARGV){ $p = () = m!/!g; s!/!(|/!g; $_.= "(|/.*))" . ")" x $p; say "--path-regex '\''^/(|$_\$'\''" }' 'mark/data/documents'

You can then restore just those files like:

btrfs restore -x -m -i  -r 290 --path-regex  '^/(|mark(|/data(|/documents(|/.*))))$' sdb3_backup.img /laptop/restore/

btrfs, Computers, Linux

Diagnosing faulty memory in Linux…

Leave a comment

For the past year I’ve had very occasional chrome crashes (segfaults in rendering process) and an occasional bit of btrfs corruption. As it was always easily repairable with btrfs check --repair I never thought much about it, although I suspected it may be an issue with the memory. I ran memtest86 overnight one time but it didn’t show up any issues. There were never any read or SMART issues logged on the disk either, and it happened to another disk within the machine as well.

Recently though I was seeing btrfs corruption on a weekly basis, especially after upgrading to ubuntu 18.04 (from ubuntu 16.04). I thought it may be a kernel issue so I got one of the latest kernels. It seemed to happen especially when I was doing something quite file-system intense, for example browsing some cache-heavy pages while running a vm with a long build process going on.

Then, earlier in the week the hard drive got corrupted again, much more seriously and after spending some time fixing, running `btrfs check –repair` a few times it suddenly started deleting a load of inodes. Force rebooting the machine I discovered that the disk was un-mountable, although later I was able to recover quite a lot of key data from btrfs restore as documented in this post.

memtest86 was still not showing any issues, and so my first thought was that assuming the hard disk was not at fault it may be something to do only when the memory had a lot of contention (memtest86 was only able to run on a single core on my box). I booted a minimal version of linux and ran a multi-process test over a large amount (not all) of the memory:

apt -y install memtester
seq $(nproc) | xargs -P1000 -n 1 bash -c 'memtester $0 10; E=$?; [[ $E != 0 ]] && { echo "FAILURE: EXIT status: $E"; exit 255; }' "$((($(grep MemAvailable /proc/meminfo | awk '{print $2}') / 1024 - 100) / $(nproc)))"

and check for FAILURE in the log messages, it likely also shows in dmesg, and may only show there if you have ECC RAM.

This will run a process per CPU aiming to consume pretty much all of your available memory. 10 is the number of test cycles to run through. In my case 8 cores and 16gb memory = 1400mb per memtester process. It took about 45 min to run once over the 16gb, or about 25 min to run over 8gb (each of the individual sodimms in my laptop).

Within about 10 minutes it started showing issues on one of the chips. I’ve done a bit of research since this and seen that if a memory chip is going to fail then it would usually do it within the first 6 months of being used. However this is a kingston chip that has been in my laptop since I bought it 2 or 3 years back. I added another 8gb samsung chip a year ago and it seemed to be after that that the issues started, however that chip works out as fine. Perhaps adding another chip in broke something, or perhaps it just wore out or overheated somehow…

btrfs, Linux, Ubuntu

Making a BTRFS read-only snapshot writable

Leave a comment
For the past few years I’ve been using btrfs on most filesystems that I create, whilst it’s pretty slow on rotating disk media now that most of my hardware is SSD-based there’s not much of a performance penalty (as long as you’re not using quotas to track filesystem usage). The massive advantage is the ability to have proper snapshot history (unlike any LVM snapshotting hacks that you may suggest) going back a long time with very little overhead. With a tool like snapper (which admittedly is tricky to get set up) you can automatically rotate your snapshots and easily recover any files that you accidentally changed or deleted. Alongside always using git for code repositories, this has saved my skin repeatedly! Anyway, by default snapper creates read-only snapshots. But when trying to diagnose some database server file corruption I recently experienced I wanted to change a btrfs snapshot from read-only to read-write so I could update some files. After spending a while looking around in the manual and on stack overflow I couldn’t see any way to do this with the kernel/toolchain versions that I was using. Then, the solution struck me. Simply create a read-write snapshot of the read-only snapshot and work off that. Sometimes it’s very easy to look at the more complicated way of doing things and forget about some of the easier solutions that there might be! UPDATE: For newer versions of btrfs tools you can toggle read-onlyness of snapshots by running the following command against the subvolume directory: 
btrfs property set -ts /path/to/snapshot ro false