Being reckless and some linux action
This week was passing pretty dull, so I wanted to install some VMs to test some stuff. After realizing my disk is almost full, I’ve fired up partitionmanager, I saw there is some place for growth towards the end of my current partition, I decided to grow it a bit. Just as a small reenactment, some screenshots:
I am using btrfs root under LUKS encrypted partition and partitionmanager was able to resize that. Since I can handle the underlying FS’ growth, I said “Apply” without thinking much. You’ll probably recognize why this was a bad idea.
2 minutes later, I’ve returned back to partitionmanager, only to see it was unresponsive. I wanted to check what is going on and fired up a terminal. The terminal didn’t open. Then I saw the part of the message which I didn’t read carefully: “Move partition .. to the right by 1,00 KiB”.
O-oh.. Switched back to an open terminal, but whatever command I give just failed with either “command not found” or “library missing” warning. I realized it might be already too late and force-reset the system.
After GRUB screen, initramfs just raised a kernel panic with “partition not found” error. I’ve booted the system with sysrescuecd, only to see an “unknown partition” on the place of my LUKS partition. Looks like the operation trimmed the beginning of the partition before finishing its other tasks, suggesting it started from the beginning instead of the end. I am not actually sure how this was planned to roll, I need to dig some manual files, but initially this method didn’t make much sense to me.
Anyway, here was my small challenge. First I’ve checked the partition table:
~> fdisk -l /dev/sda
Disk /dev/sda: 477 GiB, 512110190592 bytes, 1000215216 sectors
Disk model: SAMSUNG MZ7LN512
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0xd68e6078
Device Boot Start End Sectors Size Id Type
/dev/sda1 * 2048 907263 905216 442M 83 Linux
/dev/sda2 909310 1000214527 999305218 476.5G 5 Extended
/dev/sda5 911360 418109441 417198082 199G 83 Linux
/dev/sda6 477485056 1000214527 522729472 249.3G 83 Linux
Hmm, sector size is 512 bytes. Also the root partition (sda5) was starting from sector 911360. I was hoping to at least recover the LUKS header by reading 1KiB (2 sectors) before the “new” partition start, so I’ve tried:
~> dd if=/dev/sda of=gurkantest skip=911358 bs=512 count=2
2+0 records in
2+0 records out
1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000304071 s, 3.4 MB/s
~> file gurkantest
gurkantest: LUKS encrypted file, ver 1 ...
Phew, at least it didn’t wipe it. Let’s modify the partition table and decrease the starting sector of sda5:
~> sfdisk -d /dev/sda > prt_table
~> grep sda5 prt_table
/dev/sda5 : start= 911360, size= 417198082, type=83
~> sed -i '/sda5/s/911360/911358/' prt_table
~> sfdisk -f /dev/sda < prt_table
Time for another reboot, which passed the decryption state, but failed with filesystem errors. Apparently btrfs was not so happy either. I was able to mount the partition but not all file operations were succeeding. After digging a bit, I’ve realized the superblock was fine, but there were multiple other errors. While I was able to re-create the extent-tree and re-calculate some stuff, sadly some btrfs rescue commands were failing and overall FS was unhealthy.
Except my inability to read stuff properly before doing dangerous actions (which is happening first time surprisingly), the problem is caused by partition alignment. A feature UI disk managers tend to enable by default for some reason. As you can see in the “advanced” section:
Luckily, btrfs has a restore command which I can say “find and rescue all possible ones”. Most of the files were actually OK. But I didn’t need any files except my thunderbird settings and ssh keys anyway (because I can’t/won’t declare these under my NixOS configuration).
And since my configuration is backed-up to multiple places already, including tarsnap. I got my configuration back and built my new system, which brought EVERYTHING back in ~20 minutes :)
Now I am bored again..