Joe's notes

Runbook — joe-host thin-pool fix (one offline window)

Properties1
descriptionThe 45-60 minute offline procedure that fsck'd both filesystems, right-sized disk-1 and enabled discard.

4 min read

Runbook — joe-host thin-pool fix (one offline window)

Goal: fsck both filesystems, right-size disk-1 from 451G to 280G, enable discard on the OS disk, stop writeback caching, so the VM can never exhaust the Proxmox thin pool again.

Expected downtime: 45–60 min. Everything on joe-host is down during this window — AdGuard DNS (the whole house loses DNS unless the router has a fallback), Vaultwarden, Frigate, Paperless, Immich, cloudflared, Home Assistant’s non-HA services. HA core itself is on .155 and stays up, but anything it calls on joe-host will fail.

Current state before starting

  • Pool pve/data 65.59% (128 GB free) | disk-0 32G @ 99.86% | disk-1 451G @ 47.13%
  • Guest: / 20G/32G (67%), /data 204G/442G (49%)
  • Immich stopped, restart=no (all four containers)
  • fsck has NOT been run since the I/O failures started (15 Oct)

PHASE 1 — stop the VM and boot from the ISO

In the Proxmox node Shell:

qm shutdown 101 --timeout 180
qm status 101

Expect: status: stopped. If it hasn’t stopped after 3 minutes, qm stop 101.

Confirm the CD is attached and note its bus:

qm config 101 | grep -iE "^ide2|^sata[0-9]"

Expect something like ide2: local:iso/ubuntu-26.04...amd64.iso,media=cdrom,size=... If the bus is not ide2, use whatever it shows in the next two commands.

qm set 101 --boot "order=ide2;scsi0"
qm start 101

Open the VM Console. At the GRUB menu pick Try Ubuntu (do not install). Wait for the desktop, then open Terminal.

PHASE 2 — identify the disks, fsck, shrink (inside the live Ubuntu)

lsblk -o NAME,SIZE,FSTYPE,LABEL

Confirm two disks: ~32G and ~451G. In this runbook they are sda (32G) and sdb (451G). If the sizes land the other way round, swap them in every command below. Do not guess.

fsck the OS volume:

sudo e2fsck -f -y /dev/sda2

Expect a summary ending FILE SYSTEM WAS MODIFIED or clean. If it says modified, run it again and repeat until it reports clean.

fsck the data volume — this is the important one (204 GB, takes 10–30 min):

sudo e2fsck -f -y /dev/sdb2

Again, repeat until it comes back clean without modifications.

Only if both are clean, shrink the filesystem:

sudo resize2fs /dev/sdb2 272G

Expect: The filesystem is now 71303168 blocks long. (The filesystem must be slightly smaller than the final LV size — 272G inside a 280G LV.)

sudo poweroff

STOP here and tell me what fsck printed. If either volume reports errors that fsck could not fix, do not run the resize — we stop and plan a copy-and-swap instead.

PHASE 3 — shrink the volume and fix the disk options (host Shell)

qm resize 101 scsi1 -171G
lvs -a -o lv_name,lv_size,data_percent pve/vm-101-disk-0 pve/vm-101-disk-1

Expect vm-101-disk-1 at 280.00g (was 451.00g). If it errors, stop — the LV must be shrunk before anything else touches the disk.

Add discard to the OS disk and drop writeback caching on the data disk:

qm set 101 --scsi0 local-lvm:vm-101-disk-0,iothread=1,discard=on
qm set 101 --scsi1 local-lvm:vm-101-disk-1,discard=on,ssd=1,iothread=1,cache=none
qm set 101 --boot "order=scsi0;net0"
qm config 101 | grep -iE "^scsi0|^scsi1|^boot"

Expect both disks showing discard=on, scsi1 showing cache=none, and boot: order=scsi0;net0.

qm start 101

PHASE 4 — verify (I do this from joe-host)

df -h / /data
fstrim -v /
fstrim -v /data

Then, on the host, lvs -a -o lv_name,data_percent should show disk-0 down from 99.86% to ~63% and the pool around 63–65% with no oversubscription left (32 + 280 = 312 GB virtual vs 372.82 GB pool).

Then: restart Immich, watch the pool %, resume the import.

If it goes wrong

  • fsck cannot fix something → stop, do not resize. Plan a copy of /data to a new volume.
  • resize2fs fails → poweroff immediately, boot normally. The filesystem is untouched if the resize never started; if it started and failed, /data needs a recovery pass before anything else.
  • LV shrink errors → the VM is still on a 451G LV with a 272G filesystem; that is safe and recoverable (re-grow the LV back to 451G and resize2fs to fill it).
  • VM won’t boot after Phase 3 → boot the ISO, check blkid/lsblk, and re-run e2fsck on /dev/sdb2 before anything else.

Graph View

Backlinks