Rebuild runbook — joe-host (VM 101)

Status: draft v0.1, 18 Oct 2026. Never rehearsed. Do not treat as proven. Driver: Beetroot (HA standby, app_local_hermes_standby). Rootbeet (in-VM) dies when disk-1 is destroyed, so it cannot drive its own rebuild. Purpose: reclaim the ~54 GB of stale thin-pool mapping by rebuilding vm-101-disk-1 from a staging copy, with no B2 round-trip.

---

Access inventory (all verified 18 Oct 2026)

Path	Command	Verified result
Beetroot → Proxmox	ssh root@192.168.0.200 'qm list'	returns VM 101, running
Beetroot → guest root	ssh root@192.168.0.182 'id -u'	0
Beetroot → guest joe	ssh joe@192.168.0.182 hostname	joe
rclone binary	/config/hermes-standby/bin/rclone	v1.75.2 (arm64, persistent in /config)
rclone config	/config/hermes-standby/.config/rclone/rclone.conf	409 B, mode 600, remotes b2 + immich-crypt
B2 reach	rclone --config … lsd b2:	bucket joe-immich-backup

Beetroot’s key (hermes-standby@ha) is in /root/.ssh/authorized_keys on both Proxmox and the guest. Backups of the prior file: /root/.ssh/authorized_keys.bak-20261018 (Proxmox).

---

Pre-flight (abort if any fail)

1. Backups fresh. On the guest:
   • rclone --config ~/.hermes/state/rclone.conf lsl immich-crypt:db | sort -k2 | tail -1 → dump from the last 24h
   • ls -lt ~/.hermes/backups/homelab/ | head -3 → weekly tars from the last 7 days
   • ls -lt ~/.hermes/backups/owui/ | head -2
2. Staging disk present on the Proxmox host and empty:
   • ssh root@192.168.0.200 'lsblk -o NAME,SIZE,FSTYPE,MOUNTPOINT'
   • Confirm nothing on it is mounted and no data matters.
3. Record the truth before touching anything:
   • ssh root@192.168.0.200 'qm config 101' → save
   • ssh root@192.168.0.200 'dmsetup status pve-data-tpool' → note used blocks
   • guest: df -h /data, docker ps --format '{{.Names}}' | sort → save
4. Immich quiesced. Stop the stack so the DB dump and library are consistent: cd ~/stacks/33 && docker compose stop (or the Immich compose dir).

---

Phase 1 — attach and prepare the staging disk

On Proxmox:

lsblk                                   # find the new disk, e.g. /dev/sdc
pvcreate /dev/sdc1                      # or partition first with parted if bare
vgextend pve /dev/sdc1                  # alt: make a separate VG, see note

Note: adding to pve VG grows the pool, not the staging area. For staging, prefer a separate VG/mount so the copy does not land back on the thin pool:

vgcreate stage /dev/sdc1
lvcreate -l 100%FREE -n staging stage
mkfs.ext4 /dev/stage/staging
mkdir -p /mnt/stage && mount /dev/stage/staging /mnt/stage

Verify: df -h /mnt/stage shows the expected size.

---

Phase 2 — stage /data

From the guest, push to Proxmox over the LAN (the guest cannot mount the host’s staging directly without NFS/SMB, so use rsync over SSH):

# on the guest as root
rsync -aHAX --numeric-ids --info=progress2 \
  /data/ root@192.168.0.200:/mnt/stage/data/

This preserves ACLs, xattrs, hardlinks and ownership — required for Docker volumes.

Verify before proceeding:

du -sx --block-size=1 /data                  # on guest
du -s --block-size=1 /mnt/stage/data         # on proxmox

Sizes must match within a few MB. Do not continue if they differ.

Also capture the mount layout so /etc/fstab can be reproduced:

findmnt -no SOURCE,TARGET,FSTYPE,OPTIONS /data

---

Phase 3 — destroy and recreate the disk

This is the irreversible step. Everything must be staged and verified.

# on Proxmox
qm stop 101                             # clean shutdown first: qm shutdown 101, wait
qm config 101                           # save the output verbatim
qm set 101 --delete scsi1               # detach the old disk
lvremove pve/vm-101-disk-1              # frees the pool
lvcreate -V 320G -T pve/data -n vm-101-disk-1
qm set 101 --scsi1 local-lvm:vm-101-disk-1,discard=on,ssd=1
qm start 101

Verify: dmsetup status pve-data-tpool used blocks should drop by roughly the old disk’s allocation.

---

Phase 4 — recreate the filesystem and restore

Guest boots from disk-0 (OS intact, 32 GB). /home/joe will be missing because it is a bind mount of /data/home/joe; the mountpoint exists but is empty.

# as root on the guest
parted /dev/sdb mklabel gpt
parted /dev/sdb mkpart primary ext4 1MiB 100%
mkfs.ext4 -m 1 -L data /dev/sdb2
mount /dev/sdb2 /data
rsync -aHAX --numeric-ids --info=progress2 \
  root@192.168.0.200:/mnt/stage/data/ /data/

Then restore fstab (keep nofail, keep discard):

grep -q '/data' /etc/fstab || \
  printf 'UUID=%s /data ext4 defaults,nofail,discard 0 2\n' "$(blkid -s UUID -o value /dev/sdb2)" >> /etc/fstab
findmnt --verify

Verify: du -sx --block-size=1 /data matches Phase 2.

---

Phase 5 — bring services back

# /data/var/lib/docker is the data-root, so Docker sees its volumes again
systemctl start docker
docker ps -a | wc -l                    # expect the previous container count
cd ~/stacks && for d in */; do (cd "$d" && docker compose up -d); done

Images are not in backup. Rebuild from source where a Dockerfile exists; otherwise pull. Expect this to be the longest phase (hours).

---

Phase 6 — Hermes

~/.hermes comes back with the rsync. Then:

cd ~/.hermes/hermes-agent && python3 -m venv venv && venv/bin/pip install -e .
journalctl --user -u hermes-gateway -n 20

Crons live in ~/.hermes/cron/jobs.json and are restored by the copy.

---

Phase 7 — verification checklist

• df -h /data total ≈ 320 G, used ≈ 242 G
• ssh root@192.168.0.200 'lvs -o lv_name,data_percent pve/data' → pool ≈ 70%
• Immich: curl -s localhost:2283/api/server/ping → pong; asset count unchanged
• docker ps count matches the pre-flight list
• Vaultwarden, Paperless, CouchDB, Mealie, Frigate all responding
• Hermes gateway active; Telegram answers
• system-validate.py exits 0

---

Rollback

There is no rollback once Phase 3 runs. The safety net is:

• the staging copy on /mnt/stage/data, and
• B2: immich-crypt:db (logical DB dumps) and the weekly volume tars.

If the restore fails, do not wipe the staging copy. Re-run Phase 4 against it.

---

What is captured nightly (added 18 Oct 2026)

system-b2-backup.py (03:35) now runs host-config-capture.py first and syncs the result to immich-crypt:system-backups/host/. One tarball, newest 8 kept, containing:

• etc/ — fstab, crypttab, docker/daemon.json, sshd_config, hostname/hosts, grub, crontab, sysctl, apt sources, /etc/systemd/system and drop-ins
• user/ — joe’s systemd user units, crontab, Hermes config.yaml, CRON-MAP, ssh config
• pkg/ — dpkg --get-selections and apt-mark showmanual (replayable package list)
• disk/ — lsblk, blkid, df, mount (the layout to reproduce, not the data)
• docker/ — container, image+digest, volume and network inventory
• pve/ — 101.conf (VM hardware), storage.cfg, host.fw, fetched over SSH from 192.168.0.200

Verified 18 Oct 2026: 341 entries, fstab + daemon.json + 101.conf all present.

Known gaps (as of 18 Oct 2026)

1. No rehearsal. This has never been executed. First run is the real one.
2. Images not backed up. ~40 stacks must be rebuilt from ~/stacks (Dockerfiles present).
3. 11 Docker volumes have no tar — but the important ones are covered elsewhere:
   • immich_immich-postgres → covered by logical dumps in immich-crypt:db
   • immich_immich-library → covered by immich-crypt:library (originals only)
   • open-webui_open-webui-data → covered by ~/.hermes/backups/owui tars
   • Genuinely uncovered: nginx-proxy-manager_npm-letsencrypt, adguard_data, 16_camofox-data, 23_stirling-configs, open-terminal-data, and 3 anonymous volumes
4. Immich backup excludes derived media by design (18 Oct 2026): thumbs/ and encoded-video/ are not stored in B2 — Immich regenerates both from the originals. Restoring therefore means a re-run of the thumbnail/transcode queues, which takes hours on video and is expected, not a fault.
5. Beetroot has no evidence trail. It should write a log to /config/hermes-standby/rebuild-<date>.log and message Joe at each phase boundary.
6. /root is not captured (mode 700). Root-owned ad-hoc scripts live only on the disk.
7. Package versions drift. apt-mark showmanual records names, not pins; a replay months later pulls whatever is current.