Hands-on runbook — 16 Oct 2026

Everything in here needs a keyboard. Two jobs: the vault (5 minutes, zero risk) and the /data filesystem check (30–60 minutes, box offline). Do them in that order.

Everything else on the audit list is mine — see the last section.

---

Pre-flight (30 seconds, read-only)

Run this first so we both have a before-picture. It changes nothing.

cd ~ && df -h / /data \
  && docker ps -q | wc -l \
  && python3 ~/.hermes/scripts/root-disk-cap.py --status \
  && python3 - <<'PY'
import json
d = json.load(open('/home/joe/.hermes/cron/jobs.json'))
js = d.get('jobs', d) if isinstance(d, dict) else d
bad = [j for j in js if (j.get('last_status') or '').lower() not in ('', 'success', 'ok', 'silent')]
print(f"cron: {len(js)} jobs, {len(bad)} not ok ->", [j.get('name') for j in bad])
PY

Expect: root ~20G used / 11G free, /data ~285G used, 45 containers, cap data 13.75GiB / cap 26GiB, and cron with exactly one job not ok (rbw-vault-watch).

---

Job 1 — the vault (do this first: 5 minutes, no risk)

1.1 Re-seal the master credential against the new host key

The old master.cred is encrypted to a key that died with the reinstall — proof: systemd-creds --user decrypt fails with io.systemd.System. Re-store it:

read -rsp "Vaultwarden master password: " VP && echo && \
  printf '%s' "$VP" | systemd-creds --user encrypt --name=rbw-master - \
  ~/.config/rbw/master.cred && chmod 600 ~/.config/rbw/master.cred && unset VP && echo stored

Expect exactly: stored

Two notes:

• The password is never echoed, never in your shell history (read -rsp), and goes straight into the encrypted store.
• Your TOTP secret is intact (~/.hermes/rbw-totp, 33 bytes) and the pinentry shim already answers code prompts, so the re-login can complete unattended.

1.2 Prove the credential decrypts (shows a length, never the value)

systemd-creds --user decrypt --name=rbw-master ~/.config/rbw/master.cred | wc -c

Expect a number matching your password length + 1. If you see an error instead, the re-store didn’t take — redo 1.1.

1.3 Unlock and prove it

rbw unlocked                      # expect: agent is locked
rbw unlock                        # will now succeed silently
rbw unlocked                      # expect: agent is unlocked
rbw sync
rbw list | wc -l                  # expect ~833 entries
python3 ~/.hermes/scripts/vault-cred.py status   # expect: agent state : UNLOCKED

1.4 Prove the keeper works — this is the part that was actually broken

The keeper is what re-unlocks after a reboot, so test it the only way that counts: lock the vault and let the timer fix it.

rbw lock && rbw unlocked                       # expect: LOCKED
systemctl --user start rbw-vault-keeper.service
sleep 5
rbw unlocked                                   # expect: UNLOCKED

If it stays locked, read the reason:

journalctl --user -u rbw-vault-keeper.service -n 30 --no-pager
journalctl --user -u rbw-vault-agent.service -n 20 --no-pager 2>/dev/null
cat ~/.local/share/rbw/agent.err 2>/dev/null | tail -20

1.5 Confirm the knock-on effects are gone

hermes mcp list 2>&1 | grep -i secrets      # expect no "[secrets:command] helper failed" line
python3 ~/.hermes/scripts/vault-cred.py fields open-design.ai 2>&1 | head -5

The first one is the important one: every hermes CLI call was printing [secrets:command] helper failed; resolving no value because the vault was locked.

---

Job 2 — /data filesystem check (30–60 min, box offline)

Why: /dev/sdb2 carries ~5,456 recorded errors and 320 unreadable directory blocks (EBADMSG, “Directory block failed checksum”) from the 15 Oct corruption. Nothing is failing because of it right now — the corrupt blocks are confined to the parked store at /data/var/lib/containerd.corrupt-20261016 — so this is maintenance, not an emergency. You can skip it and do it another day.

Scope, stated plainly: this touches /dev/sdb2 only. Your root filesystem is /dev/sda2 and is not involved. That is the one that broke last time.

Safety net: all four B2 backup jobs ran green today (immich, HA, system, hermes state) and I pulled a file back out of the archive and decrypted it as a restore drill. Everything on /data except the truly-corrupt directory blocks is in B2.

2.1 Prep (normal desktop session, before switching modes)

cd /                      # so your shell isn't sitting inside /home/joe
sudo passwd root          # TEMPORARY root password — rescue mode asks for one

Why the root password: rescue.target drops you at a sulogin prompt, which asks for root; Ubuntu installs normally leave root locked, and you can’t sudo from a rescue prompt. You’ll lock it again in 2.6. SSH root login stays disabled, so this is console-only.

2.2 Switch to rescue mode

Do this at the Proxmox console, not over SSH — the network stack goes down.

Proxmox web UI → your VM → Console → log in as joe → then:

sudo systemctl isolate rescue.target

The screen clears and you get: Give root password for maintenance.

2.3 Unmount everything that lives on /data

mount | grep -E '/data|/home/joe|containerd'      # see what's still mounted
umount /home/joe 2>/dev/null; umount /var/lib/containerd 2>/dev/null
umount /data
mount | grep -c /data                             # expect: 0

If umount /data says target is busy, find the holder and stop it:

fuser -vm /data

Then re-run the two umount lines. In rescue mode docker and containerd are already down, so this should be clean. Do not run e2fsck if mount still shows /data mounted — that is the one thing that can really damage a filesystem.

2.4 Look first, change nothing

e2fsck -fn /dev/sdb2 | tail -40

-n = answer “no” to everything, so this only reports. Expect a long list of Directory block failed checksum, Entry '…' in … has deleted/unused inode, and a final *** FILE SYSTEM WAS MODIFIED ***-less summary ending in something like /dev/sdb2: ********** WARNING: Filesystem still has errors ********** — that wording is normal for a check-only run and is not a failure. Exit code 4 is expected here; anything else, stop and tell me what it printed.

2.5 Fix it

e2fsck -fyD /dev/sdb2

• -y = answer “yes” to every fix.
• -D = optimise directories (this is the flag that rebuilds the corrupt ones).
• It prints fixes as it goes. On 285GB this runs minutes, occasionally longer. Let it finish; do not kill it.
• Expect it to clear or reconnect the 320 bad directory blocks — that means the parked store’s unreadable directories get tidied. It was already unreadable, so nothing is lost that was still usable.

Then:

echo "--- final state ---"
e2fsck -fn /dev/sdb2 | tail -5      # expect: clean, exit 0
reboot

2.6 After it comes back up

sudo passwd -l root                                   # re-lock the temporary root account
sudo dumpe2fs -h /dev/sdb2 2>/dev/null | grep -iE 'errors count|filesystem state'

Expect Filesystem state: clean and Errors count: 0.

Then prove the box is whole again:

df -h / /data
findmnt -no TARGET,SOURCE /data /home/joe /var/lib/containerd
docker ps -q | wc -l                                  # expect 45
systemctl --user is-active hermes-gateway mnemobrain-gbrain
curl -s -m 8 http://127.0.0.1:3131/health; echo
dig +short @192.168.0.182 example.com | head -1       # LAN DNS still answering
python3 ~/.hermes/scripts/vault-cred.py status         # vault still UNLOCKED

If any container is missing after the reboot, tell me which and I’ll bring it back — that’s exactly the failure mode the bare-run containers have, and it’s on my list.

---

Job 3 — two clicks in the Proxmox UI (not commands)

While you’re in there, these are the reason a 6-hour outage had no recovery on 12 Oct:

1. VM → Options → Start at boot → Yes. Verifies joe-host returns by itself.
2. The physical Proxmox host’s BIOS → “Restore on AC Power Loss” → Power On. That’s a reboot-into-BIOS job, not a UI one — do it when convenient, not tonight.

---

Everything else is mine

These are all things I can do without you, and I’ll verify each one rather than assume it. Say which you want and in what order:

1. Recreate the four missing web services (pulse :8645, tennis-log :8646, health-app :8654, finance-app :8656) as systemd user units so a rebuild can’t lose them again, and reconcile their tunnel/NPM rules. decent-visualizer :8653 has no source left on disk, so it’s either rebuilt from scratch or its ingress retired — your call.
2. Install the mnemosyne memory-provider plugin into the Hermes venv and repair the ~/.hermes/plugins/mnemosyne symlink (py3.11 → py3.12). Restores the configured memory layer; needs a gateway restart to take effect.
3. Repoint the three stale HA endpoints .209 → .182 (two llmvision providers + ai_automation_suggester) — this is what’s breaking the doorbell’s “person describe (AI)” on every ring.
4. Compact USER.md (3592/3600 chars) before a write silently evicts an entry.
5. Tracker tidy: 6 wrong Cron Reminders rows, 4 expired ⏰ rows, the 36-day-stale therapy compile, the dead parcels register, plus adding the Transaction Review table to tracker-schema.json so validation actually covers it.
6. Housekeeping: stop git tracking the 136MB Chromium profile in ~/.hermes/scripts, script (or formally document) daily-fact-extraction as a third LLM-prompt cron, fix the era MCP URL, prune ~6.5GB of build cache.

If you’d rather run some of those yourself, say which and I’ll hand you the exact commands instead of doing them.