Joe's notes

RESTORE.md — Disaster Recovery Runbook (Joe's stack)

Properties1
descriptionDisaster recovery runbook: if this host or the HA Pi dies, start here.

9 min read

RESTORE.md — Disaster Recovery Runbook (Joe’s stack)

If joe-host or the HA Pi fails, the current off-site copies are in Backblaze B2, written through rclone’s client-side-encrypted immich-crypt remote. The old Google Drive instructions below were retired; do not use them as the host backup source.

Task data first — Simple Tasks has its own procedure. The app’s logical bundles and the tested restore path are in RESTORE.md; bundles are taken hourly, kept 48 hourly + 30 daily, and mirrored to system-backups/simple-tasks in B2. Prove any bundle with python3 simple-tasks/ops.py restore-check <bundle> before trusting it. That procedure covers the task database only; this runbook covers the host and services.

Bootstrap dependency: restoring from an encrypted remote requires the rclone configuration and crypt credentials before downloading system-backups/state, because that backup contains another copy of the same configuration. The current independent copy documented for the Beetroot standby is /config/hermes-standby/.config/rclone/rclone.conf (remotes b2 and immich-crypt, mode 0600; documented as checked 18 Oct 2026). That is a bootstrap path, not a rehearsed blank-host restore. If joe-host and Beetroot are both unavailable, B2 access has not yet been proven.

Off-site layout (checked 28 Oct 2026):

immich-crypt:
├── system-backups/
│   ├── workspace/       # workspace files; .git excluded
│   ├── hermes/          # Hermes state tars, config.yaml, jobs.json, Git bundles
│   ├── homelab/         # local Docker volume tars
│   ├── source-code/     # encrypted archive of live stacks and chat app source
│   ├── owui/             # Open WebUI data tars
│   ├── mnemosyne/        # Mnemosyne DB/config tars
│   ├── host/             # host-config-capture tars
│   └── state/            # ~/.hermes/state, including rclone.conf
├── ha-backups/           # HA snapshots streamed from /backup; newest three retained
├── library/              # Immich originals only (derived thumbnails/video excluded)
├── db/                   # Immich PostgreSQL logical dumps
└── heartbeats/            # Immich library mirror verification marker

The B2 API remote is b2: and the encrypted remote is immich-crypt:. The B2 account stores the encrypted objects; do not describe system-backups/state/rclone.conf as a stand-alone bootstrap because it is inside that encrypted set.

Latest checks (28 Oct 2026): system-b2-backup completed ok at 03:41 UTC; ha-b2-backup at 07:56; immich-b2-backup at 03:32; and mnemosyne-backup at 03:10. The system mirror completed with five local and five remote OWUI objects and no remote-only OWUI objects. The source-code archive was uploaded separately later and verified by exact read-back SHA-256. The revised runbook itself is not in B2 yet; the next scheduled mirror is 29 Oct at 03:35 UTC.

Open WebUI backup status (verified 28 Oct 2026): The weekly owui-backup slot on 25 Oct was missed; the live scheduler still shows last run 18 Oct and next slot 1 Nov. A manual producer run created 27 Oct archives; owui-20261027-213542.tar.gz is present in B2 and its read-back SHA-256 matched the local file. The other 27 Oct archive remains local only. The full system mirror completed at 03:41 UTC on 28 Oct; a current name comparison finds five local and five remote objects, with no remote-only items remaining.

Recovery priority: 1) money/finance data 2) conversation history 3) config/credentials 4) services.

Part A — Hermes host (this box)

0. Fresh machine prerequisites

  • Linux (Ubuntu 24.04-ish), user joe, ~/.hermes writable
  • Docker + docker compose plugin, git, python3
  • Hermes Agent installed per https://hermes-agent.nousresearch.com/docs
  • Restore ~/.hermes contents below, then Joe should run hermes gateway restart from his own SSH shell. Do not try to restart the gateway from inside the agent process.

1. Obtain B2 access and stage a copy

  • On a running joe-host, the rclone configuration is ~/.hermes/state/rclone.conf (mode 0600); the host has the rclone/rclone:latest image but no standalone rclone binary. The config includes secrets; never print it.
  • On a fresh host, obtain the independent B2/crypt bootstrap material first. The only second copy currently documented is Beetroot’s standby config at /config/hermes-standby/.config/rclone/rclone.conf; use of that copy for a full blank-host restore is not rehearsed.
  • Once an authorized rclone config is mounted at /cfg/rclone.conf, copy the needed prefix to a scratch destination. Use copy, not sync, to avoid deleting the source:
mkdir -m 700 -p "$HOME/restore-stage"
# On the running joe-host; on a fresh host, first install the independently recovered config.
docker run --rm \
  -v "$HOME/.hermes/state/rclone.conf:/cfg/rclone.conf:ro" \
  -v "$HOME/restore-stage:/out" \
  rclone/rclone:latest copy immich-crypt:system-backups/workspace /out/workspace \
  --config /cfg/rclone.conf
  • Verify the copied artefact before using it (sha256sum, archive listing, or a database integrity check). The 27 Oct file-level drill of this document matched SHA-256 exactly; that is not a substitute for a service restore.

2. Workspace (trackers)

  • Restore immich-crypt:system-backups/workspace/ into ~/workspace/ (the workspace tree is mirrored without .git). Keep a copy of the existing tree until the restored files are checked.
  • The workspace Git history is a separate bundle at immich-crypt:system-backups/hermes/workspace-git.bundle. Copy it to scratch, run git bundle verify, then initialise/fetch and inspect git bundle list-heads before checking out a branch. Do not assume a particular branch name.
  • AGENTS.md and USER.md are in the workspace root files.
  • This reconstructs the data and Git history; a full workspace restore has not been rehearsed.

3. Hermes config + secrets

  • Fetch immich-crypt:system-backups/hermes/ into scratch. It contains config.yaml, jobs.json, memories.tar.gz, skills.tar.gz, scripts-repo.tar.gz, secrets.tar.gz, state.tar.gz, and Git bundles.
  • The live host had 105 cron definitions (101 enabled) at 27 Oct 23:45 UTC. Treat this only as a point-in-time inventory: use the restored jobs.json as the snapshot, then verify the resulting list with hermes cron list. Do not expect a fixed count.
  • Extract with a restrictive umask (umask 077) and set owner-only permissions on the restored files and directories. The current local staging directory is mode 0700 and its regular files are mode 0600.
  • secrets.tar.gz contains selected credential files. It is plaintext in the local staging directory, but the immich-crypt B2 remote encrypts objects client-side. Never print its contents. Do not blindly restore obsolete tokens; compare the explicit SECRETS list in hermes-state-backup.py with the current services.
  • state.tar.gz contains state.db, response_store.db, kanban.db, SOUL.md, checkpoints/, and state/. state/ includes the rclone config, but that copy cannot bootstrap its own encrypted download.
  • The current live OpenDesign provider uses key_env: OPENDESIGN_API_KEY. Five owner-only historical config backups still contain that same key inline for rollback; the 27 Oct 03:55 B2 mirror predates the move. Do not restore an older config.yaml into service without re-checking the provider stanza. The next scheduled mirror should pick up the current config and this revised runbook.
  • scripts-repo.tar.gz contains the script tree without .git; scripts-git.bundle and workspace-git.bundle carry history. Use git bundle verify before fetching.

4. Docker homelab

  • The homelab volume archives are under immich-crypt:system-backups/homelab/; the daily system mirror copies the local staging set. Inspect the actual tar names and embedded volume labels before restoring. The old hard-coded examples in earlier revisions were stale after stack/volume renames.
  • Live stack-source backup (verified 28 Oct 2026): Joe chose the host-path Compose files as canonical. The encrypted B2 object immich-crypt:system-backups/source-code/source-code-current.tar.gz contains the current /home/joe/stacks source and /home/joe/build/chat-test source/deploy files. It is 3,224,244 bytes; a read-back through the crypt remote matched the local SHA-256 exactly. The archive includes 372 files, including ten current .env files; it excludes node_modules, dist, and two stale .env.bak files. A scratch restore checked all 372 files against the sources, confirmed all ten restored .env files were mode 0600, and the scratch restore was then trashed. No .env values were read or printed. All twelve .env and .env.bak files are ignored and untracked; their modes are now 0600. system-b2-backup.py now refreshes the source archive and syncs its source-code subdirectory on the normal nightly run; next run is 29 Oct at 03:35 UTC. No stack was redeployed, and the full system mirror was not run manually.
  • Restore a volume tar only to the exact Docker volume expected by its current compose definition. Do not start the stack until the mapping is confirmed; avoid restoring an old tar over a live volume.
  • Open WebUI has its own archive under system-backups/owui/. The latest verified B2 object is owui-20261027-213542.tar.gz; the scheduled job history still shows 18 Oct, next run 1 Nov. The 28 Oct full mirror finished; a read-only name comparison now finds five local and five remote objects, with no remote-only objects.

5. Verify

  • Check restored files, Git bundles, and databases before starting services; use git bundle verify and SQLite PRAGMA quick_check where applicable.
  • Use system-validate.py, wiki-lint.py, homelab-backup-integrity.py, and backup-freshness-watch.py for their respective checks. The weekly freshness limit is now 192 hours (8 days, 24h grace); a green status still does not prove every scheduled producer ran.
  • drive-backup.py is retired. The current host mirror is system-b2-backup; HA snapshots use ha-b2-backup.
  • Test each service from its actual client path, not merely with a host-side HTTP status. No complete clean-host or whole-service restore has been proven yet.

Part B — Home Assistant Pi

  • Install HAOS on a new Pi/SD/USB and provide a snapshot through HA’s supported restore flow.
  • ha-b2-backup streams the latest /backup snapshot to immich-crypt:ha-backups/, retaining three. The 27 Oct newest listed object is 2ddd5faa.tar (3,917,271,040 bytes). We listed it in B2, but did not download or restore this HA archive.
  • The Home Assistant Google Drive Backup add-on is partial since 19 Oct: it excludes local_hermes_standby. Do not describe it as a full HA snapshot or assume it can restore that add-on.
  • After restore: Zigbee coordinator (Z2M/Matter hub) should come back with its config; if radio hardware also died, devices need re-pairing.
  • Verify the HA MCP add-on responds on :9583, ha-mcp-url matches, ISA/GIA sensors report, and dashboards (Walney, etc.) load.

Notes & knowns

  • Retention and deletion: retention is producer-specific. system-b2-backup mirrors local staging with rclone sync; deleting a local source file can delete its remote counterpart on the next run. Inspect both sides before pruning.
  • Encryption: secrets.tar.gz is plaintext on the live host but mode 0600 inside a mode-0700 staging tree. B2 objects under immich-crypt: are client-side encrypted. The encryption config is itself secret; keep an independent bootstrap copy outside B2.
  • Freshness checks: backup-freshness-watch.py runs daily; weekly backup pillars now have a 192-hour (8-day) limit, leaving 24h beyond a normal week. backup-integrity-check.py and homelab-backup-integrity.py check archives. A green watcher does not prove every scheduled producer ran (the Open WebUI example above).
  • Post-snapshot changes: the full system-B2 mirror completed ok at 03:41 UTC on 28 Oct. The source-code archive was copied separately later and read-back verified; the normal system mirror will refresh it at the next scheduled run. Runbook changes made after the 03:41 mirror are not yet in B2.
  • Restore proof: the 28 Oct source-code archive was restored into a disposable scratch directory and all 372 files matched the live source; the scratch copy was trashed. This proves source-file recovery, not a working stack, clean-host, Open WebUI, Immich, or HA restore.
  • Git remotes: the workspace and scripts repos have no remote. The /home/joe home repo also has no remote. The live stack source is now covered by the encrypted source-code-current.tar.gz object and the nightly system-B2 source list; its .env values are not committed to Git. Do not treat this as a clean-host or full service restore proof.

Graph View

Backlinks