Whole-stack audit closeout: 28 Oct 2026

Whole-stack audit closeout

Method: evidence first, borrowing Joan Didion’s reportage: observed facts, then supported judgments.

Current handover: 28 Oct 2026, 17:33 UTC. The original audit snapshot closed at 00:48. Since then, Joe approved and I deployed the F-19 Access/destination guard, chose the live host path for stack source, and approved verified Immich transcode reclamation. The encrypted B2 source archive passed read-back and a 372-file scratch restore; scheduled state and system-B2 backups completed. After removing only audit-generated test artifacts from Trash, live LVM reads are pve/data 83.85% (about 60.21 GiB free) and vm-101-disk-1 90.70%, with /data 61 GiB free. The 14:15 watcher run succeeded but preserves pool warn by hysteresis until below 80%; the guest LV is ok. Full tests/cron: 1,482 passed, 0 failed, 2 skipped, 1 non-failing RuntimeWarning; the two former child-process failures now pass. The test fixes remain uncommitted and F-16/F-06 are not live. A host probe cannot reach Roomba 192.168.0.91; three image updates remain. Mnemosyne’s active-provider check had zero failures and no recall fallback, but found detached gist/embedding/vector references left untouched. The Life Admin spacing drift was normalized; system-validate.py and wiki-lint.py pass. The normalizer still reports 4 machine-classified changes in General Todo and 1 in Investment Gains; F-25 records the mismatch between General Todo’s human schema ownership and the normalizer’s machine exclusion. Neither file was rewritten. F-18 remains unresolved. No gateway restart, image pull, or further B2 write occurred. The CouchDB service was recreated with the unchanged local image as part of the credential rotation. Joe’s choices are recorded there: keep backups running, leave the test fix inactive, allow the managed updates, rely on Google Photos, and defer the full restore. CouchDB preflight found version 3.5.3, both system databases present, and a 26 Oct volume tar matching all 12 live files and the B2 read-back; no upgrade was tested on a copy. F-25 remains unchanged pending an ownership decision. F-27 is mitigated at the credential level. The former admin password is invalid (old auth 401; new auth 200); the new per-app Fossil/Tennis users are verified and in Vaultwarden. health-export.py uses the health-scoped account. The actual source is the host Compose project at /home/joe/stacks/28, not a Portainer-volume stack; its owner-only .env and the recreated same-image CouchDB container now use the new admin password. A verified pre-rotation volume backup covered all 12 files; no image was pulled. Existing phone sync settings still need the named Vaultwarden credentials; both remote DBs remain empty. Historical copies of the old password remain but cannot authenticate. Joe kept the route and declined 2FA; the exact Access audience remains unverified. No Cloudflare policy/route change or Git history rewrite was made.

Time tracked: the earlier closeout segment ran 23:29:04–00:48:02 UTC (1h 18m 58s). The 28 Oct follow-on was not separately time-tracked; prior audit sessions are not included in a combined total.

Current verdict

The two acute gaps Joe selected are closed: the public proxy now requires Access and permits only the pinned WebDAV target, and the live Compose/chat source is protected by a client-side-encrypted B2 archive. Historical copies of the former admin password remain in state/history but are invalid after rotation. F-27’s browser exposure is now contained: health-app v4.5 uses a same-origin API and a wellness-scoped health-app-proxy account; active JS contains no CouchDB credentials, and .bak URLs return 404. Production API probes and a disposable-browser offline/sync/delete test passed. The global admin password was rotated and verified (new authentication 200, old 401); the state file, host Compose .env, and recreated container environment now agree. health-export.py uses the scoped health account; separate Fossil/Tennis accounts are in Vaultwarden. The phone apps still need those credentials entered to restore sync, though both remote DBs are empty and Joe reports no unsynced data. The source is the host Compose project, not a Portainer-volume stack. The exact Access policy audience remains unverified; Joe chose to keep the route and not set up 2FA. CouchDB’s port is published on all host interfaces, though outside-LAN reachability is untested. No Cloudflare route/policy change was made; the global admin was rotated. The post-leak state archive is also in encrypted B2. Thin-pool pressure is mitigated: current pve/data is 83.85%, vm-101-disk-1 is 90.70%, and /data has 61 GiB free. The watcher retains a warning state until the pool drops below 80%; no resize is safe while it exceeds 75%. Only verified generated Immich outputs were removed; the 730 Motion Photo source clips under encoded-video remain in Google Photos, not in the B2 mirror. Mnemosyne passes functional health checks; detached gist/embedding/vector references remain unexplained and untouched. Roomba 192.168.0.91 is unreachable from the host, and several tracker tasks need Joe’s next call. F-25 records a low-priority mismatch: General Todo is marked human in the schema but treated as machine-owned by the normalizer, so its formatting drift remains untouched. The full cron suite has no failures, with one non-failing warning; Hermes changes remain uncommitted and not live.

Baseline findings at the 00:48 closeout

• F-19 (S1): public chat-memory proxy. A no-JWT request to the deployed Pages proxy, using a caller-supplied base URL, reached the configured WebDAV endpoint. Without Basic auth it returned upstream 401; with the configured Basic auth it returned HTTP 200 and the memory document (schema v1, 103 items, 2 tombstones). No item contents were printed. The public route has no Access JWT check or destination allowlist, so it is an open-proxy/SSRF risk and can expose the document to requests carrying those Basic credentials. I did not change the public route. Joe’s approval is required.
• F-20 (S1): live stack source is not protected. 43/44 containers point to Compose files under /home/joe/stacks; Portainer does not mount that directory. Of 35 host Compose files, 21 match a file in portainer_data, 5 differ, and 9 have no volume copy. The 26 Oct Portainer-volume tar is present in B2 and its streamed read-back SHA-256 matches the local archive. It does not back up the current host source for the 14 mismatched or missing stacks. The /home/joe Git repo has no remote and none of the 35 Compose files is tracked. The system-B2 source list excludes /home/joe/stacks. Twelve .env files had been unignored. I added narrow ignore rules and verified all 12 are ignored, with no matching path in Git history. I did not read their values, stage files, or commit.
• F-18 (S1): credential exposure and 03:35 backup. The historical CouchDB credential remains in the local session DB and protected rollback copies. A current-account check returned 401. That does not prove revocation elsewhere. The 02:45 state backup and 03:35 encrypted B2 sync remain scheduled. The latter also removes three remote-only OWUI objects unless retained locally. No pause, purge, rotation, or full sync was performed.
• PVE thin pool (S2): 86.61%. pve/data is 372.82 GiB with 49.92 GiB free. Metadata is 3.58%, and VG free space is 0. Do not resize the guest while the pool is above the 75% safety threshold. The guest’s 26 GiB root cap is healthy: 15.61 GiB used, 8.78 GiB writable, reserve complete. /data is 84% used with 51 GiB free.
• Home Assistant (S2): Supervisor/core health checks succeeded. Over the last five days, the log scan counted 2,237 Roomba connection failures and 481 Fellow authentication failures. Roomba remains setup_retry; Fellow is loaded. Thirteen of 14 add-ons are started. The Cloudflared add-on is stopped while the separate host container is running. ESPHome has an available update. No HA configuration was changed.
• Automation and updates: at 00:41, 105 cron definitions, 101 enabled and 4 paused. None lacked next_run_at or was overdue. Root-disk-cap last ran ok at 00:40:43 with next run 00:45; root-disk-tripwire last ran ok at 00:37:42 with next run 00:42. Seven jobs have no prior status, all with future first-run schedules (four one-shots, two monthly, one weekly). Nine Sunday jobs missed 25 Oct, and the F-16 fix is not live. Four image updates were classified: three Compose-managed and one standalone Cloudflared. No image was pulled. The Compose updater’s next run is 04:00.

What’s solid in the baseline

• All 44 containers were up, the root cap reserve is complete, and both disk monitors ran ok.
• The OWUI archive and 26 Oct Portainer-volume archive passed exact B2 read-back SHA-256 checks. The 14 Oct Immich database and six sample-file restore passed; no full service restore was claimed.
• All four shim listeners returned a real completion. Memory fail-closed and extraction tests passed.
• The Home Assistant Supervisor/core health probes and workspace tracker/report edits were checked read-only.

Changes made before the 00:48 handover

• Rechecked the host, Docker estate, cron metadata, HA, Mnemosyne, trackers/registers, app proxy, scripts and skill inventory.
• Verified the OWUI archive and the 26 Oct Portainer-volume archive by exact B2 read-back SHA-256. This confirms object integrity. It does not prove a whole-service restore.
• Updated RESTORE.md to state the live /home/joe/stacks recovery gap. Corrected the bills table structure and retained the £679 Klarna entry as unverified. Corrected the subscriptions receipt label and Life Admin update marker.
• Added the stack .env ignore rules to /home/joe/.gitignore. Joe has since chosen the live host-path source and protected it with an encrypted archive. The Joe-owned docker-homelab skill remains untouched; RESTORE.md now records the archive and restore limit.
• Memory extraction tests and browser/mobile layout tests passed. Script syntax sweeps passed. No dead-code deletion was justified by this pass.

At the 00:48 handover, no new B2 write, image pull, stack redeploy, gateway restart, Git commit, public Pages change, login, or outbound message had been made. Since then, the explicitly authorized source-code object was uploaded and read-back verified; the scheduled system-B2 mirror ran normally. No image pull, stack redeploy, gateway restart, or Git commit has occurred.

Decisions recorded and actions still open

1. F-18 credential/backups: Joe chose to keep both encrypted backup jobs running while I map the historical credential’s possible use. The next runs remain hermes-state-backup at 29 Oct 02:45 UTC and system-b2-backup at 03:35 UTC. No rotation or purge without a scoped plan.
2. F-16/F-06: Joe chose to leave the local fix uncommitted and inactive. The full tests/cron suite passed 1,482/0/2 with one non-failing RuntimeWarning. The 1 Nov weekly-housekeeping and OWUI-backup jobs will run with current live code unless he later chooses to activate the fix. No Git identity, commit or gateway restart is needed now.
3. Disk / recovery: no immediate disk action. Live pool/LV are 83.85%/90.70%; the 14:15 watcher retains warn until pool use drops below 80%. Keep the guard active and do not resize until the pool is at or below 75%. Joe chose to rely on Google Photos for the 730 Motion Photo source clips and defer a separate B2 or full-restore plan. The 471 removed outputs may need individual regeneration before playback.
4. HA/updates: Joe chose to allow the scheduled SearXNG/CouchDB updates at 29 Oct 04:00 and leave standalone Cloudflared unchanged. The updater remains enabled and no hold markers are present; no image was pulled. CouchDB is 3.5.3, both system databases are present, and its 26 Oct volume tar matches the current 12-file volume and B2 read-back. No upgrade test on a copy was run. Joe still needs to check Roomba’s router lease/reconnect it and reauthenticate Fellow through its normal UI. Review ESPHome 2026.10.1 release notes before updating.
5. Trackers: read the AESA DEHú notice; confirm Home Connect/Dunelm outcomes; check MCOL by 1 Nov; decide on the £250 offer; reconcile Klarna £679 from source evidence. No task-row content changed; one Life Admin summary cell received a whitespace-only normalization. Joe was unsure about F-25, so the current no-write behavior remains; the machine-classified formatting drift is recorded.
6. Recovery: Joe chose to defer a full host/service restore rehearsal. The source archive restored 372 files, but no full stack or host restore has been proved.
7. Chat app: the proxy guard is live. Model-provider hosts remain blocked until you provide the exact HTTPS origin.
8. Later: remaining skill splits and an explicit email-job tool allowlist remain undecided; Joe-owned/dirty skills were left untouched.
9. F-27: Joe kept the health-app route and chose not to set up 2FA. The exact Access audience remains unverified. The corrected scan found six local files/12 occurrences and two Git blobs in commit ff1d555ce5cd (16 Oct; workspace repo has no remote). The former admin password is treated as compromised, has been rotated, and the replacement verifies. I deployed health-app v4.5: same-origin API, server-side health-app-proxy scoped to wellness, no browser credentials, and 404 for .bak paths. Live probes and disposable-browser offline/sync/delete tests passed. The old admin password is invalid (new auth 200, old 401); host Compose .env and recreated container env are updated. health-export.py now uses the scoped account. Separate Fossil/Tennis credentials are stored in Vaultwarden; Joe still needs to enter them on the phones to restore sync. Both remote DBs are empty and he reports no unsynced data. No route/policy change, 2FA enrollment, or Git history rewrite was made.

My pick: Joe updates Fossil Log and Tennis Log on the phones using their Vaultwarden sync entries; then we verify sync via the database counters. The former admin is already invalid. Keep the route/no-2FA choice; do not rewrite Git history without a separate review. Keep encrypted backups running. Read the AESA notice first because its response deadline is unknown, then resolve Roomba’s address and Fellow authorization. Defer unrelated image updates and review Motion Photo B2 coverage separately; no B2 change has been made.

Live audit refresh — 28 Oct 2026, 15:11 UTC

Read-only verification against the 14:18 closeout. No user services, settings, backups, or tracker rows were changed. The temporary whole-tree disk scanner was stopped after it used about 60% of one core for 75 seconds; its temporary container was removed automatically. Per-directory attribution beyond Docker’s volume accounting remains unverified.

Disk and storage

• The physical NVMe (ZETTASTONE CP200 512GB) passes SMART. Health temperature is 41°C, wear used 2%, spare 100%, and the media-integrity/error-log counts are both zero. Proxmox’s 24-hour kernel scan showed no I/O, filesystem, or no-space errors.
• The current tight limit is the guest thin LV: vm-101-disk-1 is 320 GiB at 90.71%, leaving about 29.7 GiB of LV allocation headroom. The thin pool is 83.85% with 60.2 GiB free; guest /data is 80% used with about 60.5 GiB available. Therefore the practical next-write ceiling is about 29.7 GiB, not the larger df or pool number. Do not resize while the pool is above the 75% safety threshold. The guest GPT entry and Proxmox raw-disk table both report the same 671,084,511 sectors; the initial lsblk size discrepancy did not reproduce in the byte/sector check.
• The root cap remains healthy: 15.63/26 GiB of capped data, 5.32 GiB reserve complete, and 8.75 GiB writable. root-disk-cap and root-disk-tripwire report ok; the latter’s last run was 15:02. /var/lib/containerd is bind-mounted from /data with discard enabled; Docker’s data root is /data/var/lib/docker.
• The 12:37–14:59 disk-breaker samples oscillate around 83.8–84.5% pool use and 62–65 GiB free on /data, so the pool is high but not showing a sustained rise in that window. Docker reports 119.8 GB of volumes, including 114.1 GB for the Immich library, and 69.39 GB of images. Its 69.02 GB image reclaim estimate is not a safe reclaim promise; nothing was pruned.

Backup and host pressure

• Daily immich-b2-backup and system-b2-backup jobs completed ok at 03:32 and 03:41 today; ha-b2-backup completed ok at 07:56. HA’s local registry and disk both show the 28 Oct 07:00 partial snapshot, 3.77 GB.
• The weekly homelab-backup and owui-backup jobs remain enabled but their latest output files are 18 Oct; homelab-backup-integrity last ran 19 Oct. The expected 25 Oct weekly run is absent, and their next scheduled run is 1 Nov. This is still a recovery freshness gap; today’s system-B2 and HA backups do not prove those volume archives are current.
• joe-host is under memory pressure: 19/22 GiB RAM used, 3.1 GiB available, and 6.2/6.2 GiB swap used at 15:09. Kernel logs show global OOM kills of Chrome at 16:44 on 27 Oct and 13:11 on 28 Oct. Proxmox itself has 6.2 GiB available RAM and no swap configured. This is recorded as F-26; the cause has not been isolated.

Other live checks

• HA Core 2026.10.4 / OS 18.3 / Supervisor 2026.10.4 reports healthy: true; HA disk is 60.3/219.4 GB used. The Pi has 823 MiB available RAM and 924 MiB swap used; swap-in/out counters were zero. Watchman currently finds 3 unavailable and 1 unknown entity referenced by the active dashboard, plus 4 stale references in a saved dashboard backup; missing services is zero (last refreshed 27 Oct).
• system-validate.py exits 0; wiki-lint.py exits 0 with 0 findings across 123 notes. The normalizer still previews 4 General Todo formatting changes and 1 Investment Gains spacing change; F-25 remains unresolved. The workspace and Hermes scripts repositories already had uncommitted changes; those were preserved.
• Hermes state.db passes quick_check at 292.4 MB, with 34,350 messages and 143 sessions. MEMORY.md is 5,984/6,000 characters and USER.md 3,581/3,600. Mnemosyne’s SQLite DB passes quick_check at 31.2 MB; direct vector-table counts could not be queried because this shell’s SQLite lacks vec0, so the prior same-day recall/provider result remains the latest vector-health evidence.

Coverage limits

The latest full-stack audit above remains the broad service/configuration baseline. This refresh rechecked disk, backups, memory, selected HA health, and workspace validation. I did not repeat every integration, endpoint, image-registry, or restore probe. No backup was created, restored, or deleted; no cleanup or VM resize was attempted.