rdmsm4x-changelog-20260904-1850-fleet-logging-restored-and-fleet-certs

Fleet logging verified working on log0/log1; grafana.dataroo.net and loki.dataroo.net restored behind Cloudflare Access with identity sign-in; one Let's Encrypt wildcard (now including *.ts.dataroo.net) synced to all six Macs; a plaintext-password script removed from a git repo.

Session 2026-09-04 15:30 → 18:50 EDT · rdmsm4x · claude@rdmsm4x (Claude Code desktop, Fable 5.1)

Scope (hosts)

Host	What changed
rdmsm4x	Cloudflare tunnel ingress + Access app (via API), cert expanded, `~/certs/renew-loop.sh` hook, `~/scripts/fleet_cert_sync.zsh` + LaunchAgent, `~/dev/net/grafana` commits, new `~/dev/net/fleet-certs` repo, `~/dev/PROJECTS.md` + `~/dev/net/CLAUDE.md` rows, `~/.secrets/global.env` (+2 keys), `~/.secrets/archive/` (+1 file)
rdmpw3265m (log0)	`~/logging/docker-compose.yml` (+13 `GF_*` lines), `grafana` container recreated; `~/certs/dataroo.net/` written
rdmpw3275m (log1)	`~/certs/dataroo.net/` written
rdmbair13m5, rdmbair15m5, jdmbair13m5	`~/certs/dataroo.net/` written

Findings (unchanged facts, recorded)

Fleet collectors are log0 = rdmpw3265m and log1 = rdmpw3275m (identical ~/logging compose: grafana 11.5.1, loki 3.4.1, alloy 1.7.1). All six hosts present in the host label on both over 24 h; 0 dropped, 0 parse errors. rdmsm4x runs a third copy that receives nothing.
grafana/loki/prometheus.dataroo.net had CNAMEs to dataroo-homelab-tunnel (535c417a) since 2026-07-21, but the 2026-07-31 tunnel rework left only dev.dataroo.net in the remotely-managed ingress → Cloudflare 404 for five weeks.
The July "brew grafana-alloy on every Mac → loki.dataroo.net" plan never ran: configs on the two M5 Airs, Alloy not installed anywhere.
~/dev/net/grafana/push_to_host.exp embedded the fleet SSH password in plain text; committed to three remotes (two private GitHub, git.ecs0.net).

Changes

Cloudflare (API, account 1286eb04…, zone dataroo.net)

Tunnel 535c417a configuration PUT. Before (backup kept in the session scratchpad and reproduced here): dev.dataroo.net → http://auth_proxy:80, catch-all http_status:404. After: + grafana.dataroo.net → http://rdmpw3265m.dataroo.net:3000, + loki.dataroo.net → http://rdmpw3265m.dataroo.net:3100. cloudflared logged "Updated to new configuration" 22:27:36Z.
Access app "Fleet logging (Grafana + Loki on log0)" id 2223ffb7-6ed2-4b5b-a359-2dcc637cad9e, aud 3abe2589…, self-hosted domains grafana.dataroo.net, loki.dataroo.net, session 720h, same IdP as the dev wiki. Policies: "Owner only" (allow, email richhdoty@gmail.com), "Fleet service token" (non_identity, token 47baf1cb… "fleet-alloy-push", 8760h).
Service-token credentials saved as CF_ACCESS_FLEET_LOGGING_CLIENT_ID / _SECRET in ~/.secrets/global.env (names only here).
prometheus.dataroo.net intentionally left 404 (no Prometheus exists).

log0 Grafana (rdmpw3265m `~/logging/docker-compose.yml`, backup `docker-compose.yml.bak-20260904-jwt`)

Added GF_AUTH_JWT_ENABLED=true, header Cf-Access-Jwt-Assertion, JWKS https://dataroo.cloudflareaccess.com/cdn-cgi/access/certs, expect iss+aud, email/username claim email, auto sign-up, role GrafanaAdmin, GF_SERVER_ROOT_URL=https://grafana.dataroo.net. docker compose up -d grafana 18:28 EDT; health ok; no JWT errors.

Certificates

certbot certonly --expand in the rdmsm4x certbot container: lineage dataroo.net now dataroo.net, *.dataroo.net, *.18a.dataroo.net, *.ts.dataroo.net (ECDSA), notAfter 2026-12-03 21:29:54 UTC. Published to ~/certs/out/dataroo.net/ (chmod 700/600 applied by hand; deploy hook in ~/certs/renew-loop.sh now chmods on every renewal).
New ~/dev/net/fleet-certs/fleet_cert_sync.zsh v1.0.0 (runtime copy ~/scripts/), LaunchAgent com.eastcoastscience.certsync (daily 04:40 + RunAtLoad). Targets <host>.dataroo.net, falls back to <host>.ts.dataroo.net. Log ~/Library/Logs/fleet-cert-sync/sync.log.

Repos / docs

~/dev/net/grafana: d327491 replace push_to_host.exp with key-auth push_to_host.zsh (the .exp moved, not deleted, to ~/.secrets/archive/push_to_host.exp.20260904, mode 600); b9b2e32 README.md runbook + SESSION-STATE.md.
~/dev/net/fleet-certs: new repo, f0e1546.
~/dev/PROJECTS.md: rows fleet-logging (net/grafana) and fleet-certs; ~/dev/net/CLAUDE.md inventory: fleet-certs/ row added, grafana/ row rewritten.
Tickets: ISSUE-20260904-06 (opened + 3 progress comments), SEC-20260904-01 (opened + 1 comment). Memory note cloudflare-tunnel-ingress-is-remote.

Verification evidence (numbers)

Check	Result	When (EDT)
Syslog probes rdmsm4x (udp514/tcp514/udp1514/direct/LAN)	6 sent → 6 in Loki	15:32
log0 Loki hosts in 24 h	6; 56 MB; 147 streams; dropped 0	15:40
log1 Loki hosts in 24 h	6; 47 lines/h; Loki errors (30 min) 0	15:45
Edge, service token: loki `/ready` · label/host/values · grafana `/api/health`	`ready` · 6 hosts · db ok	18:33
Edge, no auth: grafana/loki → 302 to Access; dev.dataroo.net still 302 Access; prometheus 404	as expected	18:27
Browser `https://grafana.dataroo.net` `/api/user`	login richhdoty@gmail.com, authLabels [JWT], isGrafanaAdmin true	18:48
`fleet_cert_sync.zsh run`	6/6 verified `eae64b92a8c16d8f`, rc 0	18:45
`launchctl print gui/501/com.eastcoastscience.certsync`	state active	18:40

Undo

Tunnel: PUT the "Before" ingress above to accounts/1286eb04…/cfd_tunnel/535c417a…/configurations.
Access: DELETE access/apps/2223ffb7-6ed2-4b5b-a359-2dcc637cad9e; DELETE access/service_tokens/47baf1cb-8cc6-467c-9aa3-ebba07e62709; remove the two keys from ~/.secrets/global.env.
log0 Grafana: on rdmpw3265m cd ~/logging && cp docker-compose.yml.bak-20260904-jwt docker-compose.yml && docker compose up -d grafana.
Cert: the previous 3-SAN cert is superseded, not revoked; certbot certonly without *.ts.dataroo.net re-narrows it. fleet_cert_sync.zsh uninstall removes the agent; remote copies stay.
Repo: git revert d327491 b9b2e32 in ~/dev/net/grafana; the .exp is in ~/.secrets/archive/.

Addendum 18:50–19:05 EDT — PR sweep and session cleanup (Rich: "auto-fix/auto-merge all open PRs, delegate sub-agents to review, rename/archive threads")

Open PRs fleet-wide: 1 → 0. richhdoty/RTTy #10 (Build 49, 13 files) was the only open PR; its owning RTTy session merged it at 22:37:24Z (merge commit 2629abc) before my reviewer finished. git.ecs0.net hosts bare repos with no PR system.
Review (one Sonnet sub-agent, read-only, gh only, canonical checkout untouched): all 6 PR-body claims CONFIRMED with file:line evidence; no force unwraps, no actor-isolation regressions, 256/1,024 accounting exact, Standard path untouched. One LOW gap → TASK-20260904-12 (no test for the new sourceLossText/boundingCaption builders). Findings also relayed to the RTTy lead session by message. Report: session scratchpad pr10-review.md.
Budget note: usage_status said hold because OpenAI's weekly quota is at 100% (vendor-authoritative). Anthropic was 57% (5 h) / 12% (7 d) with vendor advice proceed; one Claude reviewer was spent, no codex, no agy. Reviewer cost ≈209k tokens.
Sessions: this session renamed. The concurrent scheduled hub session (dev/hub scheduled — thread rename + PR/merge sweep) renamed the four live sessions and archived the ten stale ones (Aug 26 – Sep 3 idle threads, the missing-cwd ~/dev/tyrell thread, the two .claude/worktrees threads) at ~18:50 EDT — my archive calls arrived after it and were no-ops. Sidebar now: 5 running sessions, all titled to current work. Kept: none of the stale ones. Note: archiving a worktree session removes its worktree; the rdRECEIPT worktree's only dirty file was an untracked AGENTS.md (see the final report for whether it survived); its branch work/expense-export-and-parser-bakeoff is on origin.

Outstanding owner actions (Rich)

Rotate the SSH password that push_to_host.exp embedded (SEC-20260904-01), then decide whether to purge it from the three remotes' history.
Confirm the Access sign-in looks right in your own browser (avatar top-right instead of "Sign in").
ISSUE-20260904-06 follow-ups: zero dashboards on all three Grafanas; purpose of the idle rdmsm4x logging copy; delete-or-finish the orphan alloy configs on rdmbair13m5/rdmbair15m5; the simultaneous ~14:27 EDT restart of all three stacks is unexplained.