frankie (pg standby) needs db-perms.sh re-run when reachable #4

New Issue

grenade · 2026-07-02T06:28:50Z

grenade commented

2026-07-02 06:28:50 +00:00

frankie.hanzalova.internal was unreachable (no route to host 10.6.0.70) when the app-owned pg_ident.conf.d/moments.conf migration ran against magrathea on 2026-07-02 (#3).

The standby therefore still has the legacy host-named layout — and critically never received the frootmig.kosherinata.internal → moments_rw mapping at all, so a failover today would lock the worker out.

When frankie is back: run script/db-perms.sh from a workstation; it converges both pg hosts in one idempotent pass (migrates legacy lines into moments.conf, adds missing mappings, reloads postgres only if changed).

frankie.hanzalova.internal was unreachable (`no route to host 10.6.0.70`) when the app-owned `pg_ident.conf.d/moments.conf` migration ran against magrathea on 2026-07-02 (#3). The standby therefore still has the legacy host-named layout — and critically never received the `frootmig.kosherinata.internal → moments_rw` mapping at all, so a failover today would lock the worker out. When frankie is back: run `script/db-perms.sh` from a workstation; it converges both pg hosts in one idempotent pass (migrates legacy lines into `moments.conf`, adds missing mappings, reloads postgres only if changed).

grenade commented

2026-08-17 10:34:19 +00:00

Ran and verified. Both pg hosts are reachable again, and the standby now has the app-owned file with the two mappings that matter:

frankie:  moments.conf:cert_cn nikola.kosherinata.internal   moments_ro
          moments.conf:cert_cn frootmig.kosherinata.internal moments_rw

Live, not just on disk — postgres reloaded at 13:33:11 (pg_conf_load_time() confirms it, and the journal shows received SIGHUP, reloading configuration files right after the two appends to moments.conf).

That clears the failover risk. Worth recording that it was broader than this issue described: nikola → moments_ro was missing too, so a failover would have taken the api down as well, not only the worker.

Residual divergence, not blocking this

The primary carries four mappings, the standby two:

magrathea:  frootmig -> moments_rw, nikola -> moments_ro,
            roosta -> moments_rw,   roosta -> moments_ro
frankie:    frootmig -> moments_rw, nikola -> moments_ro

The two roosta.hanzalova.internal (workstation) lines are absent on frankie, and that follows from how db-perms.sh works: it converges each host to mapping_pairs — which holds only the api and worker hosts — plus whatever legacy moments_* lines already existed on that host. magrathea had the workstation lines to migrate out of its host-named files; frankie had none, so it got only the two.

So the script converges each host against itself rather than against a common desired set, and hosts can stay permanently divergent. Consequence today is small: after a failover, direct psql from the workstation as moments_rw/moments_ro would not authenticate against frankie (the ssh + sudo -u postgres path in asset/sql/bootstrap-moments.sql still works). Filed separately if worth fixing.

Ran and verified. Both pg hosts are reachable again, and the standby now has the app-owned file with the two mappings that matter: ``` frankie: moments.conf:cert_cn nikola.kosherinata.internal moments_ro moments.conf:cert_cn frootmig.kosherinata.internal moments_rw ``` Live, not just on disk — postgres reloaded at 13:33:11 (`pg_conf_load_time()` confirms it, and the journal shows `received SIGHUP, reloading configuration files` right after the two appends to `moments.conf`). That clears the failover risk. Worth recording that it was broader than this issue described: `nikola → moments_ro` was missing too, so a failover would have taken the api down as well, not only the worker. ### Residual divergence, not blocking this The primary carries four mappings, the standby two: ``` magrathea: frootmig -> moments_rw, nikola -> moments_ro, roosta -> moments_rw, roosta -> moments_ro frankie: frootmig -> moments_rw, nikola -> moments_ro ``` The two `roosta.hanzalova.internal` (workstation) lines are absent on frankie, and that follows from how `db-perms.sh` works: it converges each host to `mapping_pairs` — which holds only the api and worker hosts — plus whatever legacy `moments_*` lines already existed *on that host*. magrathea had the workstation lines to migrate out of its host-named files; frankie had none, so it got only the two. So the script converges each host against itself rather than against a common desired set, and hosts can stay permanently divergent. Consequence today is small: after a failover, direct `psql` from the workstation as `moments_rw`/`moments_ro` would not authenticate against frankie (the `ssh` + `sudo -u postgres` path in `asset/sql/bootstrap-moments.sql` still works). Filed separately if worth fixing.

grenade closed this issue

2026-08-17 10:34:23 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: grenade/moments#4