feat(deploy): manifest-driven config, teardown + db-perms, hardening

deploy.sh:
- never rsync into /; stage to /tmp on the remote and install at final
  paths via sudo bash heredoc, closing the parent-dir attribute leak
  that broke three hosts in the earlier rsync incident
- shell-quote heredoc args via ${var@Q}
- drop -A -X on the remaining (web) rsyncs
- generic worker.secrets loop reads (env-var → pass path) from manifest;
  GITEA_TOKEN now flows through automatically
- in-memory bash substitution for templates (secrets never on argv)
- simplify semanage port labelling: --add 2>/dev/null || --modify (the
  old grep pre-check matched only the first listed port)
- restorecon back to short flags (Fedora policycoreutils has no long
  forms; --recursive errored at deploy time)
- quieter health probe loop: curl diagnostics only on final failure

manifest as source of truth:
- api.config.bind drives BIND_ADDR, firewalld port, semanage label,
  health-probe URL
- web.config.{server_name,root,api_upstream} drives nginx render,
  rsync targets, restorecon scope
- nginx config renamed to site.conf.tmpl; firewalld svc to
  moments-api.xml.tmpl; both rendered at deploy time
- topology flip: api → nikola, worker → frootmig (anjie freed)

new scripts:
- script/teardown.sh: idempotent component teardown, never rsyncs,
  shared-state cleanup gated on absence of remaining env files,
  --remove-docroot guard against shallow / system paths
- script/db-perms.sh: rewritten — fixes grep/append role mismatch that
  appended duplicates on re-run, adds postgres reload, hits primary +
  standby in a single invocation

readme: genericized; deployment topology no longer carries real host
or site names.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

asset/config/api.env.tmpl

@@ -1,6 +1,6 @@
 JOURNAL_STREAM=1
 RUST_LOG=info,sqlx=warn,tower_http=info
 
-BIND_ADDR=0.0.0.0:42424
+BIND_ADDR={{BIND}}
 
 DATABASE_URL=postgres://moments_ro@magrathea.kosherinata.internal:5432/moments?sslmode=verify-full&sslrootcert=/etc/pki/ca-trust/source/anchors/root-internal.pem&sslcert=/etc/pki/tls/misc/{{HOSTNAME}}.pem&sslkey=/etc/pki/tls/private/{{HOSTNAME}}.pem

asset/config/worker.env.tmpl

@@ -10,6 +10,7 @@ SEARCH_POLL_INTERVAL_SECS=86400
 
 GITEA_HOST=git.lair.cafe
 GITEA_USER=grenade
+GITEA_TOKEN={{GITEA_TOKEN}}
 GITEA_POLL_INTERVAL_SECS=600
 
 HG_HOST=hg-edge.mozilla.org

asset/firewalld/moments-api.xml.tmpl

@@ -2,5 +2,5 @@
 <service>
   <short>moments-api</short>
   <description>moments read-only HTTP API</description>
-  <port protocol="tcp" port="42424"/>
+  <port protocol="tcp" port="{{API_PORT}}"/>
 </service>

asset/manifest.yml

@@ -3,7 +3,7 @@ environments:
   prod:
     components:
       api:
-        hosts: [anjie.kosherinata.internal]
+        hosts: [nikola.kosherinata.internal]
         config:
           bind: 0.0.0.0:42424
           db_role: moments_ro
@@ -11,7 +11,7 @@ environments:
           db_port: 5432
           db_name: moments
       worker:
-        hosts: [anjie.kosherinata.internal]
+        hosts: [frootmig.kosherinata.internal]
         config:
           db_role: moments_rw
           db_host: magrathea.kosherinata.internal
@@ -27,10 +27,10 @@ environments:
           bugzilla_email: rthijssen@mozilla.com
         secrets:
           GITHUB_TOKEN: github.com/grenade/admin-token
-          # GITEA_TOKEN, BUGZILLA_API_KEY: optional, omit unless required.
+          GITEA_TOKEN: git.lair.cafe/grenade/admin-token
       web:
         hosts: [oolon.kosherinata.internal]
         config:
           server_name: rob.tn
           root: /var/www/rob.tn
-          api_upstream: http://anjie.kosherinata.internal:42424
+          api_upstream: http://nikola.kosherinata.internal:42424

asset/nginx/site.conf.tmpl

@@ -1,18 +1,18 @@
 upstream moments_api {
-    server anjie.kosherinata.internal:42424 max_fails=3 fail_timeout=30s;
+    server {{API_UPSTREAM_ADDR}} max_fails=3 fail_timeout=30s;
     keepalive 8;
 }
 
 server {
-    server_name rob.tn;
+    server_name {{SERVER_NAME}};
     listen 443 ssl;
     http2 on;
 
-    ssl_certificate /etc/letsencrypt/live/rob.tn/fullchain.pem;
+    ssl_certificate /etc/letsencrypt/live/{{SERVER_NAME}}/fullchain.pem;
-    ssl_certificate_key /etc/letsencrypt/live/rob.tn/privkey.pem;
+    ssl_certificate_key /etc/letsencrypt/live/{{SERVER_NAME}}/privkey.pem;
     ssl_protocols TLSv1.2 TLSv1.3;
 
-    root /var/www/rob.tn;
+    root {{DOCROOT}};
     index index.html;
 
     location / {
@@ -28,7 +28,7 @@ server {
 
     location /api/ {
         rewrite ^/api/(.*)$ /$1 break;
-        proxy_pass http://moments_api;
+        proxy_pass {{API_UPSTREAM_SCHEME}}://moments_api;
         proxy_http_version 1.1;
         proxy_set_header Connection "";
         proxy_set_header Host $host;
@@ -38,6 +38,6 @@ server {
         proxy_connect_timeout 5s;
     }
 
-    access_log /var/log/nginx/rob.tn.access.log;
+    access_log /var/log/nginx/{{SERVER_NAME}}.access.log;
-    error_log  /var/log/nginx/rob.tn.error.log;
+    error_log  /var/log/nginx/{{SERVER_NAME}}.error.log;
 }

readme.md

@@ -1,6 +1,6 @@
 # moments
 
-Personal activity timeline for [rob.tn](https://rob.tn). Polls public sources (GitHub, Gitea, hg-edge.mozilla.org, bugzilla.mozilla.org), stores raw payloads in Postgres, and serves a reshaped timeline to a static React frontend.
+Personal activity timeline. Polls public sources (GitHub, Gitea, Mercurial, Bugzilla), stores raw payloads in Postgres, and serves a reshaped timeline to a static React frontend.
 
 Successor to the now-defunct [grenade-events-react](https://github.com/grenade/grenade-events-react), which depended on MongoDB Stitch (retired by MongoDB in September 2022).
 
@@ -28,7 +28,7 @@ cargo run -p moments-api      # serves on 127.0.0.1:8080
 cargo run -p moments-worker   # one-shot ingest tick (until --interval is wired up)
 ```
 
-The API expects a Postgres reachable at `DATABASE_URL`. For magrathea, that's an mTLS connection using the host cert. For local dev against a throwaway database:
+The API expects a Postgres reachable at `DATABASE_URL`. In production this is an mTLS connection using the host cert. For local dev against a throwaway database:
 
 ```sh
 DATABASE_URL=postgres://localhost/moments cargo run -p moments-api
@@ -39,37 +39,31 @@ Migrations live in `crates/moments-data/migrations/` and run automatically on wo
 ## Deployment
 
 ```sh
-./script/deploy.sh prod all          # api + worker + web
+./script/deploy.sh <env> all          # api + worker + web
-./script/deploy.sh prod api worker   # subset
+./script/deploy.sh <env> api worker   # subset
-./script/deploy.sh prod default      # api + web only (worker untouched)
+./script/deploy.sh <env> default      # api + web only (worker untouched)
-./script/deploy.sh prod all --dry-run
+./script/deploy.sh <env> all --dry-run
 ```
 
-Topology:
+Concrete hosts, ports, and the site's `server_name` live in `asset/manifest.yml`. The shape of the deployment:
 
-| Component | Host                              | Notes                                                              |
+| Component | Notes                                                                  |
-| --------- | --------------------------------- | ------------------------------------------------------------------ |
+| --------- | ---------------------------------------------------------------------- |
-| api       | `anjie.kosherinata.internal`      | binds `0.0.0.0:42424`; firewalld service `moments-api`             |
+| api       | binds the port from `api.config.bind`; firewalld service `moments-api` |
-| worker    | `anjie.kosherinata.internal`      | no listening port; pollers only                                    |
+| worker    | no listening port; pollers only                                        |
-| web       | `oolon.kosherinata.internal`      | per-site nginx ingress for rob.tn; `/api/*` → anjie across the WG  |
+| web       | per-site nginx ingress; `/api/*` reverse-proxies to the api host       |
-| db        | `magrathea.kosherinata.internal`  | postgres mTLS, passwordless                                        |
+| db        | postgres mTLS, passwordless                                            |
 
-api and worker are co-located on `anjie` while `nikola` and `frootmig` are out for drive replacement.
+Postgres roles `moments_rw` and `moments_ro` must exist on the primary, with `pg_ident.conf.d/<host>.conf` mapping the api host's FQDN → `moments_ro` and the worker host's FQDN → `moments_rw`. See `asset/sql/bootstrap-moments.sql`, `asset/postgres/ident.conf.tmpl`, and `script/db-perms.sh` (idempotently adds the cert_cn lines on the postgres primary + standby and reloads postgres).
 
-Postgres roles `moments_rw` and `moments_ro` must exist on the primary, with `pg_ident.conf` mapping `anjie.kosherinata.internal` to **both** roles (one cert_cn line per mapping). See `asset/sql/bootstrap-moments.sql` and `asset/postgres/ident.conf.tmpl`.
+Inter-host traffic over the WG mesh: web's nginx connects to the api host in plaintext. The mesh provides the encryption layer; per-hop TLS for an internal HTTP read-only API on already-public data is deferred. If that changes, swap the api binary to rustls + the host cert pair, and update the nginx upstream to `https://`.
 
-Inter-host traffic over the WG mesh: oolon's nginx connects to `http://anjie.kosherinata.internal:42424` in plaintext. The mesh provides the encryption layer; per-hop TLS for an internal HTTP read-only API on already-public data is deferred. If that changes, swap the api binary to rustls + the host cert pair, and update the nginx upstream to `https://`.
+Secrets are resolved at deploy time via `pass`. The mapping of env-var name → pass-store path lives under `worker.secrets` in `manifest.yml`; `deploy.sh` iterates the map, fetches each secret, and substitutes the matching `{{NAME}}` placeholder in `worker.env.tmpl`. To add a secret: add a `worker.secrets` entry, add `NAME={{NAME}}` to `worker.env.tmpl`, and ensure `pass show <path>` returns the value on the deploying machine.
 
-Secrets resolved by `deploy.sh` via `pass`:
+### First-time setup
 
-- `github.com/grenade/admin-token` — GitHub PAT for events + search APIs (worker only).
+After the first successful prod deploy:
 
-Optional, set if needed in `worker.env`: `GITEA_TOKEN`, `BUGZILLA_API_KEY`.
+1. Point public DNS for the site at the web host's public IP (unproxied).
-
+2. Confirm `curl --fail --silent --show-error https://<site>/api/v1/healthz` returns `ok`.
-### DNS cutover
+3. If migrating from a predecessor, archive the old repo with a pointer to this one.
-
-`rob.tn` currently resolves to GitHub Pages. After the first successful prod deploy:
-
-1. Update Cloudflare DNS for `rob.tn` to the WAN IP that fronts `oolon` (unproxied — see architecture doc §11).
-2. Confirm `curl -fsS https://rob.tn/api/v1/healthz` returns `ok`.
-3. Add an archival notice to the top of [grenade-events-react/readme.md](https://github.com/grenade/grenade-events-react) pointing at this repo, and archive the GitHub repo.

script/db-perms.sh

@@ -0,0 +1,63 @@
+#!/usr/bin/env bash
+#
+# Idempotently add cert_cn → role mappings to pg_ident.conf.d on the moments
+# postgres primary and standby, then reload postgres so the changes take
+# effect. Re-running is a no-op (no duplicate lines, no spurious reload).
+#
+# Run from a workstation with ssh access to both pg hosts. This script ssh's
+# out; do NOT run it on magrathea/frankie directly.
+
+set -euo pipefail
+
+api_host=nikola.kosherinata.internal
+worker_host=frootmig.kosherinata.internal
+
+pg_hosts=(
+    magrathea.kosherinata.internal
+    frankie.kosherinata.internal
+)
+
+# Each (cert_cn host, role) pair becomes one cert_cn line in
+# pg_ident.conf.d/<cert_cn host>.conf on every pg host listed above.
+mapping_pairs=(
+    "$api_host"    moments_ro
+    "$worker_host" moments_rw
+)
+
+ident_dir=/var/lib/pgsql/18/data/pg_ident.conf.d
+
+for pg_host in "${pg_hosts[@]}"; do
+    printf '==> %s\n' "$pg_host"
+    ssh -o BatchMode=yes "$pg_host" "sudo bash -s -- ${ident_dir@Q} ${mapping_pairs[@]@Q}" <<'REMOTE_EOF'
+set -euo pipefail
+ident_dir="$1"; shift
+
+changed=0
+while [[ $# -gt 0 ]]; do
+    cert_cn_host="$1"
+    role="$2"
+    shift 2
+
+    line="cert_cn ${cert_cn_host} ${role}"
+    file="${ident_dir}/${cert_cn_host}.conf"
+
+    # The heredoc runs as root via sudo bash, so [[ -f ]] and grep are fine
+    # without dropping privs. tee --append runs as postgres so a newly-created
+    # file lands with the conventional postgres:postgres ownership.
+    if [[ -f "$file" ]] && grep --fixed-strings --line-regexp --quiet -- "$line" "$file"; then
+        printf '  present: %s\n' "$line"
+    else
+        printf '%s\n' "$line" | sudo -u postgres tee --append "$file" >/dev/null
+        printf '  added:   %s\n' "$line"
+        changed=1
+    fi
+done
+
+if (( changed )); then
+    systemctl reload postgresql-18
+    echo "  reloaded postgresql-18"
+else
+    echo "  no changes; reload skipped"
+fi
+REMOTE_EOF
+done

script/deploy.sh

@@ -68,10 +68,10 @@ command -v cargo >/dev/null 2>&1        || die "cargo is required"
 # Resolve component list ----------------------------------------------------
 
 env_path=".environments.${environment}"
-yq -e "${env_path}" "$manifest" >/dev/null \
+yq --exit-status "${env_path}" "$manifest" >/dev/null \
     || die "environment '$environment' not found in manifest"
 
-mapfile -t all_components < <(yq -r "${env_path}.components | keys | .[]" "$manifest")
+mapfile -t all_components < <(yq --raw-output "${env_path}.components | keys | .[]" "$manifest")
 
 if [[ ${#components[@]} -eq 0 ]]; then
     usage
@@ -108,75 +108,116 @@ deploy_api() {
     local host="$1"
     log "api -> $host"
 
+    local bind
+    bind="$(yq --raw-output "${env_path}.components.api.config.bind" "$manifest")"
+    [[ -n "$bind" && "$bind" != "null" ]] || die "api.config.bind missing in manifest"
+    [[ "$bind" == *:* ]] \
+        || die "api.config.bind must be host:port form: '$bind'"
+
+    local api_port
+    api_port="${bind##*:}"
+    [[ "$api_port" =~ ^[0-9]+$ ]] \
+        || die "api.config.bind port is not numeric: '$api_port'"
+
     if (( dry_run )); then
-        printf '\033[2m[dry-run]\033[0m render api.env (HOSTNAME=%s) + units, rsync to %s:/, run sysusers/restorecon/semanage/systemctl on %s\n' \
+        printf '\033[2m[dry-run]\033[0m render api.env (HOSTNAME=%s, BIND=%s) + firewalld svc (port=%s) + units, stage to %s:/tmp/, install via heredoc, run sysusers/restorecon/semanage/systemctl on %s\n' \
-            "$host" "$host" "$host" >&2
+            "$host" "$bind" "$api_port" "$host" "$host" >&2
         return 0
     fi
 
-    local fqdn
+    local fqdn="$host"
-    fqdn="$host"
 
     local stage
-    stage="$(mktemp -d)"
+    stage="$(mktemp --directory)"
-    trap "rm -rf '$stage'" RETURN
+    trap "rm --recursive --force '$stage'" RETURN
 
-    install -d "$stage/etc/moments" "$stage/etc/systemd/system" "$stage/etc/sysusers.d" "$stage/etc/firewalld/services" "$stage/usr/local/bin"
+    install --directory \
+        "$stage/etc/moments" \
+        "$stage/etc/systemd/system" \
+        "$stage/etc/sysusers.d" \
+        "$stage/etc/firewalld/services" \
+        "$stage/usr/local/bin"
 
-    # Render env file with hostname substitution.
+    local rendered
-    sed "s|{{HOSTNAME}}|${fqdn}|g" "${repo_root}/asset/config/api.env.tmpl" \
+    rendered="$(<"${repo_root}/asset/config/api.env.tmpl")"
-        > "$stage/etc/moments/api.env"
+    rendered=${rendered//'{{HOSTNAME}}'/$fqdn}
+    rendered=${rendered//'{{BIND}}'/$bind}
+    printf '%s\n' "$rendered" > "$stage/etc/moments/api.env"
 
-    sed "s|{{HOSTNAME}}|${fqdn}|g" "${repo_root}/asset/systemd/moments-api-cert.path" \
+    rendered="$(<"${repo_root}/asset/systemd/moments-api-cert.path")"
-        > "$stage/etc/systemd/system/moments-api-cert.path"
+    rendered=${rendered//'{{HOSTNAME}}'/$fqdn}
+    printf '%s\n' "$rendered" > "$stage/etc/systemd/system/moments-api-cert.path"
 
-    install -m 0644 "${repo_root}/asset/systemd/moments-api.service"             "$stage/etc/systemd/system/"
+    rendered="$(<"${repo_root}/asset/firewalld/moments-api.xml.tmpl")"
-    install -m 0644 "${repo_root}/asset/systemd/moments-api-cert-reload.service" "$stage/etc/systemd/system/"
+    rendered=${rendered//'{{API_PORT}}'/$api_port}
-    install -m 0644 "${repo_root}/asset/systemd/moments.sysusers.conf"           "$stage/etc/sysusers.d/moments.conf"
+    printf '%s\n' "$rendered" > "$stage/etc/firewalld/services/moments-api.xml"
-    install -m 0644 "${repo_root}/asset/firewalld/moments-api.xml"               "$stage/etc/firewalld/services/moments-api.xml"
+    chmod 0644 "$stage/etc/firewalld/services/moments-api.xml"
-    install -m 0755 "${repo_root}/target/release/moments-api"                    "$stage/usr/local/bin/moments-api"
+
+    install --mode=0644 "${repo_root}/asset/systemd/moments-api.service"             "$stage/etc/systemd/system/"
+    install --mode=0644 "${repo_root}/asset/systemd/moments-api-cert-reload.service" "$stage/etc/systemd/system/"
+    install --mode=0644 "${repo_root}/asset/systemd/moments.sysusers.conf"           "$stage/etc/sysusers.d/moments.conf"
+    install --mode=0755 "${repo_root}/target/release/moments-api"                    "$stage/usr/local/bin/moments-api"
 
-    # Permissions on the rendered env: root-owned, moments group readable.
     chmod 0640 "$stage/etc/moments/api.env"
 
-    if (( dry_run )); then
+    # Stage to a tmpdir on the remote, then `install` each file at its final
-        printf '\033[2m[dry-run]\033[0m rsync staged -> %s:/\n' "$host" >&2
+    # path via the heredoc. Never rsync into /, since rsync of staged parent
-    else
+    # dirs (etc/, usr/, ...) can leak ownership, ACLs and xattrs onto the
-        rsync \
+    # live system dirs.
-            --archive \
+    local remote_stage="/tmp/moments-deploy.api.${$}.${RANDOM}"
-            --hard-links \
-            --acls \
-            --xattrs \
-            --numeric-ids \
-            --chown root:root \
-            --rsync-path 'sudo rsync' \
-            "$stage/" \
-            "${host}:/"
-    fi
 
-    ssh_run "$host" "sudo bash -s" <<'REMOTE_EOF'
+    rsync \
+        --archive \
+        --hard-links \
+        --numeric-ids \
+        --rsh='ssh -o BatchMode=yes' \
+        "$stage/" \
+        "${host}:${remote_stage}/"
+
+    ssh_run "$host" "sudo bash -s -- ${remote_stage@Q} ${api_port@Q}" <<'REMOTE_EOF'
 set -euo pipefail
-fqdn="$(hostname -f)"
+remote_stage="$1"
+api_port="$2"
+trap 'rm --recursive --force "$remote_stage"' EXIT
 
+fqdn="$(hostname --fqdn)"
+
+install --owner=root --group=root --mode=0644 \
+    "$remote_stage/etc/sysusers.d/moments.conf" \
+    /etc/sysusers.d/moments.conf
 systemd-sysusers /etc/sysusers.d/moments.conf
 
-install -d -o root    -g moments -m 0750 /etc/moments
+install --directory --owner=root    --group=moments --mode=0750 /etc/moments
-install -d -o moments -g moments -m 0750 /var/lib/moments
+install --directory --owner=moments --group=moments --mode=0750 /var/lib/moments
-chown root:moments /etc/moments/api.env
-chmod 0640        /etc/moments/api.env
 
-# Grant the moments user read access to the host private key — required for
+install --owner=root --group=moments --mode=0640 \
-# the postgres mTLS connection.
+    "$remote_stage/etc/moments/api.env" \
-setfacl -m u:moments:r "/etc/pki/tls/private/${fqdn}.pem" || true
+    /etc/moments/api.env
+install --owner=root --group=root --mode=0644 \
+    "$remote_stage/etc/systemd/system/moments-api.service" \
+    /etc/systemd/system/moments-api.service
+install --owner=root --group=root --mode=0644 \
+    "$remote_stage/etc/systemd/system/moments-api-cert.path" \
+    /etc/systemd/system/moments-api-cert.path
+install --owner=root --group=root --mode=0644 \
+    "$remote_stage/etc/systemd/system/moments-api-cert-reload.service" \
+    /etc/systemd/system/moments-api-cert-reload.service
+install --owner=root --group=root --mode=0644 \
+    "$remote_stage/etc/firewalld/services/moments-api.xml" \
+    /etc/firewalld/services/moments-api.xml
+install --owner=root --group=root --mode=0755 \
+    "$remote_stage/usr/local/bin/moments-api" \
+    /usr/local/bin/moments-api
 
-# Label the API port. Idempotent — the -m fallback turns "already labelled"
+# Grant the moments user read access to the host private key for the
-# into a no-op.
+# postgres mTLS connection.
-if ! semanage port -l | awk '{print $1, $3}' | grep -qE "^http_port_t .*42424"; then
+setfacl --modify=u:moments:r "/etc/pki/tls/private/${fqdn}.pem" || true
-    semanage port -a -t http_port_t -p tcp 42424 || \
+
-    semanage port -m -t http_port_t -p tcp 42424
+# Idempotent label: --add fails if the port is already labelled (we suppress
-fi
+# that one stderr line); --modify is then a no-op or fixes a stale type.
+semanage port --add    --type=http_port_t --proto=tcp "$api_port" 2>/dev/null \
+    || semanage port --modify --type=http_port_t --proto=tcp "$api_port"
 
-# Firewalld: install the named service and enable it in the default zone.
 firewall-cmd --reload
 zone="$(firewall-cmd --get-default-zone)"
 if ! firewall-cmd --zone="$zone" --query-service=moments-api >/dev/null 2>&1; then
@@ -191,17 +232,18 @@ systemctl enable --now moments-api-cert.path
 systemctl enable --now moments-api.service
 systemctl restart    moments-api.service
 
-# Health probe — hit the bound interface, not loopback, so we exercise the
+# Quietly retry while the service binds; only show curl's diagnostics if
-# same path nginx will use from oolon.
+# every attempt fails. The journalctl tail on failure is the verbose source.
 for i in 1 2 3 4 5 6 7 8 9 10; do
-    if curl -fsS "http://${fqdn}:42424/v1/healthz" >/dev/null; then
+    if curl --fail --silent "http://${fqdn}:${api_port}/v1/healthz" >/dev/null 2>&1; then
         echo "moments-api healthy"
         exit 0
     fi
     sleep 1
 done
 echo "moments-api did not become healthy" >&2
-journalctl -u moments-api.service -n 50 --no-pager >&2
+curl --fail --silent --show-error "http://${fqdn}:${api_port}/v1/healthz" >/dev/null || true
+journalctl --unit=moments-api.service --lines=50 --no-pager >&2
 exit 1
 REMOTE_EOF
 }
@@ -210,69 +252,112 @@ deploy_worker() {
     local host="$1"
     log "worker -> $host"
 
+    # Manifest entries under `worker.secrets` map env-var name -> pass store path.
+    # The script fetches each via `pass` and substitutes the matching {{NAME}}
+    # placeholder in worker.env.tmpl. Adding a new secret is then a manifest +
+    # template change; no script edit required.
+    local -a secret_lines secret_keys
+    mapfile -t secret_lines < <(yq --raw-output \
+        "${env_path}.components.worker.secrets // {} | to_entries | .[] | \"\(.key)=\(.value)\"" \
+        "$manifest")
+    local line
+    for line in "${secret_lines[@]}"; do
+        [[ -n "$line" ]] && secret_keys+=("${line%%=*}")
+    done
+
     if (( dry_run )); then
-        printf '\033[2m[dry-run]\033[0m render worker.env (HOSTNAME=%s, GITHUB_TOKEN from pass) + units, rsync to %s:/, run sysusers/restorecon/systemctl on %s\n' \
+        printf '\033[2m[dry-run]\033[0m render worker.env (HOSTNAME=%s, secrets [%s] from pass) + units, stage to %s:/tmp/, install via heredoc, run sysusers/restorecon/systemctl on %s\n' \
-            "$host" "$host" "$host" >&2
+            "$host" "${secret_keys[*]:-none}" "$host" "$host" >&2
         return 0
     fi
 
-    local fqdn
+    local fqdn="$host"
-    fqdn="$host"
-
-    local github_token=""
-    if pass show github.com/grenade/admin-token >/dev/null 2>&1; then
-        github_token="$(pass show github.com/grenade/admin-token)"
-    else
-        warn "no github admin-token in pass; worker will run without GITHUB_TOKEN"
-    fi
 
     local stage
-    stage="$(mktemp -d)"
+    stage="$(mktemp --directory)"
-    trap "rm -rf '$stage'" RETURN
+    trap "rm --recursive --force '$stage'" RETURN
 
-    install -d "$stage/etc/moments" "$stage/etc/systemd/system" "$stage/etc/sysusers.d" "$stage/usr/local/bin"
+    install --directory \
+        "$stage/etc/moments" \
+        "$stage/etc/systemd/system" \
+        "$stage/etc/sysusers.d" \
+        "$stage/usr/local/bin"
 
-    sed -e "s|{{HOSTNAME}}|${fqdn}|g" \
+    # Render templates in-memory so secrets never appear on a command line
-        -e "s|{{GITHUB_TOKEN}}|${github_token}|g" \
+    # (sed would expose them to anything that can read /proc/<pid>/cmdline).
-        "${repo_root}/asset/config/worker.env.tmpl" > "$stage/etc/moments/worker.env"
+    local rendered
+    rendered="$(<"${repo_root}/asset/config/worker.env.tmpl")"
+    rendered=${rendered//'{{HOSTNAME}}'/$fqdn}
+    local key pass_path value
+    for line in "${secret_lines[@]}"; do
+        [[ -z "$line" ]] && continue
+        key="${line%%=*}"
+        pass_path="${line#*=}"
+        if pass show "$pass_path" >/dev/null 2>&1; then
+            value="$(pass show "$pass_path")"
+        else
+            warn "no secret in pass at '${pass_path}' for ${key}; worker will run without ${key}"
+            value=""
+        fi
+        rendered=${rendered//"{{${key}}}"/$value}
+    done
+    printf '%s\n' "$rendered" > "$stage/etc/moments/worker.env"
 
-    sed "s|{{HOSTNAME}}|${fqdn}|g" "${repo_root}/asset/systemd/moments-worker-cert.path" \
+    rendered="$(<"${repo_root}/asset/systemd/moments-worker-cert.path")"
-        > "$stage/etc/systemd/system/moments-worker-cert.path"
+    rendered=${rendered//'{{HOSTNAME}}'/$fqdn}
+    printf '%s\n' "$rendered" > "$stage/etc/systemd/system/moments-worker-cert.path"
 
-    install -m 0644 "${repo_root}/asset/systemd/moments-worker.service"             "$stage/etc/systemd/system/"
+    install --mode=0644 "${repo_root}/asset/systemd/moments-worker.service"             "$stage/etc/systemd/system/"
-    install -m 0644 "${repo_root}/asset/systemd/moments-worker-cert-reload.service" "$stage/etc/systemd/system/"
+    install --mode=0644 "${repo_root}/asset/systemd/moments-worker-cert-reload.service" "$stage/etc/systemd/system/"
-    install -m 0644 "${repo_root}/asset/systemd/moments.sysusers.conf"              "$stage/etc/sysusers.d/moments.conf"
+    install --mode=0644 "${repo_root}/asset/systemd/moments.sysusers.conf"              "$stage/etc/sysusers.d/moments.conf"
-    install -m 0755 "${repo_root}/target/release/moments-worker"                    "$stage/usr/local/bin/moments-worker"
+    install --mode=0755 "${repo_root}/target/release/moments-worker"                    "$stage/usr/local/bin/moments-worker"
 
     chmod 0640 "$stage/etc/moments/worker.env"
 
-    if (( dry_run )); then
+    # Stage to a tmpdir on the remote, then `install` each file at its final
-        printf '\033[2m[dry-run]\033[0m rsync staged -> %s:/\n' "$host" >&2
+    # path via the heredoc. Never rsync into /.
-    else
+    local remote_stage="/tmp/moments-deploy.worker.${$}.${RANDOM}"
-        rsync \
-            --archive \
-            --hard-links \
-            --acls \
-            --xattrs \
-            --numeric-ids \
-            --chown root:root \
-            --rsync-path 'sudo rsync' \
-            "$stage/" \
-            "${host}:/"
-    fi
 
-    ssh_run "$host" "sudo bash -s" <<'REMOTE_EOF'
+    rsync \
+        --archive \
+        --hard-links \
+        --numeric-ids \
+        --rsh='ssh -o BatchMode=yes' \
+        "$stage/" \
+        "${host}:${remote_stage}/"
+
+    ssh_run "$host" "sudo bash -s -- ${remote_stage@Q}" <<'REMOTE_EOF'
 set -euo pipefail
-fqdn="$(hostname -f)"
+remote_stage="$1"
+trap 'rm --recursive --force "$remote_stage"' EXIT
 
+fqdn="$(hostname --fqdn)"
+
+install --owner=root --group=root --mode=0644 \
+    "$remote_stage/etc/sysusers.d/moments.conf" \
+    /etc/sysusers.d/moments.conf
 systemd-sysusers /etc/sysusers.d/moments.conf
 
-install -d -o root    -g moments -m 0750 /etc/moments
+install --directory --owner=root    --group=moments --mode=0750 /etc/moments
-install -d -o moments -g moments -m 0750 /var/lib/moments
+install --directory --owner=moments --group=moments --mode=0750 /var/lib/moments
-chown root:moments /etc/moments/worker.env
-chmod 0640        /etc/moments/worker.env
 
-setfacl -m u:moments:r "/etc/pki/tls/private/${fqdn}.pem" || true
+install --owner=root --group=moments --mode=0640 \
+    "$remote_stage/etc/moments/worker.env" \
+    /etc/moments/worker.env
+install --owner=root --group=root --mode=0644 \
+    "$remote_stage/etc/systemd/system/moments-worker.service" \
+    /etc/systemd/system/moments-worker.service
+install --owner=root --group=root --mode=0644 \
+    "$remote_stage/etc/systemd/system/moments-worker-cert.path" \
+    /etc/systemd/system/moments-worker-cert.path
+install --owner=root --group=root --mode=0644 \
+    "$remote_stage/etc/systemd/system/moments-worker-cert-reload.service" \
+    /etc/systemd/system/moments-worker-cert-reload.service
+install --owner=root --group=root --mode=0755 \
+    "$remote_stage/usr/local/bin/moments-worker" \
+    /usr/local/bin/moments-worker
+
+setfacl --modify=u:moments:r "/etc/pki/tls/private/${fqdn}.pem" || true
 
 restorecon -Rv /usr/local/bin/moments-worker /etc/moments /var/lib/moments
 
@@ -281,9 +366,8 @@ systemctl enable --now moments-worker-cert.path
 systemctl enable --now moments-worker.service
 systemctl restart    moments-worker.service
 
-# Liveness probe — worker doesn't expose a port, so check is-active.
 if ! systemctl is-active --quiet moments-worker.service; then
-    journalctl -u moments-worker.service -n 50 --no-pager >&2
+    journalctl --unit=moments-worker.service --lines=50 --no-pager >&2
     exit 1
 fi
 echo "moments-worker active"
@@ -294,61 +378,93 @@ deploy_web() {
     local host="$1"
     log "web -> $host"
 
+    local server_name web_root api_upstream
+    server_name="$(yq --raw-output "${env_path}.components.web.config.server_name" "$manifest")"
+    web_root="$(yq    --raw-output "${env_path}.components.web.config.root"        "$manifest")"
+    api_upstream="$(yq --raw-output "${env_path}.components.web.config.api_upstream" "$manifest")"
+    [[ -n "$server_name"  && "$server_name"  != "null" ]] || die "web.config.server_name missing in manifest"
+    [[ -n "$web_root"     && "$web_root"     != "null" ]] || die "web.config.root missing in manifest"
+    [[ -n "$api_upstream" && "$api_upstream" != "null" ]] || die "web.config.api_upstream missing in manifest"
+    [[ "$web_root" == /* ]] \
+        || die "web.config.root must be an absolute path: '$web_root'"
+    [[ "$api_upstream" == http://*  || "$api_upstream" == https://* ]] \
+        || die "web.config.api_upstream must be a http(s) URL: '$api_upstream'"
+
+    local api_upstream_scheme api_upstream_addr api_upstream_port
+    api_upstream_scheme="${api_upstream%%://*}"
+    api_upstream_addr="${api_upstream#*://}"
+    [[ "$api_upstream_addr" == *:* ]] \
+        || die "web.config.api_upstream must include an explicit port: '$api_upstream'"
+    api_upstream_port="${api_upstream_addr##*:}"
+    [[ "$api_upstream_port" =~ ^[0-9]+$ ]] \
+        || die "extracted upstream port is not numeric: '$api_upstream_port'"
+
+    local site_conf_path="/etc/nginx/conf.d/${server_name}.conf"
+
     if (( dry_run )); then
-        printf '\033[2m[dry-run]\033[0m rsync ui/dist/ to %s:/var/www/rob.tn/ + nginx config, run nginx -t/reload on %s\n' \
+        printf '\033[2m[dry-run]\033[0m render %s (server_name=%s, docroot=%s, upstream=%s://%s) + rsync ui/dist/ to %s:%s/, run nginx -t/reload on %s\n' \
-            "$host" "$host" >&2
+            "$site_conf_path" "$server_name" "$web_root" \
+            "$api_upstream_scheme" "$api_upstream_addr" \
+            "$host" "$web_root" "$host" >&2
         return 0
     fi
 
     local stage
-    stage="$(mktemp -d)"
+    stage="$(mktemp --directory)"
-    trap "rm -rf '$stage'" RETURN
+    trap "rm --recursive --force '$stage'" RETURN
 
-    install -d "$stage/var/www/rob.tn" "$stage/etc/nginx/conf.d"
+    install --directory "${stage}${web_root}" "$stage/etc/nginx/conf.d"
 
-    rsync -a "${repo_root}/ui/dist/" "$stage/var/www/rob.tn/"
+    rsync --archive "${repo_root}/ui/dist/" "${stage}${web_root}/"
-    install -m 0644 "${repo_root}/asset/nginx/rob.tn.conf" "$stage/etc/nginx/conf.d/rob.tn.conf"
 
-    if (( dry_run )); then
+    local rendered
-        printf '\033[2m[dry-run]\033[0m rsync staged -> %s:/\n' "$host" >&2
+    rendered="$(<"${repo_root}/asset/nginx/site.conf.tmpl")"
-    else
+    rendered=${rendered//'{{SERVER_NAME}}'/$server_name}
-        rsync \
+    rendered=${rendered//'{{DOCROOT}}'/$web_root}
-            --archive \
+    rendered=${rendered//'{{API_UPSTREAM_SCHEME}}'/$api_upstream_scheme}
-            --hard-links \
+    rendered=${rendered//'{{API_UPSTREAM_ADDR}}'/$api_upstream_addr}
-            --acls \
+    printf '%s\n' "$rendered" > "${stage}${site_conf_path}"
-            --xattrs \
+    chmod 0644 "${stage}${site_conf_path}"
-            --numeric-ids \
-            --chown root:root \
-            --rsync-path 'sudo rsync' \
-            --delete \
-            "$stage/var/www/rob.tn/" \
-            "${host}:/var/www/rob.tn/"
-        rsync \
-            --archive \
-            --hard-links \
-            --acls \
-            --xattrs \
-            --numeric-ids \
-            --chown root:root \
-            --rsync-path 'sudo rsync' \
-            "$stage/etc/nginx/conf.d/rob.tn.conf" \
-            "${host}:/etc/nginx/conf.d/rob.tn.conf"
-    fi
 
-    ssh_run "$host" "sudo bash -s" <<'REMOTE_EOF'
+    # Both targets are leaf paths (the docroot itself, and a single named
+    # file) so rsync does not traverse /var or /etc parents — `--chown` is
+    # enough; -A/-X are intentionally absent.
+    rsync \
+        --archive \
+        --hard-links \
+        --numeric-ids \
+        --chown root:root \
+        --rsh='ssh -o BatchMode=yes' \
+        --rsync-path 'sudo rsync' \
+        --delete \
+        "${stage}${web_root}/" \
+        "${host}:${web_root}/"
+    rsync \
+        --archive \
+        --hard-links \
+        --numeric-ids \
+        --chown root:root \
+        --rsh='ssh -o BatchMode=yes' \
+        --rsync-path 'sudo rsync' \
+        "${stage}${site_conf_path}" \
+        "${host}:${site_conf_path}"
+
+    ssh_run "$host" "sudo bash -s -- ${web_root@Q} ${site_conf_path@Q} ${api_upstream_port@Q}" <<'REMOTE_EOF'
 set -euo pipefail
+web_root="$1"
+site_conf_path="$2"
+api_upstream_port="$3"
 
 # Allow nginx to make outbound connections to the moments-api upstream
 # across the WG mesh.
 setsebool -P httpd_can_network_connect on
 
-# Label the upstream port so httpd_t may name_connect to it.
+# Idempotent label: --add fails if the port is already labelled (we suppress
-if ! semanage port -l | awk '{print $1, $3}' | grep -qE "^http_port_t .*42424"; then
+# that one stderr line); --modify is then a no-op or fixes a stale type.
-    semanage port -a -t http_port_t -p tcp 42424 || \
+semanage port --add    --type=http_port_t --proto=tcp "$api_upstream_port" 2>/dev/null \
-    semanage port -m -t http_port_t -p tcp 42424
+    || semanage port --modify --type=http_port_t --proto=tcp "$api_upstream_port"
-fi
 
-restorecon -Rv /var/www/rob.tn /etc/nginx/conf.d/rob.tn.conf
+restorecon -Rv "$web_root" "$site_conf_path"
 
 if ! nginx -t; then
     echo "nginx config check failed" >&2
@@ -363,7 +479,7 @@ REMOTE_EOF
 
 failed=()
 for component in "${components[@]}"; do
-    mapfile -t hosts < <(yq -r "${env_path}.components.${component}.hosts[]" "$manifest")
+    mapfile -t hosts < <(yq --raw-output "${env_path}.components.${component}.hosts[]" "$manifest")
     for host in "${hosts[@]}"; do
         case "$component" in
             api)    deploy_api    "$host" || failed+=("api@$host") ;;

script/teardown.sh

@@ -0,0 +1,288 @@
+#!/usr/bin/env bash
+#
+# moments teardown script.
+#
+#   ./script/teardown.sh <environment> <host> [component...] [--dry-run]
+#   ./script/teardown.sh prod anjie.kosherinata.internal api worker
+#   ./script/teardown.sh prod oolon.kosherinata.internal web --remove-docroot
+#   ./script/teardown.sh prod anjie.kosherinata.internal all --dry-run
+#
+# Removes moments unit files, binaries, env files, firewalld service +
+# definition, SELinux port label, and (when no moments component env files
+# remain) the shared /etc/moments + /var/lib/moments dirs and the sysusers
+# entry. Idempotent — safe to re-run.
+#
+# Notes:
+#  - The host argument is explicit on purpose: you typically tear down on
+#    hosts you've already removed from manifest.components.<c>.hosts.
+#  - Manifest is still read for env-wide config (api port, server_name,
+#    docroot path), so $environment must still resolve.
+#  - The `moments` user/group is intentionally NOT removed: any leftover
+#    file owned by it would become orphan-owned. Run `userdel moments`
+#    manually if you're certain there are none.
+#  - Web docroot is left intact unless --remove-docroot is given.
+
+set -euo pipefail
+shopt -s nullglob
+
+repo_root="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
+manifest="${repo_root}/asset/manifest.yml"
+dry_run=0
+remove_docroot=0
+
+usage() {
+    cat <<EOF >&2
+usage: $(basename "$0") <environment> <host> [component...] [--dry-run] [--remove-docroot]
+       $(basename "$0") prod anjie.kosherinata.internal api worker
+       $(basename "$0") prod oolon.kosherinata.internal web --remove-docroot
+       $(basename "$0") prod anjie.kosherinata.internal all
+
+components: api | worker | web | all
+EOF
+    exit 2
+}
+
+log()  { printf '\033[1;34m[teardown]\033[0m %s\n' "$*" >&2; }
+warn() { printf '\033[1;33m[teardown]\033[0m %s\n' "$*" >&2; }
+die()  { printf '\033[1;31m[teardown]\033[0m %s\n' "$*" >&2; exit 1; }
+
+ssh_run() {
+    local host="$1"; shift
+    if (( dry_run )); then
+        printf '\033[2m[dry-run]\033[0m ssh %s -- %s\n' "$host" "$*" >&2
+    else
+        ssh -o BatchMode=yes "$host" "$@"
+    fi
+}
+
+[[ $# -ge 2 ]] || usage
+environment="$1"; shift
+target_host="$1"; shift
+
+components=()
+while [[ $# -gt 0 ]]; do
+    case "$1" in
+        --dry-run)         dry_run=1 ;;
+        --remove-docroot)  remove_docroot=1 ;;
+        *)                 components+=("$1") ;;
+    esac
+    shift
+done
+
+[[ -f "$manifest" ]]                 || die "manifest not found: $manifest"
+command -v yq    >/dev/null 2>&1     || die "yq is required"
+
+env_path=".environments.${environment}"
+yq --exit-status "${env_path}" "$manifest" >/dev/null \
+    || die "environment '$environment' not found in manifest"
+
+if [[ ${#components[@]} -eq 0 ]]; then
+    usage
+fi
+if [[ "${components[0]:-}" == "all" ]]; then
+    components=(api worker web)
+fi
+
+teardown_api() {
+    local host="$1"
+    log "api -> $host"
+
+    local bind api_port=""
+    bind="$(yq --raw-output "${env_path}.components.api.config.bind" "$manifest")"
+    if [[ -n "$bind" && "$bind" != "null" && "$bind" == *:* ]]; then
+        api_port="${bind##*:}"
+        [[ "$api_port" =~ ^[0-9]+$ ]] || api_port=""
+    fi
+
+    if (( dry_run )); then
+        printf '\033[2m[dry-run]\033[0m stop+disable moments-api units, remove unit files, /etc/moments/api.env, /usr/local/bin/moments-api, firewalld svc moments-api, SELinux label tcp/%s on %s\n' \
+            "${api_port:-<unknown>}" "$host" >&2
+        return 0
+    fi
+
+    ssh_run "$host" "sudo bash -s -- ${api_port@Q}" <<'REMOTE_EOF'
+set -euo pipefail
+api_port="$1"
+
+# Stop + disable units. `disable --now` quietly does nothing on a unit that
+# isn't loaded, but emits non-zero exit on some systemd versions when the
+# file is already gone — swallow that so re-runs are clean.
+for unit in moments-api.service moments-api-cert.path moments-api-cert-reload.service; do
+    systemctl disable --now "$unit" 2>/dev/null || true
+done
+
+rm --force \
+    /etc/systemd/system/moments-api.service \
+    /etc/systemd/system/moments-api-cert.path \
+    /etc/systemd/system/moments-api-cert-reload.service
+
+systemctl daemon-reload
+
+rm --force /etc/moments/api.env /usr/local/bin/moments-api
+
+# Firewalld: remove service from default zone, then drop service definition.
+zone="$(firewall-cmd --get-default-zone)"
+if firewall-cmd --zone="$zone" --query-service=moments-api >/dev/null 2>&1; then
+    firewall-cmd --permanent --zone="$zone" --remove-service=moments-api
+    firewall-cmd            --zone="$zone" --remove-service=moments-api 2>/dev/null || true
+fi
+rm --force /etc/firewalld/services/moments-api.xml
+firewall-cmd --reload
+
+# SELinux: remove the port label, if we know which port. --delete fails when
+# the port wasn't user-labelled — that's fine, swallow it.
+if [[ -n "$api_port" ]]; then
+    semanage port --delete --proto=tcp "$api_port" 2>/dev/null || true
+fi
+
+echo "moments-api torn down"
+REMOTE_EOF
+}
+
+teardown_worker() {
+    local host="$1"
+    log "worker -> $host"
+
+    if (( dry_run )); then
+        printf '\033[2m[dry-run]\033[0m stop+disable moments-worker units, remove unit files, /etc/moments/worker.env, /usr/local/bin/moments-worker on %s\n' \
+            "$host" >&2
+        return 0
+    fi
+
+    ssh_run "$host" "sudo bash -s" <<'REMOTE_EOF'
+set -euo pipefail
+
+for unit in moments-worker.service moments-worker-cert.path moments-worker-cert-reload.service; do
+    systemctl disable --now "$unit" 2>/dev/null || true
+done
+
+rm --force \
+    /etc/systemd/system/moments-worker.service \
+    /etc/systemd/system/moments-worker-cert.path \
+    /etc/systemd/system/moments-worker-cert-reload.service
+
+systemctl daemon-reload
+
+rm --force /etc/moments/worker.env /usr/local/bin/moments-worker
+
+echo "moments-worker torn down"
+REMOTE_EOF
+}
+
+teardown_web() {
+    local host="$1"
+    log "web -> $host"
+
+    local server_name web_root
+    server_name="$(yq --raw-output "${env_path}.components.web.config.server_name" "$manifest")"
+    web_root="$(yq    --raw-output "${env_path}.components.web.config.root"        "$manifest")"
+    [[ -n "$server_name" && "$server_name" != "null" ]] || die "web.config.server_name missing in manifest"
+    [[ -n "$web_root"    && "$web_root"    != "null" ]] || die "web.config.root missing in manifest"
+    [[ "$web_root" == /* ]] || die "web.config.root must be an absolute path: '$web_root'"
+
+    # Refuse to recursively remove a shallow or system path even if the
+    # manifest says so.
+    if (( remove_docroot )); then
+        case "$web_root" in
+            /|/bin|/bin/*|/boot|/boot/*|/dev|/dev/*|/etc|/etc/*|/home|/home/*|/lib|/lib/*|/lib64|/lib64/*|/proc|/proc/*|/root|/root/*|/run|/run/*|/sbin|/sbin/*|/srv|/srv/*|/sys|/sys/*|/tmp|/tmp/*|/usr|/usr/*|/var|/var/lib|/var/log|/var/run|/var/spool|/var/www)
+                die "refusing to recursively remove a system path: '$web_root'"
+                ;;
+        esac
+        # Require at least three path components (e.g. /var/www/<site>) to
+        # rule out things like /opt or /srv directly.
+        [[ "$web_root" =~ ^/[^/]+/[^/]+/[^/]+ ]] \
+            || die "refusing to recursively remove a path with fewer than 3 components: '$web_root'"
+    fi
+
+    local site_conf_path="/etc/nginx/conf.d/${server_name}.conf"
+
+    if (( dry_run )); then
+        if (( remove_docroot )); then
+            printf '\033[2m[dry-run]\033[0m remove %s, recursively remove %s, nginx -t/reload on %s\n' \
+                "$site_conf_path" "$web_root" "$host" >&2
+        else
+            printf '\033[2m[dry-run]\033[0m remove %s, nginx -t/reload on %s (docroot %s left intact; pass --remove-docroot to also clear it)\n' \
+                "$site_conf_path" "$host" "$web_root" >&2
+        fi
+        return 0
+    fi
+
+    ssh_run "$host" "sudo bash -s -- ${site_conf_path@Q} ${web_root@Q} ${remove_docroot@Q}" <<'REMOTE_EOF'
+set -euo pipefail
+site_conf_path="$1"
+web_root="$2"
+remove_docroot="$3"
+
+rm --force "$site_conf_path"
+
+if nginx -t 2>&1; then
+    systemctl reload nginx
+    echo "nginx reloaded without ${site_conf_path}"
+else
+    echo "nginx -t failed AFTER removing ${site_conf_path}; check other site configs" >&2
+    exit 1
+fi
+
+if [[ "$remove_docroot" == "1" && -d "$web_root" ]]; then
+    rm --recursive --force "$web_root"
+    echo "removed docroot ${web_root}"
+fi
+REMOTE_EOF
+}
+
+teardown_shared() {
+    local host="$1"
+    log "shared (post-component cleanup) -> $host"
+
+    if (( dry_run )); then
+        printf '\033[2m[dry-run]\033[0m if no api.env/worker.env remain: remove /etc/sysusers.d/moments.conf and rmdir /etc/moments + /var/lib/moments on %s (moments user left in place)\n' \
+            "$host" >&2
+        return 0
+    fi
+
+    ssh_run "$host" "sudo bash -s" <<'REMOTE_EOF'
+set -euo pipefail
+
+# If any component env still exists, leave shared state alone — another
+# moments component is still using /etc/moments and the moments user.
+if [[ -e /etc/moments/api.env || -e /etc/moments/worker.env ]]; then
+    echo "moments env files still present; leaving /etc/moments + /var/lib/moments + sysusers entry in place"
+    exit 0
+fi
+
+# rmdir refuses non-empty dirs — defensive against unknown stragglers.
+rmdir /etc/moments     2>/dev/null || true
+rmdir /var/lib/moments 2>/dev/null || true
+
+rm --force /etc/sysusers.d/moments.conf
+
+echo "shared state cleared (where empty); moments user/group intentionally left in place"
+REMOTE_EOF
+}
+
+# Dispatch ------------------------------------------------------------------
+
+failed=()
+did_app=0
+for component in "${components[@]}"; do
+    case "$component" in
+        api)    teardown_api    "$target_host" || failed+=("api@$target_host") ;;
+        worker) teardown_worker "$target_host" || failed+=("worker@$target_host") ;;
+        web)    teardown_web    "$target_host" || failed+=("web@$target_host") ;;
+        *)      warn "unknown component: $component" ;;
+    esac
+    case "$component" in
+        api|worker) did_app=1 ;;
+    esac
+done
+
+# Shared cleanup runs after api/worker teardown. It's a no-op if either
+# component still has its env file present on the host.
+if (( did_app )); then
+    teardown_shared "$target_host" || failed+=("shared@$target_host")
+fi
+
+if [[ ${#failed[@]} -gt 0 ]]; then
+    die "failed: ${failed[*]}"
+fi
+log "teardown complete on $target_host"