Run 7 shipped everything, brought api and poller up, served /v1/ready through
the proxy — and failed on the last step with exit 4.
ssh gitea_ci@"$API_HOST" "systemctl is-active \$unit.service"
The escaped \$unit is a leftover from when this ran inside a heredoc. Over a
plain ssh it sends a literal $unit for the *remote* shell to expand, against a
variable that only exists in the workflow's shell — so it ran
`systemctl is-active .service`, which is exit 4, unit not found. It was checking
nothing, and would have kept reporting healthy no matter what the units did.
Also pass -n to both probe sshs. ssh reads stdin by default, and that is a trap
waiting for whoever next pipes something into this step: the first ssh eats the
rest of the script and the remaining checks silently never run. (It bit the test
harness for this fix, which is how it was noticed.)
And say in a comment why tireless-runner is not probed: it refuses to start
until the interactive agent login exists, so probing it would make every deploy
red for something the deploy cannot fix.
Verified by extracting this step's body from the YAML and running it as a file,
the way Actions does:
{"config":"ok","database":"not_implemented","forge":"not_implemented"}
tireless-api: active
tireless-poller: active
Refs #9
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013TxK1CWPkFXqdcXMJ4hVe6
Run 5 shipped everything and started all three units, which then failed
identically:
configuration is invalid: /etc/tireless/config.toml: Permission denied
The config went out 0640 root:root, but the services run as `tireless`, which
is not in the root group. `--chown root:tireless` is not available as a fix:
on a fresh host that group does not exist until systemd-sysusers runs later in
the same deploy — the same ordering trap that produced the StateDirectory
change. So 0644 root:root, which is defensible precisely because this file
carries no secrets by design; the one that does, tireless.env, stays
0640 root:tireless and is installed by hand.
Testing that on the host turned up a second, quieter fault: `--chmod` only
applies to files rsync actually transfers, so a redeploy whose config is
byte-identical leaves the old mode in place. A mode fix would have appeared to
work on the deploy that introduced it and silently not applied afterwards. Add
-p to the shared options and switch to --chmod=F644/F755 so every push asserts
the mode it wants rather than hoping the content changed.
Also give tireless-runner an explicit start limit. Its preflight fails
permanently without an agent login, and Restart=on-failure with RestartSec=30s
never trips the default 5-starts-per-10s limiter, so it would have retried
forever. Three attempts in ten minutes, then failed, where systemctl status
says why.
Verified on bob: api and poller active, /v1/ready returns
{"config":"ok","database":"not_implemented","forge":"not_implemented"}, and
`tireless preflight` reports Subscription billing with the OpenCode lane
asserted non-Anthropic.
Refs #9
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013TxK1CWPkFXqdcXMJ4hVe6
Run 4 failed at `ship artifacts` with `sudo: unrecognized option '--server'`.
`R="--rsync-path=sudo rsync --mkpath"` expanded unquoted as `rsync $R` splits
into three arguments — `--rsync-path=sudo`, plus a stray `rsync` that rsync
reads as a source path — so the remote end ran `sudo --server`. Use an array.
The dashboard step quoted it inline and was unaffected, which is why only half
the deploy was broken.
Every rsync destination and every sudo command in `apply system state` has now
been exercised directly against bob as gitea_ci, rather than by another six
minute round trip: seven rsync targets, sysusers, restorecon, firewalld and
daemon-reload all pass.
That surfaced the second fault. restorecon was given /var/lib/tireless, which
does not exist on a fresh host: infra-setup.sh tried to create it before
systemd-sysusers had created the account to own it, so the attempt always raced
and always lost. Declare StateDirectory=tireless on all three units instead —
systemd creates the directory, owns it as the service user and labels it — and
drop the path from restorecon, the grant, and infra-setup.
Refs #9
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013TxK1CWPkFXqdcXMJ4hVe6
Three faults, any one of which would have failed the first deploy — found by
reading the house conventions in ~/git/architecture rather than by running it.
`runs-on: fedora-43-rust` is not a registered runner label. The catalogue in
gitea-runners.md §2 has `rust`; the job would never have been scheduled. Also
collapse build and deploy into one job: the `rust` image descends from
runner-fedora-44 and carries node, ssh and rsync, so the artifact upload and
download — on actions/*-artifact@v3, EOL upstream — bought nothing but a
round-trip and a directory-structure assumption. rpm.lair.cafe's working deploy
is single-job for the same reason.
The nginx vhost bound `listen 443 ssl`. TCP 443 on hanzalova belongs to the
stream SNI router (reverse-proxies.md §4-5, and the live bench.internal.conf
confirms it), so the vhost would never have been reached — the router answers
first with whichever certificate its default branch holds. `nginx -t` passes
either way, which is what makes this worth catching by reading rather than by
deploying. Now `listen 127.0.0.1:14443 ssl proxy_protocol;`.
Cert paths pointed at the host identity cert under /etc/pki/tls. A per-service
name needs its own cert (internal-tls.md §2): the host cert's SAN is bob's FQDN,
not tireless.internal, so verification would have failed. Now
/etc/nginx/tls/{cert,key}/tireless.internal.pem, minted with --san and renewed
by step@tireless.timer.
infra-setup.sh now provisions the ingress rather than describing it: mints the
cert through the JWK provisioner (removing the credential even on failure),
installs the vhost via sites-available + symlink, and registers the
split-horizon record on BOTH routers — a record on one router NXDOMAINs at the
other site. opn-cli has no reconfigure verb, so the apply is a direct API POST;
without it the name resolves only whenever Unbound next happens to reload.
Also fold the stale-bindings check into the workflow (closes the CI half of #8)
and let the runner unit fail without failing the deploy, since it correctly
refuses to start until the interactive login exists.
Refs #9
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013TxK1CWPkFXqdcXMJ4hVe6
Three artefacts disagreed about where nginx runs. design.md §6.2 and the vhost
both said the hanzalova proxy; the API bound 127.0.0.1 and the workflow rsynced
the dashboard to bob. That combination deploys green and then serves nothing,
since a proxy on another host cannot reach bob's loopback.
Resolve it the way design.md already stated: nginx on the proxy, dashboard
shipped there, API bound 0.0.0.0 behind firewalld and the mesh. The health probe
now runs from the proxy over the mesh rather than from bob's loopback, so it
fails when firewalld is closed instead of passing regardless. infra-setup.sh
grows a proxy grant scoped to static files alone, and the nginx vhost install as
a manual step — it needs a certificate, and nothing was telling the operator to
install it at all.
npm run lint had never run: eslint 9 needs a flat config and there was none. Add
it, ignoring the ts-rs generated bindings, and run it in CI so it stays true.
Untrack dashboard/tsconfig.tsbuildinfo, a build artifact that would have put a
spurious diff in every pull request tireless opens.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013TxK1CWPkFXqdcXMJ4hVe6
Autonomous issue-to-PR driver for Claude Code and OpenCode, structured per
lair/architecture generic.md.
Workspace: entities/core/data/agent library crates plus api, worker and cli
binaries. Two pieces of real logic land with tests — lane routing (cc for
judgement, oc for specification) and the limit governor.
Constraints encoded as code rather than comments:
- agents are spawned as vendor binaries; tireless never calls a provider API
- ANTHROPIC_API_KEY is never set by tireless, only passed through
- assert_not_anthropic refuses to start an OpenCode lane pointed at Anthropic
- every run passes the governor; provider rate-limit signals win over our own
accounting
Deployment assets target bob.hanzalova.internal:23296 (registered in
port-allocations.md), fronted by hanzalova at tireless.internal.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DHhHtohxcdk1PL3tfnYJdH
