lair/tireless
3
0
Fork 1
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

Verify the deploy workflow end to end, before there is anything to lose #9

New Issue
Closed
opened 2026-08-07 12:40:14 +00:00 by grenade · 1 comment
grenade commented 2026-08-07 12:40:14 +00:00
Owner
Copy Link

Goal

.gitea/workflows/deploy.yaml has never run. It is the stage 0 done-when and
the gate every later stage passes through, and it currently carries at least two
untested assumptions:

  • actions/upload-artifact@v3 and download-artifact@v3 are EOL upstream, and
    the deploy job assumes the artifact preserved the asset/ and
    dashboard/dist/ directory structure. If v3 flattens or the action is
    unavailable on this runner, every rsync path in the deploy job is wrong.
  • Ingress was just moved to the proxy host, which means WEB_HOST needs a
    gitea_ci account and the narrower sudoers grant that script/infra-setup.sh
    now provisions. That path has never been exercised.

Run it now, while a failed deploy costs nothing, rather than during the first
stage that has state worth keeping.

Files

  • .gitea/workflows/deploy.yaml — whatever the dry run shows is wrong
  • script/infra-setup.sh — same

Steps

  1. Run script/infra-setup.sh against bob and the proxy, and confirm both
    gitea_ci grants validate with visudo -cf.
  2. Trigger the workflow with workflow_dispatch.
  3. Fix whatever fails. Expect the artifact actions first; migrate to a supported
    version if v3 does not work on this runner.
  4. Confirm the health probe genuinely tests reachability: it now curls
    bob:23296/v1/ready from the proxy, so closing the firewalld service
    should make it fail. Verify that, rather than assuming it.
  5. Confirm /v1/ready reports config: "ok" and database: "not_implemented",
    which is the honest state until stage 1 lands.
  6. Confirm the dashboard loads at https://tireless.internal after the nginx
    vhost is installed by hand (infra-setup step 3).

Acceptance

  • The deploy workflow completes green from workflow_dispatch.
  • curl -fsS https://tireless.internal/v1/ready returns 200 from inside the
    mesh.
  • systemctl is-active reports all three units active on bob.
  • sudo -u tireless tireless preflight on bob reports the expected billing
    mode and a non-Anthropic OpenCode lane.

Out of scope

  • The interactive Claude Code login and the Gitea bot account. Those are manual
    one-time steps documented in script/infra-setup.sh; this issue only needs
    the runner to start, and it is expected to refuse until the login exists.
  • Postgres role creation on magrathea and frankie. Stage 1 needs it; this does
    not.
  • Any application code. If the deploy reveals a code bug, file it separately.
## Goal `.gitea/workflows/deploy.yaml` has never run. It is the stage 0 done-when and the gate every later stage passes through, and it currently carries at least two untested assumptions: - `actions/upload-artifact@v3` and `download-artifact@v3` are EOL upstream, and the deploy job assumes the artifact preserved the `asset/` and `dashboard/dist/` directory structure. If v3 flattens or the action is unavailable on this runner, every rsync path in the deploy job is wrong. - Ingress was just moved to the proxy host, which means `WEB_HOST` needs a `gitea_ci` account and the narrower sudoers grant that `script/infra-setup.sh` now provisions. That path has never been exercised. Run it now, while a failed deploy costs nothing, rather than during the first stage that has state worth keeping. ## Files - `.gitea/workflows/deploy.yaml` — whatever the dry run shows is wrong - `script/infra-setup.sh` — same ## Steps 1. Run `script/infra-setup.sh` against bob and the proxy, and confirm both `gitea_ci` grants validate with `visudo -cf`. 2. Trigger the workflow with `workflow_dispatch`. 3. Fix whatever fails. Expect the artifact actions first; migrate to a supported version if v3 does not work on this runner. 4. Confirm the health probe genuinely tests reachability: it now curls `bob:23296/v1/ready` *from the proxy*, so closing the firewalld service should make it fail. Verify that, rather than assuming it. 5. Confirm `/v1/ready` reports `config: "ok"` and `database: "not_implemented"`, which is the honest state until stage 1 lands. 6. Confirm the dashboard loads at `https://tireless.internal` after the nginx vhost is installed by hand (infra-setup step 3). ## Acceptance - The deploy workflow completes green from `workflow_dispatch`. - `curl -fsS https://tireless.internal/v1/ready` returns 200 from inside the mesh. - `systemctl is-active` reports all three units active on bob. - `sudo -u tireless tireless preflight` on bob reports the expected billing mode and a non-Anthropic OpenCode lane. ## Out of scope - The interactive Claude Code login and the Gitea bot account. Those are manual one-time steps documented in `script/infra-setup.sh`; this issue only needs the *runner* to start, and it is expected to refuse until the login exists. - Postgres role creation on magrathea and frankie. Stage 1 needs it; this does not. - Any application code. If the deploy reveals a code bug, file it separately.
grenade added the tireless/implementtireless/agent:cc labels 2026-08-07 12:41:01 +00:00
grenade commented 2026-08-07 14:07:34 +00:00
Author
Owner
Copy Link

Green: run 8.

https://tireless.internal/v1/ready  {"config":"ok","database":"not_implemented","forge":"not_implemented"}
https://tireless.internal/          HTTP 200
tireless-api                        active
tireless-poller                     active
tireless-runner                     failed  (see below — this is correct)
tireless preflight                  Subscription / lair-helexa / Qwen3.6-27B (asserted non-Anthropic)

What was wrong

Eight faults. The two the issue predicted were both real, but neither was the
one that would have cost the most to find later.

# Fault Why it stayed hidden
1 runs-on: fedora-43-rust is not a registered label The job was never scheduled. Runs 2 and 3 have started_at: 1970-01-01 and are recorded as "cancelled", which tells you nothing
2 Vhost bound :443, owned by the stream SNI router nginx -t passes; the symptom is the wrong certificate on a working handshake
3 Cert path was the host identity cert Its SAN is bob's FQDN, so it fails only for a client verifying tireless.internal
4 --rsync-path word-split by an unquoted variable Loud, but only on the half of the deploy that used the variable
5 restorecon on /var/lib/tireless, which cannot exist yet The account that owns it is created later in the same deploy
6 Config shipped 0640 root:root; services run as tireless Compounded by --chmod being a no-op without -p, so the fix would have worked once and silently stopped
7 API bound the clap default, ignoring [api] bind The service looks perfectly healthy from bob — starts, logs "listening", answers a local curl. Only the proxy fails
8 Health probe expanded $unit on the remote side It ran systemctl is-active .service: checking nothing, and would have reported healthy regardless

The artifact-action risk this issue led with turned out to be moot — build and
deploy collapsed into one job, since the rust image already carries node, ssh
and rsync, so there is nothing to hand between jobs.

Six of the eight were silent. That is the part worth carrying forward: where a
probe exists, it must be able to fail. The health probe now runs from the
proxy over the mesh rather than bob's loopback, so closing firewalld breaks it.

Faults 4–8 were found by exercising the deploy directly against the hosts as
gitea_ci — every rsync destination, every sudo command, and the health probe
step extracted from the YAML and run as a file — rather than at six minutes per
attempt.

Provisioned

infra-setup.sh now does the ingress rather than describing it: mints the
tireless.internal cert through the JWK provisioner (removing the credential
even on failure), installs the vhost via sites-available + symlink, enables
step@tireless.timer, and registers the split-horizon record on both
routers — a record on one router NXDOMAINs at the other site. opn-cli has no
reconfigure verb, so the apply is a direct API POST; without it the name
resolves only whenever Unbound next happens to reload.

The served certificate was checked against disk by serial, per
internal-tls.md §3. They match.

Remaining, and out of scope per this issue

tireless-runner is in failed because the interactive agent login has not been
done as the service account (script/infra-setup.sh step 1). That is the
invariant working — a runner with no credentials refuses to start rather than
pretending — and it is why the deploy tolerates that one unit failing.

Note the acceptance criterion "all three units active" contradicted this issue's
own Out of scope section, which said the runner is expected to refuse. The Out
of scope text was right; the criterion was over-specified when I wrote it.

Also worth fixing, outside this repo

architecture/internal-tls.md §5 says hanzalova vhosts bind :443 directly.
reverse-proxies.md §4 says both proxies use the 127.0.0.1:14443 tier and
notes hanzalova gained its router on 2026-07-30; the live bench.internal.conf
confirms it. Anyone following §5 gets fault #2 — which that same doc warns about
in §3. Worth a correction in the architecture repo.

Green: [run 8](https://git.lair.cafe/lair/tireless/actions/runs/8). ``` https://tireless.internal/v1/ready {"config":"ok","database":"not_implemented","forge":"not_implemented"} https://tireless.internal/ HTTP 200 tireless-api active tireless-poller active tireless-runner failed (see below — this is correct) tireless preflight Subscription / lair-helexa / Qwen3.6-27B (asserted non-Anthropic) ``` ## What was wrong Eight faults. The two the issue predicted were both real, but neither was the one that would have cost the most to find later. | # | Fault | Why it stayed hidden | | --- | --- | --- | | 1 | `runs-on: fedora-43-rust` is not a registered label | The job was never scheduled. Runs 2 and 3 have `started_at: 1970-01-01` and are recorded as "cancelled", which tells you nothing | | 2 | Vhost bound `:443`, owned by the stream SNI router | `nginx -t` passes; the symptom is the wrong certificate on a working handshake | | 3 | Cert path was the host identity cert | Its SAN is bob's FQDN, so it fails only for a client verifying `tireless.internal` | | 4 | `--rsync-path` word-split by an unquoted variable | Loud, but only on the half of the deploy that used the variable | | 5 | `restorecon` on `/var/lib/tireless`, which cannot exist yet | The account that owns it is created later in the same deploy | | 6 | Config shipped `0640 root:root`; services run as `tireless` | Compounded by `--chmod` being a no-op without `-p`, so the fix would have worked once and silently stopped | | 7 | API bound the clap default, ignoring `[api] bind` | The service looks perfectly healthy *from bob* — starts, logs "listening", answers a local curl. Only the proxy fails | | 8 | Health probe expanded `$unit` on the remote side | It ran `systemctl is-active .service`: checking nothing, and would have reported healthy regardless | The artifact-action risk this issue led with turned out to be moot — build and deploy collapsed into one job, since the `rust` image already carries node, ssh and rsync, so there is nothing to hand between jobs. Six of the eight were silent. That is the part worth carrying forward: **where a probe exists, it must be able to fail.** The health probe now runs from the proxy over the mesh rather than bob's loopback, so closing firewalld breaks it. Faults 4–8 were found by exercising the deploy directly against the hosts as `gitea_ci` — every rsync destination, every `sudo` command, and the health probe step extracted from the YAML and run as a file — rather than at six minutes per attempt. ## Provisioned `infra-setup.sh` now does the ingress rather than describing it: mints the `tireless.internal` cert through the JWK provisioner (removing the credential even on failure), installs the vhost via `sites-available` + symlink, enables `step@tireless.timer`, and registers the split-horizon record on **both** routers — a record on one router `NXDOMAIN`s at the other site. `opn-cli` has no reconfigure verb, so the apply is a direct API POST; without it the name resolves only whenever Unbound next happens to reload. The served certificate was checked against disk by serial, per `internal-tls.md` §3. They match. ## Remaining, and out of scope per this issue `tireless-runner` is in `failed` because the interactive agent login has not been done as the service account (`script/infra-setup.sh` step 1). That is the invariant working — a runner with no credentials refuses to start rather than pretending — and it is why the deploy tolerates that one unit failing. Note the acceptance criterion "all three units active" contradicted this issue's own Out of scope section, which said the runner is expected to refuse. The Out of scope text was right; the criterion was over-specified when I wrote it. ## Also worth fixing, outside this repo `architecture/internal-tls.md` §5 says hanzalova vhosts bind `:443` directly. `reverse-proxies.md` §4 says both proxies use the `127.0.0.1:14443` tier and notes hanzalova gained its router on 2026-07-30; the live `bench.internal.conf` confirms it. Anyone following §5 gets fault #2 — which that same doc warns about in §3. Worth a correction in the architecture repo.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
epic
Framing issue for a set of children.
 tireless
Opt-in. Without it tireless ignores the issue entirely. Applied by a human.
 tireless/agent:cc
Force the Claude Code lane. Use for constraint-bearing changes (design.md 10.2).
 tireless/agent:oc
Force the OpenCode lane.
 tireless/blocked
Needs a human; tireless has stopped.
 tireless/claimed
A job holds this issue. Written by tireless; advisory on read.
 tireless/discover
Survey this repository and propose work. Applied to a tracking issue.
 tireless/done
Delivered: children created, or PR opened.
 tireless/implement
Implement and open a pull request.
 tireless/plan
Decompose into an epic and child issues.
 tireless/proposed
Opened by a discovery run. Awaiting a human decision; deliberately not opted in.
No Label tireless/agent:cc tireless/implement
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
agent-bot grenade tireless-poller
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: lair/tireless#9
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?