The status said "nothing has been deployed to bob, the workflow has never run",
which stopped being true at run 8. Leaving it would be the same class of
misleading state this repo keeps trying to remove.
Also record the seven faults it took to get green, because six of the eight were
silent rather than loud: a runner label that meant the job was never scheduled,
a vhost that nginx -t accepts and the SNI router never reaches, a cert whose SAN
only the client checks, an ordering trap, a --chmod that stops applying after
the first deploy, an API that looks healthy from the host it is unreachable on,
and a health probe checking a unit name that expanded to nothing.
The pattern is the useful part for later stages: the expensive faults were the
ones where a check passed while measuring nothing.
Closes#9
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013TxK1CWPkFXqdcXMJ4hVe6
Run 7 shipped everything, brought api and poller up, served /v1/ready through
the proxy — and failed on the last step with exit 4.
ssh gitea_ci@"$API_HOST" "systemctl is-active \$unit.service"
The escaped \$unit is a leftover from when this ran inside a heredoc. Over a
plain ssh it sends a literal $unit for the *remote* shell to expand, against a
variable that only exists in the workflow's shell — so it ran
`systemctl is-active .service`, which is exit 4, unit not found. It was checking
nothing, and would have kept reporting healthy no matter what the units did.
Also pass -n to both probe sshs. ssh reads stdin by default, and that is a trap
waiting for whoever next pipes something into this step: the first ssh eats the
rest of the script and the remaining checks silently never run. (It bit the test
harness for this fix, which is how it was noticed.)
And say in a comment why tireless-runner is not probed: it refuses to start
until the interactive agent login exists, so probing it would make every deploy
red for something the deploy cannot fix.
Verified by extracting this step's body from the YAML and running it as a file,
the way Actions does:
{"config":"ok","database":"not_implemented","forge":"not_implemented"}
tireless-api: active
tireless-poller: active
Refs #9
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013TxK1CWPkFXqdcXMJ4hVe6
Run 6 started every unit cleanly and still failed, at the health probe. The
journal says why:
"tireless-api listening", addr: "127.0.0.1:23296"
while the config it had just loaded says `bind = "0.0.0.0:23296"`. The `--bind`
flag carried `default_value_t`, so it was always Some, so it always won — the
config field I added was never read by anything.
The failure mode is the interesting part: from bob the service looks perfect. It
starts, logs "listening", and answers a local curl. Only a request from the
proxy that actually fronts it fails, which is why the probe was moved off
loopback in the first place.
Make --bind an Option with no default and fall back to config.api.bind, and add
a test asserting the shipped template does not bind loopback — with the reason,
so that if ingress ever moves onto bob the test explains that the bind, the
firewalld service and the vhost move together.
Verified end to end on the real hosts: the API binds 0.0.0.0, the proxy reaches
/v1/ready across the mesh, https://tireless.internal serves both the dashboard
and the API, and the served certificate matches the one on disk by serial.
Refs #9
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013TxK1CWPkFXqdcXMJ4hVe6
Run 5 shipped everything and started all three units, which then failed
identically:
configuration is invalid: /etc/tireless/config.toml: Permission denied
The config went out 0640 root:root, but the services run as `tireless`, which
is not in the root group. `--chown root:tireless` is not available as a fix:
on a fresh host that group does not exist until systemd-sysusers runs later in
the same deploy — the same ordering trap that produced the StateDirectory
change. So 0644 root:root, which is defensible precisely because this file
carries no secrets by design; the one that does, tireless.env, stays
0640 root:tireless and is installed by hand.
Testing that on the host turned up a second, quieter fault: `--chmod` only
applies to files rsync actually transfers, so a redeploy whose config is
byte-identical leaves the old mode in place. A mode fix would have appeared to
work on the deploy that introduced it and silently not applied afterwards. Add
-p to the shared options and switch to --chmod=F644/F755 so every push asserts
the mode it wants rather than hoping the content changed.
Also give tireless-runner an explicit start limit. Its preflight fails
permanently without an agent login, and Restart=on-failure with RestartSec=30s
never trips the default 5-starts-per-10s limiter, so it would have retried
forever. Three attempts in ten minutes, then failed, where systemctl status
says why.
Verified on bob: api and poller active, /v1/ready returns
{"config":"ok","database":"not_implemented","forge":"not_implemented"}, and
`tireless preflight` reports Subscription billing with the OpenCode lane
asserted non-Anthropic.
Refs #9
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013TxK1CWPkFXqdcXMJ4hVe6
Run 4 failed at `ship artifacts` with `sudo: unrecognized option '--server'`.
`R="--rsync-path=sudo rsync --mkpath"` expanded unquoted as `rsync $R` splits
into three arguments — `--rsync-path=sudo`, plus a stray `rsync` that rsync
reads as a source path — so the remote end ran `sudo --server`. Use an array.
The dashboard step quoted it inline and was unaffected, which is why only half
the deploy was broken.
Every rsync destination and every sudo command in `apply system state` has now
been exercised directly against bob as gitea_ci, rather than by another six
minute round trip: seven rsync targets, sysusers, restorecon, firewalld and
daemon-reload all pass.
That surfaced the second fault. restorecon was given /var/lib/tireless, which
does not exist on a fresh host: infra-setup.sh tried to create it before
systemd-sysusers had created the account to own it, so the attempt always raced
and always lost. Declare StateDirectory=tireless on all three units instead —
systemd creates the directory, owns it as the service user and labels it — and
drop the path from restorecon, the grant, and infra-setup.
Refs #9
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013TxK1CWPkFXqdcXMJ4hVe6
sudo matches an argument vector exactly, so the sudoers line and the command
the workflow runs have to agree character for character. Moving the dashboard
to the proxy shortened the workflow's restorecon to five paths but left the
grant listing six, which denies it.
/var/www/tireless was never bob's to relabel in any case — the proxy's own
grant covers it, and that one is scoped to static files alone.
Refs #9
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013TxK1CWPkFXqdcXMJ4hVe6
Three faults, any one of which would have failed the first deploy — found by
reading the house conventions in ~/git/architecture rather than by running it.
`runs-on: fedora-43-rust` is not a registered runner label. The catalogue in
gitea-runners.md §2 has `rust`; the job would never have been scheduled. Also
collapse build and deploy into one job: the `rust` image descends from
runner-fedora-44 and carries node, ssh and rsync, so the artifact upload and
download — on actions/*-artifact@v3, EOL upstream — bought nothing but a
round-trip and a directory-structure assumption. rpm.lair.cafe's working deploy
is single-job for the same reason.
The nginx vhost bound `listen 443 ssl`. TCP 443 on hanzalova belongs to the
stream SNI router (reverse-proxies.md §4-5, and the live bench.internal.conf
confirms it), so the vhost would never have been reached — the router answers
first with whichever certificate its default branch holds. `nginx -t` passes
either way, which is what makes this worth catching by reading rather than by
deploying. Now `listen 127.0.0.1:14443 ssl proxy_protocol;`.
Cert paths pointed at the host identity cert under /etc/pki/tls. A per-service
name needs its own cert (internal-tls.md §2): the host cert's SAN is bob's FQDN,
not tireless.internal, so verification would have failed. Now
/etc/nginx/tls/{cert,key}/tireless.internal.pem, minted with --san and renewed
by step@tireless.timer.
infra-setup.sh now provisions the ingress rather than describing it: mints the
cert through the JWK provisioner (removing the credential even on failure),
installs the vhost via sites-available + symlink, and registers the
split-horizon record on BOTH routers — a record on one router NXDOMAINs at the
other site. opn-cli has no reconfigure verb, so the apply is a direct API POST;
without it the name resolves only whenever Unbound next happens to reload.
Also fold the stale-bindings check into the workflow (closes the CI half of #8)
and let the runner unit fail without failing the deploy, since it correctly
refuses to start until the interactive login exists.
Refs #9
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013TxK1CWPkFXqdcXMJ4hVe6
Three artefacts disagreed about where nginx runs. design.md §6.2 and the vhost
both said the hanzalova proxy; the API bound 127.0.0.1 and the workflow rsynced
the dashboard to bob. That combination deploys green and then serves nothing,
since a proxy on another host cannot reach bob's loopback.
Resolve it the way design.md already stated: nginx on the proxy, dashboard
shipped there, API bound 0.0.0.0 behind firewalld and the mesh. The health probe
now runs from the proxy over the mesh rather than from bob's loopback, so it
fails when firewalld is closed instead of passing regardless. infra-setup.sh
grows a proxy grant scoped to static files alone, and the nginx vhost install as
a manual step — it needs a certificate, and nothing was telling the operator to
install it at all.
npm run lint had never run: eslint 9 needs a flat config and there was none. Add
it, ignoring the ts-rs generated bindings, and run it in CI so it stays true.
Untrack dashboard/tsconfig.tsbuildinfo, a build artifact that would have put a
spurious diff in every pull request tireless opens.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013TxK1CWPkFXqdcXMJ4hVe6
design.md described what tireless does to an issue but never what it is for, and
in one place said the opposite of the intent: §1 assigned "identifying what
should be worked on" to the operator, which is the thing discovery automates.
A reader — human or agent — would have concluded the system is human-triggered
only and never built the lane.
Rewrite §1 around continuous multi-repo development. Add §2.5, the autonomy
boundary, as a table of which transitions are automatic and which are human,
with the alternatives considered and why they lose. Add §2.6 on anchoring
discovery to a tracking issue rather than making Job.issue a sum type, and what
that buys. Add stage 6 for the lane, renumbering scheduling to 7 and hardening
to 8, with the reasoning for placing a cheap stage late.
Add §10 on tireless working on tireless: that a merged PR restarts the runner
that opened it and why that is survivable but not free, which areas must be
routed to the stronger lane because they are constraint-bearing, and the four
questions dogfooding is expected to answer.
Two new invariants. Admission is inherited (§2.5), and every constraint must be
reachable from a binary's startup path — the rule the previous commit's orphaned
guards violated.
AGENTS.md symlinks CLAUDE.md: both agents look for their own filename, the
implementation prompts tell them to, and a symlink is the only version of this
that cannot drift.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013TxK1CWPkFXqdcXMJ4hVe6
The docs described a reactive executor: every entry point was a human label, and
the only issue tireless ever created was a plan child. Nothing surveyed a repo or
proposed work, which is the half that makes this continuous rather than
on-demand.
Add JobKind::Discover, routed always to Claude Code (proposing work is the
highest-judgement, lowest-volume task), the tireless/discover and
tireless/proposed labels, and prompt/discover.cc.md as a fourth member of the
versioned prompt set. The contract version does not move: the plan structure is
unchanged, and bumping for less than a shape change trains people to bump
reflexively.
With discovery comes the question of where the loop closes, which was previously
unspecified — routing inferred that plan children are auto-admitted, but nothing
said so. State it as a rule and enforce it:
Admission is inherited, never invented.
may_opt_in() lets tireless label a plan child, because a human admitted its
parent, and refuses to label a discovered issue, because nothing has been
admitted. It is a function rather than a config flag on purpose: the failure it
prevents is unbounded, not merely wrong, so relaxing it should require review.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013TxK1CWPkFXqdcXMJ4hVe6
The invariants in CLAUDE.md were tested functions nobody called. There was no
config type at all: --config was accepted and ignored by all three binaries,
figment was an unused dependency, and the 124-line config.toml.tmpl was
aspirational. assert_not_anthropic could not run at startup because nothing read
the provider it checks.
Add tireless_core::config, validated on construction so a Config in hand is a
checked one, and tireless_agent::preflight, called by the worker at startup and
by `tireless preflight` on demand — the same function, so the two cannot
disagree. Move the Anthropic guard to tireless_core::policy where validate() can
reach it; tireless_agent::opencode re-exports it so the documented path resolves.
Also make /v1/ready honest. It returned "ok" unconditionally while its own doc
comment promised dependency checks, so the deploy probe greened on a process
that could do nothing. It now reports per-dependency state, with unwired ones
saying not_implemented rather than ok.
Tests parse the shipped template rather than a fixture, so template and code
cannot drift apart silently.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013TxK1CWPkFXqdcXMJ4hVe6
Passthrough is verified and pinned by regression tests that assert on what
cortex forwarded upstream, so tireless can rely on it. The design no longer
carries it as an open dependency.
Two residual behaviours replace it, neither a passthrough defect:
Multiple system messages are forwarded unmerged and in order, last one wins.
OpenCode sends its own preamble alongside an agent's configured prompt, so the
stage 5 question becomes "did implement.oc.md arrive last", not "did it arrive".
That failure is silent -- an agent that reads as a generic coding assistant and
ignores Out of scope -- and would present as a prompt-quality problem rather
than a plumbing one, so stage 5 now asserts on the upstream request.
helexa#223: /no_think is ignored on /v1/responses, where a thinking model can
spend its whole output budget on reasoning and return "" with status
incomplete. Added RunOutcome::OutputBudgetExhausted so that is retryable but
blameless -- a mis-sized ceiling must not trip a circuit breaker on a lane with
nothing wrong with it. Config pins the chat/completions surface.
The qwen3_next gap is recorded as a gap, not a hole: the system slot is not
arch-branched, so there is no family-specific path to fail.
Fleet now offers Qwen3-Coder-Next, a better fit for executing a written spec,
but cold and feasible only on beast where the pinned 27B lives. Recorded as an
operator decision per generic.md §14 rather than taken here. Config also notes
why tireless pins a model name and never a helexa/* capability alias.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DHhHtohxcdk1PL3tfnYJdH
helexa#179's application-owned system prompts let tireless shape both ends of
the cc->oc handoff, so the "will a 27B execute an Opus plan" risk becomes a
tunable rather than a hope.
Three prompts, versioned as one set: plan.cc.md tells Claude Code it is writing
for a literal, absent reader; implement.oc.md tells OpenCode to execute exactly
that and report rather than improvise; implement.cc.md covers unplanned issues.
PromptSet::load refuses a mismatched set, and tests assert the prompts mention
every section ChildSpec requires.
The middle is validated, not trusted. plan::validate rejects a plan before any
implementation job is enqueued unless every child carries a runnable acceptance
command (a stopping condition) and a non-empty out-of-scope list (a boundary) --
the two sections a small model needs and a human reader does not. Dangling and
cyclic dependencies are caught too, and implementation_order derives the start
order.
cc uses --append-system-prompt, never --system-prompt: replacing Claude Code's
default discards the tool-use scaffolding that makes it a coding agent. Pin
bumped to 2.1.220, the version this flag surface was verified against.
The oc path depends on helexa#179's passthrough guarantee, which is still open
and unverified for qwen3 arch templating. Stage 5 now opens with a PONG probe
rather than debugging it through a failed implementation run.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DHhHtohxcdk1PL3tfnYJdH
Autonomous issue-to-PR driver for Claude Code and OpenCode, structured per
lair/architecture generic.md.
Workspace: entities/core/data/agent library crates plus api, worker and cli
binaries. Two pieces of real logic land with tests — lane routing (cc for
judgement, oc for specification) and the limit governor.
Constraints encoded as code rather than comments:
- agents are spawned as vendor binaries; tireless never calls a provider API
- ANTHROPIC_API_KEY is never set by tireless, only passed through
- assert_not_anthropic refuses to start an OpenCode lane pointed at Anthropic
- every run passes the governor; provider rate-limit signals win over our own
accounting
Deployment assets target bob.hanzalova.internal:23296 (registered in
port-allocations.md), fronted by hanzalova at tireless.internal.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DHhHtohxcdk1PL3tfnYJdH
