Files
newsfeed/.gitea/workflows/deploy.yml
rob thijssen 3db1f586a8
Some checks failed
deploy / build-web (push) Successful in 1m30s
deploy / deploy-web (push) Successful in 4s
deploy / build-api (push) Successful in 6m35s
deploy / deploy-api (push) Failing after 10s
feat(modelwatch): push producer for open-weight model releases
Add newsfeed-modelwatch: a one-shot, systemd-timer-driven producer that watches
the Hugging Face Hub for open-weight releases and pushes filtered candidates to
the ingest endpoint. Answers "how do I automate the HF firehose into my feed" —
the RSS half already works on the pull rail; this is the JSON/API half.

How it works: each run polls the HF models API (a watchlist of orgs +
trendingScore), applies an admission predicate (license allowlist, total-params
cap, safetensors present, not gated, pipeline_tag), and POSTs the survivors to
POST /v1/ingest/candidates under a bearer token, tagged source="huggingface".

Two properties of the existing ingest rail shape the design:
- Ingest is idempotent on (user, external_id). Using the repo id as external_id
  makes the producer STATELESS — no dedup table; re-runs/overlapping timers just
  re-submit and the server drops repeats.
- HF `tags` are copied onto the candidate, so the user's per-interest weights do
  the ranking. The predicate is only an ADMISSION filter (what's worth surfacing
  at all) and stays subordinate to explicit weights, per the house rule.

Layout: newsfeed-modelwatch is a client of the API, not an internal component —
it holds no core/data deps. Pure logic (HF types, predicate, mapping to
CandidateSubmission, param formatting) lives in the lib with unit tests; the bin
does the HTTP polling/posting. CandidateSubmission gains Serialize so the
in-workspace producer can build and post one.

Ops:
- asset/systemd/newsfeed-modelwatch.{service,timer}: oneshot + 3-hourly timer.
  The ingest token is a secret, kept in /etc/newsfeed/modelwatch.env
  (NEWSFEED_TOKEN, mapped to `token` by figment) so a redeploy of the config
  never clobbers it.
- asset/config/modelwatch.toml.tmpl: watchlist + predicate, no secret.
- infra-setup.sh installs the units, creates the env placeholder (never
  overwritten), enables the timer, and prints the token step.
- deploy.yml builds + ships the binary and the (secret-free) config each deploy.

Verified: unit tests (gated union, license precedence, admit accept/reject,
mapping); dry-run against live HF; full loop against a local API — minted token,
submitted a real release, confirmed it landed in the feed with the huggingface
source linked and HF tags carried through. fmt/clippy/test all green.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016fKZzDpvjiJ9eYbPGgJvUP
2026-07-08 14:29:32 +03:00

164 lines
6.1 KiB
YAML

# CI-driven deploy for newsfeed (architecture deployment-gitea-actions.md).
#
# The workflow is the source of infra truth — hosts/ports/paths live here, not in a
# manifest. Build-and-rsync model: build the binaries + the SPA bundle, then rsync
# straight to the targets as the scoped `gitea_ci` user.
#
# api + worker -> slartibartfast.kosherinata.internal (share one SQLite file)
# web (SPA) -> oolon.kosherinata.internal (nginx serves + proxies /v1)
#
# Runners (see architecture gitea-runners.md): Rust build on `rust` (cargo toolchain);
# SPA build + deploy jobs on `fedora-43` (node + pnpm; ssh via git-core's openssh-clients
# dep; rsync + curl in the base). The whole fleet is Fedora, so binaries are built native —
# no glibc skew, no musl toolchain needed (deployment-gitea-actions.md §6: build on a
# runner no newer than the oldest target).
#
# One-time host prep (gitea_ci user, sudoers, service account, dirs, cert, vhost) is done
# by script/infra-setup.sh. Only secret required here: RSYNC_SSH_KEY (the runner's private
# key). newsfeed has no app secrets — SQLite has no password and tokens are hashed at rest.
name: deploy
on:
push:
branches: [main]
workflow_dispatch:
concurrency:
group: deploy
cancel-in-progress: false
env:
API_HOST: slartibartfast.kosherinata.internal
WEB_HOST: oolon.kosherinata.internal
API_PORT: "22672"
SSH_OPTS: -o StrictHostKeyChecking=accept-new
jobs:
build-api:
runs-on: rust
steps:
- uses: actions/checkout@v4
# lint/test gate: a broken commit never deploys
- run: cargo fmt --all --check
- run: cargo clippy --workspace --all-targets -- -D warnings
- run: cargo test --workspace
- name: build binaries
run: |
cargo build --release -p newsfeed-api -p newsfeed-worker -p newsfeed-modelwatch
mkdir -p out
cp target/release/newsfeed-api target/release/newsfeed-worker \
target/release/newsfeed-modelwatch out/
- uses: actions/upload-artifact@v3
with:
name: binaries
path: out/
build-web:
runs-on: fedora-43
steps:
- uses: actions/checkout@v4
- name: build SPA
working-directory: web
# pnpm is preinstalled globally on the fedora runners (via npm i -g pnpm);
# Fedora's nodejs doesn't bundle corepack, so call pnpm directly.
run: |
pnpm install --frozen-lockfile
pnpm build
- uses: actions/upload-artifact@v3
with:
name: web
path: web/dist/
deploy-api:
needs: build-api
runs-on: fedora-43
steps:
- uses: actions/checkout@v4
- uses: actions/download-artifact@v3
with:
name: binaries
path: bin
- name: write ssh key
run: |
install -d -m 0700 ~/.ssh
echo "${{ secrets.RSYNC_SSH_KEY }}" > ~/.ssh/id_deploy
chmod 0600 ~/.ssh/id_deploy
- name: reachability check
run: ssh $SSH_OPTS -i ~/.ssh/id_deploy gitea_ci@"$API_HOST" hostname -f
- name: rsync binaries
run: |
chmod +x bin/*
for b in newsfeed-api newsfeed-worker newsfeed-modelwatch; do
rsync -az -e "ssh $SSH_OPTS -i ~/.ssh/id_deploy" \
--rsync-path='sudo rsync' --chown=root:root --chmod=0755 --mkpath \
"bin/$b" gitea_ci@"$API_HOST":/usr/local/bin/"$b"
done
- name: rsync config
run: |
rsync -az -e "ssh $SSH_OPTS -i ~/.ssh/id_deploy" \
--rsync-path='sudo rsync' --chown=root:newsfeed --chmod=0640 --mkpath \
asset/config/config.toml.tmpl gitea_ci@"$API_HOST":/etc/newsfeed/config.toml
rsync -az -e "ssh $SSH_OPTS -i ~/.ssh/id_deploy" \
--rsync-path='sudo rsync' --chown=root:newsfeed --chmod=0640 --mkpath \
asset/config/worker.toml.tmpl gitea_ci@"$API_HOST":/etc/newsfeed/worker.toml
# modelwatch config carries no secret (the token lives in modelwatch.env), so it
# is safe to overwrite on every deploy.
rsync -az -e "ssh $SSH_OPTS -i ~/.ssh/id_deploy" \
--rsync-path='sudo rsync' --chown=root:newsfeed --chmod=0640 --mkpath \
asset/config/modelwatch.toml.tmpl gitea_ci@"$API_HOST":/etc/newsfeed/modelwatch.toml
- name: apply system state
run: |
ssh $SSH_OPTS -i ~/.ssh/id_deploy gitea_ci@"$API_HOST" '
sudo restorecon -R /usr/local/bin/newsfeed-api /usr/local/bin/newsfeed-worker /usr/local/bin/newsfeed-modelwatch /etc/newsfeed /var/lib/newsfeed
sudo systemctl daemon-reload
sudo systemctl restart newsfeed-api.service
sudo systemctl restart newsfeed-worker.service
'
- name: health probe
run: |
for i in $(seq 1 10); do
code=$(curl -fsS -o /dev/null -w '%{http_code}' "http://$API_HOST:$API_PORT/health") && [ "$code" = 200 ] && exit 0
sleep 2
done
echo "api did not become healthy" >&2; exit 1
- name: capture journal
if: always()
run: |
ssh $SSH_OPTS -i ~/.ssh/id_deploy gitea_ci@"$API_HOST" \
'journalctl -u newsfeed-api.service -u newsfeed-worker.service -n 80 --no-pager' || true
deploy-web:
needs: build-web
runs-on: fedora-43
steps:
- uses: actions/download-artifact@v3
with:
name: web
path: dist
- name: write ssh key
run: |
install -d -m 0700 ~/.ssh
echo "${{ secrets.RSYNC_SSH_KEY }}" > ~/.ssh/id_deploy
chmod 0600 ~/.ssh/id_deploy
- name: reachability check
run: ssh $SSH_OPTS -i ~/.ssh/id_deploy gitea_ci@"$WEB_HOST" hostname -f
- name: rsync SPA
run: |
rsync -az --delete -e "ssh $SSH_OPTS -i ~/.ssh/id_deploy" \
--rsync-path='sudo rsync' --chown=nginx:nginx --mkpath \
dist/ gitea_ci@"$WEB_HOST":/var/www/newsfeed/
- name: relabel + reload nginx
run: |
ssh $SSH_OPTS -i ~/.ssh/id_deploy gitea_ci@"$WEB_HOST" '
sudo restorecon -R /var/www/newsfeed
sudo nginx -t
sudo systemctl reload nginx.service
'