Add newsfeed-modelwatch: a one-shot, systemd-timer-driven producer that watches
the Hugging Face Hub for open-weight releases and pushes filtered candidates to
the ingest endpoint. Answers "how do I automate the HF firehose into my feed" —
the RSS half already works on the pull rail; this is the JSON/API half.
How it works: each run polls the HF models API (a watchlist of orgs +
trendingScore), applies an admission predicate (license allowlist, total-params
cap, safetensors present, not gated, pipeline_tag), and POSTs the survivors to
POST /v1/ingest/candidates under a bearer token, tagged source="huggingface".
Two properties of the existing ingest rail shape the design:
- Ingest is idempotent on (user, external_id). Using the repo id as external_id
makes the producer STATELESS — no dedup table; re-runs/overlapping timers just
re-submit and the server drops repeats.
- HF `tags` are copied onto the candidate, so the user's per-interest weights do
the ranking. The predicate is only an ADMISSION filter (what's worth surfacing
at all) and stays subordinate to explicit weights, per the house rule.
Layout: newsfeed-modelwatch is a client of the API, not an internal component —
it holds no core/data deps. Pure logic (HF types, predicate, mapping to
CandidateSubmission, param formatting) lives in the lib with unit tests; the bin
does the HTTP polling/posting. CandidateSubmission gains Serialize so the
in-workspace producer can build and post one.
Ops:
- asset/systemd/newsfeed-modelwatch.{service,timer}: oneshot + 3-hourly timer.
The ingest token is a secret, kept in /etc/newsfeed/modelwatch.env
(NEWSFEED_TOKEN, mapped to `token` by figment) so a redeploy of the config
never clobbers it.
- asset/config/modelwatch.toml.tmpl: watchlist + predicate, no secret.
- infra-setup.sh installs the units, creates the env placeholder (never
overwritten), enables the timer, and prints the token step.
- deploy.yml builds + ships the binary and the (secret-free) config each deploy.
Verified: unit tests (gated union, license precedence, admit accept/reject,
mapping); dry-run against live HF; full loop against a local API — minted token,
submitted a real release, confirmed it landed in the feed with the huggingface
source linked and HF tags carried through. fmt/clippy/test all green.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016fKZzDpvjiJ9eYbPGgJvUP
164 lines
6.1 KiB
YAML
164 lines
6.1 KiB
YAML
# CI-driven deploy for newsfeed (architecture deployment-gitea-actions.md).
|
|
#
|
|
# The workflow is the source of infra truth — hosts/ports/paths live here, not in a
|
|
# manifest. Build-and-rsync model: build the binaries + the SPA bundle, then rsync
|
|
# straight to the targets as the scoped `gitea_ci` user.
|
|
#
|
|
# api + worker -> slartibartfast.kosherinata.internal (share one SQLite file)
|
|
# web (SPA) -> oolon.kosherinata.internal (nginx serves + proxies /v1)
|
|
#
|
|
# Runners (see architecture gitea-runners.md): Rust build on `rust` (cargo toolchain);
|
|
# SPA build + deploy jobs on `fedora-43` (node + pnpm; ssh via git-core's openssh-clients
|
|
# dep; rsync + curl in the base). The whole fleet is Fedora, so binaries are built native —
|
|
# no glibc skew, no musl toolchain needed (deployment-gitea-actions.md §6: build on a
|
|
# runner no newer than the oldest target).
|
|
#
|
|
# One-time host prep (gitea_ci user, sudoers, service account, dirs, cert, vhost) is done
|
|
# by script/infra-setup.sh. Only secret required here: RSYNC_SSH_KEY (the runner's private
|
|
# key). newsfeed has no app secrets — SQLite has no password and tokens are hashed at rest.
|
|
|
|
name: deploy
|
|
on:
|
|
push:
|
|
branches: [main]
|
|
workflow_dispatch:
|
|
|
|
concurrency:
|
|
group: deploy
|
|
cancel-in-progress: false
|
|
|
|
env:
|
|
API_HOST: slartibartfast.kosherinata.internal
|
|
WEB_HOST: oolon.kosherinata.internal
|
|
API_PORT: "22672"
|
|
SSH_OPTS: -o StrictHostKeyChecking=accept-new
|
|
|
|
jobs:
|
|
build-api:
|
|
runs-on: rust
|
|
steps:
|
|
- uses: actions/checkout@v4
|
|
# lint/test gate: a broken commit never deploys
|
|
- run: cargo fmt --all --check
|
|
- run: cargo clippy --workspace --all-targets -- -D warnings
|
|
- run: cargo test --workspace
|
|
- name: build binaries
|
|
run: |
|
|
cargo build --release -p newsfeed-api -p newsfeed-worker -p newsfeed-modelwatch
|
|
mkdir -p out
|
|
cp target/release/newsfeed-api target/release/newsfeed-worker \
|
|
target/release/newsfeed-modelwatch out/
|
|
- uses: actions/upload-artifact@v3
|
|
with:
|
|
name: binaries
|
|
path: out/
|
|
|
|
build-web:
|
|
runs-on: fedora-43
|
|
steps:
|
|
- uses: actions/checkout@v4
|
|
- name: build SPA
|
|
working-directory: web
|
|
# pnpm is preinstalled globally on the fedora runners (via npm i -g pnpm);
|
|
# Fedora's nodejs doesn't bundle corepack, so call pnpm directly.
|
|
run: |
|
|
pnpm install --frozen-lockfile
|
|
pnpm build
|
|
- uses: actions/upload-artifact@v3
|
|
with:
|
|
name: web
|
|
path: web/dist/
|
|
|
|
deploy-api:
|
|
needs: build-api
|
|
runs-on: fedora-43
|
|
steps:
|
|
- uses: actions/checkout@v4
|
|
- uses: actions/download-artifact@v3
|
|
with:
|
|
name: binaries
|
|
path: bin
|
|
- name: write ssh key
|
|
run: |
|
|
install -d -m 0700 ~/.ssh
|
|
echo "${{ secrets.RSYNC_SSH_KEY }}" > ~/.ssh/id_deploy
|
|
chmod 0600 ~/.ssh/id_deploy
|
|
- name: reachability check
|
|
run: ssh $SSH_OPTS -i ~/.ssh/id_deploy gitea_ci@"$API_HOST" hostname -f
|
|
|
|
- name: rsync binaries
|
|
run: |
|
|
chmod +x bin/*
|
|
for b in newsfeed-api newsfeed-worker newsfeed-modelwatch; do
|
|
rsync -az -e "ssh $SSH_OPTS -i ~/.ssh/id_deploy" \
|
|
--rsync-path='sudo rsync' --chown=root:root --chmod=0755 --mkpath \
|
|
"bin/$b" gitea_ci@"$API_HOST":/usr/local/bin/"$b"
|
|
done
|
|
|
|
- name: rsync config
|
|
run: |
|
|
rsync -az -e "ssh $SSH_OPTS -i ~/.ssh/id_deploy" \
|
|
--rsync-path='sudo rsync' --chown=root:newsfeed --chmod=0640 --mkpath \
|
|
asset/config/config.toml.tmpl gitea_ci@"$API_HOST":/etc/newsfeed/config.toml
|
|
rsync -az -e "ssh $SSH_OPTS -i ~/.ssh/id_deploy" \
|
|
--rsync-path='sudo rsync' --chown=root:newsfeed --chmod=0640 --mkpath \
|
|
asset/config/worker.toml.tmpl gitea_ci@"$API_HOST":/etc/newsfeed/worker.toml
|
|
# modelwatch config carries no secret (the token lives in modelwatch.env), so it
|
|
# is safe to overwrite on every deploy.
|
|
rsync -az -e "ssh $SSH_OPTS -i ~/.ssh/id_deploy" \
|
|
--rsync-path='sudo rsync' --chown=root:newsfeed --chmod=0640 --mkpath \
|
|
asset/config/modelwatch.toml.tmpl gitea_ci@"$API_HOST":/etc/newsfeed/modelwatch.toml
|
|
|
|
- name: apply system state
|
|
run: |
|
|
ssh $SSH_OPTS -i ~/.ssh/id_deploy gitea_ci@"$API_HOST" '
|
|
sudo restorecon -R /usr/local/bin/newsfeed-api /usr/local/bin/newsfeed-worker /usr/local/bin/newsfeed-modelwatch /etc/newsfeed /var/lib/newsfeed
|
|
sudo systemctl daemon-reload
|
|
sudo systemctl restart newsfeed-api.service
|
|
sudo systemctl restart newsfeed-worker.service
|
|
'
|
|
|
|
- name: health probe
|
|
run: |
|
|
for i in $(seq 1 10); do
|
|
code=$(curl -fsS -o /dev/null -w '%{http_code}' "http://$API_HOST:$API_PORT/health") && [ "$code" = 200 ] && exit 0
|
|
sleep 2
|
|
done
|
|
echo "api did not become healthy" >&2; exit 1
|
|
|
|
- name: capture journal
|
|
if: always()
|
|
run: |
|
|
ssh $SSH_OPTS -i ~/.ssh/id_deploy gitea_ci@"$API_HOST" \
|
|
'journalctl -u newsfeed-api.service -u newsfeed-worker.service -n 80 --no-pager' || true
|
|
|
|
deploy-web:
|
|
needs: build-web
|
|
runs-on: fedora-43
|
|
steps:
|
|
- uses: actions/download-artifact@v3
|
|
with:
|
|
name: web
|
|
path: dist
|
|
- name: write ssh key
|
|
run: |
|
|
install -d -m 0700 ~/.ssh
|
|
echo "${{ secrets.RSYNC_SSH_KEY }}" > ~/.ssh/id_deploy
|
|
chmod 0600 ~/.ssh/id_deploy
|
|
- name: reachability check
|
|
run: ssh $SSH_OPTS -i ~/.ssh/id_deploy gitea_ci@"$WEB_HOST" hostname -f
|
|
|
|
- name: rsync SPA
|
|
run: |
|
|
rsync -az --delete -e "ssh $SSH_OPTS -i ~/.ssh/id_deploy" \
|
|
--rsync-path='sudo rsync' --chown=nginx:nginx --mkpath \
|
|
dist/ gitea_ci@"$WEB_HOST":/var/www/newsfeed/
|
|
|
|
- name: relabel + reload nginx
|
|
run: |
|
|
ssh $SSH_OPTS -i ~/.ssh/id_deploy gitea_ci@"$WEB_HOST" '
|
|
sudo restorecon -R /var/www/newsfeed
|
|
sudo nginx -t
|
|
sudo systemctl reload nginx.service
|
|
'
|