rob thijssen
059c781422
The suite's first run on a runner reported "10 passed, 0 failed" and
exited 0. Locally it is 11. The difference is the `hf` CLI check, which
did not run: the workflow invokes .venv/bin/python directly without
putting the venv's bin on PATH, so shutil.which("hf") found nothing and
the check skipped.
A skipped check reads exactly like a passing one, and this is the suite
that runs unattended on a weekly timer specifically to notice when a
client changes underneath us -- so a silent skip defeats its purpose.
Exports the venv's bin in the workflow and asserts `hf version` before
the suite, so a missing CLI fails the job outright rather than being
quietly stepped over. Also counts skips and lists them in the summary,
where they cannot be mistaken for passes.
Verified both ways: with the venv off PATH the suite now reports
"10 passed, 0 failed, 1 skipped" and names it; with it on PATH, 11
passed and 0 skipped.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XZG2i4AmfSqE97EJGBVb64
38 lines
1.3 KiB
YAML
38 lines
1.3 KiB
YAML
name: conformance
|
|
|
|
on:
|
|
workflow_dispatch:
|
|
schedule:
|
|
# Weekly. The clients are pinned, so this catches upstream Hub changes
|
|
# rather than client upgrades; a client upgrade is a deliberate commit and
|
|
# a spec-review trigger.
|
|
- cron: "0 4 * * 1"
|
|
|
|
jobs:
|
|
conformance:
|
|
runs-on: rust
|
|
steps:
|
|
- uses: actions/checkout@v4
|
|
|
|
- name: build
|
|
run: cargo build --release --bin rustingface
|
|
|
|
- name: install pinned clients
|
|
run: |
|
|
python3 -m venv .venv
|
|
.venv/bin/pip install --upgrade pip
|
|
.venv/bin/pip install -r test/conformance/requirements.txt
|
|
|
|
# Runs against a directory rather than a bucket. The S3 path is covered
|
|
# by the `doctor` step of the deploy workflow, which exercises the real
|
|
# MinIO on caveman with the real credentials.
|
|
# The venv's bin goes on PATH so the `hf` CLI is found. Without it the
|
|
# suite reports "skipped: hf not on PATH" and still exits 0 -- a check
|
|
# that silently does not run, in the one suite whose job is to notice
|
|
# when a client changes underneath us.
|
|
- name: conformance
|
|
run: |
|
|
export PATH="$PWD/.venv/bin:$PATH"
|
|
hf version
|
|
.venv/bin/python test/conformance/conformance.py --binary target/release/rustingface
|