commit e12457cf27a6d47e128b44bacc2af822083a9eb6 Author: rob thijssen Date: Mon Aug 31 09:47:23 2026 +0300 doc: design specification diff --git a/doc/spec.md b/doc/spec.md new file mode 100644 index 0000000..d727c5f --- /dev/null +++ b/doc/spec.md @@ -0,0 +1,562 @@ +# rustingface + +**A sovereign, HuggingFace-compatible model registry.** + +Single Rust binary. One config file. S3-compatible storage. Anything a client +fetches through it lives in your bucket until you explicitly remove it. + +--- + +## 1. Purpose + +Inference operators build stacks pinned to specific model revisions, then +depend on `huggingface.co` remaining reachable, unchanged, and willing to serve +them for the life of that stack. rustingface removes that dependency without +requiring any client-side change beyond setting `HF_ENDPOINT`. + +### Guarantees + +The following are contractual. Everything else in this document is +implementation detail in service of them. + +1. **Fetch-once retention.** Any blob served through rustingface is durably + stored in the configured bucket before the response completes. It is removed + only by an explicit operator action. +2. **No eviction.** There is no TTL, no LRU, no size-triggered sweep, no + background cleaner. Storage growth is the operator's problem, by design. +3. **Offline sufficiency.** Once a `(repo, revision, file)` triple has been + served, it is servable again with no network path to any upstream, forever. +4. **Reference stability.** A mutable ref (`main`) resolves to the same commit + SHA on every subsequent request, until the operator explicitly repins it. +5. **State portability.** The bucket is the entire system state. Lose the host, + reinstall the binary, point it at the bucket, and service resumes. There is + no second datastore to back up in sync. + +### Non-goals + +- Multi-tenancy. Running this as a service for third parties would relocate the + dependency rather than remove it, and would make the operator a redistributor + of weights under licences that frequently forbid it. rustingface is software + you run, not a service you consume. +- Write/upload API, repo creation, commits, PRs, discussions. +- Spaces, Inference API, model search, trending, likes. +- Git or Git-LFS server protocol. +- Xet client or CAS implementation (see §8). +- A web UI. `hf` CLI and the admin CLI are the interfaces. + +--- + +## 2. Core principle: record and replay + +rustingface does **not** reimplement the Hub's semantics. It records the exact +HTTP metadata upstream returned at first fetch, and replays it verbatim +thereafter. + +Concretely: `ETag`, `X-Repo-Commit`, `X-Linked-Etag` and `X-Linked-Size` are +persisted as opaque strings in the manifest and echoed back byte-identically on +every later request. rustingface never computes a git blob SHA-1, never derives +an etag from content, never decides what a commit hash "should" be. + +This matters because client cache layouts are keyed on etag. Replaying upstream +values verbatim means a `~/.cache/huggingface` populated via rustingface is +interchangeable with one populated directly from the Hub, and stays correct +across upstream changes to how those values are computed. + +The corollary: rustingface cannot serve a `(repo, revision, file)` triple it has +never seen upstream. That is the intended failure mode, not a limitation to +engineer around. + +--- + +## 3. Architecture + +``` + client (transformers / vLLM / hf CLI) + │ HF_ENDPOINT=https://models.internal + ▼ + ┌──────────────────────────────────────┐ + │ rustingface │ + │ │ + │ axum router ──► resolver │ + │ │ │ + │ ├─► metadata (S3) │ + │ ├─► blobs (S3) │ + │ └─► upstream │ + │ (miss only) │ + └──────────────────────────────────────┘ + │ │ + ▼ ▼ + S3-compatible huggingface.co + (MinIO/Ceph/…) (optional, miss-only) +``` + +Single process. No database, no queue, no cache server, no sidecar. + +### Request lifecycle (resolve) + +``` +1. parse (repo_type, repo_id, revision, filename) +2. revision is a ref name? ──► look up refs/… ──► commit sha + │ miss + upstream enabled + └──► resolve upstream, freeze, persist +3. load manifests/…/.json +4. entry present + blob exists? ──► serve (proxy or 302 presign) +5. otherwise, if upstream enabled: + single-flight upstream fetch + tee: client stream ‖ S3 multipart upload + on completion: write manifest +6. otherwise: 404 with the appropriate X-Error-Code +``` + +--- + +## 4. Storage layout + +Everything lives in one bucket. Prefixes are stable and human-inspectable. + +``` +blobs/sha256/// +manifests////.json +refs////.json +repos////repo.json +``` + +- `repo_type` ∈ `models` | `datasets`. +- Two-level hex fanout on blobs to avoid hot key prefixes on backends that + shard by key range. +- Blob keys are content-addressed by SHA-256 of the file bytes. Deduplication + across repos and revisions is automatic. +- Manifests are immutable once written. A commit SHA fully determines content. +- Refs are the only mutable objects, and only under explicit operator action. + +### Manifest schema + +```jsonc +{ + "schema": 1, + "repo_type": "models", + "repo_id": "Qwen/Qwen3-32B", + "commit": "8f2a...c31", + "recorded_at": "2026-08-31T09:14:02Z", + "upstream": "https://huggingface.co", + "repo_meta": { + // verbatim subset of the upstream /api/models response, sufficient to + // answer model-info requests offline + "id": "Qwen/Qwen3-32B", + "sha": "8f2a...c31", + "lastModified": "2026-06-11T00:00:00.000Z", + "private": false, + "gated": false, + "tags": ["text-generation", "safetensors"], + "siblings": [{ "rfilename": "config.json" }, { "rfilename": "model-00001-of-00017.safetensors" }] + }, + "entries": [ + { + "path": "config.json", + "size": 1204, + "digest": "sha256:9c1e...ff", // rustingface blob key + "etag": "\"a3f9c2...\"", // upstream ETag, verbatim, incl. quotes + "linked_etag": null, // upstream X-Linked-Etag, verbatim + "linked_size": null, // upstream X-Linked-Size, verbatim + "lfs": false + }, + { + "path": "model-00001-of-00017.safetensors", + "size": 4831838208, + "digest": "sha256:71ab...09", + "etag": "\"71ab...09\"", + "linked_etag": "\"71ab...09\"", + "linked_size": 4831838208, + "lfs": true + } + ] +} +``` + +`entries` is append-only within a commit. A partially-populated manifest is +valid and expected: clients typically fetch a subset of a repo's files, and +rustingface records what it has actually seen. A file absent from `entries` is +a miss, not a 404, when upstream is available. + +### Ref schema + +```jsonc +{ + "schema": 1, + "ref": "main", + "commit": "8f2a...c31", + "pinned_at": "2026-08-31T09:14:02Z", + "pinned_by": "auto-freeze" // or "operator" after an explicit repin +} +``` + +### Write ordering and crash safety + +Strict two-phase, always in this order: + +1. Complete the S3 multipart upload of the blob. The blob is not referenced by + anything yet; a crash here leaks an orphan, recoverable by `gc`. +2. Write or update the manifest referencing it. + +A manifest must never reference a blob whose upload has not completed. This +ordering means the only possible inconsistency after a crash is an unreferenced +blob, which is safe and reclaimable. The inverse (a dangling manifest entry) +would be a silent correctness failure and must be impossible by construction. + +Concurrent manifest writes for the same commit are resolved by read-merge-write +with a bounded retry. Entries are deterministic for a given commit, so merges +cannot conflict semantically; last-writer-wins on the merged set is correct. + +Ref creation should use a conditional put (`If-None-Match: *`) where the backend +supports it, so first-write-wins freezing is atomic. Where unsupported, fall +back to read-then-write; the race window produces two identical values in the +common case and is corrected by the next explicit repin otherwise. + +--- + +## 5. Configuration + +Single TOML file, default `/etc/rustingface/config.toml`. Overridable per-key by +environment (`RUSTINGFACE__SERVER__LISTEN`). Secrets are read from files, never +inline, so they can be managed by `pass`/systemd credentials and kept out of the +config. + +```toml +[server] +listen = "127.0.0.1:8080" +external_url = "https://models.internal.example" # used to build redirect URLs +request_timeout = "300s" + +[storage] +endpoint = "https://s3.internal.example:9000" +bucket = "rustingface" +region = "us-east-1" +path_style = true +access_key_id_file = "/etc/rustingface/s3-access-key" +secret_access_key_file = "/etc/rustingface/s3-secret-key" +blob_delivery = "proxy" # proxy | redirect +presign_ttl = "15m" # redirect mode only +multipart_part_size = "16MiB" + +[upstream] +enabled = true # false = hard-sealed, serve only what is stored +endpoint = "https://huggingface.co" +token_file = "/etc/rustingface/hf-token" # optional +connect_timeout = "10s" +read_timeout = "60s" +disable_xet = true # see §8; do not change + +[policy] +ref_resolution = "freeze" # freeze | follow +allow_new_repos = true # false = only repos already in the bucket +allowlist = [] # optional glob list, e.g. ["Qwen/*", "meta-llama/*"] + +[auth] +mode = "none" # none | bearer +token_file = "/etc/rustingface/client-tokens" + +[observability] +metrics = true # /metrics, Prometheus text format +log_format = "json" # json | text +``` + +### `policy.ref_resolution` + +- `freeze` (default) — the first resolution of a mutable ref is recorded and + reused indefinitely. Upstream is never consulted for that ref again. +- `follow` — mutable refs are re-resolved upstream when reachable, falling back + to the stored pin when not. + +`follow` breaks guarantee 4 and is offered only for development. It should log a +warning at startup. In `freeze` mode, moving to a newer revision is an operator +action (`rustingface refresh`), which is the entire point: upstream cannot +change what your fleet resolves to. + +--- + +## 6. HTTP surface + +All paths accept an optional `/api/models/…` or `/api/datasets/…` repo type +prefix as the upstream does. `{repo_id}` is `namespace/name` or a bare `name`. + +### Metadata + +| Method | Path | Notes | +|---|---|---| +| `GET` | `/api/models/{repo_id}` | Serves `repo_meta` from the manifest of the current pinned ref | +| `GET` | `/api/models/{repo_id}/revision/{revision}` | As above, for a specific revision | +| `GET` | `/api/models/{repo_id}/refs` | `{"branches":[{"name","ref","targetCommit"}],"tags":[]}` from stored refs | +| `GET` | `/api/models/{repo_id}/tree/{revision}/{path}` | Supports `recursive`, `expand`; built from manifest `entries` | +| `GET` | `/api/datasets/...` | Same four shapes | +| `GET` | `/api/whoami-v2` | Static identity so `hf auth login` succeeds | + +### Blobs + +| Method | Path | +|---|---| +| `HEAD` | `/{repo_id}/resolve/{revision}/{filename}` | +| `GET` | `/{repo_id}/resolve/{revision}/{filename}` | +| `HEAD`/`GET` | `/datasets/{repo_id}/resolve/{revision}/{filename}` | + +### Response header contract + +This is the part clients actually depend on. On both `HEAD` and `GET`: + +| Header | Value | +|---|---| +| `X-Repo-Commit` | Resolved commit SHA, replayed from the manifest | +| `ETag` | Upstream `ETag`, verbatim, quotes included | +| `X-Linked-Etag` | Upstream value, verbatim; omitted if upstream omitted it | +| `X-Linked-Size` | Upstream value, verbatim; omitted if upstream omitted it | +| `Content-Length` | True byte length | +| `Accept-Ranges` | `bytes` | + +Rules: + +- **Never** set `Content-Encoding`. The client sends `Accept-Encoding: identity` + precisely to learn real sizes; compressing this path corrupts size accounting. +- The client prefers `X-Linked-Etag` and falls back to `ETag`. Emit both exactly + as upstream did rather than normalising. +- Range requests must be honoured on the proxy path for resumable downloads. +- In `redirect` mode, respond `302` with `Location` set to a presigned URL and + the metadata headers still present on the `HEAD` response. + +### Error semantics + +The client maps a custom header to typed exceptions. Emit it. + +| Condition | Status | `X-Error-Code` | +|---|---|---| +| Repo not stored, upstream disabled or unreachable | `404` | `RepoNotFound` | +| Revision unknown and unresolvable | `404` | `RevisionNotFound` | +| File not in manifest and unfetchable | `404` | `EntryNotFound` | +| Upstream returned gated/403 | `403` | `GatedRepo` | +| Upstream auth failed | `401` | — | + +> Verify the exact `X-Error-Code` strings against the `huggingface_hub` version +> you pin in the conformance suite (§11); they are client-internal constants and +> should be treated as version-coupled rather than stable API. + +Failures must be loud. A missing blob with no upstream is an error response, not +an empty file or a partial stream. Recent `huggingface_hub` raises +`IncompleteSnapshotError` rather than returning a partial directory when the Hub +is unreachable and files are missing, which is the behaviour you want to +preserve rather than paper over. + +--- + +## 7. Fetch path + +### Single-flight + +Concurrent requests for the same uncached `(repo_type, repo_id, commit, path)` +must produce exactly one upstream fetch. Key an in-flight map on that tuple; +subsequent callers subscribe to the same broadcast rather than issuing their own +request. Without this, a fleet-wide rollout of a new model produces N concurrent +40GB pulls. + +### Streaming tee + +The upstream response body is consumed once and written to two sinks +concurrently: + +- the client response stream, and +- an S3 multipart upload, buffered at `multipart_part_size`. + +Digest is computed incrementally over the same stream. On completion, verify the +digest against upstream's LFS oid where one was provided; mismatch aborts the +upload and fails the request. + +**If the client disconnects mid-transfer, the upload continues to completion in +a detached task.** Discarding 30GB of transferred bytes because a client hit +Ctrl-C is the single most annoying possible behaviour, and the retention +guarantee is better served by finishing. + +### Upstream auth + +`upstream.token_file` is used as `Authorization: Bearer` on upstream requests +only. It is never echoed downstream and never logged. Gated repos work if the +operator's token has accepted the terms; the resulting stored copy is for that +operator's own use, which is the licensing posture this design depends on. + +--- + +## 8. Xet policy + +The Hub is migrating to Xet-backed storage, where a Xet-aware client receives +file reconstruction information from a CAS rather than a URL. rustingface +implements none of it. + +- **Downstream:** never advertise Xet capability. Clients then take the ordinary + resolve-and-redirect path. +- **Upstream:** set `HF_HUB_DISABLE_XET=1` semantics on outbound fetches so + rustingface receives whole-file URLs via the Git LFS bridge, which reconstructs + the file and returns a single resource URL for legacy clients. Storing whole + blobs keeps the bucket independently readable, which is guarantee 5. + +The cost is fetch-time bandwidth efficiency on first pull. The benefit is that +the on-disk format stays "files with names", and a future maintainer of this +project cannot strand anyone's data behind a chunk-reconstruction format. That +trade is the right way round for a sovereignty tool. + +--- + +## 9. Admin CLI + +Same binary, subcommands. + +``` +rustingface serve # default; run the HTTP service +rustingface fetch [@rev] [--files …] # warm the bucket ahead of need +rustingface pin # set a ref explicitly +rustingface refresh [] # re-resolve upstream and repin +rustingface list [--repo-type models] # stored repos, revisions, sizes +rustingface show [@rev] # manifest contents +rustingface rm [@rev] # remove refs/manifests (NOT blobs) +rustingface gc [--dry-run] # delete blobs no manifest references +rustingface verify [] # re-read blobs, check digests +rustingface doctor # config, S3 reachability, permissions +``` + +`rm` and `gc` are deliberately separate. Removal detaches metadata; reclamation +is a second, explicit, dry-runnable step. Nothing deletes bytes without an +operator typing `gc`. + +--- + +## 10. Deployment + +### Crates + +| Concern | Crate | +|---|---| +| HTTP server | `axum` + `tower-http` | +| Object storage | `object_store` (S3/local/Azure behind one API, multipart handled) | +| Upstream client | `reqwest` with `rustls-tls` | +| Runtime | `tokio` | +| Config | `serde` + `toml` | +| Hashing | `sha2` | +| CLI | `clap` | +| Single-flight | `dashmap` + `tokio::sync::broadcast` | +| Telemetry | `tracing`, `tracing-subscriber`, `metrics-exporter-prometheus` | + +### TLS + +Terminate with `rustls` using the `aws-lc-rs` provider and a hybrid post-quantum +key exchange group (`X25519MLKEM768`), with certificates issued by the internal +step-ca. Alternatively terminate at a reverse proxy and run rustingface on +loopback; `server.external_url` exists so redirect URLs remain correct either +way. + +### systemd unit + +```ini +[Unit] +Description=rustingface sovereign model registry +After=network-online.target +Wants=network-online.target + +[Service] +Type=exec +ExecStart=/usr/local/bin/rustingface serve --config /etc/rustingface/config.toml +DynamicUser=yes +LoadCredential=s3-key:/etc/rustingface/s3-secret-key +LoadCredential=hf-token:/etc/rustingface/hf-token +NoNewPrivileges=yes +ProtectSystem=strict +ProtectHome=yes +PrivateTmp=yes +PrivateDevices=yes +ProtectKernelTunables=yes +ProtectKernelModules=yes +ProtectControlGroups=yes +RestrictAddressFamilies=AF_INET AF_INET6 +RestrictNamespaces=yes +LockPersonality=yes +MemoryDenyWriteExecute=yes +SystemCallFilter=@system-service +SystemCallArchitectures=native +CapabilityBoundingSet= +AmbientCapabilities= +UMask=0077 +Restart=on-failure +RestartSec=5s + +[Install] +WantedBy=multi-user.target +``` + +There is no local state directory to protect, which is what makes the hardening +this easy: the process holds no durable data. `StateDirectory=` is deliberately +absent. + +A Podman quadlet variant should mount only the config and credentials, with +`--read-only` and SELinux labels intact. + +### Sizing + +Memory is bounded by concurrent transfers × `multipart_part_size` × 2 (tee +buffers), plus a small metadata cache. A 16MiB part size with 32 concurrent +transfers is roughly 1GiB worst case. There is no reason for this service to +need more than 2GiB. + +--- + +## 11. Conformance testing + +The spec is only as good as the client compatibility, so test against real +clients rather than against this document. + +**Matrix:** `huggingface_hub` (`hf_hub_download`, `snapshot_download`, `hf` CLI), +`transformers.from_pretrained`, `diffusers`, vLLM `--model`, and a +`safetensors`-only path. Pin exact client versions in CI and treat a client +upgrade as a spec-review trigger. + +**The test that matters:** + +1. Configure a fresh bucket. Set `HF_ENDPOINT`. +2. Pull a model. Confirm bytes landed in the bucket. +3. **Null-route `huggingface.co` at the site router.** +4. Wipe every client-side cache. +5. Pull the same model again. It must succeed, identically, with no upstream + traffic — verify at the router, not by trusting the service. +6. Restart rustingface, wipe its host entirely, reinstall, re-point at the + bucket, repeat step 5. + +Anything less than step 6 tests a cache, not sovereignty. + +**Additional cases:** concurrent cold fetch of the same file from 8 clients +(exactly one upstream request); client disconnect mid-transfer (upload still +completes); resumed/ranged download; gated repo with and without a valid token; +`main` pinning stability across an upstream commit; `gc` after `rm` reclaiming +exactly the right blobs and no others. + +--- + +## 12. Phasing + +| Phase | Scope | +|---|---| +| 0 | Resolve + model info + tree, proxy delivery, freeze pinning, single-flight, tee upload. Models only. | +| 1 | Datasets, redirect delivery with presigned URLs, admin CLI (`fetch`/`list`/`show`), metrics. | +| 2 | `gc`, `verify`, `refresh`, allowlist policy, bearer auth for clients. | +| 3 | Sealed mode (`upstream.enabled = false`) as a first-class deployment target; conformance suite in CI. | +| Later | Optional OCI/ModelPack export as a second read path over the same content-addressed blobs, hedging the possibility that the ecosystem standardises away from the HF protocol. | + +--- + +## 13. Open questions + +- **Repo metadata completeness.** `repo_meta` is recorded from whatever upstream + returned at first fetch. If a client later requests a field that was not + captured, is that a miss (fetch upstream) or a partial answer? Leaning toward + recording the full upstream JSON response verbatim and filtering on read, + which makes the question moot at the cost of some stored bytes. +- **`siblings` vs `entries` divergence.** `repo_meta.siblings` lists every file + in the repo upstream; `entries` lists only what has been fetched. `tree` and + model-info responses will therefore advertise files that are not stored. + Correct in `upstream.enabled = true`, and misleading in sealed mode. Sealed + mode should probably filter `siblings` down to `entries`. +- **Symlink/alias handling** for repos renamed upstream. Probably out of scope: + a rename produces a new repo id, and the old manifest stays valid forever, + which is arguably the desired behaviour.