grenade/rustingface
1
0
Fork 0
Code Issues 2 Pull Requests Actions Packages Projects Releases Wiki Activity

doc: design specification

Browse Source
This commit is contained in:
grenade
2026-08-31 09:47:23 +03:00
commit e12457cf27
1 changed files with 562 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/spec.md
+562
View File
@@ -0,0 +1,562 @@
# rustingface
**A sovereign, HuggingFace-compatible model registry.**
Single Rust binary. One config file. S3-compatible storage. Anything a client
fetches through it lives in your bucket until you explicitly remove it.
---
## 1. Purpose
Inference operators build stacks pinned to specific model revisions, then
depend on `huggingface.co` remaining reachable, unchanged, and willing to serve
them for the life of that stack. rustingface removes that dependency without
requiring any client-side change beyond setting `HF_ENDPOINT`.
### Guarantees
The following are contractual. Everything else in this document is
implementation detail in service of them.
1. **Fetch-once retention.** Any blob served through rustingface is durably
stored in the configured bucket before the response completes. It is removed
only by an explicit operator action.
2. **No eviction.** There is no TTL, no LRU, no size-triggered sweep, no
background cleaner. Storage growth is the operator's problem, by design.
3. **Offline sufficiency.** Once a `(repo, revision, file)` triple has been
served, it is servable again with no network path to any upstream, forever.
4. **Reference stability.** A mutable ref (`main`) resolves to the same commit
SHA on every subsequent request, until the operator explicitly repins it.
5. **State portability.** The bucket is the entire system state. Lose the host,
reinstall the binary, point it at the bucket, and service resumes. There is
no second datastore to back up in sync.
### Non-goals
- Multi-tenancy. Running this as a service for third parties would relocate the
dependency rather than remove it, and would make the operator a redistributor
of weights under licences that frequently forbid it. rustingface is software
you run, not a service you consume.
- Write/upload API, repo creation, commits, PRs, discussions.
- Spaces, Inference API, model search, trending, likes.
- Git or Git-LFS server protocol.
- Xet client or CAS implementation (see §8).
- A web UI. `hf` CLI and the admin CLI are the interfaces.
---
## 2. Core principle: record and replay
rustingface does **not** reimplement the Hub's semantics. It records the exact
HTTP metadata upstream returned at first fetch, and replays it verbatim
thereafter.
Concretely: `ETag`, `X-Repo-Commit`, `X-Linked-Etag` and `X-Linked-Size` are
persisted as opaque strings in the manifest and echoed back byte-identically on
every later request. rustingface never computes a git blob SHA-1, never derives
an etag from content, never decides what a commit hash "should" be.
This matters because client cache layouts are keyed on etag. Replaying upstream
values verbatim means a `~/.cache/huggingface` populated via rustingface is
interchangeable with one populated directly from the Hub, and stays correct
across upstream changes to how those values are computed.
The corollary: rustingface cannot serve a `(repo, revision, file)` triple it has
never seen upstream. That is the intended failure mode, not a limitation to
engineer around.
---
## 3. Architecture
```
client (transformers / vLLM / hf CLI)
│ HF_ENDPOINT=https://models.internal
▼
┌──────────────────────────────────────┐
│ rustingface │
│ │
│ axum router ──► resolver │
│ │ │
│ ├─► metadata (S3) │
│ ├─► blobs (S3) │
│ └─► upstream │
│ (miss only) │
└──────────────────────────────────────┘
│ │
▼ ▼
S3-compatible huggingface.co
(MinIO/Ceph/…) (optional, miss-only)
```
Single process. No database, no queue, no cache server, no sidecar.
### Request lifecycle (resolve)
```
1. parse (repo_type, repo_id, revision, filename)
2. revision is a ref name? ──► look up refs/… ──► commit sha
│ miss + upstream enabled
└──► resolve upstream, freeze, persist
3. load manifests/…/<commit>.json
4. entry present + blob exists? ──► serve (proxy or 302 presign)
5. otherwise, if upstream enabled:
single-flight upstream fetch
tee: client stream ‖ S3 multipart upload
on completion: write manifest
6. otherwise: 404 with the appropriate X-Error-Code
```
---
## 4. Storage layout
Everything lives in one bucket. Prefixes are stable and human-inspectable.
```
blobs/sha256/<aa>/<bb>/<full-64-hex>
manifests/<repo_type>/<namespace>/<name>/<commit-sha>.json
refs/<repo_type>/<namespace>/<name>/<ref-name>.json
repos/<repo_type>/<namespace>/<name>/repo.json
```
- `repo_type` ∈ `models` | `datasets`.
- Two-level hex fanout on blobs to avoid hot key prefixes on backends that
shard by key range.
- Blob keys are content-addressed by SHA-256 of the file bytes. Deduplication
across repos and revisions is automatic.
- Manifests are immutable once written. A commit SHA fully determines content.
- Refs are the only mutable objects, and only under explicit operator action.
### Manifest schema
```jsonc
{
"schema": 1,
"repo_type": "models",
"repo_id": "Qwen/Qwen3-32B",
"commit": "8f2a...c31",
"recorded_at": "2026-08-31T09:14:02Z",
"upstream": "https://huggingface.co",
"repo_meta": {
// verbatim subset of the upstream /api/models response, sufficient to
// answer model-info requests offline
"id": "Qwen/Qwen3-32B",
"sha": "8f2a...c31",
"lastModified": "2026-06-11T00:00:00.000Z",
"private": false,
"gated": false,
"tags": ["text-generation", "safetensors"],
"siblings": [{ "rfilename": "config.json" }, { "rfilename": "model-00001-of-00017.safetensors" }]
},
"entries": [
{
"path": "config.json",
"size": 1204,
"digest": "sha256:9c1e...ff", // rustingface blob key
"etag": "\"a3f9c2...\"", // upstream ETag, verbatim, incl. quotes
"linked_etag": null, // upstream X-Linked-Etag, verbatim
"linked_size": null, // upstream X-Linked-Size, verbatim
"lfs": false
},
{
"path": "model-00001-of-00017.safetensors",
"size": 4831838208,
"digest": "sha256:71ab...09",
"etag": "\"71ab...09\"",
"linked_etag": "\"71ab...09\"",
"linked_size": 4831838208,
"lfs": true
}
]
}
```
`entries` is append-only within a commit. A partially-populated manifest is
valid and expected: clients typically fetch a subset of a repo's files, and
rustingface records what it has actually seen. A file absent from `entries` is
a miss, not a 404, when upstream is available.
### Ref schema
```jsonc
{
"schema": 1,
"ref": "main",
"commit": "8f2a...c31",
"pinned_at": "2026-08-31T09:14:02Z",
"pinned_by": "auto-freeze" // or "operator" after an explicit repin
}
```
### Write ordering and crash safety
Strict two-phase, always in this order:
1. Complete the S3 multipart upload of the blob. The blob is not referenced by
anything yet; a crash here leaks an orphan, recoverable by `gc`.
2. Write or update the manifest referencing it.
A manifest must never reference a blob whose upload has not completed. This
ordering means the only possible inconsistency after a crash is an unreferenced
blob, which is safe and reclaimable. The inverse (a dangling manifest entry)
would be a silent correctness failure and must be impossible by construction.
Concurrent manifest writes for the same commit are resolved by read-merge-write
with a bounded retry. Entries are deterministic for a given commit, so merges
cannot conflict semantically; last-writer-wins on the merged set is correct.
Ref creation should use a conditional put (`If-None-Match: *`) where the backend
supports it, so first-write-wins freezing is atomic. Where unsupported, fall
back to read-then-write; the race window produces two identical values in the
common case and is corrected by the next explicit repin otherwise.
---
## 5. Configuration
Single TOML file, default `/etc/rustingface/config.toml`. Overridable per-key by
environment (`RUSTINGFACE__SERVER__LISTEN`). Secrets are read from files, never
inline, so they can be managed by `pass`/systemd credentials and kept out of the
config.
```toml
[server]
listen = "127.0.0.1:8080"
external_url = "https://models.internal.example" # used to build redirect URLs
request_timeout = "300s"
[storage]
endpoint = "https://s3.internal.example:9000"
bucket = "rustingface"
region = "us-east-1"
path_style = true
access_key_id_file = "/etc/rustingface/s3-access-key"
secret_access_key_file = "/etc/rustingface/s3-secret-key"
blob_delivery = "proxy" # proxy | redirect
presign_ttl = "15m" # redirect mode only
multipart_part_size = "16MiB"
[upstream]
enabled = true # false = hard-sealed, serve only what is stored
endpoint = "https://huggingface.co"
token_file = "/etc/rustingface/hf-token" # optional
connect_timeout = "10s"
read_timeout = "60s"
disable_xet = true # see §8; do not change
[policy]
ref_resolution = "freeze" # freeze | follow
allow_new_repos = true # false = only repos already in the bucket
allowlist = [] # optional glob list, e.g. ["Qwen/*", "meta-llama/*"]
[auth]
mode = "none" # none | bearer
token_file = "/etc/rustingface/client-tokens"
[observability]
metrics = true # /metrics, Prometheus text format
log_format = "json" # json | text
```
### `policy.ref_resolution`
- `freeze` (default) — the first resolution of a mutable ref is recorded and
reused indefinitely. Upstream is never consulted for that ref again.
- `follow` — mutable refs are re-resolved upstream when reachable, falling back
to the stored pin when not.
`follow` breaks guarantee 4 and is offered only for development. It should log a
warning at startup. In `freeze` mode, moving to a newer revision is an operator
action (`rustingface refresh`), which is the entire point: upstream cannot
change what your fleet resolves to.
---
## 6. HTTP surface
All paths accept an optional `/api/models/…` or `/api/datasets/…` repo type
prefix as the upstream does. `{repo_id}` is `namespace/name` or a bare `name`.
### Metadata
| Method | Path | Notes |
|---|---|---|
| `GET` | `/api/models/{repo_id}` | Serves `repo_meta` from the manifest of the current pinned ref |
| `GET` | `/api/models/{repo_id}/revision/{revision}` | As above, for a specific revision |
| `GET` | `/api/models/{repo_id}/refs` | `{"branches":[{"name","ref","targetCommit"}],"tags":[]}` from stored refs |
| `GET` | `/api/models/{repo_id}/tree/{revision}/{path}` | Supports `recursive`, `expand`; built from manifest `entries` |
| `GET` | `/api/datasets/...` | Same four shapes |
| `GET` | `/api/whoami-v2` | Static identity so `hf auth login` succeeds |
### Blobs
| Method | Path |
|---|---|
| `HEAD` | `/{repo_id}/resolve/{revision}/{filename}` |
| `GET` | `/{repo_id}/resolve/{revision}/{filename}` |
| `HEAD`/`GET` | `/datasets/{repo_id}/resolve/{revision}/{filename}` |
### Response header contract
This is the part clients actually depend on. On both `HEAD` and `GET`:
| Header | Value |
|---|---|
| `X-Repo-Commit` | Resolved commit SHA, replayed from the manifest |
| `ETag` | Upstream `ETag`, verbatim, quotes included |
| `X-Linked-Etag` | Upstream value, verbatim; omitted if upstream omitted it |
| `X-Linked-Size` | Upstream value, verbatim; omitted if upstream omitted it |
| `Content-Length` | True byte length |
| `Accept-Ranges` | `bytes` |
Rules:
- **Never** set `Content-Encoding`. The client sends `Accept-Encoding: identity`
precisely to learn real sizes; compressing this path corrupts size accounting.
- The client prefers `X-Linked-Etag` and falls back to `ETag`. Emit both exactly
as upstream did rather than normalising.
- Range requests must be honoured on the proxy path for resumable downloads.
- In `redirect` mode, respond `302` with `Location` set to a presigned URL and
the metadata headers still present on the `HEAD` response.
### Error semantics
The client maps a custom header to typed exceptions. Emit it.
| Condition | Status | `X-Error-Code` |
|---|---|---|
| Repo not stored, upstream disabled or unreachable | `404` | `RepoNotFound` |
| Revision unknown and unresolvable | `404` | `RevisionNotFound` |
| File not in manifest and unfetchable | `404` | `EntryNotFound` |
| Upstream returned gated/403 | `403` | `GatedRepo` |
| Upstream auth failed | `401` | — |
> Verify the exact `X-Error-Code` strings against the `huggingface_hub` version
> you pin in the conformance suite (§11); they are client-internal constants and
> should be treated as version-coupled rather than stable API.
Failures must be loud. A missing blob with no upstream is an error response, not
an empty file or a partial stream. Recent `huggingface_hub` raises
`IncompleteSnapshotError` rather than returning a partial directory when the Hub
is unreachable and files are missing, which is the behaviour you want to
preserve rather than paper over.
---
## 7. Fetch path
### Single-flight
Concurrent requests for the same uncached `(repo_type, repo_id, commit, path)`
must produce exactly one upstream fetch. Key an in-flight map on that tuple;
subsequent callers subscribe to the same broadcast rather than issuing their own
request. Without this, a fleet-wide rollout of a new model produces N concurrent
40GB pulls.
### Streaming tee
The upstream response body is consumed once and written to two sinks
concurrently:
- the client response stream, and
- an S3 multipart upload, buffered at `multipart_part_size`.
Digest is computed incrementally over the same stream. On completion, verify the
digest against upstream's LFS oid where one was provided; mismatch aborts the
upload and fails the request.
**If the client disconnects mid-transfer, the upload continues to completion in
a detached task.** Discarding 30GB of transferred bytes because a client hit
Ctrl-C is the single most annoying possible behaviour, and the retention
guarantee is better served by finishing.
### Upstream auth
`upstream.token_file` is used as `Authorization: Bearer` on upstream requests
only. It is never echoed downstream and never logged. Gated repos work if the
operator's token has accepted the terms; the resulting stored copy is for that
operator's own use, which is the licensing posture this design depends on.
---
## 8. Xet policy
The Hub is migrating to Xet-backed storage, where a Xet-aware client receives
file reconstruction information from a CAS rather than a URL. rustingface
implements none of it.
- **Downstream:** never advertise Xet capability. Clients then take the ordinary
resolve-and-redirect path.
- **Upstream:** set `HF_HUB_DISABLE_XET=1` semantics on outbound fetches so
rustingface receives whole-file URLs via the Git LFS bridge, which reconstructs
the file and returns a single resource URL for legacy clients. Storing whole
blobs keeps the bucket independently readable, which is guarantee 5.
The cost is fetch-time bandwidth efficiency on first pull. The benefit is that
the on-disk format stays "files with names", and a future maintainer of this
project cannot strand anyone's data behind a chunk-reconstruction format. That
trade is the right way round for a sovereignty tool.
---
## 9. Admin CLI
Same binary, subcommands.
```
rustingface serve # default; run the HTTP service
rustingface fetch <repo>[@rev] [--files …] # warm the bucket ahead of need
rustingface pin <repo> <ref> <commit> # set a ref explicitly
rustingface refresh <repo> [<ref>] # re-resolve upstream and repin
rustingface list [--repo-type models] # stored repos, revisions, sizes
rustingface show <repo>[@rev] # manifest contents
rustingface rm <repo>[@rev] # remove refs/manifests (NOT blobs)
rustingface gc [--dry-run] # delete blobs no manifest references
rustingface verify [<repo>] # re-read blobs, check digests
rustingface doctor # config, S3 reachability, permissions
```
`rm` and `gc` are deliberately separate. Removal detaches metadata; reclamation
is a second, explicit, dry-runnable step. Nothing deletes bytes without an
operator typing `gc`.
---
## 10. Deployment
### Crates
| Concern | Crate |
|---|---|
| HTTP server | `axum` + `tower-http` |
| Object storage | `object_store` (S3/local/Azure behind one API, multipart handled) |
| Upstream client | `reqwest` with `rustls-tls` |
| Runtime | `tokio` |
| Config | `serde` + `toml` |
| Hashing | `sha2` |
| CLI | `clap` |
| Single-flight | `dashmap` + `tokio::sync::broadcast` |
| Telemetry | `tracing`, `tracing-subscriber`, `metrics-exporter-prometheus` |
### TLS
Terminate with `rustls` using the `aws-lc-rs` provider and a hybrid post-quantum
key exchange group (`X25519MLKEM768`), with certificates issued by the internal
step-ca. Alternatively terminate at a reverse proxy and run rustingface on
loopback; `server.external_url` exists so redirect URLs remain correct either
way.
### systemd unit
```ini
[Unit]
Description=rustingface sovereign model registry
After=network-online.target
Wants=network-online.target
[Service]
Type=exec
ExecStart=/usr/local/bin/rustingface serve --config /etc/rustingface/config.toml
DynamicUser=yes
LoadCredential=s3-key:/etc/rustingface/s3-secret-key
LoadCredential=hf-token:/etc/rustingface/hf-token
NoNewPrivileges=yes
ProtectSystem=strict
ProtectHome=yes
PrivateTmp=yes
PrivateDevices=yes
ProtectKernelTunables=yes
ProtectKernelModules=yes
ProtectControlGroups=yes
RestrictAddressFamilies=AF_INET AF_INET6
RestrictNamespaces=yes
LockPersonality=yes
MemoryDenyWriteExecute=yes
SystemCallFilter=@system-service
SystemCallArchitectures=native
CapabilityBoundingSet=
AmbientCapabilities=
UMask=0077
Restart=on-failure
RestartSec=5s
[Install]
WantedBy=multi-user.target
```
There is no local state directory to protect, which is what makes the hardening
this easy: the process holds no durable data. `StateDirectory=` is deliberately
absent.
A Podman quadlet variant should mount only the config and credentials, with
`--read-only` and SELinux labels intact.
### Sizing
Memory is bounded by concurrent transfers × `multipart_part_size` × 2 (tee
buffers), plus a small metadata cache. A 16MiB part size with 32 concurrent
transfers is roughly 1GiB worst case. There is no reason for this service to
need more than 2GiB.
---
## 11. Conformance testing
The spec is only as good as the client compatibility, so test against real
clients rather than against this document.
**Matrix:** `huggingface_hub` (`hf_hub_download`, `snapshot_download`, `hf` CLI),
`transformers.from_pretrained`, `diffusers`, vLLM `--model`, and a
`safetensors`-only path. Pin exact client versions in CI and treat a client
upgrade as a spec-review trigger.
**The test that matters:**
1. Configure a fresh bucket. Set `HF_ENDPOINT`.
2. Pull a model. Confirm bytes landed in the bucket.
3. **Null-route `huggingface.co` at the site router.**
4. Wipe every client-side cache.
5. Pull the same model again. It must succeed, identically, with no upstream
traffic — verify at the router, not by trusting the service.
6. Restart rustingface, wipe its host entirely, reinstall, re-point at the
bucket, repeat step 5.
Anything less than step 6 tests a cache, not sovereignty.
**Additional cases:** concurrent cold fetch of the same file from 8 clients
(exactly one upstream request); client disconnect mid-transfer (upload still
completes); resumed/ranged download; gated repo with and without a valid token;
`main` pinning stability across an upstream commit; `gc` after `rm` reclaiming
exactly the right blobs and no others.
---
## 12. Phasing
| Phase | Scope |
|---|---|
| 0 | Resolve + model info + tree, proxy delivery, freeze pinning, single-flight, tee upload. Models only. |
| 1 | Datasets, redirect delivery with presigned URLs, admin CLI (`fetch`/`list`/`show`), metrics. |
| 2 | `gc`, `verify`, `refresh`, allowlist policy, bearer auth for clients. |
| 3 | Sealed mode (`upstream.enabled = false`) as a first-class deployment target; conformance suite in CI. |
| Later | Optional OCI/ModelPack export as a second read path over the same content-addressed blobs, hedging the possibility that the ecosystem standardises away from the HF protocol. |
---
## 13. Open questions
- **Repo metadata completeness.** `repo_meta` is recorded from whatever upstream
returned at first fetch. If a client later requests a field that was not
captured, is that a miss (fetch upstream) or a partial answer? Leaning toward
recording the full upstream JSON response verbatim and filtering on read,
which makes the question moot at the cost of some stored bytes.
- **`siblings` vs `entries` divergence.** `repo_meta.siblings` lists every file
in the repo upstream; `entries` lists only what has been fetched. `tree` and
model-info responses will therefore advertise files that are not stored.
Correct in `upstream.enabled = true`, and misleading in sealed mode. Sealed
mode should probably filter `siblings` down to `entries`.
- **Symlink/alias handling** for repos renamed upstream. Probably out of scope:
a rename produces a new repo id, and the old manifest stays valid forever,
which is arguably the desired behaviour.