grenade/rustingface

Fork 0

Files

rob thijssen e12457cf27

doc: design specification

2026-08-31 09:47:23 +03:00

22 KiB

Raw Permalink Blame History

rustingface

A sovereign, HuggingFace-compatible model registry.

Single Rust binary. One config file. S3-compatible storage. Anything a client fetches through it lives in your bucket until you explicitly remove it.

1. Purpose

Inference operators build stacks pinned to specific model revisions, then depend on huggingface.co remaining reachable, unchanged, and willing to serve them for the life of that stack. rustingface removes that dependency without requiring any client-side change beyond setting HF_ENDPOINT.

Guarantees

The following are contractual. Everything else in this document is implementation detail in service of them.

Fetch-once retention. Any blob served through rustingface is durably stored in the configured bucket before the response completes. It is removed only by an explicit operator action.
No eviction. There is no TTL, no LRU, no size-triggered sweep, no background cleaner. Storage growth is the operator's problem, by design.
Offline sufficiency. Once a (repo, revision, file) triple has been served, it is servable again with no network path to any upstream, forever.
Reference stability. A mutable ref (main) resolves to the same commit SHA on every subsequent request, until the operator explicitly repins it.
State portability. The bucket is the entire system state. Lose the host, reinstall the binary, point it at the bucket, and service resumes. There is no second datastore to back up in sync.

Non-goals

Multi-tenancy. Running this as a service for third parties would relocate the dependency rather than remove it, and would make the operator a redistributor of weights under licences that frequently forbid it. rustingface is software you run, not a service you consume.
Write/upload API, repo creation, commits, PRs, discussions.
Spaces, Inference API, model search, trending, likes.
Git or Git-LFS server protocol.
Xet client or CAS implementation (see §8).
A web UI. hf CLI and the admin CLI are the interfaces.

2. Core principle: record and replay

rustingface does not reimplement the Hub's semantics. It records the exact HTTP metadata upstream returned at first fetch, and replays it verbatim thereafter.

Concretely: ETag, X-Repo-Commit, X-Linked-Etag and X-Linked-Size are persisted as opaque strings in the manifest and echoed back byte-identically on every later request. rustingface never computes a git blob SHA-1, never derives an etag from content, never decides what a commit hash "should" be.

This matters because client cache layouts are keyed on etag. Replaying upstream values verbatim means a ~/.cache/huggingface populated via rustingface is interchangeable with one populated directly from the Hub, and stays correct across upstream changes to how those values are computed.

The corollary: rustingface cannot serve a (repo, revision, file) triple it has never seen upstream. That is the intended failure mode, not a limitation to engineer around.

3. Architecture

   client (transformers / vLLM / hf CLI)
        │  HF_ENDPOINT=https://models.internal
        ▼
  ┌──────────────────────────────────────┐
  │            rustingface               │
  │                                      │
  │  axum router ──► resolver            │
  │                    │                 │
  │                    ├─► metadata (S3) │
  │                    ├─► blobs (S3)    │
  │                    └─► upstream      │
  │                        (miss only)   │
  └──────────────────────────────────────┘
        │                        │
        ▼                        ▼
   S3-compatible            huggingface.co
   (MinIO/Ceph/…)           (optional, miss-only)

Single process. No database, no queue, no cache server, no sidecar.

Request lifecycle (resolve)

1. parse (repo_type, repo_id, revision, filename)
2. revision is a ref name?  ──► look up refs/…  ──► commit sha
                                     │ miss + upstream enabled
                                     └──► resolve upstream, freeze, persist
3. load manifests/…/<commit>.json
4. entry present + blob exists?  ──► serve (proxy or 302 presign)
5. otherwise, if upstream enabled:
      single-flight upstream fetch
      tee: client stream  ‖  S3 multipart upload
      on completion: write manifest
6. otherwise: 404 with the appropriate X-Error-Code

4. Storage layout

Everything lives in one bucket. Prefixes are stable and human-inspectable.

blobs/sha256/<aa>/<bb>/<full-64-hex>
manifests/<repo_type>/<namespace>/<name>/<commit-sha>.json
refs/<repo_type>/<namespace>/<name>/<ref-name>.json
repos/<repo_type>/<namespace>/<name>/repo.json

repo_type ∈ models | datasets.
Two-level hex fanout on blobs to avoid hot key prefixes on backends that shard by key range.
Blob keys are content-addressed by SHA-256 of the file bytes. Deduplication across repos and revisions is automatic.
Manifests are immutable once written. A commit SHA fully determines content.
Refs are the only mutable objects, and only under explicit operator action.

Manifest schema

{
  "schema": 1,
  "repo_type": "models",
  "repo_id": "Qwen/Qwen3-32B",
  "commit": "8f2a...c31",
  "recorded_at": "2026-08-31T09:14:02Z",
  "upstream": "https://huggingface.co",
  "repo_meta": {
    // verbatim subset of the upstream /api/models response, sufficient to
    // answer model-info requests offline
    "id": "Qwen/Qwen3-32B",
    "sha": "8f2a...c31",
    "lastModified": "2026-06-11T00:00:00.000Z",
    "private": false,
    "gated": false,
    "tags": ["text-generation", "safetensors"],
    "siblings": [{ "rfilename": "config.json" }, { "rfilename": "model-00001-of-00017.safetensors" }]
  },
  "entries": [
    {
      "path": "config.json",
      "size": 1204,
      "digest": "sha256:9c1e...ff",     // rustingface blob key
      "etag": "\"a3f9c2...\"",           // upstream ETag, verbatim, incl. quotes
      "linked_etag": null,                // upstream X-Linked-Etag, verbatim
      "linked_size": null,                // upstream X-Linked-Size, verbatim
      "lfs": false
    },
    {
      "path": "model-00001-of-00017.safetensors",
      "size": 4831838208,
      "digest": "sha256:71ab...09",
      "etag": "\"71ab...09\"",
      "linked_etag": "\"71ab...09\"",
      "linked_size": 4831838208,
      "lfs": true
    }
  ]
}

entries is append-only within a commit. A partially-populated manifest is valid and expected: clients typically fetch a subset of a repo's files, and rustingface records what it has actually seen. A file absent from entries is a miss, not a 404, when upstream is available.

Ref schema

{
  "schema": 1,
  "ref": "main",
  "commit": "8f2a...c31",
  "pinned_at": "2026-08-31T09:14:02Z",
  "pinned_by": "auto-freeze"    // or "operator" after an explicit repin
}

Write ordering and crash safety

Strict two-phase, always in this order:

Complete the S3 multipart upload of the blob. The blob is not referenced by anything yet; a crash here leaks an orphan, recoverable by gc.
Write or update the manifest referencing it.

A manifest must never reference a blob whose upload has not completed. This ordering means the only possible inconsistency after a crash is an unreferenced blob, which is safe and reclaimable. The inverse (a dangling manifest entry) would be a silent correctness failure and must be impossible by construction.

Concurrent manifest writes for the same commit are resolved by read-merge-write with a bounded retry. Entries are deterministic for a given commit, so merges cannot conflict semantically; last-writer-wins on the merged set is correct.

Ref creation should use a conditional put (If-None-Match: *) where the backend supports it, so first-write-wins freezing is atomic. Where unsupported, fall back to read-then-write; the race window produces two identical values in the common case and is corrected by the next explicit repin otherwise.

5. Configuration

Single TOML file, default /etc/rustingface/config.toml. Overridable per-key by environment (RUSTINGFACE__SERVER__LISTEN). Secrets are read from files, never inline, so they can be managed by pass/systemd credentials and kept out of the config.

[server]
listen         = "127.0.0.1:8080"
external_url   = "https://models.internal.example"   # used to build redirect URLs
request_timeout = "300s"

[storage]
endpoint               = "https://s3.internal.example:9000"
bucket                 = "rustingface"
region                 = "us-east-1"
path_style             = true
access_key_id_file     = "/etc/rustingface/s3-access-key"
secret_access_key_file = "/etc/rustingface/s3-secret-key"
blob_delivery          = "proxy"        # proxy | redirect
presign_ttl            = "15m"          # redirect mode only
multipart_part_size    = "16MiB"

[upstream]
enabled         = true                  # false = hard-sealed, serve only what is stored
endpoint        = "https://huggingface.co"
token_file      = "/etc/rustingface/hf-token"   # optional
connect_timeout = "10s"
read_timeout    = "60s"
disable_xet     = true                  # see §8; do not change

[policy]
ref_resolution = "freeze"               # freeze | follow
allow_new_repos = true                  # false = only repos already in the bucket
allowlist = []                          # optional glob list, e.g. ["Qwen/*", "meta-llama/*"]

[auth]
mode       = "none"                     # none | bearer
token_file = "/etc/rustingface/client-tokens"

[observability]
metrics = true                          # /metrics, Prometheus text format
log_format = "json"                     # json | text

`policy.ref_resolution`

freeze (default) — the first resolution of a mutable ref is recorded and reused indefinitely. Upstream is never consulted for that ref again.
follow — mutable refs are re-resolved upstream when reachable, falling back to the stored pin when not.

follow breaks guarantee 4 and is offered only for development. It should log a warning at startup. In freeze mode, moving to a newer revision is an operator action (rustingface refresh), which is the entire point: upstream cannot change what your fleet resolves to.

6. HTTP surface

All paths accept an optional /api/models/… or /api/datasets/… repo type prefix as the upstream does. {repo_id} is namespace/name or a bare name.

Metadata

Method	Path	Notes
`GET`	`/api/models/{repo_id}`	Serves `repo_meta` from the manifest of the current pinned ref
`GET`	`/api/models/{repo_id}/revision/{revision}`	As above, for a specific revision
`GET`	`/api/models/{repo_id}/refs`	`{"branches":[{"name","ref","targetCommit"}],"tags":[]}` from stored refs
`GET`	`/api/models/{repo_id}/tree/{revision}/{path}`	Supports `recursive`, `expand`; built from manifest `entries`
`GET`	`/api/datasets/...`	Same four shapes
`GET`	`/api/whoami-v2`	Static identity so `hf auth login` succeeds

Blobs

Method	Path
`HEAD`	`/{repo_id}/resolve/{revision}/{filename}`
`GET`	`/{repo_id}/resolve/{revision}/{filename}`
`HEAD`/`GET`	`/datasets/{repo_id}/resolve/{revision}/{filename}`

Response header contract

This is the part clients actually depend on. On both HEAD and GET:

Header	Value
`X-Repo-Commit`	Resolved commit SHA, replayed from the manifest
`ETag`	Upstream `ETag`, verbatim, quotes included
`X-Linked-Etag`	Upstream value, verbatim; omitted if upstream omitted it
`X-Linked-Size`	Upstream value, verbatim; omitted if upstream omitted it
`Content-Length`	True byte length
`Accept-Ranges`	`bytes`

Rules:

Never set Content-Encoding. The client sends Accept-Encoding: identity precisely to learn real sizes; compressing this path corrupts size accounting.
The client prefers X-Linked-Etag and falls back to ETag. Emit both exactly as upstream did rather than normalising.
Range requests must be honoured on the proxy path for resumable downloads.
In redirect mode, respond 302 with Location set to a presigned URL and the metadata headers still present on the HEAD response.

Error semantics

The client maps a custom header to typed exceptions. Emit it.

Condition	Status	`X-Error-Code`
Repo not stored, upstream disabled or unreachable	`404`	`RepoNotFound`
Revision unknown and unresolvable	`404`	`RevisionNotFound`
File not in manifest and unfetchable	`404`	`EntryNotFound`
Upstream returned gated/403	`403`	`GatedRepo`
Upstream auth failed	`401`	—

Verify the exact X-Error-Code strings against the huggingface_hub version you pin in the conformance suite (§11); they are client-internal constants and should be treated as version-coupled rather than stable API.

Failures must be loud. A missing blob with no upstream is an error response, not an empty file or a partial stream. Recent huggingface_hub raises IncompleteSnapshotError rather than returning a partial directory when the Hub is unreachable and files are missing, which is the behaviour you want to preserve rather than paper over.

7. Fetch path

Single-flight

Concurrent requests for the same uncached (repo_type, repo_id, commit, path) must produce exactly one upstream fetch. Key an in-flight map on that tuple; subsequent callers subscribe to the same broadcast rather than issuing their own request. Without this, a fleet-wide rollout of a new model produces N concurrent 40GB pulls.

Streaming tee

The upstream response body is consumed once and written to two sinks concurrently:

the client response stream, and
an S3 multipart upload, buffered at multipart_part_size.

Digest is computed incrementally over the same stream. On completion, verify the digest against upstream's LFS oid where one was provided; mismatch aborts the upload and fails the request.

If the client disconnects mid-transfer, the upload continues to completion in a detached task. Discarding 30GB of transferred bytes because a client hit Ctrl-C is the single most annoying possible behaviour, and the retention guarantee is better served by finishing.

Upstream auth

upstream.token_file is used as Authorization: Bearer on upstream requests only. It is never echoed downstream and never logged. Gated repos work if the operator's token has accepted the terms; the resulting stored copy is for that operator's own use, which is the licensing posture this design depends on.

8. Xet policy

The Hub is migrating to Xet-backed storage, where a Xet-aware client receives file reconstruction information from a CAS rather than a URL. rustingface implements none of it.

Downstream: never advertise Xet capability. Clients then take the ordinary resolve-and-redirect path.
Upstream: set HF_HUB_DISABLE_XET=1 semantics on outbound fetches so rustingface receives whole-file URLs via the Git LFS bridge, which reconstructs the file and returns a single resource URL for legacy clients. Storing whole blobs keeps the bucket independently readable, which is guarantee 5.

The cost is fetch-time bandwidth efficiency on first pull. The benefit is that the on-disk format stays "files with names", and a future maintainer of this project cannot strand anyone's data behind a chunk-reconstruction format. That trade is the right way round for a sovereignty tool.

9. Admin CLI

Same binary, subcommands.

rustingface serve                          # default; run the HTTP service
rustingface fetch <repo>[@rev] [--files …] # warm the bucket ahead of need
rustingface pin <repo> <ref> <commit>      # set a ref explicitly
rustingface refresh <repo> [<ref>]         # re-resolve upstream and repin
rustingface list [--repo-type models]      # stored repos, revisions, sizes
rustingface show <repo>[@rev]              # manifest contents
rustingface rm <repo>[@rev]                # remove refs/manifests (NOT blobs)
rustingface gc [--dry-run]                 # delete blobs no manifest references
rustingface verify [<repo>]                # re-read blobs, check digests
rustingface doctor                         # config, S3 reachability, permissions

rm and gc are deliberately separate. Removal detaches metadata; reclamation is a second, explicit, dry-runnable step. Nothing deletes bytes without an operator typing gc.

10. Deployment

Crates

Concern	Crate
HTTP server	`axum` + `tower-http`
Object storage	`object_store` (S3/local/Azure behind one API, multipart handled)
Upstream client	`reqwest` with `rustls-tls`
Runtime	`tokio`
Config	`serde` + `toml`
Hashing	`sha2`
CLI	`clap`
Single-flight	`dashmap` + `tokio::sync::broadcast`
Telemetry	`tracing`, `tracing-subscriber`, `metrics-exporter-prometheus`

TLS

Terminate with rustls using the aws-lc-rs provider and a hybrid post-quantum key exchange group (X25519MLKEM768), with certificates issued by the internal step-ca. Alternatively terminate at a reverse proxy and run rustingface on loopback; server.external_url exists so redirect URLs remain correct either way.

systemd unit

[Unit]
Description=rustingface sovereign model registry
After=network-online.target
Wants=network-online.target

[Service]
Type=exec
ExecStart=/usr/local/bin/rustingface serve --config /etc/rustingface/config.toml
DynamicUser=yes
LoadCredential=s3-key:/etc/rustingface/s3-secret-key
LoadCredential=hf-token:/etc/rustingface/hf-token
NoNewPrivileges=yes
ProtectSystem=strict
ProtectHome=yes
PrivateTmp=yes
PrivateDevices=yes
ProtectKernelTunables=yes
ProtectKernelModules=yes
ProtectControlGroups=yes
RestrictAddressFamilies=AF_INET AF_INET6
RestrictNamespaces=yes
LockPersonality=yes
MemoryDenyWriteExecute=yes
SystemCallFilter=@system-service
SystemCallArchitectures=native
CapabilityBoundingSet=
AmbientCapabilities=
UMask=0077
Restart=on-failure
RestartSec=5s

[Install]
WantedBy=multi-user.target

There is no local state directory to protect, which is what makes the hardening this easy: the process holds no durable data. StateDirectory= is deliberately absent.

A Podman quadlet variant should mount only the config and credentials, with --read-only and SELinux labels intact.

Sizing

Memory is bounded by concurrent transfers × multipart_part_size × 2 (tee buffers), plus a small metadata cache. A 16MiB part size with 32 concurrent transfers is roughly 1GiB worst case. There is no reason for this service to need more than 2GiB.

11. Conformance testing

The spec is only as good as the client compatibility, so test against real clients rather than against this document.

Matrix: huggingface_hub (hf_hub_download, snapshot_download, hf CLI), transformers.from_pretrained, diffusers, vLLM --model, and a safetensors-only path. Pin exact client versions in CI and treat a client upgrade as a spec-review trigger.

The test that matters:

Configure a fresh bucket. Set HF_ENDPOINT.
Pull a model. Confirm bytes landed in the bucket.
Null-route huggingface.co at the site router.
Wipe every client-side cache.
Pull the same model again. It must succeed, identically, with no upstream traffic — verify at the router, not by trusting the service.
Restart rustingface, wipe its host entirely, reinstall, re-point at the bucket, repeat step 5.

Anything less than step 6 tests a cache, not sovereignty.

Additional cases: concurrent cold fetch of the same file from 8 clients (exactly one upstream request); client disconnect mid-transfer (upload still completes); resumed/ranged download; gated repo with and without a valid token; main pinning stability across an upstream commit; gc after rm reclaiming exactly the right blobs and no others.

12. Phasing

Phase	Scope
0	Resolve + model info + tree, proxy delivery, freeze pinning, single-flight, tee upload. Models only.
1	Datasets, redirect delivery with presigned URLs, admin CLI (`fetch`/`list`/`show`), metrics.
2	`gc`, `verify`, `refresh`, allowlist policy, bearer auth for clients.
3	Sealed mode (`upstream.enabled = false`) as a first-class deployment target; conformance suite in CI.
Later	Optional OCI/ModelPack export as a second read path over the same content-addressed blobs, hedging the possibility that the ecosystem standardises away from the HF protocol.

13. Open questions

Repo metadata completeness. repo_meta is recorded from whatever upstream returned at first fetch. If a client later requests a field that was not captured, is that a miss (fetch upstream) or a partial answer? Leaning toward recording the full upstream JSON response verbatim and filtering on read, which makes the question moot at the cost of some stored bytes.
siblings vs entries divergence. repo_meta.siblings lists every file in the repo upstream; entries lists only what has been fetched. tree and model-info responses will therefore advertise files that are not stored. Correct in upstream.enabled = true, and misleading in sealed mode. Sealed mode should probably filter siblings down to entries.
Symlink/alias handling for repos renamed upstream. Probably out of scope: a rename produces a new repo id, and the old manifest stays valid forever, which is arguably the desired behaviour.

22 KiB Raw Permalink Blame History Unescape Escape