Commit Graph

58 Commits

Author SHA1 Message Date
886e19717b chore: remove redundant metric 2026-03-30 10:57:51 +03:00
731e3688e7 Fix deploy script for oneshot service state detection
systemctl is-active returns "activating" (exit 3) for Type=oneshot
services while ExecStart is running. Match both "active" and
"activating" states so the deploy script correctly stops the index
service. Also stop the cluster service to break the index↔cluster
cycle cleanly during deploys.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-30 10:44:22 +03:00
c0bcf63daa Format progress log as progress=checked/total
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-30 10:34:57 +03:00
294cc10cf8 Cycle index and cluster as oneshot services
Replace OnSuccess with ExecStartPost so cluster only starts after
index genuinely completes, not when stopped by a deploy. Cluster's
ExecStartPost restarts index, creating a continuous cycle that breaks
cleanly when either service is stopped externally.

Add Conflicts= on both units so they never run simultaneously.
Change index to Type=oneshot and remove Restart=always.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-30 10:03:02 +03:00
a9413012a8 Include checked/total/indexed counts in progress log
Show how many galleries have been checked, how many were skipped vs
indexed, and the total so overall progress is visible at a glance.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-30 09:55:05 +03:00
4acf189562 Log progress every 100 skipped galleries
Emit an info log with the running skip count every 100 unchanged
galleries so long runs show progress rather than silence.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-30 09:49:31 +03:00
617fa34a23 Pre-warm connection pool and size it to match concurrency
Configure sqlx pool with min_connections = max_connections so all
connections are established at startup, avoiding slow-acquire warnings
from lazy mTLS handshakes. Add idle_timeout (5 min) to recycle stale
connections from prior runs, and reduce acquire_timeout to 10s for
faster failure.

Size the pool to io_concurrency + ml_concurrency + 2 to accommodate
the worst case where all IO tasks call image_exists concurrently.
Reduce default io_concurrency from 4× to 2× ML concurrency to keep
pool size within PostgreSQL's default max_connections.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-30 09:40:35 +03:00
154d46f5a0 Batch skip-path DB writes and reduce pool pressure
Skip paths in process_image no longer call upsert_gallery_image
individually. Instead they return the GalleryImage and the gallery
loop batch-upserts all skipped rows in a single query. This eliminates
~100 concurrent DB calls per gallery during the common case where most
images are unchanged.

Pool size reduced from io_concurrency + ml_concurrency + 4 to
ml_concurrency + 6, since only ML transactions and the occasional
image_exists fallback need connections during task execution.

Reduce service concurrency from 32 to 24 for index and cluster.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-30 09:37:03 +03:00
eed2184ed2 Fix DB connection pool exhaustion during image processing
Move IO semaphore acquisition to the task spawn site so at most
io_concurrency tasks are active at any time. Previously all ~100
images per gallery were spawned eagerly and raced for DB connections
on their skip-path upserts outside any semaphore, causing pool
timeouts when tasks exceeded the connection limit.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-30 09:31:37 +03:00
83cd55bdcb Remove --include/--exclude filters and unused dependencies
Drop FilterConfig and the glob-based include/exclude filtering that was
never used in practice and complicated gallery stats computation.

Remove 17 unused crate dependencies flagged by cargo-machete across
rbv-data, rbv-cluster, rbv-ingest, rbv-auth, rbv-search, rbv-cli, and
rbv-api. Remove glob, hyper-util, tokio-rustls, and tower from the
workspace root as no crate references them.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-30 09:29:25 +03:00
0dbc932d4a Optimize index pipeline with 4-layer skip logic and backfill command
Gallery-level skip: store file_count/total_bytes per gallery, skip
entirely when unchanged. Batch existence check: one query per gallery
instead of per image. File-size fast path: stat-only skip when filename
and size match stored values. Two-stage pipeline: separate IO and ML
concurrency (--io-concurrency flag).

Adds `rbv backfill` to populate file_size and gallery stats from disk
without ML, so skip optimisations are effective immediately after upgrade.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-30 09:14:07 +03:00
1e7bc916e1 Fix duplicate VALUES keyword in assign_faces_batch query
QueryBuilder::push_values() already prepends VALUES, but the initial
query string also included it, producing invalid SQL: FROM (VALUES VALUES ...).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-30 08:48:48 +03:00
4326b1cd51 fix: tag/subject ui 2026-03-30 08:34:46 +03:00
1d7a505fdb Separate tags and subjects into independent concepts
Tags and subjects were mixed in SQL queries (tags || subjects,
ANY(tags) OR ANY(subjects)) and UI rendering. Now each has its own
data queries, API endpoints, browse pages (/tags, /subjects), detail
pages (/tag/:tag, /subject/:subject), and add buttons in the gallery
footer. The gallery footer shows two labelled rows for subjects and
tags with independent autocomplete and add controls.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-27 15:41:27 +02:00
542b793a70 doc: subject list plan 2026-03-27 15:35:15 +02:00
1548010bd1 Sync pagination state with URL query params
Add usePageParam hook that reads/writes ?page=N via useSearchParams,
making all paginated pages bookmarkable and shareable. Applied to
People, Person (gp/fp params), MergePerson, Tag, and Tags pages.
Page 1 omits the param for clean default URLs.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-27 15:24:32 +02:00
326a6ea89f Add paginated /tags page with random gallery thumbnails per tag
New /tags route shows a grid of tag cards, each with a randomly
selected thumbnail from a gallery that has that tag. Uses a single
LATERAL join query paginated to 24 tags at a time to avoid timeout
on large tag sets. Refresh re-rolls the random images. Tags link
added to nav header.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-27 15:05:00 +02:00
58f91678c2 doc: tag list plan 2026-03-27 14:31:36 +02:00
64ea4305ee fix: cdn acls 2026-03-27 13:53:17 +02:00
a10436880c Add CDN support for serving images via nginx with API fallback
Adds --cdn-map CLI arg (e.g. --cdn-map /tank/data/rbv/vault=/vault)
to configure filesystem-to-URL prefix mappings. A GET /api/config
endpoint exposes these mappings to the frontend. Gallery viewer,
home page, and tag page resolve image/thumbnail URLs through nginx
when a matching CDN map exists, falling back to the API otherwise.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-27 13:46:32 +02:00
6d0ce945ed Add clickable tags and tag management to gallery viewer
Tags and subjects in the gallery view now link to /tag/:tag, which
shows a paginated grid of all galleries with that tag. A + button
lets users add new or existing tags to a gallery, with autocomplete
from all known tags. Backend adds by-tag query, tag listing, and
add-tag endpoints.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-27 12:58:55 +02:00
8d52abd932 fix(deploy): shell quoting 2026-03-27 12:49:24 +02:00
fb56428494 Add per-user gallery favourites with heart toggle
Adds a favourites table, API endpoints (list/add/remove), and UI
integration: heart icons on gallery cards (home page) and in the
gallery viewer header, with a favourites section on the home page.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-27 12:48:58 +02:00
1bf3ac1303 feat(deploy): enable rbv-api on index host 2026-03-27 11:50:52 +02:00
b2e85b479e fix(deploy): enable rbv-index@vault.service if disabled 2026-03-27 07:34:48 +02:00
f3cea035fc Add --face-cache to index command and clear cache on --ml-purge
Also add --face-cache to rbv-api and rbv-index@ systemd service files.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-26 10:07:15 +02:00
f60f66d6e2 chore: service tweaks 2026-03-26 08:39:03 +02:00
0fcb94e0b8 Replace remote ML API with local ONNX inference, add person image counts and gallery persons
- Remove MlClient HTTP client and reqwest dependency; all inference now runs
  locally via rbv-infer (CLIP visual+textual, SCRFD, ArcFace)
- Make --model-dir required, remove --ml-uri from CLI and API server
- Add denormalized image_count column to persons table with trigger to keep
  it updated; order /people route by image count descending
- Add GET /galleries/:id/persons endpoint and display person links on gallery page
- Fix face crop endpoint to resize source image to match detection coordinates
- Add --ml-purge flag to index command for wiping all ML data before re-index
- Add --face-score-thresh CLI arg (default 0.7, was hardcoded 0.5) to tune
  face detection sensitivity
- Lower default cluster threshold from 0.65 to 0.55 for more aggressive grouping
- Include gallery ID in pipeline log output and image path in error messages
- Update docs and systemd service files

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-26 08:38:24 +02:00
409c8fd86b chore: deployment semantics 2026-03-26 06:18:40 +02:00
f5fa5f4f6b Batch cluster DB writes: ~268k round trips → two bulk queries
Generate all PersonIds upfront, then bulk-insert all persons with a
chunked QueryBuilder INSERT and bulk-update all face assignments with a
chunked QueryBuilder UPDATE FROM VALUES. Reduces the 40-minute write
phase to seconds. Also fixes NaiveDateTime/TIMESTAMPTZ decode in person.rs.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-23 16:46:49 +02:00
23c3836510 Add VectorChord installation to db-cluster.md
Manual install from GitHub release zip using exact paths confirmed from
zip contents (sharedir/extension/* → /usr/pgsql-17/share/extension/,
pkglibdir/vchord.so → /usr/pgsql-17/lib/). Adds shared_preload_libraries
and CREATE EXTENSION for both rbv and immich databases.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-23 15:21:20 +02:00
166e682880 Use setfacl for postgres cert access instead of copying files
Leaves certs root-owned; ACL grants postgres read access so renewals
take effect without re-copying. Use absolute paths in postgresql.conf.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-23 14:25:03 +02:00
cc0a59995a Update db-cluster.md: cert auth throughout, no passwords
Replace scram-sha-256 with mTLS cert auth using the step CA convention
already in use on all infra hosts. Documents cert installation, pg_hba
hostssl/cert rules, pg_ident CN mapping, passwordless role creation,
and cert-auth connection strings for rbv and immich.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-23 14:22:30 +02:00
2c3192fea9 Add doc/db-cluster.md: standalone setup for frankie + Patroni HA plan
Documents Phase 1 (PG17 + pgvector standalone on frankie.hanzalova.internal)
and outlines Phase 2 (Patroni three-node cluster) for future reference.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-23 14:13:45 +02:00
caf60c0ff0 Fix cluster DB pool size to 2 (sequential access, not concurrency-bound)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-23 13:03:08 +02:00
81a9044870 Add --concurrency to cluster command, parallelise dot products with rayon
Mirrors the index command convention: defaults to 4, tunable per host in
the service unit. The inner similarity loop is parallelised via a rayon
thread pool of the configured size; union-find remains sequential.
rayon added to workspace and rbv-cluster dependencies.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-23 12:58:09 +02:00
7f273fdfe3 Add progress logging to cluster_faces every 1% of the outer loop
Emits pct/unions/eta_secs at INFO so large runs show meaningful progress
rather than silence. Also makes union() return bool so merges are counted.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-23 12:50:56 +02:00
beeef9dc33 Add rbv-cluster.service and trigger it OnSuccess of index
Each time rbv-index@.service completes successfully, systemd fires
rbv-cluster.service (a oneshot) to assign unprocessed faces to persons.
Keeping them as separate units gives each its own status and journal,
and allows cluster to be triggered independently. deploy_index now
deploys and password-substitutes both unit files.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-23 12:24:15 +02:00
8e28028b5f Auto-center active thumbnail on image navigation
Smoothly scroll the thumb strip to center the active thumbnail when
navigating between images. Remove unused variables that broke the build.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-23 12:12:39 +02:00
7ba0f038c7 Add pointer-aware zoom and drag-to-pan for gallery image viewer
Zoom now recenters around the mouse pointer position instead of the
image center. When zoomed in, click-drag pans the image. Cursor
changes to grab/grabbing when zoomed. Reset clears both zoom and pan.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-23 11:59:11 +02:00
8de6f0c78c Fix thumb strip scroll by deferring wheel listener until gallery loads
The wheel event listener was attached on mount when the thumb strip
hadn't rendered yet (early return before gallery data). Re-run the
effect when gallery loads so the ref is available.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-23 11:54:05 +02:00
ee60e5d480 Refactor deploy script into per-component functions with arg selection
Split deploy.sh into deploy_db, deploy_index, deploy_api, deploy_ui
functions. Components are selected via script args (e.g. "db index"),
"all" deploys everything, and no args defaults to index+api+ui.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-23 11:34:39 +02:00
716f51afad chore: decouple index and api 2026-03-23 11:32:40 +02:00
10e360a8d1 Add mouse wheel zoom to gallery image viewer
Scroll to zoom in/out (0.5x–5x) on the main gallery image, with a
reset button that appears when zoomed. Zoom resets automatically on
image navigation.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-23 11:19:49 +02:00
1b3473c100 chore: deployment semantics 2026-03-23 11:07:54 +02:00
539e23a035 Shuffle galleries by default so unprocessed ones are reached sooner
A sorted root-level index always replays already-processed galleries
before reaching new ones, delaying their availability in the UI.
Shuffling (the new default) distributes the probability evenly so new
galleries surface early in any run. Pass --sort-galleries to restore
deterministic ordering.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-23 09:17:23 +02:00
32e04e3ecf Size DB connection pool to concurrency to eliminate acquire contention
With --concurrency 24 the hardcoded pool of 10 meant 14 workers always
queued for a connection, causing >2s acquire warnings. Pool size is now
derived from concurrency (concurrency + 4 spare) for the index command
so it always matches the workload. Other commands use fixed small values
appropriate to their access patterns.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-23 07:28:59 +02:00
a96affe510 Update --concurrency doc comment for local ONNX inference
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-23 07:23:29 +02:00
7e2ce30c1f Move pre-processing log to trace, emit per-gallery summary at info
'Processing gallery: ...' was a generic pre-action statement repeated
thousands of times at INFO. Replace with a post-completion summary line
carrying actual data: images processed/skipped, faces detected, errors.
Falls back to the filesystem path when source_name is empty.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-23 06:30:30 +02:00
ae3eecbf11 Default all index.json fields so missing values skip images, not galleries
id defaults to 0, collection/source/source.name/source.url default to ""
so a gallery with an incomplete or partially valid index.json still has its
images ingested rather than being skipped entirely.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-23 05:46:24 +02:00