Files
moments/crates/moments-entities/src/lib.rs
rob thijssen 815bfa7deb
All checks were successful
deploy / Build api + worker (static musl) (push) Successful in 5m24s
deploy / Deploy moments-worker to frootmig (push) Successful in 17s
deploy / Deploy moments-api to nikola (push) Successful in 24s
deploy / Build prerendered web (push) Successful in 4m36s
deploy / Deploy web to oolon (push) Successful in 23s
fix: reconcile repo visibility instead of trusting the ingest-time flag
`events.public` was decided once, when a row was ingested, from whatever
the forge reported at that moment — and every poller is incremental (the
github events feed caps at 90 days, search at its top-1000 window, the
per-repo scanner at a `since` cursor, the gitea feed at page 1 after the
first run). Nothing ever revisited a repo, so flipping one to private
upstream only relabelled whatever activity happened afterwards: its
history kept serving commit messages, issue titles and the repo name
indefinitely. The reverse flip was equally frozen.

The github and gitea sources now run a reconciliation pass before
ingesting. github reuses its existing repo discovery, which already
re-reads `private`/`isPrivate` for everything reachable, and only spends
a request on repos we're still exposing that discovery didn't return;
gitea has no equivalent bulk endpoint, so it asks per repo, once a day
rather than once a tick. Repos already hidden are skipped — they can't
leak, and staying hidden is the safe direction to err in. A 404 counts
as private (with the user's own token, a repo still in reach answers 200
even when private, so 404 means gone or transferred), while rate limits
and transient errors flip nothing; the pass runs in both directions, so
a spurious hide is undone by the next successful poll.

That needs a repo key the worker can UPDATE against, hence `events.repo`
— a stored generated column, and now the single definition of the
payload -> repo mapping that list_events, list_projects,
activity_summary and language_daily_counts each carried their own copy
of. Consolidating them fixes an attribution gap along the way:
/search/issues items carry neither `repo.name` nor
`repository.full_name`, only `repository_url`, so every issue and PR
backfilled through search resolved to NULL in all four queries. Those
events now attach to their repo, which both makes them reconcilable and
means they show up in /projects and /activity/summary.

Also closes a leak that predates the flip problem: `/v1/languages/repos`
had no visibility gate at all, and repo_languages is populated for every
repo the worker discovers, private ones included. Repo names were on the
wire (and baked into the prerendered HTML via the dehydrated query
cache) regardless of what `events.public` said. Rather than a second
visibility column to keep in sync, the response now derives it — a
repo's languages are exposed exactly when at least one of its events is.

Verified against postgres 16: the generated column extracts every
payload shape the four sources produce (and NULLs a non-github
`repository_url`), the reconciliation UPDATE flips all three github
event shapes for a repo in one statement and is a no-op on re-run, the
gitea host filter excludes other hosts while treating rows predating the
`_host` stamp as local, and /v1/languages/repos drops a repo once its
events go private.

Closes #6
2026-08-15 19:41:33 +03:00

296 lines
8.3 KiB
Rust

use chrono::{DateTime, Utc};
use serde::{Deserialize, Serialize};
#[derive(Debug, Clone, Copy, Serialize, Deserialize, PartialEq, Eq, Hash)]
#[serde(rename_all = "lowercase")]
pub enum Source {
Github,
Gitea,
Hg,
Bugzilla,
Blog,
}
impl Source {
pub const ALL: &'static [Source] = &[
Source::Github,
Source::Gitea,
Source::Hg,
Source::Bugzilla,
Source::Blog,
];
pub fn as_str(&self) -> &'static str {
match self {
Source::Github => "github",
Source::Gitea => "gitea",
Source::Hg => "hg",
Source::Bugzilla => "bugzilla",
Source::Blog => "blog",
}
}
}
impl std::str::FromStr for Source {
type Err = ParseSourceError;
fn from_str(s: &str) -> Result<Self, Self::Err> {
match s {
"github" => Ok(Source::Github),
"gitea" => Ok(Source::Gitea),
"hg" => Ok(Source::Hg),
"bugzilla" => Ok(Source::Bugzilla),
"blog" => Ok(Source::Blog),
other => Err(ParseSourceError(other.to_string())),
}
}
}
#[derive(Debug, thiserror::Error)]
#[error("unknown source: {0}")]
pub struct ParseSourceError(pub String);
/// Raw event as stored. The presentation reshape lives in `moments-core`
/// and runs at API request time.
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct Event {
pub id: String,
pub source: Source,
pub action: String,
pub occurred_at: DateTime<Utc>,
/// True when the upstream marks this event as visible to anyone (e.g.
/// GitHub's top-level `public` flag). The DB stores everything; the API
/// uses this to gate what gets surfaced on the public timeline.
pub public: bool,
pub payload: serde_json::Value,
}
/// Filters accepted by `GET /v1/events`.
#[derive(Debug, Clone, Default)]
pub struct EventQuery {
pub from: Option<DateTime<Utc>>,
pub to: Option<DateTime<Utc>>,
pub sources: Option<Vec<Source>>,
/// Filter to events matching a specific repo, against the `repo` column
/// the DB derives from each payload.
pub repo: Option<String>,
/// When false (default), only `public = true` rows are returned. The API
/// pins this to false today; a future authenticated path can flip it.
pub include_private: bool,
pub limit: u32,
}
/// Per-source rollup returned by `GET /v1/sources`.
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct SourceSummary {
pub source: Source,
pub count: i64,
pub earliest: Option<DateTime<Utc>>,
pub latest: Option<DateTime<Utc>>,
}
/// Per-day event count for the contribution graph.
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct DailyCount {
pub date: chrono::NaiveDate,
pub count: i64,
}
/// Bucketing granularity for `GET /v1/activity/summary`.
#[derive(Debug, Clone, Copy, Serialize, Deserialize, PartialEq, Eq)]
#[serde(rename_all = "lowercase")]
pub enum SummaryBucket {
Day,
Week,
Month,
Year,
}
impl SummaryBucket {
/// Postgres `date_trunc` field name.
pub fn as_str(&self) -> &'static str {
match self {
SummaryBucket::Day => "day",
SummaryBucket::Week => "week",
SummaryBucket::Month => "month",
SummaryBucket::Year => "year",
}
}
}
impl std::str::FromStr for SummaryBucket {
type Err = ParseBucketError;
fn from_str(s: &str) -> Result<Self, Self::Err> {
match s {
"day" => Ok(SummaryBucket::Day),
"week" => Ok(SummaryBucket::Week),
"month" => Ok(SummaryBucket::Month),
"year" => Ok(SummaryBucket::Year),
other => Err(ParseBucketError(other.to_string())),
}
}
}
#[derive(Debug, thiserror::Error)]
#[error("unknown bucket: {0} (expected day, week, month or year)")]
pub struct ParseBucketError(pub String);
/// Per-repo event count within one summary period. `period` is the UTC
/// start date of the bucket (day, ISO week Monday, first of month/year).
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct RepoPeriodCount {
pub period: chrono::NaiveDate,
pub source: Source,
pub repo: String,
pub count: i64,
}
/// Average events per day at a given hour of the day, computed in a
/// caller-supplied IANA timezone. 24 entries (0..=23).
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct HourlyAvg {
pub hour: i32,
pub avg: f64,
}
/// Per-repo activity rollup for the dashboard.
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct ProjectSummary {
pub repo: String,
pub source: Source,
pub host: String,
pub commit_count: i64,
pub issue_count: i64,
pub pr_count: i64,
pub first_activity: Option<DateTime<Utc>>,
pub last_activity: Option<DateTime<Utc>>,
}
/// Per-language daily commit count for the language stream graph.
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct LanguageDailyCount {
pub date: chrono::NaiveDate,
pub language: String,
pub color: Option<String>,
pub commits: i64,
}
/// Current exposure of one repo. Used in both directions by the worker's
/// visibility reconciliation: read back from the store as "is any event for
/// this repo still public", and written as "this is what upstream says now".
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct RepoVisibility {
pub repo: String,
pub public: bool,
}
/// Per-repo language breakdown from the forge.
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct RepoLanguage {
pub source: Source,
pub repo: String,
pub language: String,
pub bytes: i64,
pub color: Option<String>,
}
// ---------------------------------------------------------------------
// Presentation shape — what `GET /v1/events` actually returns.
// The API reshapes raw payloads into these so the frontend stays dumb.
// ---------------------------------------------------------------------
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct TimelineItem {
pub id: String,
pub source: Source,
pub action: String,
pub occurred_at: DateTime<Utc>,
pub icon: TimelineIcon,
/// Primary headline. Mixed plain text + inline links so the UI can
/// render the right anchors without parsing.
pub title: Vec<TitleSegment>,
pub subtitle: Option<Vec<TitleSegment>>,
pub body: Option<TimelineBody>,
}
#[derive(Debug, Clone, Serialize, Deserialize, PartialEq, Eq)]
#[serde(tag = "kind", rename_all = "lowercase")]
pub enum TitleSegment {
Text { text: String },
Link { text: String, url: String },
}
impl TitleSegment {
pub fn text(s: impl Into<String>) -> Self {
Self::Text { text: s.into() }
}
pub fn link(text: impl Into<String>, url: impl Into<String>) -> Self {
Self::Link {
text: text.into(),
url: url.into(),
}
}
}
#[derive(Debug, Clone, Serialize, Deserialize)]
#[serde(tag = "kind", rename_all = "lowercase")]
pub enum TimelineBody {
Markdown { text: String },
Commits { commits: Vec<CommitSummary> },
Links { items: Vec<TitleSegment> },
}
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct CommitSummary {
pub sha: String,
pub short_sha: String,
pub message: String,
pub url: String,
pub author: Option<String>,
}
/// UI icon hint. The frontend maps these to its own icon set; new variants
/// here require a frontend update but never break existing renders (the UI
/// falls back to the generic icon for unknown values).
#[derive(Debug, Clone, Copy, Serialize, Deserialize, PartialEq, Eq)]
#[serde(rename_all = "kebab-case")]
pub enum TimelineIcon {
GitPush,
GitCommit,
GitMerge,
GitFork,
GitBranchCreate,
GitBranchDelete,
PullRequest,
Issue,
Comment,
Star,
Release,
Bug,
Post,
Generic,
}
/// Blog index entry returned by `GET /v1/blog`.
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct BlogPostSummary {
pub slug: String,
pub title: String,
pub published_at: DateTime<Utc>,
pub excerpt: String,
}
/// Full blog post returned by `GET /v1/blog/{slug}`. The host/repo/branch
/// triple lets the UI resolve relative image srcs to forge raw URLs.
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct BlogPost {
pub slug: String,
pub title: String,
pub published_at: DateTime<Utc>,
pub markdown: String,
pub host: String,
pub repo: String,
pub branch: String,
}