# Plan: Decouple Candle Retention from Trade Retention

## Context

Currently `ingest_configs` has a single `retention_days` column governing both trade and candle purging/backfill windows. Since DSL strategies only use candles and candles are ~500x smaller than trades, the user wants to keep trades at e.g. 365 days while retaining candles indefinitely (or at a separate, longer window). The UI also doesn't surface this distinction — it shows one "Retention" field.

Two additional robustness issues exist in the current ingester that must be fixed for unlimited retention to work:

1. **No-data-available stall**: When a daily file is missing (instrument didn't exist yet), the ingester `break`s and never advances the cursor. With unlimited retention starting from 2017-01-01, an instrument that launched in 2023 would stall permanently on the first missing date.

2. **Retention expansion ignored**: The start date is `cursor + 1` when a cursor exists, with `retention_start` only as a fallback for no cursor. Changing retention from 365→730 days has no effect — the ingester never goes back to fill the expanded window.

## Design

### New column
Add nullable `candle_retention_days INTEGER` to `ingest_configs`. NULL = unlimited (no purge, backfill all available). `retention_days` becomes trade-only.

### Fix: Skip missing dates instead of stalling
When a daily file returns 404, advance the cursor past that date instead of breaking. Use a consecutive-miss counter — after N consecutive misses (e.g. 30 days), assume we've reached the instrument's earliest available data and stop. Log a summary.

This applies to both `process_config()` (trades) and `process_candles()` (candles). The logic:
```
cursor = start_date
consecutive_misses = 0
while cursor <= yesterday:
    if fetch(cursor) → None:
        consecutive_misses += 1
        if consecutive_misses >= MAX_CONSECUTIVE_MISSES:
            // No data exists this far back — advance cursor to here and stop
            update_cursor(cursor)
            break
        cursor += 1 day
        continue
    consecutive_misses = 0
    insert(data)
    update_cursor(cursor)
    cursor += 1 day
```

The monthly path already falls through to daily on a miss, so no change needed there. The key change is that daily misses skip forward instead of breaking.

### Fix: Backfill when retention window expands
Change start date logic from `cursor.unwrap_or(retention_start)` to `min(cursor + 1, retention_start)`:
```rust
let start_date = match last_cursor {
    Some(d) if d + Duration::days(1) > retention_start => retention_start,
    Some(d) => d + Duration::days(1),
    None => retention_start,
};
```
Wait — this is wrong. The cursor tracks the *most recent* successfully ingested date. If we set start_date earlier than the cursor, we'd re-fetch dates we already have (harmless due to ON CONFLICT DO NOTHING, but wasteful).

Better approach: track `earliest_ingested_date` separately. But that adds complexity.

Simpler approach: When the retention window expands past the cursor's known range, we need to backfill from `retention_start` to wherever data already exists. Since we use `ON CONFLICT DO NOTHING` on inserts, re-fetching already-ingested dates is safe (just a bit wasteful for the overlap period). The real issue is that the cursor must not block backfill.

**Solution**: Use two cursors — `last_ingested_date` (forward, existing) and `earliest_ingested_date` (backward, new). On each poll:
- Forward pass: continue from `last_ingested_date + 1` → yesterday (existing behavior)
- Backward pass: if `retention_start < earliest_ingested_date`, walk backward from `earliest_ingested_date - 1` → `retention_start`, skipping consecutive misses

Actually, this is over-engineered. The simplest correct approach:

**Solution**: Compare `retention_start` with the cursor. If `retention_start` is earlier than the cursor, set `start_date = retention_start` and rely on `ON CONFLICT DO NOTHING` for the overlap. The consecutive-miss logic handles the "no data exists" case. This only happens once when retention is expanded — subsequent polls will have `retention_start <= cursor` again and start from `cursor + 1` as normal.

```rust
let start_date = match last_cursor {
    Some(cursor) => {
        let next = cursor + Duration::days(1);
        // If retention window expanded past our earliest data, backfill from there.
        // ON CONFLICT DO NOTHING handles re-fetching existing dates.
        std::cmp::min(next, retention_start)
    }
    None => retention_start,
};
```

This is clean: one-time cost of re-checking dates during the overlap window (which the consecutive-miss counter handles efficiently), and self-correcting — once the backward gap is filled (or misses are exhausted), the cursor is at `yesterday` and subsequent polls are fast.

## Steps

### 1. Migration
**New file:** `crates/swym-dal/migrations/YYYYMMDD000001_candle_retention.sql`
```sql
ALTER TABLE ingest_configs ADD COLUMN candle_retention_days INTEGER;
```
All existing rows get NULL = unlimited. Backward-compatible.

### 2. DAL models
**File:** `crates/swym-dal/src/models/ingest_config.rs`
- Add `pub candle_retention_days: Option<i32>` to `IngestConfigRow`, `EnrichedIngestConfigRow`, `ActiveIngestConfigRow`

### 3. DAL repo queries
**File:** `crates/swym-dal/src/repo/ingest_config.rs`

- **`list_all()`**: Add `ic.candle_retention_days` to SELECT
- **`create()`**: Add param `candle_retention_days: Option<i32>`, add to INSERT columns/values/ON CONFLICT SET/RETURNING
- **`update()`**: Add two params: `candle_retention_days: Option<i32>` and `candle_retention_unlimited: Option<bool>`. Use CASE expression:
  ```sql
  candle_retention_days = CASE
      WHEN $6 = true THEN NULL
      WHEN $5 IS NOT NULL THEN $5
      ELSE candle_retention_days
  END
  ```
  This handles three states: don't change (both None), set to value (Some(n), None), set to unlimited (None, Some(true)).
- **`list_active()`**: Add `ic.candle_retention_days` to SELECT
- All RETURNING clauses: add `candle_retention_days`

### 4. API handlers
**File:** `services/api/src/handlers/ingest_configs.rs`
- `CreateIngestConfigRequest`: add `candle_retention_days: Option<i32>`
- `UpdateIngestConfigRequest`: add `candle_retention_days: Option<i32>` + `candle_retention_unlimited: Option<bool>`
- `IngestConfigResponse`: add `candle_retention_days: Option<i32>`
- `EnrichedIngestConfigResponse`: add `candle_retention_days: Option<i32>`
- Both `From` impls: map the field
- `create_ingest_config()`: pass `req.candle_retention_days` to repo
- `update_ingest_config()`: pass both new fields, include in the nothing-to-update check

### 5. Ingest worker — separate retention + robustness fixes
**File:** `services/ingest-binance/src/worker.rs`

**5a. Add constant:**
```rust
const MAX_CONSECUTIVE_MISSES: u32 = 30;
```

**5b. `process_config()` — trade ingestion loop:**
- Change `start_date` to use `min(cursor + 1, retention_start)` for retention expansion support
- Replace the daily-miss `break` with skip-and-count logic:
  ```rust
  } else {
      consecutive_misses += 1;
      if consecutive_misses >= MAX_CONSECUTIVE_MISSES {
          info!(symbol, cursor, "no data found for 30 consecutive days, stopping");
          ingest_config::update_last_ingested_date(pool, cfg.id, cursor).await.ok();
          break;
      }
  }
  cursor += Duration::days(1);
  ```
- Monthly miss already falls through to daily — no change needed there

**5c. `process_config()` — candle loop:** Replace shared `cutoff` with candle-specific cutoff:
```rust
if let Some(candle_ret) = cfg.candle_retention_days {
    let candle_cutoff = Utc::now() - Duration::days(candle_ret as i64);
    // purge_old_candles with candle_cutoff
}
// else: unlimited, skip candle purge entirely
```

**5d. `process_candles()` — candle ingestion:**
- Change `retention_start` to use `cfg.candle_retention_days`:
  ```rust
  let retention_start = match cfg.candle_retention_days {
      Some(days) => today - Duration::days(days as i64),
      None => NaiveDate::from_ymd_opt(2017, 1, 1).unwrap(),
  };
  ```
- Change `start_date` to use `min(cursor + 1, retention_start)` for retention expansion
- Replace daily-miss `break` with same skip-and-count logic as trades
- Advance the candle cursor past consecutive misses so we don't re-check them on next poll

### 6. Dashboard types
**File:** `dashboard/src/types/api.ts`
- `IngestConfig`: add `candle_retention_days: number | null`
- `CreateIngestConfigRequest`: add `candle_retention_days?: number | null`
- `UpdateIngestConfigRequest`: add `candle_retention_days?: number | null` + `candle_retention_unlimited?: boolean`

### 7. Dashboard list page
**File:** `dashboard/src/pages/IngestionPage.tsx`
- Rename "Retention" column header to "Trade Ret."
- Add "Candle Ret." column with:
  - Toggle switch for unlimited (sends `candle_retention_unlimited: true`)
  - Number input when not unlimited (inline-editable like trade retention)
  - Italic "unlimited" text when NULL

### 8. Dashboard add page
**File:** `dashboard/src/pages/AddIngestionPage.tsx`
- Rename "Retention (days)" label to "Trade Retention (days)"
- Add "Candle Retention" form group with unlimited switch (default: on) + number input when off
- Include `candle_retention_days` in submit payload (null when unlimited)

## Verification
1. `cargo check` — compile-time SQL validation against migrated schema
2. `pnpm tsc --noEmit` — dashboard type-check
3. Deploy to dev (`./script/deploy.sh dev`), verify:
   - Existing configs show NULL/unlimited candle retention
   - Can set a specific candle retention (e.g. 730 days)
   - Can toggle back to unlimited
   - Worker logs: "no data found for 30 consecutive days, stopping" for instruments that don't go back that far
   - Worker logs show separate purge behavior (trade purge runs, candle purge skipped when unlimited)
   - After expanding retention from 365→730 on an existing config, worker backtracks to fill the gap
   - New config creation with unlimited candle retention works
