Add source_type (spot/futures_um/futures_cm) throughout the ingestion
pipeline so the worker fetches from the correct S3 path prefix:
data/spot/, data/futures/um/, or data/futures/cm/
- Migration: add source_type column to ingest_configs (default 'spot')
- DAL: add source_type to all ingest_config row structs and queries;
add reset_cursor parameter to update() for forced re-ingestion
- Fetcher: parameterise S3 paths via s3_base(source_type); add
source_type param to all four fetch methods
- Worker: replace BINANCE_EPOCH constant with binance_epoch(source_type)
function; fix min→max in start_date calculation (was causing full
retention window to be re-fetched every poll cycle); add source_type
field to every log site
- Parser: skip non-numeric header rows in trade CSV files (Binance
added headers to 2025+ files, breaking parse on Field("price"))
- API handler: accept source_type and reset_cursor in ingest config
create/update endpoints; filter futures exchange pairs to PERPETUAL
contract type only
- Dashboard: add binance_futures_usd option to ingestion form; wire
source_type through to API mutation
- backfill.sh: extend to cover futures_um instruments (BTCUSDC,
ETHUSDC, SOLUSDC); refactor into backfill_group() helper
- docs/api.md: document source_type field and futures support
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The sell-quantity override in rule_strategy.rs was asymmetric: sell
orders always closed the full position, but buy orders had no equivalent
logic for closing a short. This made shorts broken in practice.
Replace 18 lines with 8 lines of symmetric logic:
- Order side opposite to current position → use full position quantity
(sell closes long, buy closes short)
- Otherwise (flat or same-side) → use configured base_quantity
Behavior is identical for existing long-only strategies. Shorts were
already supported by the DSL types and barter engine — only the
executor's quantity logic needed fixing.
Also updates docs/api.md:
- Replaces "Sell order sizing" notes with symmetric "Closing orders" note
- Removes spot-only caveat from the position condition docs
- Adds Example 7: long/short Bollinger mean reversion
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Adds richer per-run metrics to result_summary and a new multi-run
comparison endpoint to support strategy iteration workflows.
Part A — enriched flat fields in result_summary:
- Extended PositionAggregateStats SQL query with avg_win, avg_loss,
max_win, max_loss, avg_hold_duration_secs (single-pass aggregation)
- All three runner.rs paths (live, candle backtest, tick backtest) now
extract sortino_ratio, calmar_ratio, max_drawdown, pnl_return from
the barter TearSheet and include them as flat fields
- Backtest paths additionally include position aggregate stats
Part B — GET /api/v1/paper-runs/compare?ids=<uuid,...>:
- get_by_ids() DAL function using ANY($1) for batch fetch
- RunMetricsSummary response type with all flat metrics
- compare_paper_runs handler: parses comma-separated ids, returns
results ordered to match the request, silently omits missing runs
- Route registered before /{id} to prevent "compare" matching as UUID
Updated docs/api.md: new flat fields table entries, compare endpoint
section with example response, TOC entry.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Two gaps between docs/api.md and implementation:
1. result_summary shape was fabricated — docs showed flat fields
(total_positions, net_pnl, win_rate, etc.) but the actual output
was nested barter TradingSummary with instruments/assets maps.
Fix: add PositionAggregateStats DAL query that computes winning,
losing, total_fees, total_pnl_net from paper_run_positions after
insert, then surface flat fields at the top of the result_summary
JSON alongside the full barter output.
2. SizingMethod (fixed_sum, percent_of_balance, fixed_units) was
implemented but absent from the Quantity Sizing section of the docs.
Fix: add a "Sizing methods" subsection with field tables and examples.
Also update the result_summary docs section to match the real shape.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Accepts a strategy config JSON, runs the full deserialization pipeline,
and returns all errors as a structured list. Always returns HTTP 200
(`valid: false` is a validation result, not an HTTP error).
Two-stage validation:
1. Structural — serde_path_to_error deserialization; returns one error
with dotted field path (e.g. "rules[0].then.quantity") on failure.
2. Semantic — walks the full condition/expression/action tree and
collects all errors simultaneously:
- candle_interval and timeframe values checked against allowed set
- ema_crossover: fast_period < slow_period enforced
- apply_func: ATR/ADX/Supertrend/RSI blocked (require OHLC internals)
- Sizing method parameters validated (positive amounts/percents,
percent_of_balance ≤ 100)
- rules array must be non-empty
Also documents the endpoint in docs/api.md.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Adds a new SizingMethod discriminated union to QuantitySpec so that
scout (and other LLM clients) can specify sizing intent declaratively
instead of constructing expression trees:
{ "method": "fixed_sum", "amount": "500" }
{ "method": "percent_of_balance", "percent": "2", "asset": "usdc" }
{ "method": "fixed_units", "units": "0.01" }
Changes:
- swym-dal: SizingMethod enum (tagged by "method"), Sizing variant added
to QuantitySpec between Fixed and Expr so untagged serde tries it first
- paper-executor: resolve_sizing() computes base-asset quantity from
live price + balances map at candle close; integrated into the existing
quantity match in RuleStrategy::generate_algo_orders
- schema.json: SizingFixedSum / SizingPercentOfBalance / SizingFixedUnits
definitions, Action.quantity.oneOf updated
- docs/strategy-dsl.md: "Position sizing methods" section with examples
Sell orders continue to override quantity to close the full position
regardless of the sizing method specified.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- docs/api.md: add "Backfill strategy" subsection explaining nginx timeout
risk by interval, how to detect a truncated backfill (non-JSON response),
the quarterly-chunking approach with a self-contained shell example, and
a coverage_pct interpretation table for post-backfill verification
- script/backfill.sh: rewrite to iterate in quarterly chunks per
(instrument, interval), accumulate inserted counts, and print ERR on
non-JSON responses so truncated requests are immediately visible
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Rejects backtest submissions where the requested date range has fewer
than 95% of the expected candles, rather than silently queuing a run
against sparse data. The 400 error includes actual vs expected counts,
coverage percentage, and an ingestion status hint derived from the
per-interval candle cursor (caught up / lagging / never ingested).
Also enriches GET /api/v1/market-candles/coverage with expected_count
and coverage_pct fields so callers can pre-check readiness before
submitting a backtest. Documents the full incomplete-data workflow in
docs/api.md.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Covers the full iteration workflow — data preparation, strategy DSL
authoring, backtest submission, and result analysis — with field-level
schemas, validation rules, and six complete strategy examples.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Make it explicit that bars_since_entry is i64/i64 integer division, so
the result is always an exact integer Decimal (0, 1, 2, ...) and >= N
comparisons are never affected by rounding or fractional edge cases.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Exposes live position data to the rule-based DSL and allows quantity to
be a dynamic expression evaluated at candle close.
New Expr variants (all backward-compatible, additive):
entry_price — VWAP average entry price of open position
position_quantity — absolute quantity held (base asset units)
unrealised_pnl — estimated unrealised PnL (quote asset)
bars_since_entry — bars elapsed since position opened
balance { asset } — free balance of a named asset
QuantitySpec: Action.quantity is now Fixed(Decimal) | Expr(Box<Expr>).
All existing configs with string quantities continue to deserialize via
the Fixed variant (serde untagged, Fixed tried first).
EvalCtx gains: entry_price, position_quantity, unrealised_pnl, time_enter,
current_time, balances — populated from the barter PositionManager and
AssetStates in generate_algo_orders.
Enables DSL patterns: 5% stop-loss, 10% take-profit, time exit after N
bars, ATR-based position sizing, percent-of-balance sizing.
Schema and strategy-dsl.md updated with new expression definitions and
common pattern examples.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Add BollingerUpper and BollingerLower as composable FuncName variants,
enabling Bollinger Bands in any expression context (compare, cross_over,
cross_under, apply_func). The multiplier field carries num_std_dev (default
2.0). Chart auto-detects bollinger_upper/lower func nodes and the legacy
bollinger condition, rendering three lines (middle solid, upper/lower dashed).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Adds `timeframe` field to DSL expressions so a single strategy can
reference candles from multiple intervals simultaneously. The primary
`candle_interval` still drives evaluation cadence; additional timeframes
are read-only context evaluated in parallel.
**DSL / DAL (`strategy_config.rs`)**
- `timeframe: Option<String>` on `Expr::Field`, `Expr::Func`,
`Expr::ApplyFunc`, and all five legacy shorthand conditions
- `parse_interval_secs(interval)` helper
- `collect_timeframes(params)` walks the rule tree, returns all
referenced interval strings — used by API validation and executor
**Executor (`paper-executor`)**
- `SwymInstrumentData` now keyed by interval: `candle_histories`,
`ema_states`, `trade_prices` are all `HashMap<u64, …>`
- `next_candle_interval_hint` side-channel routes each incoming
`DataKind::Candle` to the correct per-timeframe history
- `candle_ready` is still gated exclusively on the primary timeframe
- `EvalCtx` carries `primary_interval_secs`; resolver helpers
(`candle_history`, `ema_state`, `trade_prices`) translate an
`Option<String>` timeframe to the correct map entry
- `ema_registrations() -> Vec<(u64, usize)>` replaces the old
single-timeframe `ema_periods()` for pre-warming EMA state
- Backtest runner merge-sorts candles from all required intervals by
`time_exchange` and feeds them in chronological order
**API (`paper_runs.rs`)**
- At backtest creation, calls `collect_timeframes` on rule-based
strategies and validates each additional timeframe: must be a known
interval and must have candle data covering the requested range
**Dashboard**
- `AddStrategyPage`: expanded DSL reference panel — added `event_count`,
`apply_func`, `unary_op`, `bars_since`, all func names (`wma`, `atr`,
`adx`, `supertrend`, `sum`), `multiplier`, and a new Multi-timeframe
section with a worked example
- `AddPaperRunPage`: shows per-additional-timeframe coverage chips and
backfill buttons alongside the primary-interval coverage indicator
**Docs / assets**
- `docs/strategy-dsl.md`: added Multi-timeframe expressions section and
updated all path references
- `docs/strategy.schema.json` → `assets/strategy/schema.json` (new
location; updated `$id`, added `TimeframeInterval` definition, added
`timeframe` property to six schema nodes)
- `assets/strategy/emmanuel-ma-v{1..8}.json` →
`assets/strategy/emmanuel-ma/v{1..8}.json` (grouped into subfolder);
seeder `include_str!` paths updated accordingly
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Closes the gap between 'what the market looks like' (indicator values)
and 'what the market has done' (discrete event history).
## New primitives
**event_count** (Condition): re-evaluates a sub-condition at each of
the last N bars (offsets 1..=period) and counts how many bars it was
true, then applies a comparison operator against an integer threshold.
Primary use case: "has cross_under(close, sma20) fired at least once in
the last 20 bars?" — the 'not-first-touch' guard in pullback strategies.
**bars_since** (Expr): scans back up to N bars and returns how many bars
ago the sub-condition last fired. Returns None (→ false) if the
condition never fired. Use inside compare to express recency constraints:
"the last touch was more than 2 bars ago."
## Implementation
Both primitives work by re-running the condition evaluator at shifted
offsets against the existing candle ring buffer — no additional state
storage required. A new evaluate_at_offset() function propagates offsets
through Compare, CrossOver, CrossUnder, AllOf, AnyOf, Not, and nested
EventCount. Position, EmaCrossover, EmaTrend, Rsi, and Bollinger return
false at non-zero offsets (documented caveat: no historical data).
cross_over/cross_under inside event_count is fully offset-aware: at
offset K it compares bar[-K] vs bar[-(K+1)].
## Coverage
5 new unit tests in signal.rs covering warm-up guard, cross_under
detection in history, zero-count guard, bars_since distance, and
bars_since returning None.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Schema: add optional 'comment' property to all 12 Condition types.
The LLM correctly annotated conditions but the schema had
additionalProperties:false without listing 'comment' — serde accepts it
fine (unknown fields are ignored), but the schema rejected it.
seed_batch: round-trip seed configs through StrategyConfig serde before
computing the hash. Seeds with redundant defaults (e.g. "offset": 0
written explicitly) would otherwise produce a hash that diverges from
what the backend computes at paper-run submission time. This is the same
class of bug fixed previously for Expr field serialization.
Seed: add 'Emmanuel MA Pullback' (v1) — a 20/200 SMA pullback strategy
generated by an LLM from docs/strategy-dsl.md, loaded from
assets/strategy/emmanuel-ma.json via include_str!.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Adds a Validation section covering check-jsonschema CLI, Python jsonschema,
VSCode JSON Schema association, and the recommended LLM workflow. Also
updates the introduction to link to strategy.schema.json.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Covers all Condition and Expr variants (including apply_func, unary_op,
cross_over, cross_under, compare) with additionalProperties: false for
strict validation. Decimal fields are typed as strings. The ApplyFuncName
definition is a restricted subset that excludes atr/adx/supertrend/rsi,
which are not valid inside apply_func.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
