Every tool was treated the same way: real data, out-of-sample evaluation, costs applied, parameter grids reported as distributions, and a pre-registered pass bar matched to the tool's role. This page documents each stage and the discipline that keeps the conclusions honest.
src/signals/<element>/, implementing exactly the catalogued formula. One runner per tool in scripts/run_<tool>.py. Twenty methodology patterns accumulated across the elements (simple directional, continuous/IC, stateful entry/exit, regime-filter classifier, cross-sectional, risk-input overlay, forecast-error input, comparative-hypothesis, etc.) — new tools reuse the closest pattern rather than inventing a new engine.runs/<tool>.json (the full audit trail) and prints a summary. The user reviews before anything is written back to Notion — no batch auto-writing.The engine (src/backtest/engine.py) is deliberately small. A position decided at timestamp t is shifted one bar internally, so a bar's return only ever uses information known before that bar started. Costs are charged on position turnover (round-trip), and perp funding is applied from the real funding-rate history when available.
# src/backtest/engine.py (core)
def run_backtest(close, position, cost_fn, funding_rate=None):
ret = close.pct_change()
pos_lagged = position.reindex(close.index).ffill().fillna(0.0).shift(1).fillna(0.0)
gross_ret = pos_lagged * ret
turnover = pos_lagged.diff().abs()
cost = cost_fn(turnover)
net_ret = gross_ret - cost
if funding_rate is not None:
net_ret = net_ret - bybit_funding_cost(pos_lagged, funding_rate)
return BacktestResult(gross_ret, net_ret, turnover, pos_lagged)Folds are expanding-window: each fold trains on everything up to a point and tests on the next window, then the train window grows. Defaults are 365 train / 90 test / 90 step bars in the series' own frequency (daily for perp 1d, scaled ×24 for hourly). A typical alpha-signal tool therefore runs 16–27 parameter cells × 22 OOS folds, and the reported metric is the median across all cell-fold combinations.
# src/backtest/walkforward.py — expanding windows
def expanding_walkforward(index, train_bars, test_bars, step_bars):
folds = []
train_end = train_bars
while train_end + test_bars <= len(index):
folds.append(Fold(train_idx=index[:train_end],
test_idx=index[train_end: train_end + test_bars]))
train_end += step_bars
return foldsEvery tool's own "Parameters to Sweep" grid is a Cartesian product swept through every fold. run_sweep returns the full per-cell, per-fold metric distribution; verdicts use the grid median. The single best cell is stored as best_cell_*_reference_only — visible in the run JSONs, never decisive. This is the project's main defense against data-mining: a grid of 24 MA crossovers will always contain one that backtests beautifully; the median across all of them will not.
# metrics reported per OOS fold (scripts/run_moving_average_crossover.py)
def fold_metric_fn(net_returns, position, close):
return {
"sharpe": ev.sharpe_ratio(net_returns),
"ic": ev.information_coefficient(position.shift(1), close.pct_change()),
"hit_rate": ev.hit_rate(position, net_returns),
"max_drawdown": ev.max_drawdown(net_returns),
"turnover": float(position.diff().abs().sum()),
}| Metric | Definition | Used for |
|---|---|---|
| IC (Information Coefficient) | Spearman rank correlation between lagged signal and forward return | Alpha signals — is the ranking predictive at all? |
| Hit rate | Fraction of active (nonzero-position) bars with position × return > 0 | Secondary alpha metric |
| Sharpe ratio | Mean / std × √periods-per-year, annualized from the bar frequency | Primary metric for most alpha tools (bar > 0.3 net) |
| Sortino ratio | Mean / downside std (negative returns only) | Risk-input & allocation comparisons |
| Max drawdown | Peak-to-trough equity decline | Risk-input bar (must be reduced vs. baseline) |
| Classification accuracy | Fraction of correctly labelled regime bars vs. a naive baseline | Regime filters |
| Cost prediction error (bps) | Mean absolute error of a cost/impact forecast vs. realized proxy | Execution-input tools, vs. a flat-bps baseline |
| Turnover-adjusted return | Total return / total turnover | Comparative-hypothesis tools (e.g. holding period) |
Pooled out-of-sample returns are tested with a 2,000-resample bootstrap against H0: mean ≤ 0. Because ~17 tools carry a p-value (alpha-role tools), the Benjamini-Hochberg FDR correction (α=0.05) is applied across the set. Running 45 tools guarantees some will look significant by chance; BH is the explicit accounting for that. (Result: the correction changed zero verdicts — see Findings.)
A trend signal and a stop-loss overlay answer different questions, so they get different bars (src/evaluation/pass_bar.py).
| Role | Pass bar (all criteria must hold) |
|---|---|
| Alpha signal | OOS median IC > 0 AND net Sharpe > 0.3 AND significant after multiple-testing correction AND positive in ≥2/3 regimes |
| Regime filter | Classification accuracy > naive baseline AND (gated Sharpe > ungated OR gated drawdown > ungated) |
| Risk input | Max drawdown reduced vs. baseline sizing AND Sortino ratio ≥ 0.9× baseline |
| Execution input | Cost-prediction error < flat-bps baseline error |
| Portfolio construction | Blended (e.g. risk-parity / MVO) Sharpe > equal-weight Sharpe |
(position != 0).mean(), which is now routine for every risk-input tool.One script per tool → one JSON per tool in runs/; 9 test modules (85 tests) cover the engine, costs, metrics, pass bars, significance, walk-forward, sweep, and schema. The data cache is regenerable (though re-pulling BTC 1h history takes ~10 min against Bybit rate limits — keep data/cache/ when you can). The full workflow for adding a new tool is on the Repo page.