A backtesting engine built around not fooling yourself.
Most backtesting libraries help you produce a result. The hard part is deciding whether the result means anything. This one is organised around the three things that make backtests lie — look-ahead bias, ignored costs, and selection across many attempts — and refuses to let you skip them.
# The most common defect in published backtests
pnl = signal * returns # signal[t] computed from bar t's closeThat one line makes almost any signal profitable. signal[t] uses bar t's
close, so it cannot be traded until t+1. This engine raises LookAheadError
if you configure lag=0, and the test suite quantifies the bias:
Same perfect signal sign(return[t]) |
Sharpe |
|---|---|
Executed at bar t (look-ahead) |
> 15 |
Executed at bar t+1 (honest) |
< 3, and negative after costs |
Both numbers come from tests/test_engine_and_metrics.py.
1. Execution lag. lag >= 1 is required, not a default you can quietly
change.
2. Costs on turnover. Charged when exposure changes, not per bar — a position held for 100 bars pays once. Presets for retail FX, institutional FX, Vietnam equity and crypto spot, plus a session-aware spread multiplier (rollover spreads run ~4x the London midpoint, and a flat assumption flatters anything trading thin hours).
3. Forward-only validation. Plain k-fold on a time series trains on the future. Provided instead: anchored walk-forward, rolling walk-forward, purged k-fold with embargo, and combinatorial purged CV.
4. Trial-aware significance. Try 70 variations on noise and the best will show a Sharpe near 2. The deflated Sharpe ratio takes the number of attempts as an argument and tells you what that best-of-70 is actually worth.
pip install git+https://github.com/PTHAICAP/backtest-engine.gitSingle dependency: numpy.
import numpy as np
from backtest_engine import run_backtest, RETAIL_FX, deflated_sharpe_ratio
result = run_backtest(signal, asset_returns, costs=RETAIL_FX, lag=1)
print(result.stats())
# {'sharpe': 0.84, 'gross_sharpe': 1.71, 'cost_drag_sharpe': 0.87,
# 'max_drawdown': -0.183, 'calmar': 0.42, 'ulcer_index': 0.061,
# 'total_turnover': 412.0, ...}cost_drag_sharpe is reported on every run because it is the number that
decides whether a strategy is real. A gross Sharpe of 1.71 falling to 0.84
net means costs are eating half the edge.
Walk-forward with leakage control:
from backtest_engine import purged_kfold, check_no_leakage
splits = purged_kfold(len(returns), n_folds=6, purge=10, embargo=5)
check_no_leakage(splits, purge=5) # raises if any fold leaks
for split in splits:
model.fit(features[split.train], labels[split.train])
oos.append(model.score(features[split.test]))Then ask whether the result survives the search that produced it:
from backtest_engine import (
deflated_sharpe_ratio, probability_of_backtest_overfitting,
permutation_test, block_bootstrap_sharpe_ci,
)
deflated_sharpe_ratio(result.returns, n_trials=70)
# {'sharpe': 1.42, 'deflated_threshold': 0.98, 'psr': 0.71, 'passes': False}
block_bootstrap_sharpe_ci(result.returns, n_boot=1000)
# {'sharpe': 1.42, 'ci_lower': -0.11, 'ci_upper': 2.88, 'excludes_zero': False}A lower bound below zero means the strategy is not distinguishable from no edge, whatever the point estimate says.
| Tool | Answers | Blind to |
|---|---|---|
deflated_sharpe_ratio |
Is this Sharpe real given n_trials attempts? |
Costs, regime change |
probability_of_backtest_overfitting |
Does in-sample ranking predict out-of-sample? | Needs many variants |
permutation_test |
Does the timing carry information? | Selection bias |
block_bootstrap_sharpe_ci |
How wide is the uncertainty? | Selection bias |
That last column matters and is asserted in the test suite. The headline test mines 70 random signals on pure noise, keeps the in-sample winner, and checks every diagnostic:
- the deflated Sharpe rejects it — it is told 70 attempts were made;
- the permutation test passes it — it only asks whether the series beats its own shuffles, which the winner does by construction, because it was selected for being best on exactly that sample.
Neither the permutation test nor the bootstrap knows a search happened, so
neither can correct for one. Running only those two on a mined strategy gives
a green light on noise. That is the single most useful thing in this
repository, and it is a passing test rather than a paragraph of advice:
test_diagnostics_reject_a_strategy_selected_from_noise.
Sharpe, Sortino, Calmar, max drawdown, Ulcer index, time underwater, profit
factor, tail ratio, win rate. Annualisation is always explicit — silent
annualisation is an easy way to publish a Sharpe wrong by a factor of sqrt(24).
Two details that are bugs in a surprising number of implementations, both with a regression test here:
- Drawdown includes starting capital as the first peak. Without it, a series that falls from the very first bar reports a maximum drawdown of zero — the metric fails silently on exactly the strategies it should flag.
- Sortino divides downside deviation by the full sample, not by the count of losing periods. Dividing by the loss count inflates the ratio for strategies that lose rarely but severely, which is the profile that blows up.
pip install -e ".[dev]"
pytest --cov=src/backtest_engine54 tests, 90% branch coverage. Deterministic — every random draw is seeded, so the numbers in this README are reproducible.
tests/test_engine_and_metrics.py look-ahead, cost accounting, metric edge cases
tests/test_validation_and_overfitting.py split validity, leakage, the four diagnostics
Single-asset, vectorised, position-based. No data loaders, no live execution, no portfolio optimiser. For the execution side see mt5-python-bridge; for live risk enforcement, trading-risk-monitor.
Nothing here is investment advice, and a backtest that passes every diagnostic in this package is still a backtest.
- Bailey & López de Prado (2014), The Deflated Sharpe Ratio
- Bailey, Borwein, López de Prado & Zhu (2017), The Probability of Backtest Overfitting
- López de Prado (2018), Advances in Financial Machine Learning — purging, embargo, CPCV
- Politis & Romano (1994), The Stationary Bootstrap
MIT — see LICENSE.