Skip to content

About

A backtesting engine built around not fooling yourself: costs, walk-forward validation, overfitting diagnostics

Topics

Resources

Stars

0 stars

Watchers

0 watching

Forks

Latest commit

 

History

3 Commits

Folders and files

Repository files navigation

backtest-engine

A backtesting engine built around not fooling yourself.

Most backtesting libraries help you produce a result. The hard part is deciding whether the result means anything. This one is organised around the three things that make backtests lie — look-ahead bias, ignored costs, and selection across many attempts — and refuses to let you skip them.

CI Python License


The problem

# The most common defect in published backtests
pnl = signal * returns          # signal[t] computed from bar t's close

That one line makes almost any signal profitable. signal[t] uses bar t's close, so it cannot be traded until t+1. This engine raises LookAheadError if you configure lag=0, and the test suite quantifies the bias:

Same perfect signal sign(return[t]) Sharpe
Executed at bar t (look-ahead) > 15
Executed at bar t+1 (honest) < 3, and negative after costs

Both numbers come from tests/test_engine_and_metrics.py.


What it enforces

1. Execution lag. lag >= 1 is required, not a default you can quietly change.

2. Costs on turnover. Charged when exposure changes, not per bar — a position held for 100 bars pays once. Presets for retail FX, institutional FX, Vietnam equity and crypto spot, plus a session-aware spread multiplier (rollover spreads run ~4x the London midpoint, and a flat assumption flatters anything trading thin hours).

3. Forward-only validation. Plain k-fold on a time series trains on the future. Provided instead: anchored walk-forward, rolling walk-forward, purged k-fold with embargo, and combinatorial purged CV.

4. Trial-aware significance. Try 70 variations on noise and the best will show a Sharpe near 2. The deflated Sharpe ratio takes the number of attempts as an argument and tells you what that best-of-70 is actually worth.


Install

pip install git+https://github.com/PTHAICAP/backtest-engine.git

Single dependency: numpy.


Usage

import numpy as np
from backtest_engine import run_backtest, RETAIL_FX, deflated_sharpe_ratio

result = run_backtest(signal, asset_returns, costs=RETAIL_FX, lag=1)

print(result.stats())
# {'sharpe': 0.84, 'gross_sharpe': 1.71, 'cost_drag_sharpe': 0.87,
#  'max_drawdown': -0.183, 'calmar': 0.42, 'ulcer_index': 0.061,
#  'total_turnover': 412.0, ...}

cost_drag_sharpe is reported on every run because it is the number that decides whether a strategy is real. A gross Sharpe of 1.71 falling to 0.84 net means costs are eating half the edge.

Walk-forward with leakage control:

from backtest_engine import purged_kfold, check_no_leakage

splits = purged_kfold(len(returns), n_folds=6, purge=10, embargo=5)
check_no_leakage(splits, purge=5)      # raises if any fold leaks

for split in splits:
    model.fit(features[split.train], labels[split.train])
    oos.append(model.score(features[split.test]))

Then ask whether the result survives the search that produced it:

from backtest_engine import (
    deflated_sharpe_ratio, probability_of_backtest_overfitting,
    permutation_test, block_bootstrap_sharpe_ci,
)

deflated_sharpe_ratio(result.returns, n_trials=70)
# {'sharpe': 1.42, 'deflated_threshold': 0.98, 'psr': 0.71, 'passes': False}

block_bootstrap_sharpe_ci(result.returns, n_boot=1000)
# {'sharpe': 1.42, 'ci_lower': -0.11, 'ci_upper': 2.88, 'excludes_zero': False}

A lower bound below zero means the strategy is not distinguishable from no edge, whatever the point estimate says.


The diagnostics, and what each one cannot do

Tool Answers Blind to
deflated_sharpe_ratio Is this Sharpe real given n_trials attempts? Costs, regime change
probability_of_backtest_overfitting Does in-sample ranking predict out-of-sample? Needs many variants
permutation_test Does the timing carry information? Selection bias
block_bootstrap_sharpe_ci How wide is the uncertainty? Selection bias

That last column matters and is asserted in the test suite. The headline test mines 70 random signals on pure noise, keeps the in-sample winner, and checks every diagnostic:

  • the deflated Sharpe rejects it — it is told 70 attempts were made;
  • the permutation test passes it — it only asks whether the series beats its own shuffles, which the winner does by construction, because it was selected for being best on exactly that sample.

Neither the permutation test nor the bootstrap knows a search happened, so neither can correct for one. Running only those two on a mined strategy gives a green light on noise. That is the single most useful thing in this repository, and it is a passing test rather than a paragraph of advice: test_diagnostics_reject_a_strategy_selected_from_noise.


Metrics

Sharpe, Sortino, Calmar, max drawdown, Ulcer index, time underwater, profit factor, tail ratio, win rate. Annualisation is always explicit — silent annualisation is an easy way to publish a Sharpe wrong by a factor of sqrt(24).

Two details that are bugs in a surprising number of implementations, both with a regression test here:

  • Drawdown includes starting capital as the first peak. Without it, a series that falls from the very first bar reports a maximum drawdown of zero — the metric fails silently on exactly the strategies it should flag.
  • Sortino divides downside deviation by the full sample, not by the count of losing periods. Dividing by the loss count inflates the ratio for strategies that lose rarely but severely, which is the profile that blows up.

Testing

pip install -e ".[dev]"
pytest --cov=src/backtest_engine

54 tests, 90% branch coverage. Deterministic — every random draw is seeded, so the numbers in this README are reproducible.

tests/test_engine_and_metrics.py           look-ahead, cost accounting, metric edge cases
tests/test_validation_and_overfitting.py   split validity, leakage, the four diagnostics

Scope

Single-asset, vectorised, position-based. No data loaders, no live execution, no portfolio optimiser. For the execution side see mt5-python-bridge; for live risk enforcement, trading-risk-monitor.

Nothing here is investment advice, and a backtest that passes every diagnostic in this package is still a backtest.

References

  • Bailey & López de Prado (2014), The Deflated Sharpe Ratio
  • Bailey, Borwein, López de Prado & Zhu (2017), The Probability of Backtest Overfitting
  • López de Prado (2018), Advances in Financial Machine Learning — purging, embargo, CPCV
  • Politis & Romano (1994), The Stationary Bootstrap

License

MIT — see LICENSE.

About

A backtesting engine built around not fooling yourself: costs, walk-forward validation, overfitting diagnostics

Topics

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages