A backtest is a claim about the past that depends on a set of conditions it rarely states. The equity curve is the visible part. Underneath it sit decisions about what data was known when, which instruments were included, how orders were filled, what trading cost, and how many other versions were tried before this one was shown. Change any of them and the curve changes too, sometimes completely.
This article goes through those assumptions one at a time and ends with a checklist you can apply to any result, whether you produced it yourself, someone handed it to you, or it came from an ai trading bot's performance screen.
Why the assumptions matter more than the curve
Two backtests with the same headline return can mean very different things. One may use data as it was available at the time, include instruments that later disappeared, charge realistic costs, and be the first configuration tested. The other may use revised data, a list of today's survivors, free fills at the closing price, and be the best of several hundred variations. The curves can look identical; only the first carries much evidence.
The useful question is therefore not "how good is the result?" but "under which conditions would this result still hold?" Each assumption below is one such condition.
Look-ahead: using information before it existed
Look-ahead bias occurs when a simulated decision uses information that was not available at the moment the decision was supposedly made. It is rarely deliberate. Common forms:
- Same-bar signals and fills. A signal computed from a day's closing price, then filled at that same close. In live conditions the close is only known once it has happened.
- Revised data. Fundamental or macro series that were later restated, used in their final form rather than as first published.
- Full-sample normalisation. Z-scores, volatility targets or thresholds computed over the entire history, including the future part of the test.
- Index membership as of today. Selecting a universe by current constituents and testing it back in time.
A practical test: for each input, write down the timestamp at which it became known, and check that it precedes the timestamp of the decision. If you cannot write that timestamp, you cannot rule look-ahead out.
Survivorship: testing only the winners
If the universe is built from instruments that exist today, everything that was delisted, merged or went to zero along the way is missing. A strategy tested on that universe is tested on a population that was filtered by its outcome.
The fix is a point-in-time universe: the list of instruments as it stood on each date, including the ones that later disappeared. If such data is not available, the limitation belongs in the assumptions list, stated plainly.
Fills, costs and the shape of an order
Most simple backtests assume an order is filled in full at a known reference price, often the close. Real execution is less cooperative. The SEC's investor guidance on order execution notes that prices can change quickly, especially in fast-moving markets, and that by the time an order reaches the market the price may be slightly or very different from the quote (Investor.gov (external site)).
In a backtest this shows up as three separate assumptions:
- Reference price. Close, next open, a VWAP, or a midpoint. Each gives a different result.
- Fill certainty. Whether the full size is assumed to trade, regardless of volume or gaps.
- Cost model. Commission, spread and slippage, often collapsed into one fixed number per trade.
A fixed per-trade cost is a reasonable simplification for review, as long as it is visible and its sensitivity is tested. The companion article on slippage and fees works through how quickly cost assumptions dominate a high-turnover result.
Data snooping and the number of trials
The least visible assumption is the one about search. A backtest shown in isolation implicitly claims to be a single test. In practice it is usually the survivor of many: different lookbacks, thresholds, instruments, periods and filters.
Bailey, Borwein, López de Prado and Zhu show that high simulated performance is easily achievable after backtesting a relatively small number of alternative configurations, and that because the number of configurations tried is rarely reported, investors cannot evaluate the degree of overfitting in most proposals (Notices of the AMS (external site)). In a companion paper they propose a framework for estimating the probability of backtest overfitting, and note that standard hold-out techniques tend to be unreliable in this setting (Journal of Computational Finance (external site)).
Harvey and Liu reach a related conclusion from the side of significance testing: a Sharpe ratio that looks significant as a single test may not be once multiple testing is accounted for, and the appropriate haircut is nonlinear, with marginal results penalised far more than strong ones (Journal of Portfolio Management (external site)).
A worked example: the best of 100 strategies with no edge
The following numbers are illustrative, not market data.
Suppose you test 100 configurations that, by construction, have no edge at all: their true Sharpe ratio is zero. Over five years of data, the estimated annual Sharpe of each has a standard error of roughly 1 / √5 ≈ 0.45. The expected maximum of 100 independent standard normal draws is about 2.5. So the best configuration out of 100 is expected to show an estimated Sharpe of roughly 2.5 × 0.45 ≈ 1.1, purely from noise.
| Configurations tried | Expected best z-score (approx.) | Expected best Sharpe, 5 years, zero true edge |
|---|---|---|
| 1 | 0.0 | 0.0 |
| 10 | 1.5 | 0.7 |
| 100 | 2.5 | 1.1 |
| 1,000 | 3.2 | 1.4 |
Illustrative calculation under simplifying assumptions (independent trials, normal estimation error). Real trials are correlated, which changes the numbers but not the direction.
A Sharpe of 1.1 would pass many casual screens. Without the trial count, nobody can tell a discovery from the top of a pile of noise.
Parameter selection and in-sample tuning
Closely related to the trial count is how the final parameters were chosen. If a 37-day lookback was picked because it scored best over the same period used to report performance, the reported result is in-sample by definition. Two cheap checks help:
- Neighbourhood stability. Do 30, 35, 40 and 45 days produce broadly similar results, or does performance collapse a few steps away from the chosen value? A sharp peak suggests fitting to noise.
- Selection period versus reporting period. Write down which dates were used to choose parameters and which are used to report results. If they overlap, say so.
A checklist for any backtest you are asked to trust
| Assumption | Question to ask | Evidence that settles it | Red flag |
|---|---|---|---|
| Look-ahead | Was every input known before the decision it fed? | Timestamps per input vs decision time | Signal and fill on the same bar |
| Survivorship | Does the universe include instruments that later disappeared? | Point-in-time constituent lists | Universe defined by today's members |
| Reference price | Which price is each fill assumed at? | Stated fill rule | "Filled at close" with no sensitivity test |
| Fill certainty | Is full size assumed to trade? | Volume or participation limit | No size constraint at all |
| Cost model | What is charged per trade, and is it varied? | Cost per side in bps, with a sensitivity run | Zero or unstated costs |
| Trial count | How many configurations were tested? | A log of every run | Only the final run exists |
| Parameter choice | Were parameters chosen on the reporting period? | Separate selection and reporting dates | Optimised and reported on the same data |
| Stability | Do nearby parameters behave similarly? | A small parameter grid | Performance peak at one value |
| Data source | Which source, which version, which adjustments? | Named dataset and retrieval date | "Historical data" with no provenance |
A result can fail one line and still be informative, as long as the failure is written next to the number.
Making assumptions visible
Before discussing performance, write the assumptions list. In Strategy Review, the simulation records its own assumptions automatically, such as synthetic daily bars, fills at the close and a fixed cost model, and you add the ones specific to your strategy. The tool is a simplified simulation rather than a full backtest, and that limitation is itself the first entry on the list. Our methodology page describes what the simulation does and does not model, and the risk disclosure covers what no simulation can tell you.