The first thing most people see from an ai trading bot is a return figure, and the first thing they do is compare it with other return figures. That order is backwards. A return is the output of data, assumptions, parameter choices and risk settings. Reviewing the output before the inputs means anchoring on the number you are supposed to be testing.
This article sets out a review order that leaves the headline return until last, with a worked example of what tends to happen to it along the way.
Why the return should be read last
Regulators have been blunt about the marketing side. The CFTC warns that AI technology cannot predict the future or sudden market changes, and describes schemes that promoted AI-created algorithms with claims of huge returns (CFTC (external site)). The SEC and FINRA, in a joint alert on automated investment tools, advise users to consider a tool's limitations and key assumptions, and note that a tool may rely on assumptions that are incorrect or that do not react to shifts in the market (Investor.gov (external site)).
Those warnings are aimed at outright fraud and at retail users, but the underlying problem is general. A headline number invites comparison, and comparison invites selection. Experienced reviewers fall into it too, simply by looking at the best-performing configuration first and then searching for reasons to believe it.
The procedure below is ordered so that each step can invalidate everything after it. If step one fails, there is no point in step six.
Start with data and assumptions
Step 1: Establish what data the result rests on
Before anything else, write down:
- Source and version. Which dataset, retrieved when, with which adjustments (splits, dividends, roll rules).
- Period. Start and end dates, and whether they include at least one regime the strategy was not designed for.
- Frequency. Daily, hourly, tick. A strategy that trades intraday but was tested on daily bars is being reviewed on the wrong data.
- Universe. Which instruments, and whether that list was fixed at the start of the period or chosen with hindsight.
- Real or synthetic. Synthetic data is useful for checking mechanics and assumptions; it says nothing about whether an edge exists in real markets.
If any of these cannot be answered, record it as an open item. Do not proceed to interpretation with unknown data.
Step 2: List the assumptions explicitly
Turn the implicit into a list. At minimum: fill price rule, cost model in bps per side, position sizing rule, treatment of gaps and missing data, how signals are timed relative to fills, and how parameters were chosen. The companion article on hidden backtest assumptions has a full checklist.
The most important line is the one most often missing: how many configurations were tried before this one was presented. Bailey, Borwein, López de Prado and Zhu propose a method for estimating the probability that a backtest is overfit, precisely because the selection process behind a result is usually invisible (Journal of Computational Finance (external site)). If the bot or its vendor cannot say how its final parameters were selected, treat the result as the best of an unknown number of trials.
Step 3: Test robustness before you trust the shape
Robustness checks ask whether the result depends on a narrow set of choices:
- Parameter neighbourhood. Shift each key parameter by 10–25% in both directions. Results should degrade gradually, not collapse.
- Cost stress. Double the cost assumption. Note whether the conclusion changes.
- Sub-periods. Split the history into halves or thirds. A result carried entirely by one stretch is a result about that stretch.
- Regime contrast. Compare a calm and a volatile scenario. A strategy that only works in one should be described that way.
Harvey and Liu argue that the common practice of halving backtested Sharpe ratios is only a rule of thumb, and that a proper multiple-testing haircut penalises marginal results far more than strong ones (Journal of Portfolio Management (external site)). Robustness checks are the practical counterpart: they show whether a result is marginal.
Read risk and drawdown behaviour together
Step 4: Read the risk, not just the return
Before looking at the return, look at:
- Maximum drawdown and how long it lasted.
- Largest single-day loss as a share of equity.
- Exposure. Maximum and average position size as a percentage of capital.
- Trade frequency, including the busiest day.
Then check these against stated risk limits. If the review has no stated limits, write them now, before seeing whether the run breached them. Limits set after looking at the outcome tend to sit just outside it. Our article on designing auditable risk rules covers how to word them.
Step 5: Examine behaviour during drawdown
Aggregate statistics hide sequence. Take the worst drawdown and read it trade by trade:
- Did position size increase while losses were accumulating?
- Did trade frequency spike?
- Did any risk limit trigger, and did the simulation actually stop when it should have?
- How long did recovery take, and was recovery driven by a few trades or many?
A strategy that adds risk into a drawdown can show an attractive full-period return and still be one bad sequence away from a very different outcome.
Step 6: Reproduce, then read the return
Finally, rerun the result from the recorded inputs. Same data version, same parameters, same assumptions. If it does not reproduce exactly, something undocumented is influencing it. Only once the run reproduces should you read the return, and you should read it alongside everything recorded in steps 1–5.
A worked example: a headline number under review
The following figures are illustrative, not market data and not the output of any real product.
A bot reports an annualised return of 31% on a five-year simulation. The review proceeds in order:
| Review step | Finding | Effect on the reported figure |
|---|---|---|
| 1. Data | Universe chosen from current constituents | Flag survivorship; result likely optimistic |
| 2. Assumptions | Costs set to zero; 240 configurations tested | Rerun at 5 bps per side: return falls to 19% |
| 3. Robustness | Lookback ±20% gives 6% to 11% | Chosen parameter sits on a narrow peak |
| 4. Risk | Max drawdown 34%; stated limit was 20% | Limit breached in year three |
| 5. Drawdown behaviour | Position size doubled during the drawdown | Sizing rule needs review |
| 6. Reproducibility | Rerun matches to the cent | Mechanics are sound; interpretation is not |
The return was never wrong as arithmetic. It was the answer to a question nobody should have been asking: what is the best result available under zero costs on a survivor universe? After review, the honest summary is a single-digit return in neighbouring parameters, a breached drawdown limit and an unresolved sizing issue.
Keeping the review itself on record
A review is only useful if someone else can follow it later. In Strategy Review, parameters, the assumptions list, risk-rule checks and scenario results sit in a single report that is saved as a versioned revision, with an audit trail of what changed between versions. Monoqant is a simplified simulation on synthetic daily bars with fills at the close and a fixed cost model. It does not connect to brokers, hold funds or place trades, and it cannot evaluate a bot's live behaviour. It gives the review a fixed structure and a record. Our methodology and risk disclosure set out the limits in full.