1nTest · strategy backtester and trading bot builder

Why the best backtest disappoints: selection, luck and a placebo test

We ranked 54 strategy-and-asset combinations by their 2022–2023 results after costs. The top ten averaged +0.51 R per trade. In 2024–2025 the same ten made −0.05 R. On data with the edge removed, the top ten of the same search still shows +0.43 R in-sample. The lead of the winners was about the size that luck alone produces.

What we tested

Five published strategy templates (fair value gap after a liquidity sweep, order block with break of structure, level raids, range deviation and the Asian session range) on twenty assets: forex majors and crosses, gold, silver, oil, US and European indices and bitcoin. Every combination is an ordinary engine backtest with commission, swap and bid/ask prices.

We split each backtest in two halves: 2022–2023 to choose, 2024–2025 to check. A combination took part only with at least 10 trades in each half, which left 54 of 100.

The result

Top ten by 2022–2023
2022–2023 (selection)+0.51 R
2024–2025 (check)−0.05 R
Top five by 2022–2023
2022–2023 (selection)+0.73 R
2024–2025 (check)+0.21 R
All combinations
2022–2023 (selection)−0.11 R
2024–2025 (check)−0.10 R
Placebo: top ten with no edge
2022–2023 (selection)+0.43 R
2024–2025 (check)—

Average net result per trade, in R (one R is the amount risked on the trade). In the selection half 19 of 54 combinations were profitable after costs; in the check half, 20.

How to read it

  • The best single combination made +1.11 R per trade in 2022–2023. The best of a search with no edge at all makes +0.87 R on average. A result that impressive is what a search of this size produces by itself.
  • Little carried over: 4 of the top ten stayed profitable in 2024–2025, but together they made −0.05 R per trade, and the rank correlation between the halves is only 0.11.
  • Most combinations lose after costs in both halves. A search across strategies and assets mostly finds the few that were lucky in the window you looked at.

What to do instead

  • Count how many variants you tried, and compare the winner with the best of the same number of placebo variants, not with zero.
  • Keep a period you did not use for choosing, and judge the strategy only there.
  • Prefer a rule that works across many assets over a combination that is the best on one.

How we measured

  • Engine backtests on the production server, period 2022-01-01 to 2026-01-01, starting balance $10,000, fixed risk of 1% per trade.
  • Costs: commission and swap measured for one commission-charging broker; spread through bid/ask prices of entry and exit.
  • Net R per trade = (profit − commission − swap) ÷ amount risked. All closed trades count, including those closed by time.
  • Placebo: every combination's 2022–2023 trades are shifted so their average is zero and resampled with replacement; the search is repeated 2,000 times.

Backtests run on October 24, 2026.

Questions

Does this mean backtesting is useless?

No. It means a backtest picked as the best of many will look better than it is. Judge it on data not used to pick it and against a placebo of the same search size.

How many trades are enough?

There is no single number, but the fewer trades and the more variants tried, the larger the lead that luck alone produces. Here combinations needed at least 10 trades per half, and luck still explained the lead.

For education, not investment advice. Past results, including backtests, do not guarantee future results.

← All research