ForgeXBacktesting
Overfitting: how to tell a real edge from a curve fit
Overfitting is what happens when you tune a strategy until it fits the past perfectly. Here is what causes it and how to catch it in a result before trusting it.
Overfitting, sometimes called curve fitting, is when a strategy has been adjusted so many times against the same data that it now describes that exact stretch of history and nothing else. It is the single most common way a backtest lies to you.
How it happens without you noticing
Nobody sets out to overfit. It happens one small step at a time, and every step feels reasonable.
You test a moving average. It is fine. You try a different length and it is better, so you keep it. You add a filter for the session, and that helps too. You skip Mondays because Mondays looked bad. Each change improves the result on the data in front of you.
After eight of those you have a strategy with a beautiful equity curve and no future. Every rule was chosen because of something that already happened.
FTMO put the warning plainly, advising caution about "over-optimising by adding too many conditions, which can lead to curve-fitting—a strategy that works well on historical data but poorly in live trading" [1].
The signs
Your strategy has a lot of rules. Every extra condition is another chance to fit noise. A strategy with three rules is far harder to overfit than one with nine.
The settings are oddly specific. A 47 period moving average is not a discovery. It is the one that happened to score best.
Small changes wreck it. New Trading name this test directly. If "tweaking a parameter dramatically changes results, that's a red flag" [2]. A real edge is a broad hill, so nearby settings all work about as well. An overfit one is a narrow spike.
It falls apart out of sample. This is the definitive test and it is why you hold data back.
How to avoid it
Decide your rules before you look, then change as few of them as you can. Every change costs you some of the honesty of the test.
Keep the rule count low. If you cannot explain why a rule should work, in terms of what other people in the market are doing, it is probably fitting noise.
Test the neighbours. If 20 periods works, check 15 and 25. If those collapse, you do not have an edge at 20 either.
Hold data back, and only look once.
The honest way to think about it
Every backtest is fitted to some degree. You cannot avoid it entirely, because you are choosing what to test based on things you have already seen.
The goal is not zero fitting. The goal is knowing how much of your result is fitting, so you size accordingly and are not surprised when live trading comes in below the test.
Sources
- FTMO Academy, "How to Backtest Trading Strategies". https://academy.ftmo.com/lesson/how-to-backtest-trading-strategies/
- New Trading, "How To Backtest A Trading Strategy: A Step-by-Step Guide". https://www.newtrading.io/backtesting-guide/