ForgeXBacktesting
In sample and out of sample testing, explained simply
Splitting the data is the cheapest way to catch a strategy that only works on the history it was built from. Here is the method and how to read the outcome.
If you build a strategy by looking at some data, then test it on that same data, you have not tested anything. You have checked your own homework. Splitting the data fixes that, and it takes about five minutes.
What the two terms mean
In sample is the data you used while you were building. You looked at it, you adjusted your rules against it, you decided your settings on it.
Out of sample is data you set aside at the start and did not look at. You only test on it once, at the end, with the rules already fixed.
The gap between the two results is the useful part. It tells you how much of your performance was the strategy and how much was you fitting it to what you had already seen.
How to split it
Split by time, not at random. Take the earliest 70 percent to build on and keep the most recent 30 percent sealed.
Time order matters because markets change. A random split lets you build using next month and test on last month, which is not something you can do in real life.
New Trading describe the same idea, recommending testing on "fresh data that wasn't used in the initial backtest", and going further with "walk-forward testing" across "multiple rolling time periods" [1].
Reading the result
If out of sample looks close to in sample, that is the good outcome. The edge held up on trades it was never fitted to.
If out of sample is a bit worse, that is normal and expected. Some drop is almost always there.
If out of sample falls apart completely, you found a pattern in the noise. That is a genuinely useful thing to learn, and it cost you nothing but time.
Walk forward, if you want to go further
Instead of one split, you roll it. Build on months one to six, test on month seven. Move forward one month and repeat. Do that across the whole record.
You end up with many out of sample results instead of one, which is far harder to fool yourself with. It is more work, and it is the strongest version of this check that a discretionary trader can realistically run.
Sources
- New Trading, "How To Backtest A Trading Strategy: A Step-by-Step Guide". https://www.newtrading.io/backtesting-guide/