It has been a while since I discussed testing for overfitting in backtests. Since then, Marcos López de Prado and coauthors have done some very thoughtful work (see the bottom), and they even started a blog. Their newest paper builds on discoveries they made in their earlier work, and is an absolute must-read.
Bailey, David H. and Borwein, Jonathan M. and Lopez de Prado, Marcos and Zhu, Qiji Jim
Pseudo-Mathematics and Financial Charlatanism: The Effects of Backtest Overfitting on Out-of-Sample Performance (October 7, 2013)
Available at SSRN: http://ssrn.com/abstract=2308659
Translating scientific papers into code is not always easy, but I spent some time implementing some of the concepts in R, so that I can understand this more fully. Just as a word of encouragement to others out there, I am no math genius nor have any advanced math education, so please don’t be intimidated by formulas. Below you will see a slidify/rCharts discussion demonstrating these first steps. I plan to research this much more thoroughly. As always, I blog to interact, so please let me know what you are thinking.
I took a swing at the other part of this story, the probability of backtest overfitting, performance degradation, and stochastic dominance. R code at https://github.com/mrbcuda/pbo
ReplyDeleteoh, very nice, I can't wait to play with it. Hopefully, I can understand enough to write a follow up post. Thanks so much for sharing.
ReplyDeletehey, I saw that you blogged about the TTRTests package - did the Hansen SPA or the White Reality Check help you out at all? I've also have been looking at ways to avoid data snooping bias with technical trading models. another paper I came across is this one from Hsu, wondering if you've seen this and had any experience with it: http://papers.ssrn.com/sol3/papers.cfm?abstract_id=1087044
ReplyDelete