The dangers of data snooping
Data snooping is the generic term of the danger that the best forecasting model found in a given data set by a certain specification search is just the result of chance instead of the result of truly superior forecasting power. Jensen (1967) already argued that the good results of the relative-strength trading rule used by Levy (1967) could be the result of survivorship bias. That is, strategies that performed well in the past get the most attention by researchers. Jensen and Benington (1969, p.470) go a step further and argue: ``Likewise given enough computer time, we are sure that we can find a mechanical trading rule which works on a table of random numbers - provided of course that we are allowed to test the same rule on the same table of numbers which we used to discover the rule. We realize of course that the rule would prove useless on any other table of random numbers, and this is exactly the issue with Levy's results.''Another form of data snooping is the publication bias. It is a well-known fact that studies presenting unusual results are more likely to be published than the studies that just confirm a well-known theory. The problem of data snooping was addressed in most of the work on technical analysis, but for a long time there was no test procedure to test for it. Finally White (2000), building on the work of Diebold and Mariano (1995) and West (1996), developed a simple and straightforward procedure for testing the null hypothesis that the best forecasting model encountered in a specification search has no predictive superiority over a given benchmark model. The alternative is of course that the best forecasting model is superior to the benchmark. Summarized in simple terms, the procedure bootstraps the original time series a great number of times, preserving the key characteristics of the time series. White (2000) recommends the stationary bootstrap of Politis and Romano (1994a, 1994b). Next, the specification search for the best forecasting model is executed for each bootstrapped series, which yields an empirical distribution of the performance of the best forecasting model. The null hypothesis is rejected at the α percent significance level if the performance of the best forecasting model on the original time series is greater than the α percent cut off level of the empirical distribution. This procedure was called White's Reality Check (RC) for data snooping.
Sullivan, Timmermann and White (1999, 2001) utilize the RC to evaluate simple technical trading strategies and calendar effects applied to the DJIA in the period 1897-1996. Sullivan et al. (1999) take the study of Brock et al. (1992) as starting point and construct an extensive set of 7846 trading rules, consisting of filters, moving averages, support-and-resistance, channel break-outs and on-balance volume averages. It is demonstrated that the results of Brock et al. (1992) hold after correction for data snooping, but that the forecasting performance tends to have disappeared in the period after the end of 1986.