Effectively training neural networks for stock index prediction: Predicting the S&P 500 index without using its index data

Abstract

We propose a novel method for training neural networks to predict the future prices of stock indexes. Unlike previous works, we do not use target stock index data for training neural networks for index prediction. Instead, we use only the data of individual companies to obtain sufficient amount of data for training neural networks for stock index prediction. As a result, our method can avoid various problems due to training complex machine learning models on a small amount of data. We performed numerous types of experiments to test methods designed for predicting the future price of the S&P 500 which is one of the most commonly traded stock indexes. Our experiments show that neural networks trained using our method outperform neural networks trained on stock index data. Compared with other state-of-the-art methods, our method is conceptually simpler and easier to apply, and achieves better results. We obtained approximately a 5-16% annual return before transaction costs during the test period (2006-2018).