Shallow Fully Connected Neural Network Training by Forcing Linearization into Valid Region and Balancing Training Rates

Abstract

A new supervisory training rule for a shallow fully connected neural network (SFCNN) is proposed in this present study. The proposed training rule is developed based on local linearization and analytical optimal solutions for linearized SFCNN. The cause of nonlinearity in neural network training is analyzed, and it is removed by local linearization. The optimal solution for the linearized SFCNN, which minimizes the cost function for the training, is analytically derived. Additionally, the training efficiency and model accuracy of the trained SFCNN are improved by keeping estimates within a valid range of the linearization. The superiority of the proposed approach is demonstrated by applying the proposed training rule to the modeling of a typical nonlinear pH process, Boston housing prices dataset, and automobile mileage per gallon dataset. The proposed training rule shows the smallest modeling error and the smallest iteration number required for convergence compared with several previous approaches from the literature for the case study.