Optimal activation function subset selection via forward and backward search for enhanced model performance

초록

Previous studies have improved neural network performance by combining multiple activation functions through hierarchical architectures or output concatenation strategies that increase the model depth. This paper proposes a heterogeneous activation assignment framework that selects an optimal subset of activation functions from a predefined pool to enhance both expressive ca pacity and learning stability. Our method employs unit-wise activation function selection, using forward selection and backward elimination to identify subsets that minimize validation loss un der a fixed computational budget. Instead of increasing depth or parameter count, we partition each hidden layer into parallel activation channels, assigning one selected function to each chan nel without introducing additional trainable parameters. We evaluate the proposed framework on a nanophotonics transmittance prediction task and eleven regression benchmarks, demon strating practical utility, strong generalization, and consistent performance gains across diverse domains. On average, it reduces MAE by 16.08 +/- 15.33 % with forward selection and 16.13 +/- 15.51 % with backward elimination compared to Rectified Linear Unit (ReLU). These results indicate that search-based activation selection enhances network expressivity through functional diversity and provides a robust foundation for future research in activation design and neural architecture optimization.

키워드