TINIEE: Traffic-Aware Adaptive In-Network Intelligence via Early-Exit Strategy

초록

In-network inference over programmable data planes (PDPs) allows fast and low-overhead inference using deep neural networks (DNNs). To alleviate massive processing and deployment costs of in-network inference, distributed deployment on multiple programmable network devices is mainly adopted. However, it is likely to produce a considerable amount of network traffic due to the long distance of the forwarding path and the intermediate data between submodels. In this work, we propose a traffic-aware adaptive in-network inference scheme, TINIEE, to significantly reduce the network traffic of in-network inference without causing a considerable reduction in classification performance. To this end, we first devise an adaptive inference method on the data plane that strikes the balance between the classification performance and the network traffic cost. Furthermore, we formulate a traffic minimization problem to decide a proper location of each submodel considering each flow's exit tendency with a predefined confidence threshold. Since the problem is excessively complicated, we devise a low-complexity practical submodel placement algorithm. We implement TINIEE on both software and hardware programmable switches, and the evaluation results demonstrate that TINIEE reduces network traffic by up to 34.48%, maximum link utilization up to 52.7%, and flow completion time up to 17.8% compared to the state-of-the-art, while maintaining a high classification performance.

키워드