Multi-Agent Q-Learning Based Multi-UAV Wireless Networks for Maximizing Energy Efficiency: Deployment and Power Control Strategy Design

Abstract

In air-to-ground communications, the network lifetime depends on the operation time of unmanned aerial vehicle-base stations (UAV-BSs) owing to the restricted battery capacity. Therefore, the maximization of energy efficiency and the minimization of outage ground users are important metrics of network performance. To achieve these two objectives, the location and transmit power of the UAV-BSs in the network must be optimized. This optimization problem may not be tractable in the conventional optimization framework because multiple UAV-BSs interact in a complicated manner. Hence, we formulate the problem as a Markov decision process and develop an algorithm to obtain a solution in a reinforcement learning framework. To avoid a central controller and high computational complexity, we employ a multi-agent distributed Q-learning algorithm to obtain a solution. Specifically, we propose a multi-agent Q-learning-based UAV-BS deployment and power control strategy to maximize energy efficiency and minimize the number of outage users in multi-UAV wireless networks. Through intensive simulations, it is demonstrated that the proposed algorithm can outperform benchmark algorithms in terms of average energy efficiency and number of average outage users in multi-UAV wireless networks. IEEE