Enhancement of hydrologic model optimization with single-step reinforcement learning

초록

Efficient calibration of hydrological models is essential for accurate water resource management but is often limited by the high computational demands of traditional methods. This study proposes a reinforcement learning (RL) approach using a single-step Proximal Policy Optimization (PPO-1) algorithm for models with static parameters. Most hydrological models require fixed parameters throughout simulation runs, presenting challenges for RL implementations relying on frequent adjustments. PPO-1 addresses this by learning from single interactions per episode, aligning with the constraints of static parameter models. The approach is evaluated using the Soil and Water Assessment Tool (SWAT) in the Tuckahoe Creek Watershed (TCW, 220.7 km2, U.S.) and the Miho River Watershed (MRW, 1,855 km2, South Korea). The RL method was tested for 1,000 episodes, with performance at 500 and 1,000 episodes compared to 1,500 simulations of SUFI-2. In TCW, RL with 500 episodes achieved Nash-Sutcliffe Efficiency (NSE) values of 0.67-0.72 for calibration and 0.70-0.80 for validation, outperforming SUFI-2 (NSE: 0.62 for calibration and 0.61-0.63 for validation). Simulation time was also reduced by 69 %, requiring 3.3 h for RL in contrast to 12.5 h for SUFI-2. In MRW, RL with 500 episodes yielded NSE values of 0.63-0.65 for both calibration and validation, comparable to SUFI-2, while reducing runtime from 575 h to 260 h. These findings demonstrate that single-step RL offers better or comparable calibration accuracy using fewer computational resources, making it effective for hydrological models with static parameter structures and transferable to broader environmental modeling applications.

키워드