Area-wide traffic signal control based on a deep graph Q-Network (DGQN) trained in an asynchronous manner

초록

Value-based reinforcement learning (RL) algorithms have been widely applied in traffic signal studies. There are, however, several problems in jointly controlling traffic lights for a large transportation network. First, the discrete action space exponentially explodes as the number of intersections to be jointly controlled increases. With its model structure, the original deep Q-network (DQN) could not accommodate a large action space. The problem was resolved by revising the output structure of a DQN holding the framework of a single-agent RL algorithm Second, when mapping traffic states into an action value, it is difficult to consider spatio-temporal correlations over a large transportation network. A deep graph Q-network (DGQN) was devised to efficiently accommodate spatio-temporal dependencies on a large scale. Finally, training the proposed DGQN with a large number of joint actions requires much time to converge. An asynchronous update methodology with multiple actor learners was devised for a DGQN to quickly reach an optimal policy. By combining these three remedies, a DGQN succeeded in jointly controlling the traffic lights in a large transportation network in Seoul. This approach outperformed other “state-of-the-art” RL algorithms as well as an actual fixed-signal operation. The proposed DGQN decreased the average delay of the current fixed operation to 55.7%, whereas those of reference models DQN-OGCN and DQN-FC were 72.5 and 92.0%, respectively.

키워드