Decision-making for the autonomous navigation of USVs based on deep reinforcement learning under IALA maritime buoyage system. (15th December 2022)

• Record Type: Journal Article
• Authors: Zhao, Yiming; Han, Fenglei; Han, Duanfeng; Peng, Xiao; Zhao, Wangyuan
• Is Part Of: Ocean engineering. Volume 266(2022) Part 1
• Publication Date: 2022-12-15
• Subjects: Decision-making -- Autonomous navigation -- Hybrid collision avoidance -- IALA Maritime buoyage system -- Deep reinforcement learning; Ocean engineering -- Periodicals; Ocean engineering; Periodicals; 620.4162
• DOI: 10.1016/j.oceaneng.2022.112557
• Languages: English
• ISSNs: 0029-8018

Abstract

Abstract: It is extremely challenging to carry out an advanced adaptive navigation system in the complex environment of a port. Especially in the waters that are about to be berthed, the unmanned surface vehicles (USVs) should autonomously identify and avoid dense buoys according to the rules. Each country has corresponding mandatory documents that strictly regulate the shape, color and orientation of water buoys in coastal and estuary channels. Aiming at the problem of USV collision avoidance navigation under the constraints of the IALA (International Association of Marine Aids and Lighthouse Authorities) Maritime buoyage system, this paper designs a mixed environment model including multiple ocean buoys according to this rule. By using the deep neural network to extract the state features of the target, and setting the reward function reasonably, the USV can not only navigate to the target autonomously, but also identify the corresponding buoy and give the corresponding decision. Using different DQN optimization algorithms to conduct comparative experiments, the stability of the algorithm's learning of the optimal strategy is improved. The results show that the algorithm can accurately avoid obstacles ahead, identify buoys effectively, and realize effective autonomous collision avoidance decision-making in complex environments with static obstacles and buoys with indicating function. This research can provide theoretical basis and method reference for USV's autonomous navigation in port. Highlights: Through the in-depth analysis of the IALA Maritime Buoyage System, the rule information of each navigation aid is designed and embedded into the environment definition. For autonomous navigation and collision avoidance decision-making, the reward function is optimized and a deep reinforcement learning-based USV autonomous navigation decision-making algorithm is proposed. By using the three variant algorithms of Q-Learning to conduct comparative experiments, the problems that autonomous navigation decision-making algorithms tend to fall into local optimal solutions and low iterative efficiency are solved. The simulation results based on pytorch and Unity3d show that USV can finally judge whether its behavior violates the IALA Buoyage System. And the navigation task can also be successfully completed at the new initial point.