ICL-RCTAC hierarchical learning algorithm optimizes trajectory tracking and environmental disturbance adaptation for AUV in the horizontal plane
DOI:
https://doi.org/10.54939/1859-1043.j.mst.114.2026.3-11Keywords:
Autonomous underwater vehicle; Reinforcement learning; Integral concurrent learning; Lyapunov stability; Adaptive control; Optimal control.Abstract
ICL-RCTAC is a hierarchical reinforcement learning algorithm for adaptive optimal trajectory tracking of an Autonomous Underwater Vehicle (AUV) in the horizontal plane. It addresses two key limitations of existing continuous-time reinforcement learning: the warm-up phase and the persistent excitation (PE) condition, via a two-phase framework. In Phase I, Integral Concurrent Learning (ICL) exploits stored historical data to identify stable Critic weights under a finite-excitation (FE) condition, with convergence formally proven in Proposition 1. Phase II then activates an Actor–Critic loop under relaxed conditions for efficient policy optimization. Lyapunov analysis confirms uniform ultimate boundedness (UUB) of the closed-loop system. Simulations in MATLAB/Simulink 2024a show faster convergence, lower tracking error, and reduced control effort than PID and RCTAC, confirming the algorithm's practical potential for AUV applications.
References
[1]. Dong Ma, Ye Li, et al., “The state of the art in key technologies for autonomous underwater vehicles: a review.” Ocean Engineering, (2025). DOI: https://doi.org/10.1016/j.eng.2025.08.002
[2]. Wibisono, A., Piran, M.J., Song, H.-K., Lee, B.M., “A Survey on Unmanned Underwater Vehicles: Challenges, Enabling Technologies, and Future Research Directions,” Sensors, 23, 7321, (2023).
[3]. Daoliang Li, and Ling Du, “AUV Trajectory Tracking Models and Control Strategies: A Review,” Journal of Marine Science and Engineering, 9, 1020, (2021). DOI: https://doi.org/10.3390/jmse9091020
[4]. Bach Nhat Hoang, Doan Trung Thanh, Phan Huy Anh, and Dang Tien Sy, “Research on solutions to enhance trajectory quality in underwater autonomous navigation of unmanned underwater vehicles using IEKF SLAM algorithm for SONAR data”, J. Mil. Sci. Technol., IITE, 27–34, (2025). DOI: https://doi.org/10.54939/1859-1043.j.mst.IITE.2025.27-34
[5]. Kairong Duan, Simon Fong, C.L. Philip Chen, “Reinforcement learning-based model-free optimized trajectory tracking strategy design for an AUV”, Neurocomputing, 469, 289-297, (2022).
[6]. Nguyen Van Duc and Sai Van Cuong, “Reinforcement learning algorithm for adaptive intelligent control”, J. Mil. Sci. Technol., CAPITI, 62–68, (2024). DOI: https://doi.org/10.54939/1859-1043.j.mst.CAPITI.2024.62-68
[7]. Duan, J., Li, S.E., Guan, Y., Sun, Q., and Cheng, B, “Relaxed actor-critic with convergence guarantees for continuous-time optimal control of nonlinear systems,” IEEE Transactions on Intelligent Vehicles, 8, 5, 3299–3311. DOI: https://doi.org/10.1109/TIV.2023.3255264
[8]. Parikh, A., Kamalapurkar, R., and Dixon, W.E, “Integral concurrent learning: Adaptive control with parameter convergence using finite excitation,” International Journal of Adaptive Control and Signal Processing, 33, 12, 1775–1787. DOI: https://doi.org/10.1002/acs.2945
[9]. T. I. Fossen, “Handbook of Marine Craft Hydrodynamics and Motion Control,” John Wiley & Sons, 2nd Edition, (2021).
[10]. Chunbin Qin, et al., “Adaptive Dynamic Programming without Persistence Excitation for Tracking in A Special Class of Linear Systems with Input Constraints,” 33rd Chinese Control and Decision Conference, (2021). DOI: https://doi.org/10.1109/CCDC52312.2021.9602844
[11]. S. Bhasin, et al., “A novel actor-critic-identifier architecture for approximate optimal control of uncertain nonlinear systems,” Automatica, 49, 1, 82-92, (2013).
[12]. H. K. Khalil, “Nonlinear Systems,” Prentice-Hall, 3rd ed., (2002).
