An improved ant colony algorithm based on Q-Learning for route planning of autonomous vehicle


  • Liping Zhao Systems Engineering Institute, Academy of Military Sciences, Peoples Liberation Army, Beijing, China
  • Feng Li Systems Engineering Institute, Academy of Military Sciences, Peoples Liberation Army, Beijing, China
  • Dongye Sun National Engineering Research Center for Transportation Safety and Emergency informatics, China Transport Telecommunications & Information Center, Beijing, China
  • Zihan Zhao Systems Engineering Institute, Academy of Military Sciences, Peoples Liberation Army, Beijing, China



Autonomous vehicle, Path planning, Q-Learning, Improved ant colony algorithm


In view of the problems existing in the path planning algorithms of unmanned vehicles, such as low search efficiency, slow convergence speed and easy to fall into the local optimal. Based on the characteristics of route planning for unmanned vehicles, this paper introduces Q-Learning into the traditional ant colony algorithm to enhance the learning ability of the algorithm in dynamic environment, so as to improve the overall efficiency of route search. By mapping pheromones into Q values in Q-learning, rapid search in complex environments is realized, and a collection-free path satisfying constraints is quickly found. The results of case analysis show that compared with the traditional ant colony algorithm and the improved ant colony algorithm considering reward and punishment factors, the improved ant colony algorithm based on Q-Learning can effectively reduce the number of iterations, shorten the path optimization time and path length and other performance indicators, and has many advantages in jumping out of the local optimal, improving the global search ability and improving the convergence speed, and has good adaptability and robustness in complex environments. It ensures the safety and stability of unmanned vehicles in complex environments.


Katrakazas C; Quddus M; Chen W H; et al. (2015). Real-time motion planning methods for autonomous on-road driving: State-of-the-art and future research directions, Transportation Research Part C, 60, 416-442, 2015.

Patle B; Pandey A; Parhi D; et al. (2019). A review: On path planning strategies for navigation of mobile robot, Defence Technology, 15(4), 582-606, 2019.

LI T C; SUN S D; GAO Y. (2010). Fan-shaped Grid Based Global Path Planning for Mobile Robot, ROBOT, 32(4), 547-552, 2010.

GUO L J; SHI W X; LI Y; LI F X; et al. (2011). Mapping algorithm using adaptive size of occupancy grids based on quadtree, Control and Decision, 26(11), 1690-1694, 2011.

Y Yang; K He; Y P Wang; Z Z Yuan; Y H Yin; M Z Guo. (2022). Identification of dynamic traffic crash risk for cross-area freeways based on statistical and machine learning methods, Physica A: Statistical Mechanics and its Applications, 595(2022), 127083, 2022.

Azim E; Chaoxian W; Chuanyang S. (2021). Research Advances and Challenges of Autonomous and Connected Ground Vehicles, IEEE Transactions on Intelligent Transportation Systems, 22(2), 683-711, 2021.

LI D L,WANG P, DU L. (2019). Path planning technologies for autonomous underwater vehicles-a review, IEEE Access, 7, 9745-9768, 2019.

Özgur C; Sarikovanlik V. (2022). Forecasting BIST100 and NASDAQ Indices with Single and Hybrid Machine Learning Algorithms, Economic Computation And Economic Cybernetics Studies And Research, DOI: 10.24818/18423264/, 56(3), 235-250, 2022.

Y. Yang; N. Tian; Y. Wang; Z. Yuan. (2022). A Parallel FP-Growth Mining Algorithm with Load Balancing Constraints for Traffic Crash Data, International Journal of Computers Communications & Control, 17(4), 4806, 2022.

Liu J.-Y.; Liu S.-F.; Gong D.-Q. (2021). Electric Vehicle Charging Station Layout Based on Particle Swarm Simulation, Int. Journal of Simulation Modelling, 20(4), 754-765, 2021.

Yang Y; Yuan Z; Meng R. (2022). Exploring Traffic Crash Occurrence Mechanism toward Cross- Area Freeways via an Improved Data Mining Approach, Journal of Transportation Engineering Part A Systems, 148(9), 04022052, 2022.

Bacha A; Bauman C; Faruque R; et al. (2008). Odin: Team Victor Tango's entry in the DARPA Urban Challenge, Journal of Field Robotics, 25(8), 467-92, 2008.

Zhang X Y; Zou Y S. (2021). Collision-free path planning for automated guided vehicles based on improved A* algorithm, Systems Engineering-Theory & Practice, 41(1), 240-246, 2021.

Carreras M; Hernandez J D; Vidal E; et al. (2016). Online motion planning for underwater inspection, Autonomous Underwater Vehicles. IEEE, 336-341, 2016.

Jalalmaab M; Fidan B; Jeon S; et al. (2015). Model predictive path planning with timevarying safety constraints for highway autonomous driving, International Conference on Advanced Robotics (ICAR), 213-217, 2015.

Receveur J-B; Victor S; Melchior P. (2020). Autonomous car decision making and trajectory tracking based on genetic algorithms and fractional potential fields, Intelligent Service Robotics, 13(2), 315-330, 2020.

Afify, H.M.; Mohammed, K.K.; Hassanien, A.E. (2020). Multi-Images Recognition of Breast Cancer Histopathological via Probabilistic Neural Network Approach, Journal of System and Management Sciences, 10(2), 53-68, 2020.

Miao C W; Chen G Z; Yan C L; et al. (2021). Path planning optimization of indoor mobile robot based on adaptive ant colony algorithm, Computers & Industrial Engineering, 156(1), 1-12, 2021.

XU L; FU W H; JIANG W H; LI Z T. (2021). mobile robots path planning based on 16-directions 24-neighborhoods improved ant colony algorithm, Control and Decision, 36(05), 1137-1146, 2021.

LI T; ZHAO H S. (2022). Path optimization for mobile robot based on evolutionary ant colony algorithm, Control and Decision, DOI:10.13195/j.kzyjc.2021.1324, 1-9, 2022.

LI S D; XU X; ZUO L. (2015). Dynamic path planning of a mobile robot with improved Qlearning algorithm, In Proceedings of 2015 IEEE International Conference on Information and Automation, 409-414, 2015.

Yang Y; Yuan Z; Chen J; Guo M. (2017). Assessment of osculating value method based on entropy weight to transportation energy conservation and emission reduction, Environmental Engineering & Management Journal, 16(10), 2413-2424, 2017.

Yang Y; Yang B; Yuan Z; et al. (2023). Modeling and Comparing Two Modes of Sharing Parking Spots at Residential Area: Real-time and Fixed-time Allocation, IET Intelligent Transport Systems, 2023.

Yuan Z; Yuan X; Yang Y; et al. (2023). Greenhouse Gas Emission Analysis and Measurement for Urban Rail Transit: A Review of Research Progress and Prospects, Digital Transportation and Safety, 1(1), 37-52, 2023.

Tan B; Peng Y Y; Lin J G. (2021). A local path planning method based on q-learning, In International Conference on Signal Processing and Machine Learning, 80-84, 2021.

TIAN X H; HUO X; ZHOU D L; ZHAO H. (2022). Ant colony pheromone aided Q-learning path planning algorithm, Control and Decision, DOI:, 2022.

MEERZA S I A; ISLAM M; UZZAL M M. (2019). Q-learning based particle swarm optimization algorithm for optimal path planning of swarm of mobile robots, Proceedings of 2019 International Conference on Advances in Science, Engineering and Robotics Technology, 1-5, 2019.

YAO Q F; ZHENG Z Y; QI L; et al. (2020). Path planning method with improved artificial potential field- a reinforcement learning perspective, IEEE Access, 8, 135513-135523, 2020.

LIU Z Y; LAN F; YANG H B. (2019). Partition heuristic RRT algorithm of path planning based on Q-learning, Proceedings of 2019 Advanced Information Technology, Electronic and Automation Control Conference, 386-392, 2019.

SHI Z G; TU J; ZHANG Q; et al. (2013). The improved Q-Learning algorithm based on pheromone mechanism for swarm robot system, Proceedings of the 32nd Chinese Control Conference, 6033- 6038, 2013.

Zhu J Y; GAO M T. (2021). AUV Path Planning Based on Particle Swarm Optimization and Improved Ant Colony Optimization, Computer Engineering and Applications, 57(06), 267-273, 2021.

HU C Y; JIANG P; ZHOU G R. (2020). Application of improved ant colony algorithm in AGV path planning, Computer Engineering and Applications, 56(8), 270-278, 2020.

MA Y N; GONG Y J; XIAO C F; et al. (2019). Path planning for autonomous underwater vehicles: an ant colony algorithm incorporating alarm pheromone, IEEE Transactions on Vehicular Technology, 68(1), 141-154, 2019.

HE X L; JIANG H; SONG Y; et al. (2019). Routing selection with reinforcement learning for energy harvesting multi-hop CRN, IEEE Access, 7, 54435-54448, 2019.

ARUNITA K; LOBIYAL D K. (2021). Q-learning based routing protocol to enhance network lifetime in WSNs, International Journal of Computer Networks & Communications, 13(2), 67- 80, 2021.

Additional Files



Most read articles by the same author(s)

Obs.: This plugin requires at least one statistics/report plugin to be enabled. If your statistics plugins provide more than one metric then please also select a main metric on the admin's site settings page and/or on the journal manager's settings pages.