Champion-level sim-to-real: empowering a deep reinforcement learning agent's control of differential-drive robots using feedback control navigation and domain randomization

Abstract

Applying Deep Reinforcement Learning (DRL) for motion control in dynamic robotic environments remains a significant challenge. The heavy reliance on simulated environments for training introduces the critical “Sim-to-Real” gap, often hindering the direct transfer of policies to physical hardware. This work addresses this challenge by presenting two practical approaches to enhance the training process: (1) Hierarchical Control Architecture, a method that assists the agent using a classical navigation heuristic during training, and (2) Acceleration Filter Randomization, a domain randomization technique designed to diversify the agent's experience against dynamic variations. These strategies were implemented using the Deep Deterministic Policy Gradient algorithm, and their performance was evaluated in the IEEE Very Small Size Soccer (VSSS) domain. In both simulation and real-world scenarios, metrics such as goal success rate, steps per episode, and accumulated reward of each method were compared to demonstrate the methods’ effectiveness in the context of the sim-to-real transfer challenge. To quantify the effectiveness in bridging the sim-to-real gap, the proposed approaches were benchmarked against a naive Baseline agent lacking specific transfer strategies. Finally, our experimental results demonstrate a successful policy transfer to a physical robot, achieving a goal-scoring success rate exceeding 88% in both proposed methods, thereby confirming the efficacy of these strategies.