MENTOR: Mixture-of-Experts Network with Task-Oriented Perturbation for Visual Reinforcement Learning

[00:00] Introduction to Mentor system for visual RL [00:29] Problem: Sample inefficiency in robotic learning [00:59] Innovation: Mixture of Experts (MoE) architecture [01:55] Results: MoE achieves 100% success in multi-task testing [02:33] Feature: Task-oriented perturbation for exploration [03:55] Real-world testing: 83% success in robotic tasks [04:33] Study: MoE and perturbation each boost performance by 30% [05:14] Future work: Optimizing MoE implementation [05:59] Challenge: Bridging simulation-to-real-world gap [06:45] Impact: Advancing practical robotics applications Authors: Suning Huang, Zheyu Zhang, Tianhai Liang, Yihan Xu, Zhehao Kou, Chenhao Lu, Guowei Xu, Zhengrong Xue, Huazhe Xu Affiliations: Tsinghua University, Shanghai Qi Zhi Institute, Shanghai AI Lab Abstract: Visual deep reinforcement learning (RL) enables robots to acquire skills from visual input for unstructured tasks. However, current algorithms suffer from low sample efficiency, limiting their practical applicability. In this work, we present MENTOR, a method that improves both the architecture and optimization of RL agents. Specifically, MENTOR replaces the standard multi-layer perceptron (MLP) with a mixture-of-experts (MoE) backbone, enhancing the agent's ability to handle complex tasks by leveraging modular expert learning to avoid gradient conflicts. Furthermore, MENTOR introduces a task-oriented perturbation mechanism, which heuristically samples perturbation candidates containing task-relevant information, leading to more targeted and effective optimization. MENTOR outperforms state-of-the-art methods across three simulation domains -- DeepMind Control Suite, Meta-World, and Adroit. Additionally, MENTOR achieves an average of 83% success rate on three challenging real-world robotic manipulation tasks including peg insertion, cable routing, and tabletop golf, which significantly surpasses the success rate of 32% from the current strongest model-free visual RL algorithm. These results underscore the importance of sample efficiency in advancing visual RL for real-world robotics. Experimental videos are available at this https URL. Link: https://arxiv.org/abs/2410.14972