SkillMimicGen: Automated Demonstration Generation for Efficient Skill Learning and Deployment

[00:00] SkillGen: AI system for robotic learning and automation. [00:19] Core Innovation: Automated dataset generation from minimal human input. [01:12] Skill Segmentation: Smart system for breaking down and adapting complex tasks. [01:59] Hybrid Skill Policy: Framework for controlling robot actions and task completion. [02:50] Performance Results: 75.4% success rate, generating 24,000+ demonstrations. [04:18] Real-World Testing: 35% success in direct simulation-to-reality transfer. [05:01] Current Limitations: Preset sequences and object tracking requirements. [06:47] HSP Variants: Different approaches to robot control and motion planning. [07:43] Practical Applications: Successful implementation in pick-and-place tasks. Authors: Caelan Garrett, Ajay Mandlekar, Bowen Wen, Dieter Fox Affiliation: NVIDIA Abstract: Imitation learning from human demonstrations is an effective paradigm for robot manipulation, but acquiring large datasets is costly and resource-intensive, especially for long-horizon tasks. To address this issue, we propose SkillMimicGen (SkillGen), an automated system for generating demonstration datasets from a few human demos. SkillGen segments human demos into manipulation skills, adapts these skills to new contexts, and stitches them together through free-space transit and transfer motion. We also propose a Hybrid Skill Policy (HSP) framework for learning skill initiation, control, and termination components from SkillGen datasets, enabling skills to be sequenced using motion planning at test-time. We demonstrate that SkillGen greatly improves data generation and policy learning performance over a state-of-the-art data generation framework, resulting in the capability to produce data for large scene variations, including clutter, and agents that are on average 24% more successful. We demonstrate the efficacy of SkillGen by generating over 24K demonstrations across 18 task variants in simulation from just 60 human demonstrations, and training proficient, often near-perfect, HSP agents. Finally, we apply SkillGen to 3 real-world manipulation tasks and also demonstrate zero-shot sim-to-real transfer on a long-horizon assembly task. Videos, and more at this https URL. Link: https://arxiv.org/abs/2410.18907