Learning Latent Action World Models In The Wild

This research explores how to model **"latent actions"** in unpredictable, real-world videos where specific movement commands are not pre-defined. The authors compare three primary methods for organizing these hidden actions: **sparsity-based constraints**, **noise addition**, and **discrete quantization**. By testing these techniques on diverse datasets like **YouTube** and **robotics footage**, the study examines how much information these models should capture to be effective. Results indicate that **sparse and noisy latents** generally outperform discrete ones in visualizing movement and executing **goal-based planning**. The findings emphasize a critical trade-off between **model capacity** and the ability to generalize across different environments. Ultimately, the work demonstrates that learning actions directly from raw video can serve as a powerful interface for **autonomous robotic control**.