VIT-LENS: Towards Omni-modal Representations

The paper, "VIT-LENS: Towards Omni-modal Representations," introduces a novel approach to enable Artificial Intelligence (AI) agents to perceive information from various modalities beyond just vision and language. It proposes a method that leverages a pre-trained visual transformer (ViT) to efficiently encode information from diverse modalities, such as 3D point clouds, depth, audio, tactile, and electroencephalograms (EEG). By aligning these modalities with a shared embedding space, VIT-LENS unlocks a range of capabilities for AI agents, including any-modality captioning, question answering, and image generation. The paper presents extensive experimental results demonstrating that VIT-LENS achieves state-of-the-art performance on various benchmark datasets and outperforms prior methods in understanding and interacting with diverse modalities.