EP048: BLIP-2 Teaches Frozen Models to See

The paper "BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models" introduces a highly compute-efficient framework for vision-language pre-training.Here is a short summary of its key concepts and findings:The Problem: Most state-of-the-art vision-language models require expensive end-to-end training on massive datasets, which incurs an extremely high computation cost.The Solution: BLIP-2 drastically reduces this cost by leveraging off-the-shelf, frozen pre-trained image encoders and frozen large language models (LLMs). Freezing these models during pre-training saves computation and prevents catastrophic forgetting.Bridging the Gap: Because the frozen LLMs have never seen images, BLIP-2 introduces a lightweight Querying Transformer (Q-Former) to bridge the modality gap between vision and language. The Q-Former acts as an information bottleneck that extracts the most useful visual features from the frozen image encoder and feeds them to the LLM.Two-Stage Pre-training: The Q-Former is trained using a novel two-stage strategy:Results: BLIP-2 achieves state-of-the-art performance on various vision-language tasks—such as visual question answering (VQA), image captioning, and image-text retrieval—despite having significantly fewer trainable parameters than existing models. For instance, it outperforms the 80-billion parameter Flamingo model by 8.7% on zero-shot VQA while using 54 times fewer trainable parameters. Furthermore, it enables powerful zero-shot instructed image-to-text generation, allowing for tasks like visual reasoning and visual conversation.