Semantic Operators: A Declarative Model for Rich, AI-based Data Processing

This paper introduces semantic operators, a declarative model for AI-powered data processing that leverages the capabilities of large language models (LLMs) for complex data transformations. The core concept is to provide a structured way to perform operations like filtering, joining, and aggregating data using natural language descriptions. By defining a "gold algorithm" for each operator, the system ensures accuracy while an optimization framework enables significant performance improvements with statistical accuracy guarantees. The authors present LOTUS, an open-source system implementing these operators and demonstrate its effectiveness and efficiency on various real-world applications, outperforming existing methods.