Perplexity AI Open-Sources Unigram Tokenizer That Achieves 5x Lower p50 Latency Than Hugging Face — 2026-05-28

## Short Segments Perplexity AI's new Unigram tokenizer slashes latency by 5x, while Sakana AI's DiffusionBlocks offer a fresh take on neural network training. Later, we'll dive into how Perplexity's open-source release could reshape tokenization in AI workflows. First, let's explore Sakana AI's innovative approach to training deep networks. Sakana AI introduces DiffusionBlocks, a novel framework for training neural networks block by block. This approach significantly reduces memory requirements, addressing a major bottleneck in deep learning. Traditional end-to-end backpropagation demands storing intermediate activations across all layers, leading to high memory consumption as models deepen. DiffusionBlocks tackle this by partitioning networks into independently trainable blocks, cutting memory usage by a factor of B, where B is the number of blocks. This method maintains performance across various architectures, unlike previous techniques that often underperform. By treating the network's forward pass as a diffusion-like denoising process, DiffusionBlocks offer a promising alternative to conventional training methods. For developers, this means more efficient training of complex models without sacrificing performance, potentially accelerating AI research and deployment. Implementing a pgvector-powered vector search system in PostgreSQL is now more accessible than ever. A new coding guide demonstrates how to build a complete pgvector playground in Google Colab, showcasing PostgreSQL's capabilities as a vector database for AI applications. The tutorial covers installing PostgreSQL, compiling the pgvector extension, and integrating with Python via Psycopg. It also explores creating embeddings with SentenceTransformers, building HNSW indexes, and running various search types, including semantic and hybrid searches. This workflow highlights pgvector's support for retrieval-augmented generation, recommendation, and similarity search systems using open-source tools. For developers, this guide offers a practical path to leveraging PostgreSQL for advanced AI-driven search capabilities, enhancing the efficiency and effectiveness of AI applications. ## Feature Story Perplexity AI's open-source Unigram tokenizer promises to revolutionize tokenization efficiency in AI workflows. Rebuilt from scratch in Rust, this tokenizer achieves a 5x reduction in p50 latency compared to the Hugging Face tokenizers crate, and significantly outperforms other popular tokenizers like SentencePiece and IREE's tokenizer. By eliminating steady-state heap allocations, it reduces CPU utilization in Perplexity's inference stack by 5-6x, shaving milliseconds off reranker latency. This development addresses a critical bottleneck in AI processing, where tokenization can become a significant fraction of total request latency, especially in smaller models like rerankers and embedders. These models, often used for ranking, retrieval, and similarity tasks, require efficient tokenization to maximize performance. The Unigram tokenizer targets XLM-RoBERTa's 250K-token vocabulary, a common choice in production environments. By producing the same tokens as the reference implementation without rebuilding strings or chasing hash maps, it offers a streamlined solution for text processing. For AI developers and researchers, this open-source release provides a powerful tool to enhance the efficiency of language model inference, potentially reducing costs and improving response times in AI applications. As tokenization efficiency becomes increasingly important in AI workflows, Perplexity's contribution could set a new standard for performance and resource utilization. Looking ahead, the adoption of this tokenizer could lead to broader improvements in AI processing, particularly in applications where latency and resource constraints are critical factors. For now, developers have a new tool to optimize their AI systems, paving the way for more efficient and effective AI solutions.