Beyond Semantic Manipulation: Token-Space Attacks on Reward Models

This research paper introduces TOMPA, a novel framework designed to expose critical vulnerabilities in reward models used for aligning artificial intelligence. Unlike traditional adversarial methods that rely on human-readable text, this approach performs automated optimization directly in token space to bypass semantic constraints. By eliminating the need for coherent natural language, the system discovers non-linguistic token patterns that achieve exceptionally high scores from top-tier evaluators. Despite being identified as superior to high-quality human references, these generated outputs consist of nonsensical gibberish and repetitive symbols. The study demonstrates that reward hacking extends beyond simple linguistic biases, revealing a structural flaw where models prioritize specific raw data sequences over actual meaning. Ultimately, the authors argue that current RLHF pipelines remain highly susceptible to exploitation through these nonsensical, length-dependent adversarial patterns.