EPISODE · Sep 30, 2025 · 23 MIN
Language Models Can Learn from Verbal Feedback Without Scalar Rewards
from Daily Paper Cast · host Jingwen Liang, Gengyu Wang
🤗 Upvotes: 48 | cs.CL, cs.AI, cs.LG Authors: Renjie Luo, Zichen Liu, Xiangyan Liu, Chao Du, Min Lin, Wenhu Chen, Wei Lu, Tianyu Pang Title: Language Models Can Learn from Verbal Feedback Without Scalar Rewards Arxiv: http://arxiv.org/abs/2509.22638v1 Abstract: LLMs are often trained with RL from human or AI feedback, yet such methods typically compress nuanced feedback into scalar rewards, discarding much of their richness and inducing scale imbalance. We propose treating verbal feedback as a conditioning signal. Inspired by language priors in text-to-image generation, which enable novel outputs from unseen prompts, we introduce the feedback-conditional policy (FCP). FCP learns directly from response-feedback pairs, approximating the feedback-conditional posterior through maximum likelihood training on offline data. We further develop an online bootstrapping stage where the policy generates under positive conditions and receives fresh feedback to refine itself. This reframes feedback-driven learning as conditional generation rather than reward optimization, offering a more expressive way for LLMs to directly learn from verbal feedback. Our code is available at https://github.com/sail-sg/feedback-conditional-policy.
What this episode covers
🤗 Upvotes: 48 | cs.CL, cs.AI, cs.LG Authors: Renjie Luo, Zichen Liu, Xiangyan Liu, Chao Du, Min Lin, Wenhu Chen, Wei Lu, Tianyu Pang Title: Language Models Can Learn from Verbal Feedback Without Scalar Rewards Arxiv: http://arxiv.org/abs/2509.22638v1 Abstract: LLMs are often trained with RL from human or AI feedback, yet such methods typically compress nuanced feedback into scalar rewards, discarding much of their richness and inducing scale imbalance. We propose treating verbal feedback as a conditioning signal. Inspired by language priors in text-to-image generation, which enable novel outputs from unseen prompts, we introduce the feedback-conditional policy (FCP). FCP learns directly from response-feedback pairs, approximating the feedback-conditional posterior through maximum likelihood training on offline data. We further develop an online bootstrapping stage where the policy generates under positive conditions and receives fresh feedback to refine itself. This reframes feedback-driven learning as conditional generation rather than reward optimization, offering a more expressive way for LLMs to directly learn from verbal feedback. Our code is available at https://github.com/sail-sg/feedback-conditional-policy.
NOW PLAYING
Language Models Can Learn from Verbal Feedback Without Scalar Rewards
No transcript for this episode yet
Similar Episodes
No similar episodes found.