Skip to content
Training a Model on Your Own Taste episode artwork

Oct 24, 2026 21 min

Training a Model on Your Own Taste

from My Weird Prompts

Daniel wants a model that generates episode titles and descriptions, learns from side-by-side button presses, and improves each checkpoint. Herman walks through what that pipeline actually is: an iterative preference optimization loop built on DPO (Direct Preference Optimization), with an implicit reward derived from a frozen reference model rather than a separate reward model or RL rollout. Along the way: what one saved record must contain, why ties deserve their own objective (DPO-RK, DPO-D), and why a single transcript's four candidates are still just one opinion. Key sources: • Rafailov et al., Direct Preference Optimization . — https://arxiv.org/abs/2305.18290 • Chen et al., On Extending DPO to Accommodate Ties . — https://arxiv.org/abs/2409.17431 • author's NeurIPS 2025 summary . — https://www.jinghong-chen.net/my-on-extending-direct-preference-optimization-to-accommodate-ties-neurips-2025/ Episode 6242 — myweirdprompts.com/n/6242 (permanent short link: myweirdprompts.com/171958) Listen on Android: MWP FM Player — myweirdprompts.com/apk

Episode details come from the podcast's RSS feed.

Embed this episode

Ready to play

Training a Model on Your Own Taste

0:00 21:55

Chapters

  1. 0:00 Intro
  2. 0:24 AI Disclaimer
  3. 0:33 Discussion
  4. 20:26 Send a Prompt
  5. 20:45 Credits
  6. 20:53 Outro

No transcript yet.

Requests help us decide what to transcribe next.

Frequently asked questions

How long is this episode of My Weird Prompts?

This episode is 21 minutes long.

When was this My Weird Prompts episode published?

This episode was published on October 24, 2026.

Can I download this My Weird Prompts episode?

Yes. Use the download control on the episode player to save the publisher-provided media file.