Skip to content
 Distribution-calibrated inference time compute for thinking llm-as-a-judge episode artwork

Dec 11, 2025 11 min

Distribution-calibrated inference time compute for thinking llm-as-a-judge

from Best AI papers explained Hosted by Enoch H. Kang

This paper discusses the Distribution-Calibrated Aggregation scheme designed to improve the reliability of "Thinking-LLM-as-a-Judge" systems, which are often used for evaluating generative AI outputs. The core problem addressed is that simply aggregating multiple, noisy individual judgments (e.g., via majority vote) is suboptimal, especially when the judge is allowed to declare a tie. The proposed method utilizes Inference-Time Compute (ITC) to generate multiple independent samples and then models the three-way preference outcomes (A preferred, B preferred, or Tie) using a Bradley–Terry–Davidson formulation that accounts for both the margin of preference and the decisiveness of the vote (non-tie rate). Extensive experiments across machine translation and reward model benchmarks demonstrate that this distribution-aware aggregation consistently reduces the Mean Absolute Error (MAE) and increases accuracy, frequently matching or exceeding individual human rater performance. The authors emphasize that this calibration step is crucial for turning stochastic, individual LLM judgments into robust and accurate final ratings.

Episode details come from the podcast's RSS feed.

Embed this episode

Ready to play

Distribution-calibrated inference time compute for thinking llm-as-a-judge

0:00 11:48

No transcript yet.

Requests help us decide what to transcribe next.

Frequently asked questions

How long is this episode of Best AI papers explained?

This episode is 11 minutes long.

When was this Best AI papers explained episode published?

This episode was published on December 11, 2025.

Can I download this Best AI papers explained episode?

Yes. Use the download control on the episode player to save the publisher-provided media file.