EPISODE · May 25, 2025 · 14 MIN
FisherSFT: Data-Efficient Supervised Fine-Tuning of Language Models Using Information Gain
from Best AI papers explained · host Enoch H. Kang
This paper introduces FisherSFT, a method for making supervised fine-tuning (SFT) of large language models (LLMs) more data-efficient by selecting the most informative training examples. The key concept is to choose examples that maximize information gain, which is approximated by evaluating the Hessian of the LLM's log-likelihood. This approach uses a computationally efficient approximation based on linearizing the LLM's last layer and employs a greedy algorithm to select sentences with the highest information gain. The authors provide a theoretical analysis bounding the prediction error and empirical results demonstrating FisherSFT's superiority over baseline sampling methods on synthetic and real-world datasets, including GPT-2.
What this episode covers
This paper introduces FisherSFT, a method for making supervised fine-tuning (SFT) of large language models (LLMs) more data-efficient by selecting the most informative training examples. The key concept is to choose examples that maximize information gain, which is approximated by evaluating the Hessian of the LLM's log-likelihood. This approach uses a computationally efficient approximation based on linearizing the LLM's last layer and employs a greedy algorithm to select sentences with the highest information gain. The authors provide a theoretical analysis bounding the prediction error and empirical results demonstrating FisherSFT's superiority over baseline sampling methods on synthetic and real-world datasets, including GPT-2.
NOW PLAYING
FisherSFT: Data-Efficient Supervised Fine-Tuning of Language Models Using Information Gain
No transcript for this episode yet
Similar Episodes
Mar 31, 2026 ·54m
Mar 27, 2026 ·14m
Mar 24, 2026 ·42m
Mar 20, 2026 ·42m
Mar 17, 2026 ·41m
Mar 13, 2026 ·44m