ALFA: Aligning LLMs to Ask Good Questions A Case Study in Clinical Reasoning

This academic paper introduces ALFA (ALignment via Fine-grained Attributes), a new framework designed to enhance how large language models (LLMs) ask questions, particularly in complex fields like clinical reasoning. The authors highlight the current limitations of LLMs in proactive information-gathering, which is crucial for decision-making in high-stakes environments. ALFA addresses this by decomposing the concept of a "good" question into specific, theory-backed attributes such as clarity, relevance, and diagnostic accuracy. The framework then synthesizes attribute-specific question variations and aligns models using preference-based optimization to learn these improved question-asking behaviors. Through a case study in clinical reasoning using the MediQ-AskDocs dataset, ALFA-aligned models demonstrated a significant reduction in diagnostic errors compared to existing state-of-the-art LLMs, showcasing the effectiveness of explicitly guiding question-asking with structured attributes.