gpt3.5 paper

Making language models bigger does not inherently make them better at following a user's intent. For example, large language models can generate outputs that are untruthful, toxic, or simply not helpful to the user. In other words, these models are not aligned with their users. In this paper, we show an avenue for aligning language models with user intent on a wide range of tasks by fine-tuning with human feedback. Starting with a set of labeler-written prompts and prompts submitted through the OpenAI API, we collect a dataset of labeler demonstrations of the desired model behavior, which we use to fine-tune GPT-3 using supervised learning. We then collect a dataset of rankings of model outputs, which we use to further fine-tune this supervised model using reinforcement learning from human feedback. We call the resulting models InstructGPT. In human evaluations on our prompt distribution, outputs from the 1.3B parameter InstructGPT model are preferred to outputs from the 175B GPT-3, despite having 100x fewer parameters. Moreover, InstructGPT models show improvements in truthfulness and reductions in toxic output generation while having minimal performance regressions on public NLP datasets. Even though InstructGPT still makes simple mistakes, our results show that fine-tuning with human feedback is a promising direction for aligning language models with human intent. 扩大语言模型的规模，本质上并不会让它们在理解用户意图方面表现得更好。例如，大语言模型可能会生成不实、有害或对用户毫无帮助的内容。换句话说，这些模型与用户的需求并不契合。在本文中，我们展示了一种通过利用人类反馈进行微调，使语言模型在广泛任务中与用户意图保持一致的方法。我们从一组标注员编写的提示以及通过 OpenAI API 提交的提示入手，收集了一个数据集，其中包含标注员对模型期望行为的示范，我们利用这些数据通过监督学习对 GPT-3 进行微调。随后，我们收集了一个模型输出排名的数据集，并利用这些数据通过人类反馈强化学习对这个经过监督学习微调的模型进行进一步优化。我们将最终得到的模型称为 InstructGPT。在针对我们提示分布的人工评估中，尽管 13 亿参数的 InstructGPT 模型参数数量比 1750 亿参数的 GPT-3 少 100 倍，但其输出却更受青睐。 此外，InstructGPT 模型在真实性方面有所提升，有害输出的生成有所减少，同时在公开自然语言处理数据集上的性能退化极小。尽管 InstructGPT 仍然会犯一些简单错误，但我们的结果表明，利用人类反馈进行微调是使语言模型与人类意图保持一致的一个有前景的方向。 https://arxiv.org/abs/2203.02155 前往小宇宙评论区与主播互动