Better Text Generation With Science And Engineering

Current text generators, such as ChatGPT, are highly unreliable, difficult to use effectively, unable to do many things we might want them to, and extremely expensive to develop and run. These defects are inherent in their underlying technology. Quite different methods could plausibly remedy all these defects. Would that be good, or bad? https://betterwithout.ai/better-text-generators John McCarthy's paper "Programs with common sense": http://www-formal.stanford.edu/jmc/mcc59/mcc59.html Harry Frankfurt, "On Bullshit": https://www.amazon.com/dp/B001EQ4OJW/?tag=meaningness-20 Petroni et al., "Language Models as Knowledge Bases?": https://aclanthology.org/D19-1250/ Gwern Branwen, "The Scaling Hypothesis": gwern.net/scaling-hypothesis Rich Sutton's "Bitter Lesson": www.incompleteideas.net/IncIdeas/BitterLesson.html Guu et al.'s "Retrieval augmented language model pre-training" (REALM): http://proceedings.mlr.press/v119/guu20a/guu20a.pdf Borgeaud et al.'s "Improving language models by retrieving from trillions of tokens" (RETRO): https://arxiv.org/pdf/2112.04426.pdf Izacard et al., "Few-shot Learning with Retrieval Augmented Language Models": https://arxiv.org/pdf/2208.03299.pdf Chirag Shah and Emily M. Bender, "Situating Search": https://dl.acm.org/doi/10.1145/3498366.3505816 David Chapman's original version of the proposal he puts forth in this episode: twitter.com/Meaningness/status/1576195630891819008 Lan et al. "Copy Is All You Need": https://arxiv.org/abs/2307.06962 Mitchell A. Gordon's "RETRO Is Blazingly Fast": https://mitchgordon.me/ml/2022/07/01/retro-is-blazing.html Min et al.'s "Silo Language Models": https://arxiv.org/pdf/2308.04430.pdf W. Daniel Hillis, The Connection Machine, 1986: https://www.amazon.com/dp/0262081571/?tag=meaningness-20 Ouyang et al., "Training language models to follow instructions with human feedback": https://arxiv.org/abs/2203.02155 Ronen Eldan and Yuanzhi Li, "TinyStories: How Small Can Language Models Be and Still Speak Coherent English?": https://arxiv.org/pdf/2305.07759.pdf Li et al., "Textbooks Are All You Need II: phi-1.5 technical report": https://arxiv.org/abs/2309.05463 Henderson et al., "Foundation Models and Fair Use": https://arxiv.org/abs/2303.15715 Authors Guild v. Google: https://en.wikipedia.org/wiki/Authors_Guild%2C_Inc._v._Google%2C_Inc. Abhishek Nagaraj and Imke Reimers, "Digitization and the Market for Physical Works: Evidence from the Google Books Project": https://www.aeaweb.org/articles?id=10.1257/pol.20210702 You can support the podcast and get episodes a week early, by supporting the Patreon: https://www.patreon.com/m/fluidityaudiobooks If you like the show, consider buying me a coffee: https://www.buymeacoffee.com/mattarnold Original music by Kevin MacLeod. This podcast is under a Creative Commons Attribution Non-Commercial International 4.0 License.