Currents 088: Melanie Mitchell on AI Measurement and Understanding

Jim talks with Melanie Mitchell about her critique of applying standardized exams to LLMs and the debate over understanding in AI. They discuss ChatGPT and GPT-4's performance on standardized exams, questioning the underlying assumptions, OpenAI's lack of transparency, soon-to-be-released open-source LLMs, prompt engineering, making GPT its own skyhook to reduce hallucinations, the number of parameters in GPT-4, why LLMs should be probed differently than humans, how LLMs lie differently than humans, Stanford's holistic assessment for LLMs, a College Board for LLMs, why the term "understanding" is overstressed today, consciousness vs intelligence, the human drive for compression, working memory limitations as the secret to human intellectual abilities, episodic memory, embodied emotions, the idea that AIs don't care, calling for a new science of intelligence, the effects of differing evolutionary pressures, whether a model of physics could emerge from language learning, how little we understand these systems, and much more.