EP119: HuggingGPT Turns LLMs Into AI Managers

HuggingGPT is a collaborative system that leverages Large Language Models (LLMs), such as ChatGPT, as a central controller to manage and integrate various expert AI models from machine learning communities like Hugging Face. The paper addresses the limitation of current LLMs in handling complex, multi-modal information (such as vision and speech) by using language as a generic interface to connect the LLM with external expert models.The system operates as an autonomous agent through a four-stage workflow:Task Planning: The LLM acts as the "brain" to analyze user requests, understand the user's intent, and disassemble the request into a sequence of manageable sub-tasks.Model Selection: The system chooses the most appropriate expert models hosted on Hugging Face based on their functional descriptions.Task Execution: The selected models are invoked to execute their specific sub-tasks, effectively handling any resource dependencies generated by previous steps.Response Generation: The LLM synthesizes the predictions and inference results from all the executed models to generate a comprehensive final response for the user.By combining the reasoning and planning capabilities of LLMs with the specialized expertise of multimodal models, HuggingGPT can autonomously tackle a wide range of sophisticated tasks across language, vision, and speech domains, paving a new pathway toward artificial general intelligence.