From Copilots to Culprits: The Legal Reality of AI | Ep. 20

Autonomous AI agents promise massive productivity gains, but with autonomy comes severe financial and legal risk. If an AI system deployed in banking falls victim to a prompt injection attack, it could falsify compliance records or authorize illegal transactions. A human might notice a mistake after one or two errors, but an AI could execute thousands of false transactions per second, potentially bankrupting a financial institution.ㅤHost Daniel Kazani sits down with Patrick Munro to examine the legal implications of this technology. Patrick shares a cautionary tale about the dangers of disabling safety features and granting full autonomy without human oversight. They discuss the difference between human-in-the-loop systems and autonomous agents from a legal perspective.ㅤThe conversation also touches on data sovereignty, the challenges of running on-premise servers, and why the healthcare sector remains hesitant to adopt automated tools. Daniel and Patrick highlight the need for a pragmatic approach to risk and the importance of establishing clear usage policies.ㅤGuest BioPatrick Munro is a technology lawyer operating at the intersection of artificial intelligence, cybersecurity, and IT regulatory compliance. He serves as Legal Counsel for Financial Services at Capgemini and as Of Counsel at the IT law boutique planit//legal. Patrick acts as a dual Subject Matter Expert for AI and Cybersecurity, actively developing legal technology tools like compliance dashboards and contract review assistants. He believes that AI gives lawyers better tools to focus on judgment-intensive work rather than replacing them completely.ㅤWhat We CoverThe financial risks of agentic AI and the dangers of prompt injection attacks.How a single hijacked AI agent could impact compliance records and sanction screenings.The monoculture risk of multiple banks deploying the same vulnerable AI models.Legal liability differences between human-in-the-loop decisions and fully autonomous smart contracts.Personal liability implications for managing directors under the NIS2 Directive.Data privacy concerns with US hyperscalers versus the logistical hurdles of on-premise AI servers.Why highly regulated sectors like healthcare are hesitant about AI adoption due to strict criminal implications for data leaks.The need to define clear use cases and usage policies before implementing AI within a company.ㅤResources MentionedCapgeminiplanit//legalSoftupMINDMASH MunichAnthropic Claude and Claude CodeCOBOLAmazon BedrockMistralNIS2 DirectiveDigital Operational Resilience Act (DORA)EU AI ActEU Omnibus Proposals