Why Enterprise RAG Systems Still Produce AI Hallucinations

Podcast: Don’t Panic! It’s Just DataGuest: Michael Marolda, Senior Product Marketing Manager for Agentic RAG at Progress SoftwareHost: Shubhangi Dua, Podcast Producer and B2B Tech Journalist at EM360TechGenerative AI has been brewing in the enterprise tech industry for at least three years now. AI pilots are launching every other day, internal copilots are deployed across enterprise divisions, and now teams themselves are experimenting with large language models (LLMs) to automate business workflows. Such additions have sped up research and notably improved productivity.While the excitement is valid, the truth beneath is often disregarded. Many enterprise AI systems produce answers that sound convincing, even when they are completely wrong.In the recent episode of the Don’t Panic! It’s Just Data podcast, Michael Marolda, Senior Product Marketing Manager for Agentic RAG at Progress Software, sat down with host Shubhangi Dua, Podcast Producer and B2B Tech Journalist at EM360Tech.Marolda argued that the problem is not necessarily with the AI models themselves. The real issue is with the enterprise data foundations supporting them.“Your AI is only as good as the knowledge it has access to,” Marolda explained during the conversation. The question is what the gap is alluded to in the AI enterprise tech space.Also Read: Build vs. Buy: The Reality of Production-Grade RAGWhat’s the Hidden Risk Costing Enterprises?According to Marolda, around 80 per cent of enterprise data remains unstructured. This includes PDFs, contracts, emails, audio files, presentations, scanned documents, videos, and handwritten notes. This is the kind of information that traditional AI systems struggle to process reliably.While enterprises are heavily investing in AI infrastructure and model testing, many still do not have systems capable of organising, retrieving, and validating this scattered knowledge. The outcome often turns into a situation where AI tools begin to generate responses without the necessary business context, despite excellent prompt engineering.“We’ve seen enterprises rush into AI implementations,” Marolda said. “But many pilots fail to scale because the information isn’t grounded in real business data.” It ultimately poses major operational risks for companies, especially in highly regulated industries.During the podcast, Marolda mentioned a high-profile case involving an airline chatbot that provided customers with incorrect policy information, leading to legal consequences for the company. The issue was not due to malicious intent or a technical failure at the model level — it was due to unreliable data grounding.For enterprises using AI in customer service, HR, legal operations, finance, or internal knowledge systems, such errors are not rare. In fact, they’ve become a governance issue.Is Modern RAG the Solution?Enterprises tend to rely on data lakes as centralised storage for vast amounts of information. However, Marolda makes a point about how storage is no longer enough in the age of AI. “A data lake is just cheap storage,” he explained. “A knowledge layer is what actually activates that information for AI.”This difference is increasingly important as enterprises move from testing to operational AI deployment. Traditional storage systems can hold documents, but they cannot interpret relationships between data points, retrieve context semantically, or validate AI-generated outputs against source material.An enterprise knowledge layer, on the other hand, is designed to fill that gap. Marolda tells Dua that modern retrieval-augmented generation (RAG) systems can process unstructured data, apply optical character recognition (OCR), convert speech to text from video and audio, and build semantic connections across enterprise content.This enables AI systems to retrieve not just documents, but highly specific pieces of contextual information, including paragraph-level citations and timestamped video references.For enterprise leaders, the implications are significant. Rather than viewing AI as a separate assistant, enterprises are increasingly seeing AI as a retrieval and reasoning layer built on top of their knowledge ecosystems.How Should Enterprises Prioritise Efficiency Over Hype?The economics of AI was a critical discussion Marolda had with Dua. He noted that while many AI providers continue to push for higher token consumption and larger workloads, enterprises such as Progress Software are now beginning to value efficiency instead.Unlike NVIDIA’s enterprise philosophy, as proposed by its CEO Jensen Huang, is a new compensation model where engineers receive annual AI token budgets worth half their base salary on top of regular pay. During a live interview on the All-In Podcast, recorded in San Jose, California, in March 2026 at Nvidia's GPU Technology Conference (GTC), Huang stated: "If a $500,000 Engineer Did Not Consume At Least $250,000 Worth of Tokens, I'm Going To Be Deeply Alarmed."“We’re actually trying to reduce token consumption,” he explained. Such an approach contrasts with broader industry trends focused on maximising AI use at scale. As enterprise AI budgets become more established, CIOs and CFOs are scrutinising infrastructure costs, energy consumption, and long-term operational sustainability.It’s particularly relevant as enterprises pit multiple LLMs against each other for quality, relevance, and cost efficiency. According to Progress’s Sr. Product Marketing Manager, the next phase of enterprise AI adoption won’t be driven by model capability alone. It will be guided by practical governance, meaning identifying which systems produce the best results at reasonable costs.Overall, successful AI adoption is not just about selecting the right model but, in fact, pivoting towards building the right knowledge architecture.For instance, enterprises continue to invest in generative AI; the enterprises that thrive may be the ones that can effectively structure, govern, retrieve, and validate their institutional knowledge.Key TakeawaysEnterprise AI hallucinations increase without grounded enterprise data.Agentic RAG helps enterprises reduce AI hallucinations and improve accuracy.Unstructured data is the biggest challenge in enterprise AI adoption.Enterprise knowledge layers improve AI governance and traceability.AI token reduction lowers enterprise AI infrastructure costs.RAG architecture helps enterprises scale trustworthy AI systems.Chapters00:00 Introduction to Enterprise AI and Knowledge Layer02:13 Challenges with Unstructured Data in AI08:11 The Importance of a Knowledge Layer12:04 Trust and Governance in AI Solutions16:48 Progress's Unique Approach to AI Solutions19:15 Agentic RAG: A New Paradigm in AI Retrieval24:52 Real-World Applications of Agentic RAG26:39 Maintaining Quality and Performance in AI Systems28:01 Key Takeaways for IT Decision MakersFor more enterprise AI, Agentic RAG, data governance, and enterprise knowledge layer insights, follow Progress Software across its official channels:Website: Progress SoftwareYouTube: @ProgressSWLinkedIn: Progress SoftwareX: @ProgressSWFor more