Beyond Copilot: Building AI That Works in the Real World with Azure AI Foundry and Agentic AI for Frontline Workers with Fergus Kidd [MVP]

In this episode of the M365 Show, Mirko Peters speaks with Fergus Kidd, Microsoft MVP, Co-Founder and CTO of FieldPal AI, about moving enterprise AI beyond the desk. While much of the conversation around Microsoft Copilot, generative AI and productivity focuses on knowledge workers, Fergus makes the case for the people maintaining infrastructure, repairing equipment, carrying out inspections and keeping essential services running. These frontline teams often work with limited connectivity, limited screen time and an overwhelming amount of paperwork—yet they have some of the clearest opportunities for meaningful AI impact.FROM EXOMARS TO EDGE AIFergus shares how his early work on the camera software for the ExoMars rover helped shape his approach to AI. A rover operating far from Earth has to work with constrained hardware, communication delays and a need to make decisions close to where the work happens. Those principles are surprisingly relevant for today’s field workers: technicians at wind farms, engineers on industrial sites and service teams on the road cannot rely on a stable connection or a laptop at every moment. The conversation explores why edge and on-device AI can be faster, more resilient and more useful when it supports people directly where the job is done.WHY FRONTLINE WORKERS NEED A DIFFERENT AI EXPERIENCEA frontline worker may not have an email address, a Microsoft Teams account or the time to type detailed updates into a tablet. They are fixing machinery, inspecting a site, installing windows or dealing with a customer. Traditional business software often fails because it adds another complicated tool to learn instead of removing friction from the work itself. Fergus explains why voice-first interaction, simple mobile experiences and rugged wearable computers can make the difference between a proof of concept and a tool people genuinely want to use.WEARABLE COMPUTE, VOICE AND THE REALITY OF THE FIELDThe episode looks at practical wearable technology from providers such as RealWear and Vuzix. Rather than treating smart glasses as a futuristic novelty, Fergus describes them as wearable computers that can give workers hands-free access to information, cameras and voice controls. A technician can speak to an AI assistant, call a manager through Teams, capture evidence, scan a code or document a repair without stopping the physical task. The key is not putting technology in front of people—it is making the technology disappear into the workflow.WHY HOLOLENS, VR AND CONSUMER GLASSES STRUGGLEDFergus and Mirko discuss why many highly visible mixed-reality and virtual-reality projects did not become standard enterprise tools. Cost, hardware fragility, difficult app integration and unclear business value all matter. In industries where a device can be dropped, damaged or used while wearing PPE, a premium headset with a steep learning curve may be the wrong answer. The conversation highlights the importance of choosing technology that integrates with enterprise data, SharePoint, Azure and existing operational processes rather than creating a disconnected consumer experience.TURNING CONVERSATIONS INTO BUSINESS VALUEFieldPal AI was built around two recurring problems: frontline workers need to look up information, and they need to complete reports. Instead of asking an installer or technician to type long forms after a full day on the road, the platform enables a short natural conversation at the point of work. The AI can capture key details, structure reports, identify missing information and ensure photos, serial numbers and job data are collected when they are still available. This does not only save time—it can improve data quality and prevent the expensive rework caused by incomplete or inaccurate paperwork.AGENTIC AI: MORE THAN A CHATBOTAgentic AI is the foundation that turns a simple chatbot into a useful operational system. Fergus explains how different agents can handle distinct tasks—such as searching a knowledge base, retrieving live data from an API, taking notes or generating a report—while an orchestrator presents one simple conversational interface to the user. For a garage, an agent may retrieve parts and pricing information. For a wind-farm inspector, it may access safety procedures and inspection workflows. The worker does not have to understand the architecture; they simply ask for help and continue their work.BUILDING WITH AZURE AI FOUNDRYFergus explains why FieldPal AI is built on Azure AI Foundry and Azure AI services. The platform combines models, speech capabilities, Azure AI Search, retrieval-augmented generation, APIs, Kubernetes and other Azure services to create a full product rather than a single assistant experience. Azure AI Foundry gives the team the flexibility to build an end-to-end application for workers who may never use Teams every day, while still keeping open the option to connect the same backend capabilities to Copilot Studio and Microsoft 365 in the future.COPILOT STUDIO OR AZURE AI FOUNDRY?The discussion makes an important distinction: Copilot Studio is powerful when the workforce already lives in Microsoft Teams and Microsoft 365. For many frontline scenarios, however, the user may need an AI assistant in a headset, a custom tablet app or even a phone call rather than inside Teams. Azure AI Foundry offers the flexibility to build those specialised experiences. The two approaches are not competitors in every situation—an organisation can use Azure AI Foundry as the intelligence layer and surface it through Copilot Studio where that makes sense.SMALL LANGUAGE MODELS, SEARCH AND TRUSTWORTHY ANSWERSOne of the strongest insights in this conversation is that bigger is not always better. FieldPal AI uses smaller language models combined with Azure AI Search and carefully scoped customer data. Instead of asking a general model to answer anything about an air-conditioning problem, the system retrieves the relevant approved documentation and presents a focused answer. This reduces irrelevant responses, helps control hallucinations and keeps the AI centred on the organisation’s actual knowledge. The trade-off is that information architecture and content quality become essential.AGENTIC RAG AND CONNECTED ENTERPRISE KNOWLEDGEFergus describes an agentic RAG approach where Azure AI Search retrieves relevant information and agents use tools to access the right sources. Depending on the scenario, that could include SharePoint repositories, APIs, databases or custom connectors. Different agents can switch between knowledge retrieval, notes and reporting in the same session. This is where enterprise AI becomes operational: it does not merely generate text, it connects people with the right data and helps them complete real tasks.COMPUTER VISION AND MULTIMODAL AI IN THE FIELDComputer vision has immediate practical value for frontline work. OCR can capture long serial numbers without requiring a technician to read them aloud, while barcode and QR scanning can quickly identify equipment and retrieve the right records. More advanced visual quality checks—such as verifying a window installation from a photo—are possible but require significant high-quality training data. Fergus discusses a pragmatic middle ground: use multimodal models to assess an image against clear criteria while keeping humans in the loop for decisions that need judgement and accountability.GOVERNANCE, SECURITY AND RESPONSIBLE DEPLOYMENTAI operating in the real world must be built on a secure and governed foundation. Fergus explains why FieldPal AI is hosted in Azure and relies on Microsoft’s security, monitoring and platform services. The episode also reinforces that AI quality begins with the information provided to the system. A good model cannot compensate for poor data, unclear prompts or missing content ownership. Organisations need to think about data access, relevant knowledge sources, secure integrations and the right level of human oversight from the startBecome a supporter of this podcast: https://www.spreaker.com/podcast/m365-fm-modern-work-security-and-productivity-with-microsoft-365--6704921/support.