Skip to content
How Data Platforms Affect ML & AI // Jake Watson // #207 episode artwork

Jan 26, 2024 39 min

How Data Platforms Affect ML & AI // Jake Watson // #207

from Agentic Conversations (formally mlops.community) Hosted by Demetrios

Jake Watson is the writer of thedataplatform.substack.com⁠ and Principal Data Engineer at The Oakland Group. MLOps podcast #207 with Jake Watson, Principal Data Engineer at The Oakland Group, How Data Platforms Affect ML & AI. // Abstract I’ve always told my clients and colleagues that traditional rule-based software is difficult, but software containing Artificial Intelligence (AI) and/or Machine Learning (ML)* is even more difficult, sometimes impossible. Why is this the case? Well, software is difficult because it’s like flying a plane while building it at the same time, but because AI and ML make rules on the fly based on various factors like training data, it’s like trying to build a plane in flight, but some parts of the plane will be designed by a machine, and you have little idea what that is going to look like till the machine finishes. This double goes for more cutting-edge AI models like GPT, where only the creators of the software have a vague idea of what it will output. This makes software with AI / ML more of a scientific experiment than engineering, which is going to make your project manager lose their mind when you have little idea how long a task is going to take. But what will make everyone’s lives easier is having solid data foundations to work from. Learn to walk before running. // Bio Jake has been working in data as an Analyst, Engineer, and/or Architect for over 10 years. Started as an analyst in the UK National Health Service, converting spreadsheets to databases tracking surgical instruments. Then continued as an analyst at a consultancy (Capita), reporting on employee engagement in the NHS and dozens of UK Universities. There, Jake moved reporting from Excel and Access to SQL Server, Python with frontend websites in d3.js. At Oakland Group, a data consultancy, Jake worked as a Cloud Engineer, Data Engineer, Tech Lead, and Architect depending on the project for dozens of clients, both big and small (mostly big). Jake has also developed and productionised ML solutions as well in the NLP and classification space. Jake has experience in building Data Platforms in Azure, AWS, and GCP (though mostly in Azure and AWS) using Infrastructure as Code and DevOps/DataOps/MLOps. In the last year, Jake has been writing articles and newsletters for my blog, including a guide on how to build a data platform: https://thedataplatform.substack.com/p/how-to-build-a-data-platform // MLOps Jobs board jobs.mlops.community // MLOps Swag/Merch https://mlops-community.myshopify.com/ // Related Links Website: https://thedataplatform.substack.com/ How Data Platform Foundations Impact AI and ML Applications blog: https://thedataplatform.substack.com/p/issue-29-how-data-platform-foundations AI in Production Conference: https://home.mlops.community/public/events/ai-in-production-2024-02-15 How to Build a Data Platform blog: https://thedataplatform.substack.com/p/how-to-build-a-data-platform --------------- ✌️Connect With Us ✌️ ------------- Join our Slack community: https://go.mlops.community/slack Follow us on Twitter: @mlopscommunity Sign up for the next meetup: https://go.mlops.community/register Catch all episodes, blogs, newsletters, and more: https://mlops.community/ Connect with Demetrios on LinkedIn: https://www.linkedin.com/in/dpbrinkm/ Connect with Jake on LinkedIn: https://www.linkedin.com/in/jake-watson-data/ Timestamps: [00:00] Jake's preferred coffee [00:26] AI in Production Conference teaser [02:38] Takeaways [04:00] Please like, share, and subscribe to our MLOps channels! [04:17] Data Engineer's Crucial Role [05:44] Jake's background [06:44] Data Platform Foundations blog [10:34] Data mesh organizational side of things [17:58] Importance of data modeling [20:13] Dealing with the sprawl [22:03] Data quality [23:59] Data hierarchy on building a platform [29:34] ML Platform Team Structure [31:47] Don't reinvent the wheel [34:04] Data pipelines synergy [37:31] Wrap up

Episode details come from the podcast's RSS feed.

Embed this episode

Ready to play

How Data Platforms Affect ML & AI // Jake Watson // #207

0:00 39:11

No transcript yet.

Requests help us decide what to transcribe next.

Frequently asked questions

How long is this episode of Agentic Conversations (formally mlops.community)?

This episode is 39 minutes long.

When was this Agentic Conversations (formally mlops.community) episode published?

This episode was published on January 26, 2024.

Can I download this Agentic Conversations (formally mlops.community) episode?

Yes. Use the download control on the episode player to save the publisher-provided media file.