Skip to content
Building A Community And Technology Stack For Scalable Big Data Geoscience At Pangeo episode artwork

Mar 28, 2022 52 min

Building A Community And Technology Stack For Scalable Big Data Geoscience At Pangeo

from The Python Podcast.__init__ Hosted by Tobias Macey

Summary Science is founded on the collection and analysis of data. For disciplines that rely on data about the earth the ability to simulate and generate that data has been growing faster than the tools for analysis of that data can keep up with. In order to help scale that capacity for everyone working in geosciences the Pangeo project compiled a reference stack that combines powerful tools into an out-of-the-box solution for researchers to be productive in short order. In this episode Ryan Abernathy and Joe Hamman explain what the Pangeo project really is, how they have integrated a combination of XArray, Dask, and Jupyter to power these analytical workflows, and how it has helped to accelerate research on multidimensional geospatial datasets. Announcements Hello and welcome to Podcast.__init__, the podcast about Python’s role in data and science. When you’re ready to launch your next app or want to try a project you hear about on the show, you’ll need somewhere to deploy it, so take a look at our friends over at Linode. With the launch of their managed Kubernetes platform it’s easy to get started with the next generation of deployment and scaling, powered by the battle tested Linode platform, including simple pricing, node balancers, 40Gbit networking, dedicated CPU and GPU instances, and worldwide data centers. Go to pythonpodcast.com/linode and get a $100 credit to try out a Kubernetes cluster of your own. And don’t forget to thank them for their continued support of this show! So now your modern data stack is set up. How is everyone going to find the data they need, and understand it? Select Star is a data discovery platform that automatically analyzes & documents your data. For every table in Select Star, you can find out where the data originated, which dashboards are built on top of it, who’s using it in the company, and how they’re using it, all the way down to the SQL queries. Best of all, it’s simple to set up, and easy for both engineering and operations teams to use. With Select Star’s data catalog, a single source of truth for your data is built in minutes, even across thousands of datasets. Try it out for free and double the length of your free trial today at pythonpodcast.com/selectstar. You’ll also get a swag package when you continue on a paid plan. Your host as usual is Tobias Macey and today I’m interviewing Ryan Abernathy and Joe Hamman about Pangeo, a community platform for Big Data geoscience Interview Introductions How did you get introduced to Python? Can you describe what Pangeo is and the story behind it? What is your role in the project/community and how did you get involved? What are the goals of the project and community? What are the areas of effort and how are they organized? What are the scientific domains that Pangeo is focused on supporting? What are the primary challenges associated with data management and analysis in these scientific communities? What are the forms that these data take and how have they been evolving? (e.g. formats/sources) What are some of the challenges introduced by the widespread adoption of cloud resources and the associated architectural patterns? Can you describe the technical components that fall under the Pangeo umbrella? How do they come together to form a functional workflow for geo sciences? How has the scope of the Pangeo project changed or evolved since it started? What are the most interesting, innovative, or unexpected ways that you have seen Pangeo used? What are the most interesting, unexpected, or challenging lessons that you have learned while working on Pangeo? When is Pangeo the wrong choice? What do you have planned for the future of Pangeo? Keep In Touch Joe @HammanHydro on Twitter Ryan @rabernat on Twitter rabernat on GitHub Website Picks Tobias Mountain Biking Ryan Klara And The Sun by Kazuo Ishiguro Joe Range by David Epstein Closing Announcements Thank you for listening! Don’t forget to check out our other show, the Data Engineering Podcast for the latest on modern data management. Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes. If you’ve learned something or tried out a project from the show then tell us about it! Email [email protected]) with your story. To help other people find the show please leave a review on iTunes and tell your friends and co-workers Links Pangeo Pangeo Forge CarbonPlan M2LInES LEAP Columbia University XArray MIT MatLab PHP Ruby Java NumPy SciPy Matplotlib C Fortran Perl Dask Data Engineering Podcast Episode Jupyter IDL HDF5 Unidata NetCDF CF Metadata Conventions Intake Podcast Episode FSSpec Parquet Data Engineering Podcast Episode Zarr Data Engineering Podcast Pangeo Forge Airbyte Data Engineering Podcast Episode Fivetran Data Engineering Podcast Episode Stitch TileDB Data Engineering Podcast Episode Pythia The intro and outro music is from Requiem for a Fish The Freak Fandango Orchestra / CC BY-SA

Episode details come from the podcast's RSS feed.

Embed this episode

Ready to play

Building A Community And Technology Stack For Scalable Big Data Geoscience At Pangeo

0:00 52:08

Chapters

  1. 1:45 Introduction to Ryan Abernathy and Joe Hammond
  2. 2:57 Ryan's Journey with Python
  3. 4:11 Joe's Introduction to Python
  4. 5:18 Overview of the Pangio Project
  5. 8:21 The Birth of Pangio
  6. 10:35 Challenges and Community Building
  7. 14:41 Core Elements and Extensions of the Pangio Stack
  8. 19:10 Scientific Domains Using Pangio
  9. 24:05 Managing Source Formats and Data Abstraction
  10. 27:29 Cloud Computing and Its Impact on Geosciences
  11. 33:54 Modern Data Stack and Scientific Data
  12. 36:14 Exploring Cloud Native Storage Formats
  13. 38:37 Evolution and Future Goals of Pangio
  14. 42:06 Innovative Uses and Success Stories
  15. 44:00 Lessons Learned in Community Building
  16. 45:47 Future Plans for Pangio
  17. 48:17 Community Resources and Education
  18. 50:03 Picks and Recommendations

No transcript yet.

Requests help us decide what to transcribe next.

Frequently asked questions

How long is this episode of The Python Podcast.__init__?

This episode is 52 minutes long.

When was this The Python Podcast.__init__ episode published?

This episode was published on March 28, 2022.

Is there a transcript available for this episode?

Yes, a full transcript is available for this episode. You can read the complete transcript on the episode page.

Can I download this The Python Podcast.__init__ episode?

Yes. Use the download control on the episode player to save the publisher-provided media file.