528: Goldfish Chunks

Tyler Hoffman returns to the show to discuss diagnostics and observability data in embedded systems. We catch up on his life after startup acquisition, explore the hows and whys of keeping product data separate from operational data, and consider the realities of fleet management at scale.  Tyler is the co-founder of Memfault. Memfault was acquired by Nordic Semiconductor about a year ago. While Nordic has nRF Cloud as a smaller scale solution for Nordic devices (~100 devices), Memfault will continue to maintain support for non-Nordic platforms as well. During the discussion, Tyler advocates for a "device-in-control" philosophy, emphasizing that edge devices should retain the intelligence to manage their own firmware updates and telemetry. We also discuss the practical constraints of remote fleet debugging, outlining why tools built for high-bandwidth web infrastructure will quickly bankrupt an IoT company, and identifying exactly when a project is too low-bandwidth, or too small, to justify an external observability platform. Christopher shares his recent experiences with Memfault which leads to a discussion of chunks, flash memory buffers and MDS. The Memfault Diagnostic Service (MDS) is a standardized way for BLE devices to send the chunk payloads to a gateway device (mobile phone) which can then forward the data to the Memfault cloud. If you want a deep dive into the reasoning around starting Memfault, Tyler was on Embedded.fm episodes 390: Irresponsible At the Time and 395: I Can No Longer Play Ping Pong. Reaching back into the archives, Elecia, Tyler, and Phillip Johnston were on the Memfault Coredump Sessions podcast, a special crosspost with Embedded.fm, episode 451: From Concept to Launch  You can also find technical deep dives on Memfault's Interrupt blog. "What we do makes a difference and you have to decide what kind of difference you want to make."  – Dr. Jane Goodall, Reason for Hope: A Spiritual Journey. Transcript