Expensive Observability: The Cardinality Challenge - OpenObservability Talks S3E02

We all collect logs, metrics and perhaps traces and other data types, in support of our observability. But this can get expensive pretty quickly, especially in microservices based systems, in what is commonly known as “the cardinality problem”.   On this episode of OpenObservability Talks I’ll host Ben Sigelman, co-founder and the GM of Lightstep, to discuss this data problem and how to overcome it. Ben architected Google’s own planet-scale metrics and distributed tracing systems (still in production today), and went on to co-create the open-source OpenTracing and OpenTelemetry projects, both part of the CNCF. The episode was live-streamed on 12 July 2022 and the video is available at https://youtu.be/gJhzwP-mZ2k OpenObservability Talks episodes are released monthly, on the last Thursday of each month and are available for listening on your favorite podcast app and on YouTube. We live-stream the episodes on Twitch and YouTube Live - tune in to see us live, and pitch in with your comments and questions on the live chat.https://www.twitch.tv/openobservabilityhttps://www.youtube.com/channel/UCLKOtaBdQAJVRJqhJDuOlPg Have you got an interesting topic you'd like to share in an episode? Reach out to us and submit your proposal at https://openobservability.io/ Show Notes: The difference between monitoring, observability and APM What comprises the cost of observability How common is the knowledge of cardinality and how to add metrics Controlling cost with sampling, verbosity and retention Lessons from Google’s metrics and tracing systems Using metric rollups and aggregations intelligently Semantic conventions for logs, metrics and traces OpenCost project New research paper by Meta on schema-first approach to application telemetry metadata OTEL code contributions - published stats Resources: Monitoring vs. observability: https://twitter.com/el_bhs/status/1349406398388400128 The two drivers of cardinality: https://twitter.com/el_bhs/status/1360276734344450050 Sampling vs verbosity: https://twitter.com/el_bhs/status/1440750741384089608 Observing resources and transactions: https://twitter.com/el_bhs/status/1372636288021524482 Socials: Twitter: https://twitter.com/OpenObserv Twitch: https://www.twitch.tv/openobservability YouTube: https://www.youtube.com/channel/UCLKOtaBdQAJVRJqhJDuOlPg