The Analytics Power Hour

What do diamond ring shopping, Uber pricing psychology, and active user metrics gone wrong have in common? They all highlight our complicated relationship with precision versus accuracy—and how that relationship can either build or destroy trust in our data. Arik Friedman from Atlassian joins us to unpack why being “about right” often beats being “exactly wrong,” and why your nagging feeling that something’s off might be a useful insight in and of itself. From the discipline of documenting assumptions to the art of knowing when to round your numbers, we tackle the very human challenge of working with data that’s supposed to be objective but rarely is. Plus, we explore Twyman’s Law (if data looks too good to be true, it probably is) and why sometimes your intuition is your last line of defense against embarrassing mistakes. This episode is brought to you by Prism from Ask-Y—your agentic analytics platform for automating analytics, exploring data, creating repeatable workflows, and delivering accurate insights—all without the need for manual query writing. Links to Resources Mentioned in the Show (Free Books: separate R & Python versions) An Introduction to Statistical Learning (Video) Hamel Husain | The Revenge of the Data Scientist | PyAI Conf 2026 (X Article) The Revenge of the Data Scientist (Video) Lenny’s Podcast: Why AI evals are the hottest new skill for product builders | Hamel Husain & Shreya Shankar The Monarch App (Article) Vibe Coding Will Bite You. Here’s Exactly Where… by Cassie Kozyrkov A Checklist for Making Better Decisions: A Glossary of All the Topics We’ve Featured on Choiceology Photo by Sarah Kilian on Unsplash Episode Transcript00:00:00.00 [Tim Wilson]: Welcome to the Analytics Power Hour. 00:00:08.92 [Announcer]: Analytics topics covered conversationally and sometimes with explicit language. 00:00:15.20 [Tim Wilson]: Hi everyone, welcome to the Analytics Power Hour. This is episode number 290, oh wait, no, wait, let me check, this is episode number 296. 00:00:26.04 [Tim Wilson]: That was a close one. 00:00:28.32 [Tim Wilson]: I mean, how embarrassing would that have been? We’re all set to have a discussion about data accuracy and getting business partners to trust the data and I almost whiffed on something as simple as the episode number. Already though, perhaps I’ve undermined your trust in me, Tim Wilson, sitting in the host chair that is usually occupied by Michael Helbling. Perhaps I have. Let’s ask my co-hosts. Moe Kiss from Canva, welcome to the show. What do you think? Did my little gaff destroy my credibility? 00:00:57.68 [Moe Kiss]: Yeah, trust is completely lost. No, I’m joking, you nailed it. 00:01:02.44 [Tim Wilson]: It’s been lost, it was lost years ago with you, so, and Julie Hoyer, from further, have I already destroyed our audience’s confidence in me? 00:01:14.16 [Julie Hoyer]: What’s a mistake among friends, right? I’m just your pre-read. 00:01:17.20 [Tim Wilson]: Okay, I like it. Well, that’s the topic for this episode. I mean, sort of. All data can be digitized and an overwhelming amount of it is and digitized data breaks down into like discrete little ones and zeros somewhere down the chain. It should be cold and objective and to use it effectively, we need to get it to be as accurate and precise as possible, right? I mean, well, maybe. So joining us for a discussion on just that topic is Arik Friedman. 00:01:48.88 [Tim Wilson]: Arik started his career as a software engineer, then took a turn and did a PhD in computer 00:01:54.28 [Tim Wilson]: science focused on privacy-preserving data mining, which is a tongue twister, then went on to work a few years as a program manager for Microsoft R&D, then he popped back over to research for a while, and now he’s been at Atlassian for over 10 years where he is currently a senior principal data scientist focused on product analytics. And today he is our guest. 00:02:18.40 [Tim Wilson]: So welcome to the show, Arik. 00:02:21.52 [Arik Friedman]: Thank you very much. Long time listener. Happy to be here. And I assure you, like all of these career moves made a lot of sense, like it, it made a lot of sense and at the time, it was a logical progression at the time. Absolutely. 00:02:36.34 [Tim Wilson]: I went architecture, technical writing, marcom analytics. So I, I sympathize and I actually, I don’t know that any of those made any sense. 00:02:45.84 [Tim Wilson]: But that’ll be something for me to ponder on my deathbed, but so Arik, you presented 00:02:53.40 [Tim Wilson]: at the last measure camp in Sydney and in that presentation, I think you had a, a fun or maybe it’s relatable at least, or maybe it’s a tragic story about an active users metric that looked like almost too good. Like the results were brilliant. Everyone was thrilled, but you had this like nagging feeling that something was off and you couldn’t really nail down the specific problem. So maybe that’s a good way to sort of start the show. Cause I think it gets to something really human about our relationship with data. So maybe we’ll kick things off by having this, having you walk us through kind of what happened 00:03:28.88 [Tim Wilson]: there. 00:03:29.88 [Arik Friedman]: Sure. So like this is a story from a while back, but you know, at the time the product team was working on a product feature, a big change in the user interface and a lot of investment went into that. So we put a lot of work on actually testing things, you know, A, B testing. I worked with another data scientist to make sure that we got things right. And I remember back then, you know, us standing in the boardroom with all the PMS and it’s looked like really a big change. Like it moved the needle quite significantly. So that was, that was a big deal. 00:04:11.24 [Tim Wilson]: Everybody were happy. And then as we went on and rolled out this feature, we saw what we expected to see. Active users went up. 00:04:22.96 [Arik Friedman]: But then, you know, over time, I’m starting to develop this, you know, old feeling, you know, you look at data over time, you get a sense of how it looks, how growth looks 00:04:34.28 [Tim Wilson]: like. 00:04:36.04 [Arik Friedman]: And I was looking at the curse and they looked a bit suspicious, right? Like it’s not supposed to go like that. And I started looking into that, okay? Because it felt a bit odd. And I remember, you know, trying to go down to specific trace logs, trying to find what went wrong, I found like everything checked. So you know, after putting some time into that, I said everything was good. We actually have a reason to believe that, you know, active users goes up, all is fine, right? I cannot say anything about this. I don’t have a smoking gun. Fast forward, sometime later, another data scientist and a software engineer, of course, we keep looking into this and they actually found a bug. 00:05:23.32 [Tim Wilson]: It turned out that because of a bug, it caused inflation of active usage monitoring. 00:05:30.00 [Arik Friedman]: And that was an unpleasant surprise for the product team. So for me, it definitely caused a lot of thinking, you know, like, how come I didn’t find that? 00:05:40.24 [Tim Wilson]: What could I have done different? 00:05:43.00 [Arik Friedman]: And that, you know, that causes a lot of reflection. 00:05:47.60 [Tim Wilson]: Well, did you go tell them, like, immediately, abruptly, did you ease it out? Like how did you actually communicate, like, what’s the rest of the story on that? 00:06:01.08 [Tim Wilson]: Yeah, so I think that’s probably the biggest mistake I made at the time because, you know, 00:06:08.36 [Arik Friedman]: I didn’t find an issue. And you know, I, at least back then, I thought, you know, we are the data people, we are the evidence people. And if we don’t have the data to back up what we say, we should shut up. And what I learned from that is actually, you know, our intuition are, you know, that’s part of our expert opinion. 00:06:31.08 [Tim Wilson]: And we should sometimes just go with it. 00:06:36.12 [Arik Friedman]: And I think there are a lot of things we can do ahead of time, you know, to prevent mistakes or to check things. But at least for me, like, for this story, like, when I look, you know, what could I’ve done different, I actually knew to do all the checks. And eventually, when your intuition is your last line of defense, and sometimes you just have to go with that. 00:06:58.04 [Moe Kiss]: So sometimes, I don’t know, I, firstly, I just want to say thank you. I’m really grateful that you’re sharing, you know, straight off the bat, an experience of where you made a mistake that’s incredibly humble and wonderful for folks to be able to learn from. So a very big thank you. And I suppose I just, I’m curious to understand this intuition piece, right? Like I feel we all have it. And I know you and I have had conversations previously about like when we’re making decisions, you know, we have data, we have intuition, we have, you know, maybe previous experience, we have all these different factors and part of our role is helping pull those things together. But I’m curious to understand. So how has this changed how you would show up? Like, let’s say this happens next time, you can’t find a smoking gun, you can’t find a data point, but you had your, you know, your intuition in your gut that’s like something’s not right here. Like, do you think this time you would say something and how would you frame it? 00:07:49.92 [Arik Friedman]: Yeah. 00:07:50.92 [Tim Wilson]: I think, first of all, it’s about being more opinionated. 00:07:56.88 [Arik Friedman]: And in the, like over time, that’s part of my growth journey. I think in the past I would, I tended to be a bit more, you know, impartial, right? We let the data speak. Like we’re just there to give the data its voice. And I think over time I feel a bit more confident, you know, to be opinionated, like I have my own opinion of things. And I think we’re actually expected to be opinionated. So I think that’s, that’s one thing, just like be more confident in our own expertise. And then I think it’s probably, I mean, the opinion on its own is not enough, right? Like a probably, even if we have just this intuition, we will still be expected to, you know, okay, dig into that. Okay, can you actually find the evidence, but I think it’s a start of a conversation, right? 00:08:50.52 [Tim Wilson]: Like this is what I think. Maybe the data looks a bit odd and we maybe want to dig a bit more into that. 00:08:59.16 [Arik Friedman]: And then the business can decide, you know what, no, like we did all our checks and balances. It’s good. We did our due diligence. 00:09:05.20 [Tim Wilson]: Let’s go on. 00:09:06.36 [Arik Friedman]: Or they could say, you know, let’s, let’s take more time and actually give it more space 00:09:10.24 [Tim Wilson]: to dig into that. 00:09:13.00 [Arik Friedman]: And at the time I basically made my own decision and said, you know, I looked enough into that and I kept it with myself. And this is something where we need to be more open and just share what we think with a business partners about that. 00:09:28.08 [Tim Wilson]: So I will sometimes find myself not doing this necessarily with a ton of structure and rigor, but trying to think what I expect to see before I actually look at the data, or sometimes asking if I really don’t think I have context and my business partner says, I think it’ll go up. I’m like, what is, like, what does up mean for this metric or what is, what would that mean? Like I don’t give me, use it as an opportunity to try to get a little more context about their intuition about their business, like there’s, there’s some psychological trick or something where you, you sort of force yourself to set what you think you’re going to see as opposed to waiting until you see it and then you immediately rationalize why it makes sense. Like I had an example from years ago where a product marketing manager came racing in because he had already told the whole company that this tiny little change he’d made to a webpage had like drastically driven the track traffic up, which made no sense. And that was one where I dug in and found that it was, it was a bot, there was a company that was selling us software and they pointed literally at his page to gather some data for the pitch in, in a few days. But it was one where it was like, he saw it, he was a good news, kind of like the active users. But if I’d gone back, if I’d known he was making that change, I would say, what do you 00:10:59.28 [Tim Wilson]: realistically think this is going to do? 00:11:03.24 [Tim Wilson]: Because if it goes way past that, maybe we want to, you know, apply a little bit of climate law or, you know, something to it. But I think there’s, I don’t know, I feel like that’s a whole, it’s a challenge, especially for people just entering a field or the space to have that intuition and to have the confidence on top of it. Like you said that a few times that it’s like, as you get more experience, the more faith you have in your intuition and your conviction. And that’s like, how do you, how do you develop that if other than letting time pass? 00:11:42.00 [Arik Friedman]: I think a lot of this is getting to live a bit in your business partner’s world, like getting to speak the language and, you know, getting to know what they care about, what they’re after. I think that adds a lot of context. And for example, I remember at some point, you know, I started hearing PMs talk all 00:12:01.48 [Tim Wilson]: the time about like JTBD, JTBD this, JTBD that, like jobs to be done. 00:12:06.72 [Arik Friedman]: I had no idea what it was all about. And, you know, I started digging into that, reading a bit about that. And, and, you know, what I found, like, I didn’t really get what they were talking about. So getting into the world, learning about that, I think helps create this common language and thinking about it from their perspective. So I think that definitely helps us, you know, to get, so to speak, like in their head. 00:12:34.20 [Tim Wilson]: When that was one where you actually, if I, I had this happen with OKRs once where I was working with someone who was, and I knew what OKRs were, I’d lived in kind of 00:12:43.72 [Tim Wilson]: an OKRs training, but it was like, oh, this client is really into it. 00:12:49.12 [Tim Wilson]: I’m going to go get a book. And didn’t you do that? Didn’t you wind up going and like read, like whatever the JTBD Bible is? Like, OK, if we’re going to use this, I want to understand it. I mean, you probably found out that they actually weren’t using it correctly. It’s like Agile or any other number of things that use the acronym, but maybe aren’t applying the process. 00:13:10.36 [Arik Friedman]: Yeah. So I went on and I actually read about jobs to be done. And I, it was actually weird because like what I was reading and what I was hearing from the PMs were not exactly the same things. And at the time, like in our confluence, like internal wiki, I wrote a blog post. Hey, like, are we doing JTBD wrong? 00:13:32.04 [Tim Wilson]: And it was a bit of a clickbait, but it started a conversation 00:13:36.96 [Arik Friedman]: because I mean, then I started like through the conversation with them. I started to find out, oh, actually there are different kinds of definitions of JTBD out there and having this conversation and trying to understand, you know, what they really mean when they talk about that, that really helped. And I did it from the perspective of, you know, how it can be more useful as a data scientist and what is it that they actually after when they talk about understanding the JTBD. So I think it was first, you know, just to catch up with them, but also to see, you know, how can I answer their questions and how can I better understand the question and, you know, improve myself as a data scientist 00:14:18.68 [Tim Wilson]: that helps them. Michael, I have news. 00:14:23.48 [Tim Wilson]: The AI analyst is emerging. 00:14:27.08 [Tim Wilson]: Oh, that’s big coming from the quintessential analyst. What do you mean, like a cryptid? Well, I mean, more like a more like a job promotion, but, you know, 00:14:35.32 [Tim Wilson]: with more existential dread, you know, how foundation models created the AI engineer role. Yeah. Developers got all these cool titles and analysts got. Can you pull this by in today? Exactly. But now we’re watching the birth of the AI analyst, someone who uses LOMs to multiply their capabilities without, you know, multiplying their stress rash. Nice. So an analyst, but with superpowers and fewer open tabs. Exactly. And the tool for that is Prism by ask-y.ai. 00:15:07.92 [Tim Wilson]: Yeah. Prism is basically the interface between what I mean and the 900 steps I don’t want to do. You ask in plain English and it helps you get from question to analysis really fast. 00:15:19.72 [Tim Wilson]: And it doesn’t forget your world. Prism’s jam memory as J.A.M. their jam memory remembers your definition like what your means by conversion. Which table is the source of truth? And that July data, don’t forget, it’s, you know, cursed. 00:15:37.64 [Tim Wilson]: Yes. Thank you. I’m tired of explaining our metrics like it’s the extended 00:15:43.92 [Tim Wilson]: MCU universe. Plus, you can capture like repeatable workflows as skills, portable expertise, like, you know, clean UTMs or fixed GA for channel grouping or standardized campaign naming and reuse those across different data 00:16:01.84 [Tim Wilson]: sets. I like that because I do a lot of copy paste. This is can’t continue type of feeling. And I feel like it would be nice to be like run skill, look smart, drink water. 00:16:15.64 [Tim Wilson]: Nice. Want to become the AI analyst before your coworker does? Go to ask-y.ai and join the wait list. 00:16:24.16 [Tim Wilson]: Yeah. And use code APH and ask why it pushes you to the top of the wait list. That’s ask-y, the letter Y.ai and use code APH. Yeah. The AI analyst is here. This product is in beta, but you can get in on the ground floor. And it’s coming for your busy work, not your job. 00:16:44.64 [Moe Kiss]: So, you know, relax, chill out. So, Arik, you and I have spent a lot of time talking about this whole like accuracy versus precision playoff. And it’s something that has just really resonated with me because I would say I always lean to kind of like the best possible answer with the time that we have in the business to help make a decision. Like, I suppose I would say I’m like quite pragmatic, but a lot of what I hear coming from data scientists is this number is wrong. This one’s right. We can’t do it this way. This, you know, we have to do it this way because this is the right way. And I guess I just wanted to hear your framing about this like playoff between accuracy and precision. And like, I don’t know, you have such an elevated way of thinking about it. 00:17:31.40 [Arik Friedman]: Yes. So I think like straight from primary school, the way we’re taught things is that accurate and precision are like being accurate and precise are the same things, right? So at school, you get like this big equation. And like in the data science world, you can think about a business question, 00:17:51.20 [Tim Wilson]: like, you know, what, what kind of features correlate with user satisfaction 00:17:55.96 [Arik Friedman]: or something like that? Or how can we predict those kind of things in parallel? 00:18:00.60 [Tim Wilson]: Like, like at school, you might get a question, like, you know, 00:18:06.24 [Arik Friedman]: how much is like 2,124 times 3,926? Like you get this big equation. And what you’re taught is that you need to go through the methods, right? Like you have to multiply this digit by that digit, carry over. If you get anything off, like any single digit is off, you lose all your marks. So if your answer is not precise, it’s not accurate. You lose all your points. And I think we kind of like carry on this mindset with us into our jobs. And in the business domain, actually precision and accuracy are not the same thing because, you know, because, oh, like these big numbers, it’s about, you know, 2k times 4k, it’s about 8 million. And like about 8 million is accurate. It’s not precise, but it’s accurate. And I think that also, like when we get, you know, business questions, there are many ways we can go and approach and solve them. You know, we can throw them, I don’t know, like hidden Markov models or, you know, clustering algorithms. So there’s all this arsenal of like all these methodologies that we learned. And like, you know, that’s kind of like even part of our pride in the craft, right? Like we want to show that we know to do all those things. But sometimes you can get a quite a good answer just by writing like a very simple SQL query. And, you know, it’s maybe not the best answer, but it’s good enough and it’s faster. And there is this quote from John Tukey from like his, he had like this paper about the future of data analysis. 00:19:43.44 [Tim Wilson]: And like this is from like the sixties. 00:19:46.44 [Arik Friedman]: And he says, like far better and approximate answer to the right question, which is often vague, than an exact answer to the wrong question, which can always be made precise. 00:19:56.92 [Tim Wilson]: And I think it really captures well that, you know, it’s first of all, 00:20:02.28 [Arik Friedman]: it’s better to answer the right questions. And I think that part of our job is to help get to those right questions. But also when we answer, it’s not just about precision. It’s, you know, the answer can be accurate without being precise. 00:20:18.24 [Tim Wilson]: And I think that’s a way for us to be more fast and effective 00:20:23.24 [Arik Friedman]: and be focused on business impact. 00:20:24.84 [Tim Wilson]: But there’s there’s a there’s a challenge, right? 00:20:27.56 [Tim Wilson]: That that our business partners accessing tools and data, they’re working with spreadsheets, they’re working with dashboards because they’re in this digital world and processing and multiplication is is is cheap. Like they are conditioned to see things that are precise. I mean, I love the like the the accuracy versus precision. Like a dark archery thing that sort of shows accurate the grid, the two by two of accurate and precise, accurate, not precise, precise, not accurate, not accurate, not precise. And it shows like the spread around the target. 00:21:07.80 [Tim Wilson]: And I think that’s good for us to be aware of. And part of it is, I mean, can that be handled a little bit with the language 00:21:17.72 [Tim Wilson]: of saying, try to get yourself out of the boat of providing. Precise answers, like get comfortable with even if you have a precise answer, better to kind of back off the precision of it a little bit so that somebody isn’t looking at you’re not arming them with ways to. To find another person, it may be too reasonably accurate, 00:21:44.64 [Tim Wilson]: but the precision means that the the first three digits kind of differ. 00:21:49.92 [Tim Wilson]: Like there’s there’s a challenge with us understanding that and our business partners thinking that way. And the levers we can pull to try to help them think in language of. Of this is this is accurate, you know, it’s about it’s about eight million, 00:22:08.88 [Tim Wilson]: you know, or whatever the answer is. 00:22:11.92 [Tim Wilson]: And they’re probably not going to object to that. They’re not going to come back and say, what do you mean about a million? 00:22:16.08 [Tim Wilson]: I need that down to the to the first digit, right? I mean, it’s the final line to walk. 00:22:22.76 [Arik Friedman]: Yeah, and I think it really depends on the context and the question being cast because in some contexts, like if you deal with financial data, you definitely want to be precise. It’s important, but most like a lot of the business questions are more direct directional in nature. And when you deal with directional questions, it doesn’t matter as much. You know, if you’re precise and actually precision can kind of draw the attention to the wrong thing because you don’t care about the position necessarily in these kind of questions. So this is about getting to the bottom of what are the decisions that they are about to take 00:22:58.80 [Tim Wilson]: and what really matters for answering those decisions. 00:23:02.36 [Moe Kiss]: So one of the things I wanted to like challenge and have a discussion on. So I’m trying to find a specific article. And of course, I can’t. But there was some research that was done with pricing and Uber. And I often quote it when I’m counselling people on buying engagement rings. Buying a two-carat diamond ring is more expensive than a one point nine nine nine carat ring, right? Like we have this bias and heuristic to think that the two carat, the rounded number is better. So hold on. 00:23:30.16 [Tim Wilson]: Just wait, wait a minute. How many people are you counselling on buying rings? Like, what is your world? 00:23:37.52 [Moe Kiss]: I my team are quite young. And so a lot of them are going through that life stage. 00:23:43.76 [Tim Wilson]: Those are just one on ones. They’re like, hey, what’s going on? I could really do some help. 00:23:48.36 [Moe Kiss]: Anyway, OK, carry on. So also, I don’t know who’s buying your two-carat diamond ring. 00:23:54.52 [Tim Wilson]: It’s a it’s a anyway. 00:23:59.32 [Moe Kiss]: Is that that’s a that’s a big one? Three carat is quite big. Yes, that that’s not for America, but for Australia. Americans tend to buy much bigger rings. Anyway, back to the point of the story. Well, and now the lab, you should see these rings. 00:24:12.16 [Julie Hoyer]: People are thrown around. 00:24:13.76 [Tim Wilson]: Sorry. You guys are living in a different world. 00:24:18.44 [Moe Kiss]: But what I was going to back to my point, one of my concerns is like, I hear you and I agree with everything that you’re saying, right? But especially, OK, the same thing happens when you’re trying to counsel people on like a day right. Sometimes like people will intentionally like make a number, not a round number, because people are more inclined to believe it, that you’ve put more work into it, because we have this bias that say if you say $1,600 a day, that like it’s not it’s you you’ve just picked a number from the sky versus having put thought into it. So like I’m just trying to think about how that plays off with our discussion here about not necessarily always wanting that precision. So you’re saying if we don’t go precise enough, 00:25:04.72 [Julie Hoyer]: they won’t trust the number. 00:25:06.48 [Moe Kiss]: Yes. So if we say, oh, the number is eight million versus eight million two thousand and ninety seven, will people just be like, is that bias going to come to life? Where they’re like, oh, maybe that number is not right. And like I can kind of disregard it because it’s come from nowhere. Or like, do you see what I’m saying about like, does the precision give any extra credibility or build more trust? 00:25:29.84 [Arik Friedman]: Right. So in those cases, like precision sometimes is kind of like the signal that you did the work. Right. And again, I think these kind of things are context dependent because 00:25:42.84 [Tim Wilson]: ideally, if you have established a good like 00:25:47.72 [Arik Friedman]: trust trust relationship with your business partner, then they trust your judgment and said you made the call the right call about what’s a good methodology to approach this. And in some cases, yeah, maybe precision does matter. Like maybe it’s a signal that’s important to them. And in that case, yeah, maybe you do need to put a bit more work 00:26:11.44 [Tim Wilson]: and and give that kind of signal. 00:26:15.60 [Arik Friedman]: So that definitely is context dependent. 00:26:18.28 [Julie Hoyer]: And part of the context, I’m just thinking like, Tim, like back to some of our discussions we’ve had with clients when they have really low volume, right? The difference of one is greater than if you have a large volume. The difference of 10 is like minuscule or, you know, like you can kind of ignore it. So I do feel like, too, it’s like, are you speaking of a really small end? Because then you don’t want to round compared to when you’re speaking large volumes, you can round to eight million and it’s probably OK. The other thing with precision that I think is interesting as we get sucked into a lot is when the stakeholders want to compare systems, numbers from systems, which we know are not going to give you the exact same one. Like directionally, they should be close enough. But I do feel like that is a scenario where they really get stuck on. Like, why is this one percent different? And it’s like, well, it’s OK. 00:27:16.12 [Tim Wilson]: It’s like eight million. 00:27:18.56 [Julie Hoyer]: It’s all right. And we’re not talking dollars and you know, which one’s your source of truth. So we should be OK to continue to use, you know, the other number in our decision making. But that’s really hard, I think, for stakeholders and even for analysts 00:27:33.68 [Tim Wilson]: to navigate. But I think we tend to make, I mean, if it’s like, you’re going to err on one side or the other. I mean, I look at somebody makes a line chart to show the results and they label every single data point down to the first value. And that’s not so like there’s there there are choices made in the presentation of the information where I think you can say there’s precision precision there. I’m showing a dashboard, but this is rounded to the nearest million or the nearest thousand. It’s not I’m not giving you eight zero zero zero zero zero zero. I’m saying eight million, you know, or eight point one million. You know, give them a give them a decimal place. So I think there are choices. I mean, certainly are to your point that like the context and actually Julia point as well, like the context does matter. When does it matter? But I I feel like there’s like the trap we can fall into on the analytics side is we have the precision precision might as well show it. You know, our P value is not only is a less than point oh five. It’s every model spits out. It’s the P values point oh oh oh one three four seven two. Like please don’t put that, you know, just kind of paste. Yeah. P values less than point oh five. 00:28:51.64 [Arik Friedman]: And Julia, I think, by the way, it’s a great point because it matters a lot like why the numbers are different. And, you know, scenario one, the numbers are different. We have no idea why. OK, that’s that’s not a good place to be. And that’s that’s a place where you probably do want to ask, you know, questions why why do these things don’t match versus a different scenario where, you know, we actually expect those numbers not to be perfectly the same because maybe we measure slightly different things or, you know, there are known reasons why they should mismatch. So I think usually it’s more about the confidence. Like we know what’s going what’s going on here and team to your point. Yeah, like maybe if we think that this will just distract and raise, you know, questions that are not relevant, then yeah, maybe it’s better off to reduce the position to just avoid that altogether. 00:29:44.04 [Tim Wilson]: So I think the other I think business our business partners and then data teams get sucked into it when it’s the different systems aren’t going to reconcile and I think to your point, like you should understand not down to the perfect we can net out everything, but say these are the the big movers of the differences. A lot of times there’s the ability to say over time, look, these move in the same direction, I watch companies say, well, we just need to pick the system of record for that metric, which feels like my the hairs on the back of my neck go up. Like that’s a cool idea that somebody in the C-suite or one level down said, we have the solution. We just pick our system of record. But the reality is all those systems exist for different reasons and it gets back to context. So I think there’s a part that says we need to put this to rest at some point by doing a little bit of an analysis of saying these differ. These are the main reasons they differ. Let us show you that they generally move in the same direction over time. And now we’re just going to have our standard little footnote that these, you know, move differently. But I think also we’re going to go look at the system that gives us the most what we need because it’s often kind of like upstream system in some process has this data, has this data, hands off to another system, which goes downstream from that. So you have to pick the system where you can get the slice. I mean, I’m thinking in a CRM digital analytics CRM sales world, you can necessarily track the marketing channel all the way through to a sale. And in the middle, you’ve got something like a lead that the downstream ability to follow it comes out of one system. The upstream to follow it comes from another system and heading off and understanding, and maybe this gets back to your point earlier. You’re like, you really need to be figuring out is the question being asked very clear and precise and let that drive everything as to where you look, what level of precision, how you pull it, what the intuition is. And now it just went on a bit of a tirade. 00:32:05.88 [Julie Hoyer]: Well, no, but you bring up a point that, you know, maybe this is me still like wrapping my head totally around the accuracy versus precision. But I do keep thinking of what you were calling out like the target. So we were talking about people being most comfortable with precision of systems matching, but we just discussed, you know, thoroughly, like why that’s, we know it doesn’t happen. But to your point, Tim, if they are, if we can explain in at least broad strokes, Arik, as you said, like why these things are differing. And if we then can say, well, as long as we’re pointing this at the right like target, the right question, the right problem to be focused on, then can we be comfortable with accuracy without precision between the tools? Like, am I taking this way off? But you’re saying, you know, like, if they’re always about 1% off, they’re 00:33:03.28 [Tim Wilson]: all hitting the right target, maybe not closely bunched together, that would 00:33:08.16 [Julie Hoyer]: make people happy with accuracy and precision. 00:33:14.08 [Arik Friedman]: Yeah. So I think that data science teams usually have a good opportunity to collaborate and aim to standardize measurements. And, you know, one example, like I’ve seen in the past where, you know, growth teams and product teams had different measurements of impact. 00:33:31.52 [Tim Wilson]: And that can be very confusing because, you know, when one team says, oh, 00:33:35.68 [Arik Friedman]: like we had that impact, and then the other team kind of like interprets it in their own language. So I think definitely for data scientists, we can be in a role where we, you know, coordinate between our teams to see, you know, can we actually standardize and measure things the same way and ensure that we actually get the same numbers all over the place. But as you said, like sometimes it’s not possible, right? Sometimes, you know, there are actually very good reasons why things should be different. 00:34:08.60 [Tim Wilson]: And maybe then we should just use different names for them. 00:34:11.96 [Arik Friedman]: So it’s very clear that, you know, when this team talks about something, it’s not the same thing that the other team talks about. So I think we definitely have a role to play with this. 00:34:20.88 [Tim Wilson]: So can we shift a little bit, because some of this talking about kind of getting trust in the data, and it feels like there is a, there’s a big one we covered, which is like, we’ll get people to trust in the accuracy. We’ve covered two things. If I’m going to mid-show recap, which we don’t really do recaps and I don’t know what the hell I’m doing. But we’ve got kind of the develop and trust your intuition. We’ve got the think about accuracy and precision differently. We haven’t actually covered like, how do you not push bad info out? And the intuition is kind of like one hook into that, but I like the frenetic pace everybody in modern business is working in. There is a drive to get the task completed off my desk over the wall and into the hands, which just like begs for mistakes to be made. And the cost of making a mistake with the query, with the report is of damaging like trust. So like how, what other like ways do we have? We, we, we briefly referenced like time is law, which really goes to the intuition and doing checks, but like, what are other ways to maintain trust in the data from our business partners? How do we prevent ourselves from undermining trust? It’s like a long ramble with a broad question. 00:36:01.00 [Arik Friedman]: Yeah. 00:36:01.36 [Tim Wilson]: I think first thing and something that I definitely would like to see people do 00:36:08.16 [Arik Friedman]: more is just stop and ask, does this make sense? And I don’t think people do that enough. And it’s like at every stage, right? Like, you know, you look at the raw data, does it make sense? Is there anything off there? If you apply methodology, does it make sense in this business context? You get a certain outcome. Does this outcome make sense? So I think this is kind of like a first line of defense. You know, does it make sense? 00:36:38.60 [Tim Wilson]: But aren’t there, there are times, there are times where it makes sense, but it’s actually wrong. Like if you’re like, oh, it made sense. Therefore, I’m going to go full steam ahead. And then you find out later that. Oops. Just because it passed that hurdle, it was plausible, but. 00:36:57.44 [Moe Kiss]: But you’re making the best decision you have with the information available at the time. I feel like the dust doesn’t make sense. Actually, it’s just a like, it’s a spot check, right? I actually think the thing that we need to do better at is. Does it make sense with our stakeholders? Like actually bringing them into some of those checkpoints versus kind of like, does it make sense to me individually? 00:37:21.44 [Arik Friedman]: Absolutely. I mean, I guess like this is the first check that you need to do with yourself before you engage with others. And I think that’s already a checkpoint that I think can probably spot some of the issues. Beyond that, yeah, I mean. Involving other people in this thought process. And like in general, you want to do peer review process for everything. And by the way, I’m coming from a computer science background. My brain is wired as a software engineer. So basically any time that I do anything that involves statistical modeling or math, I will always pull someone else that has like, you know, strong. Math jobs and statistical modeling, you know, hey, check by work. Like, did I actually apply the methodology correctly? Because I know that they think about this in a different way than I do. And they can spot things that I want. 00:38:17.40 [Tim Wilson]: Will the question come up in that context? Because you said like, does the raw data make sense? And from like the EDA of saying, I’ve made my initial query. 00:38:30.12 [Tim Wilson]: Now I’ve got a table of data that has say 25 columns and 150,000 rows. 00:38:40.72 [Tim Wilson]: Do have I gone through and checked that one is 150,000 rows seem like the right number of rows? Like that’s like a saying to like, is the, does the size of the data make sense? But going kind of checking for gaps, distributions, the values, like checking the columns, doing kind of a, doing that step of saying, this should be the right data, but have I actually done, have I checked all the values, all the variables to see if the distributions and values are reasonable? Like not just trust the query, because that adds time and it’s probably a judgment. 00:39:25.36 [Tim Wilson]: It depends on how solid and simple the SQL is. 00:39:29.24 [Tim Wilson]: Does that make sense? And would somebody, if you’re having, if you’re running it by, is this the right model, how likely are they to say, did you double check that the data that you’re 00:39:38.64 [Tim Wilson]: feeding into this model is the data that you think it is? 00:39:43.80 [Arik Friedman]: Yeah. And I think it’s probably both those things. And it depends also if it’s like, is this the first time I’m answering this kind of question or not? And I’ll give us an example. Like there were cases, you know, at some point we worked on, say, internal developer effectiveness metrics, and we tried to understand, you know, flow of work and things like that. And this was the first time we went to calculate these kind of things. So, you know, I worked on this, there was another data scientist 00:40:15.16 [Tim Wilson]: calculating the same metric in parallel, and we got completely different results. 00:40:22.32 [Arik Friedman]: And the thing is that none of us was wrong. I mean, technically, none of us, we didn’t make any mistakes or technical mistakes in the process, but we started working through the calculations step by step and really as we realized that, oh, we actually made different assumptions about what the data meant, and we made different assumptions about what we wanted to measure. So just by working through this process until the numbers matched, it allowed us to align on the assumption and it gave us the confidence that we were actually measuring the right thing. And so, like, it’s definitely not something you will do with any metric or any calculation that you do, but definitely, like, the first time that you do something, it’s good to have, like, this cross check. And once you got these right ones, okay, the next time you already know what you’re doing and you don’t need the same level of, you know, due diligence. 00:41:18.56 [Tim Wilson]: So Julie’s, like, lighting up on the, like, oh, yeah, yeah, assumptions. Like, that’s another thing we haven’t, we’ve talked about it in the past, but the discipline of documenting the assumptions that you made, and it may be, like, the list of assumptions I made were these 20. If I’m having another analyst review it, I’m going to show them all 20. When I present this to my business partner, maybe I even pre-read it, I’m like, these are the three or four that I want to make sure they understood that I made these assumptions. Like, we’ve talked about that with, as a practice of analytics of you are making these decisions and just, like, making an assumption and moving ahead is, like, pretty dangerous actually having in the process a place to write down those assumptions, one from a repeatability, somebody checking your work, but also to actually go back to the business and actually include that in the results. Not the exhaustive, here’s how we made the sausage, but, hey, I just want you to know, going into this, and if that’s your opening slide and they say, well, that’s a bad assumption, then you actually screwed up because you should have done a pre-read with, hey, before we present this, these are like the assumptions we made and here’s why we made them. But how much, how much do you think the business do that? 00:42:41.20 [Moe Kiss]: That they actually, like, I feel like assumptions are included, they’re often, like, at the bottom of the slide, in the text somewhere, and, like, the business just glosses over them. Like, I would love a business stakeholder to be like, and I’ve had that at previous companies where, like, I’ve had very senior folks, like the founders or CEO, be like, I don’t agree with this assumption, like, let’s, let’s debate this, let’s talk it out. But I don’t see, and I would love to see more of that. 00:43:06.72 [Tim Wilson]: I mean, I find it as a way to, you know, on the building trust to not have that in the delivery, but it’s like, I mean, say there’s an executive you’re presenting to, but there’s this business partner that you’re actually collaborating with, it’s a great way to throw in slack, say, hey, is it safe to assume this and have them say, yes, is it safe to assume that? 00:43:29.20 [Tim Wilson]: And then I, I sometimes will put those assumptions, like, front and center. Like, I want to be confident that, and, but, but I probably wouldn’t say I made 00:43:39.36 [Tim Wilson]: the assumption, I would say we made the assumptions. And hey, just so you know, we assumed, you know, that, that holiday has no impact on our bottom line yet. No, that’s, that would be a bad assumption to make. But I mean, it’s not, it’s not just like a CYA of like, throw this in the, in the footnotes, it’s, I think it’s as much of a show that you’re trying to understand the context and the operating environment, which is building trust upstream, which also means that there will be more trust in the output. No? 00:44:14.60 [Julie Hoyer]: Honestly, though, I feel like sometimes that type of documentation and assumptions are like, definition stuff too. I’m with Yuma, like it’s refreshing and encouraging when you are stakeholders key in on it and understand the value of it or want to debate it and can call it out if it’s wrong. But like, honestly, I think a lot of times it is more useful for yourself or others to replicate the work later on, or if you have to build on the current work. 00:44:42.92 [Tim Wilson]: Like, I think sometimes if you’re not kind of obsessed with like documenting 00:44:48.88 [Julie Hoyer]: all of those things, I mean, I’ve been the analyst that’s been given a spreadsheet with random numbers, supposedly where they pulled it. And I have no other definitions or assumptions. And they’re like, oh, recreate this and do it again for the new time period. And you’re like, F my life, I’m never going to get this. And the person that made it originally is gone. So all that to say, like, sometimes if it’s not building trust directly with stakeholders, I do think it helps like maybe the analytics team broadly or the data scientist team broadly, more for the replicability is how I see it. 00:45:26.32 [Arik Friedman]: Yeah, I think definitely like documenting the analysis or the assumptions. And like, that’s probably not the document that you’re going to eventually show as your final result, but it’s probably good to always have like, here’s the technical document with all the details, all the assumptions. So it’s available there. And so you can reproduce this. And yeah, but like referring to most point, I think you have this conversation with your business partner is part of the process, right? Like you get a question. Like, here is how I understand it. This is my interpretation. This is how it translates to the methodology. So you definitely want to be sure that, you know, you’re on the same page and you’re actually answering the same question. 00:46:10.32 [Julie Hoyer]: The other thing I think of too, like if it does get put in the footnotes 00:46:14.92 [Tim Wilson]: of the document or this the presentation, like sent to your business stakeholders 00:46:20.76 [Julie Hoyer]: mode, like I have spent so much time talking to people and on my team or being so worried about myself, what happens when I give this deck to my stakeholders and then they pass it to their stakeholders and they pass it to their stakeholders and then they start asking follow up questions and nobody knows to ask me for the clarification. So like, honestly, I think my safeguard and anxiety is what makes me put a lot in the final documentation in the appendix or in the footnotes. Because I’m like, hey, if this gets passed like three degrees away from me, which again, that’s a win if your work gets passed that far along. 00:46:54.84 [Moe Kiss]: I see this happen all the time with experiment results where like it, you know, a data scientist has pulled something together and then someone like summarizes it and someone summarizes it for a slack message and like these steps further and further away, which look, do I want stakeholders to be able to interpret experiments and communicate it? Absolutely. Lutely. But I think sometimes that’s like just the the the understanding of like the core tradeoffs kind of gets watered down or there’s like different incentives and it just gets really tricky. Arik, I know like you’ve done a lot of thinking about this, about like once the analysis is out, the experiment result is out and you kind of lose control of the narrative, so to speak, like how do you approach that? 00:47:41.76 [Arik Friedman]: Yeah, so it definitely happens like sometimes you you get a chart or something that gets copy pasted and then someone puts it on a slide deck and it’s completely out of context and you lost that. And like one one thing that I try is again, like if you have a documentation 00:48:02.40 [Tim Wilson]: of your analysis, like you should definitely have a link to that. 00:48:06.28 [Arik Friedman]: So if it’s copy pasted, at least the link to the document is still there. And, you know, there’s the footnote in the chart where you can highlight the main details, so it’s definitely a question of a balance, right? Like you don’t want to overload your visuals with all the assumptions and all the details, it’s distracting, but at least have this kind of, you know, selective context or pointer to where the information is. So it kind of like travels together with the visuals. And, you know, sometimes, you know, you just lose control and you have nothing to do about that. But at least you can take some mitigating steps. So, you know, the evidence is there. Like if anyone is really like will want to find this information, there is a way for them to get there. So that’s the least we can do to at least control that. 00:48:59.24 [Tim Wilson]: Yeah. I mean, I think like pointing to like having the footnote of like, what was the what was the right with the source? I mean, you’re well stated that it’s like you don’t want to overload it with all the assumptions, but you want to provide the breadcrumb to say 00:49:12.88 [Tim Wilson]: and the more you can put that proximate to the chart. So they’re not likely, I mean, that’s pretty gross misbehavior 00:49:21.92 [Tim Wilson]: if somebody says, oh, there are footnotes on this slide, but I’m going to pull just the chart and drop it in an email. There’s a little bit of that’s on them if it gets its own legs. If you’re giving it like, hey, this is probably important reference information that should go along with it. I think that’s good. But this, I feel like we could go on for multiple hours, but unfortunately, we need to head to rap or we will lose trust with our audience by having a two hour episode, which when you’re called the analytics power hour, that would be a disconnect between data sources. No, that’s a that’s a stretch. But before we leave, the last thing we like to do on the show always is go around the horn and share a last call, something that might be of interest to our users. And Arc, you’re our first guest. Do you have a last call or maybe a couple last calls you’d like to share? 00:50:26.00 [Arik Friedman]: The first one is, I guess, a classic, the ICLR, like the introduction to statistical learning book, which is actually a free resource. And today, there’s also a version for Python introduction to statistical learning in Python. And, you know, at times when, you know, AI sucks all the attention, I think that actually going back to the basics, to the foundations is, you know, just as important as ever, if not more so. And I know that at least from my experience, even just going over the first chapters of the books, you know, linear regression, it’s like a very practical oriented book. So, yeah, that’s that’s a good recommendation that I usually provide. Like it landed for me. 00:51:12.96 [Tim Wilson]: They have an R version and a Python version with, like, examples on it. 00:51:16.40 [Arik Friedman]: Is that right? Yeah. So there are two versions of the book. The original one was with R. Oh, it’s ISLR and ISL. 00:51:22.44 [Tim Wilson]: And ISLB. Introduction to stuff. OK, gotcha. 00:51:24.68 [Arik Friedman]: And about the available at www.startlearning.com. And like I noted, at least for me, it landed a lot of concepts that I didn’t really get before. So I really recommend that. 00:51:36.80 [Tim Wilson]: And if I’m allowed a second and a second call, 00:51:41.84 [Arik Friedman]: which is maybe more related. So I saw recently an article from Hamel Kussein called Revenge of the Data Scientist, and it’s actually both available as a YouTube talk. It’s a Pi AI talk from March. And he also posted it as a Twitter article. And he actually talks about, I think it’s about, specifically about mindsets, you know, we have the mindset that you go with to, you know, in this agent-first world. And I think that actually his point is that the data scientists approach, their mindset, their core skills, like a exploratory data analysis, metric design, model evaluation, all these things are as critical as ever and try to translate really well to this world. So I definitely recommend giving this a read. And also Hamel Kussein and Trash and Carr had a terrific episode about AI evils in Lenny’s podcast. So that’s a great resource as well. 00:52:41.64 [Tim Wilson]: Awesome. So that was three, really, sort of. But those all sound amazing. I feel like I’ve read the first three chapters of like more. Those are the books I’m most likely to abandon, but still get a lot of value because it’s like chapter four, where I start to I’m like, what, we’re at the area under the curve. I’m like, oh, boy, oh, boy, the equations are getting pretty in-depth here. So I kind of want to check those out. Julie, what’s your last call? 00:53:12.40 [Julie Hoyer]: Well, my last call is accurate and precise. And I’m feeling like maybe I have called this one out before. But you know what? It’s such a good one that I’m just going to say it again. It’s called the Moenarch app. And I’ve actually been using it now for quite a while for my own family budgeting and financial like tool for the family spenders. And it’s really nice because you can connect everything directly into it. So on top of having all your different credit cards, bank accounts, you get really nice cash flow visuals. You can recategorize like any of your expenses that come through. And it will notify you like, hey, this is a recurring one or hey, this is one that’s not categorized. You want to come in and quickly do it. But budgeting is not easy to do on your own. And I feel like this is the first app and different thing I’ve tried that I can actually keep up with it. I can quickly get to it. You know, there’s no delay like I spent. It shows there by paycheck hits. It shows there. So it’s been awesome. You can also set a lot of your budgets and goals in the app as well. So like for all your different categories, you can customize categories or not. And then you can say, you know, I’m looking to spend X amount each month in each category. And it just, I feel like makes it a lot easier to actually live out the financial plan and budget that we’re going for. So check it out if you’re needing a tool. 00:54:44.20 [Tim Wilson]: Have you figured out a way to turn off the turn off the net worth? The words because because there are periods where I want to not look at that 00:54:51.48 [Tim Wilson]: when the yeah, I’m a monarch user. 00:54:56.44 [Tim Wilson]: So I’m a fan. 00:54:57.48 [Julie Hoyer]: But you are. So you guys, it must be good if Tim’s into it. If he approves of the visuals and the tools. 00:55:03.24 [Tim Wilson]: Yeah, judge, judge your trust in the source. Moe, what’s your last call? 00:55:11.20 [Moe Kiss]: I know we talk about her a lot. But Cassie, Cassie has her cough has a really great new. I mean, it’s just her regular newsletter, but she’s doing a couple of back to back newsletters on why the vibe coding will bite you. And here is exactly where and she’s talked through a few like text scenarios of where things have gone really wrong. Like prod systems being completely white, things like that. So definitely go have a listen to that. But I think the thing like the real takeaway is that the stories are all about the same thing, which is just like misplaced trust and the speed at which it computes. And so it’s not like the nobody got hacked. AI didn’t go rogue. It’s just like people let their guard down. And I think the thing that’s been on my mind the most, which she’s so she’s just so damn articulate, but it’s like expertise won’t save you guardrails might. And I think guardrails is the topic that just keeps looping on my brain at the moment. And so, yeah, I’ve really enjoyed that newsletter and she’s got a couple more coming out on the same topic. So check it out. 00:56:17.64 [Tim Wilson]: I think that series like motivated me to dive into some new vibe coding project specifically. So far, I’m safe. I haven’t crashed the podcast because that’s where most of them happen. Starting to wonder if Riverside, our podcast recording app might actually be leaning a little too much on vibe coding is it’s having some of the joys it’s been bringing us of late. But so I’m going to pander to mo here a little bit because with the new season of choiceology that Katie milkman came out with. Um, one of the things she has is this checklist, which is this mapping of now she said she had a couple of undergraduate students do it. And that means it’s like in a PDF for some insane reason, but she’s basically gone through and looked at like, what are the different topics like attribution bias or Dunning Kruger or left digit bias, like the stuff that her episodes cover, this kind of reverses it. And it’s a guide to broken down by these are all the topics of kind of 00:57:20.56 [Tim Wilson]: cognitive biases, um, then which are the episodes you can actually listen to. 00:57:26.52 [Tim Wilson]: So if you’re not a, um, Katie milkman choiceology completionist, like I am, but you’re like, I wonder if she’s ever had an episode about, um, you know, mean reversion neglect, then this little guide will pop you to it. It is comically in a PDF, which I’m like, this is great. You guys really work to format this thing to one page. And she’s about to start a new season, which means this, this of all things that should be a vibe coded website where it gets updated and maintained. This is it, but you know what the undergrads, they’ll learn. That’s how it was scoped academia. They’re going to do their little thing. So, um, with that, um, our thanks so much for coming on. I feel like this is a case where we actually have like the show prep documents that have like even more gold in it that we were not able to get to. So, so we will have a lot of fun with that content our own ourselves. We may figure out how to bring you back for more of that. So thanks so much for coming on. 00:58:31.64 [Arik Friedman]: Thank you very much. Awesome. 00:58:33.76 [Tim Wilson]: So if you, uh, listeners, um, we love to hear from you. So if you’d love to have you leave a review or rating on whatever platform you listen to us on, if you’d like a free sticker of the podcast, uh, you can go to analytics hour.io and request one. I will, if you’ve gotten this far in the episode and you think, wow, that was a smooth conversation and these guys are professional, I will just call out now that we have dealt with, uh, tornado warning. They led to a power outage and two young children and a dog sheltering in place and with one co-host, we’ve dealt with a busted internet, um, that has been busted for the entire episode. But of course the repair team showed up for that during the episode. Um, and we’ve dealt with various cases of people dropping off and returning and not even realizing that we were still recording the show. So I encourage you to stick around for the outtakes because there might be some real doozies in those. Um, I would like to Tony, please leave this in. Thank you so much. 00:59:39.92 [Tim Wilson]: Cause if you pulled this thing together, you’re a good on your mate. 00:59:45.28 [Tim Wilson]: Uh, so it’s been fun. It’s been a fun discussion. Uh, we’ve been at this for four hours to get this one hour pulled together. No, it hadn’t been quite that bad, but we would love to hear from you. We’d love to hear if you thought that the edit was pretty smooth. If you’ve got your own thoughts on how to build, maintain, recover, trust, uh, you can reach out to us on LinkedIn. You can reach out to us on the measure slack. You can just send us an email at contact at analyticshour.io. So for Julie, for Moe, for all of the conspiring mother nature and construction projects that tried to not allow us to record this show about trust and accuracy 01:00:31.72 [Tim Wilson]: and precision, keep analyzing. 01:00:34.80 [Announcer]: Thanks for listening. Let’s keep the conversation going with your comments, suggestions and questions on Twitter at, at analytics hour on the web at analyticshour.io, our LinkedIn 01:00:46.24 [Tim Wilson]: group and the measure chat slack group music for the podcast by Josh Crowhurst. 01:00:52.36 [Announcer]: Those smart guys wanted to fit in. So they made up a term called analytics. Analytics don’t work. 01:00:59.04 [Charles Barkley]: Do the analytics say go for it? No matter who’s going for it. So if you and I want to feel the analytics, they go for it. It’s the stupidest, laziest, lamest thing I’ve ever heard for reasoning in competition. Fuck my life. 01:01:15.48 [Moe Kiss]: So the work that’s being done out the front of my house. Knocked the NBN cable out. 01:01:24.48 [Tim Wilson]: I’m a rap now. 01:01:26.12 [Moe Kiss]: Oh, shit. Were we still recording? Oh, fuck my life. 01:01:33.84 [Tim Wilson]: I had to smoke this, but I was, I was trying to make so many notes that I, uh, on other aspects that I doubted. 01:01:40.56 [Moe Kiss]: There’s a lot going on today. Oh, my God. Well, you gave me so much. 01:01:46.56 [Tim Wilson]: I just, there was no writing. 01:01:48.88 [Julie Hoyer]: Nobody has the heart to tell you if I’m like, well, then the second one, one is like, so can I wrap? Oh, guys. All right, guys. 01:02:03.32 [Arik Friedman]: I definitely want to take Tony in advance because like, yeah, like, that’s probably some work to do here. Yeah. 01:02:16.12 [Julie Hoyer]: My hot mic was the least of our concerns. I don’t know if you would have thought, Jesus. 01:02:21.08 [Tim Wilson]: Yeah, but I mean, I think it’s, I think it’s going to come together well. 01:02:26.04 [Tim Wilson]: And my mid-show recap was like me organizing my thoughts because I was like, 01:02:29.76 [Moe Kiss]: I think we’re actually hitting on some, I actually feel like Arik, we need like another two hours with you because there’s so much stuff here. It’s such gold. 01:02:39.40 [Julie Hoyer]: Seriously. Yeah. You had so much in the show prep doc that I was like, oh, I want to talk about that. 01:02:45.32 [Moe Kiss]: Oh, and this is why I was like, I knew that the two of you would really like 01:02:48.56 [Tim Wilson]: Arik because like you’re the same with the very good at prep and 01:02:53.80 [Moe Kiss]: organization and all those things. I literally bought a new microphone and it’s still. Why have I still got this shitty one that doesn’t even have a proper stand and it still works great and everyone keeps buying all these fancy ones. It sucks. 01:03:08.16 [Julie Hoyer]: I got to return my hundred dollar one, I guess, and try a twenty dollar one. Maybe I need it to be less sensitive. 01:03:14.64 [Tim Wilson]: I think the show might need to buy you an audio interface. I think it’s I don’t don’t don’t don’t make any moves. Don’t do anything drastic. 01:03:25.12 [Moe Kiss]: Yeah, the audio interfaces and changing microphones. 01:03:33.04 [Tim Wilson]: It is because it’s moving where the preamp is. It moves it into its separate. It moves it from a little thing, crappy thing in the microphone. It then takes it out and puts it in a dedicated box. 01:03:44.40 [Tim Wilson]: OK. No, this is so above my head. 01:03:48.20 [Julie Hoyer]: I just want to buy a microphone that works. 01:03:59.24 [Tim Wilson]: Rock flag and accuracy versus precision. The post #296: Avoiding Major Oopsies: Twyman’s Law, Intuition, and Valuing Accuracy Over Precision appeared first on The Analytics Power Hour: Data and Analytics Podcast.

Research and analytics: are they more like peanut butter and chocolate, or more like oil and water? On this episode, we dig into the surprisingly common (and surprisingly unfortunate) divide between these two disciplines with Stefanie Zammit, Global Director of Analytics and Insights at Bang & Olufsen. Stefanie has spent her career bridging the qual and quant worlds, and she makes a compelling case that the best insights come from putting both methodologies to work on the same business problems. From the “never ask a survey question you already have the answer to” rule to why personas are usually terrible (spoiler: it’s not the clustering, it’s the storytelling), we explore how organizations can break down the silos between research and analytics teams. Turns out, the fear of the unknown and a bunch of fancy terminology might be keeping us from some pretty powerful insights. Also, apparently 100% soundproof rooms are absolutely terrifying. This episode is brought to you by Prism from Ask-Y—your agentic analytics platform for automating analytics, exploring data, creating repeatable workflows, and delivering accurate insights—all without the need for manual query writing. Links to Resources Mentioned in the Show The Practice of Market Research: An Introduction Training crucial leadership skills through serious games The Bang & Olufsen Factory Tour Women in Research (WIRe) Women in Product How Quantum Computing Works How to Make Sense of AI Photo by Vardan Papikyan on Unsplash Episode Transcript00:00:00.00 [Announcer]: Welcome to the Analytics Power Hour. Analytics topics covered conversationally and sometimes with explicit language. 00:00:13.24 [Michael Helbling]: Hi everybody, welcome to the Analytics Power Hour and this is episode 295. You know a common phrase we hear in our industry is that the data tells us what happened, but not necessarily why. And by and large that’s true. We keep getting better at inference and some patterns of data are pretty well understood in terms of their meaning, but still there is simply something so compelling about observing how people interact with the things we’ve built, websites, products, etc. Personally I remember what an eye-opening experience it was for me 15 years ago, the first time I was sitting in a usability lab watching from behind a one-way mirror as people use the website that I measured every day with my digital data. So we wanted to talk about it, get into the topic bridging between research and more traditional analytics. And I want to introduce my co-hosts, Val Kroll. Welcome. 00:01:12.56 [Val Kroll]: Hello. 00:01:13.56 [Announcer]: Hello. 00:01:14.56 [Michael Helbling]: And I know this is a special topic for you because if you’re background into customer research. 00:01:20.08 [Val Kroll]: Oh my gosh, doing back flips. Very excited for this. 00:01:22.08 [Michael Helbling]: Yeah, I’m excited too. And Julie Hoyer, welcome. Hello. Hi. Have you done much like market research or customer research? 00:01:32.04 [Julie Hoyer]: Not myself, but I have gotten a chance to like utilize the outputs of some of those studies, which has been nice. So I’m really excited to talk about it more. 00:01:40.36 [Michael Helbling]: Yeah. Excellent. All right. And I’m Michael Helbling. And to bring additional expertise to this topic, I’m pleased to introduce our guest, Stefanie Zammit. She is the Global Director of Analytics and Insights at Bang & Olufsen. Prior to that, she led research and analytics teams at companies like Starbucks and Marks and Spencer’s. She’s worked both as a consultant in the space for many years as well. And today she is our guest. Welcome to the show, Stefanie. 00:02:05.24 [Stefanie Zammit]: Hello. 00:02:06.24 [Michael Helbling]: Happy to be here. Awesome. We’re so glad to have you. And I think this is a topic that while we cover sort of like data and analytics, research is one of those things that we really like. And so we’re really excited when we met you to sort of dig into this topic more. But to kind of catch everybody up to speed, I thought it’d be great to kick off with just you explain a little bit about your background and career and kind of how it bridged these two things and sort of, you know, what your journey has been across research and analytics. 00:02:40.40 [Stefanie Zammit]: Yeah. Absolutely. I’m very much from a pure hardcore research background. That’s where I started my career too many years ago. I started as, actually, my first job was at university. I didn’t even know what research was. I took a part-time job doing the national student survey. That’s like a thing here in the UK. It’s how the university rankings are put together. And when I graduated, I think a lot of researchers would have a similar story to this, that they sort of ended up accidentally in the field. And so I graduated during a recession, dark times, and I was desperate. And I thought back to that part-time job I had at university and like, what is that? Like, is that an industry? Is that a thing I could do forever? So I did some research and I was in the UK at the time where, luckily, there’s an amazing research industry here. There’s so many great consultancies. It’s a thriving industry. I took my first job at a company called Quadrangle, which was a management consultancy that leaned very heavily on their own in-house research function. And it was amazing, an eye-opening, and I immediately fell in love with so many aspects of it. I then went to Ipsos, which is one of the big five. That’s where I was like, I need some hardcore research skills. I need to learn about statistics and hit me with the heavy quant stuff, go to a big powerhouse. So I was there for a couple of years. And then after that, I built my own. I co-founded a research consultancy at this time I was in the Middle East, was a company called Intelligence Qatar, which is still very much there today, although sadly, I don’t get to play a part in it anymore. And I think that was where previously at research consultancies, they really divide the teams up. So you had your market researchers, and then you had your analytics department with all the smart people, the statisticians were all there, then you’d have your fieldwork teams, your data processing teams. And it was very siloed, and even market research was pretty siloed. You’d have your qual team, and then your quant team, and something that was really hard for me as I progressed agency side, you’d have recruiters say to you, like, do you want a qual role or a quant role? And I struggled so hard to give them the answer because I genuinely loved both. And my favorite projects were the multi-phase projects where you’d get the best of both. But I was also super kind of looking over the shoulder of my analytics colleagues and statistician colleagues, what are you guys up to? What are you doing? So I always naturally was interested in the entire spectrum. And then when I had my own agency, again, really investing in the analytics side, as well as the research side just brought me closer to those worlds. And then finally went client side, I thought, all right, I had enough of this consulting game, joined Marks and Spencer’s. And I was really lucky that they had research analytics together in one department. That’s all I’ve known because it was like that in that first company. And we had a great leader at the time that was very adamant that the best deliverables are worked on by both of these teams. So I took that to Starbucks as well, where again, the organization is huge in Starbucks, you’re looking at around 250 people, but they are all in the same department together, research and analytics, if subteams, at least it’s still in the same family. So it’s something that I’ve been so passionate about today, and I think really set me up for success to do what I do now, which is to lead both the analytics function and the market research function in one team. 00:06:09.76 [Val Kroll]: I love that. The one thing that you mentioned there about not being able to pick which you liked better, the qual or the quant in your favorite, where the multi-phase projects that start with the qual and then lead into quant, I feel like that’s like a little bit novel, like a little bit inside baseball for researchers. Can you describe what that is because I have a follow up question after you talk about that a little bit, but just kind of describe like why someone would have like a multi-phased approach to like their research project? 00:06:40.60 [Announcer]: Absolutely. 00:06:42.20 [Stefanie Zammit]: So every methodology has its benefit. Qualitative research is where you start when you don’t know much about a subject and you need to explore. It’s very exploratory. You are using it to figure out where your hypotheses even are. You might have one or two hypotheses, but they’re fluffy and you need to explore the topic more. So qualitative is where you do that in this kind of limitless way of a very open data generation phase. And then you need to validate because you’ve spoken to like, I don’t know, max 40 people if you’re doing a really big qualitative project. So you want to validate that and you need some statistics and some numbers behind it. So you want to do your quant. So I have my hypotheses now, I’ll run a survey to measure and size those truths and see how statistically significant they are. And methodologically, these require two different expertise because qualitatively you’re trained in moderation, projective techniques, how to read between the lines, how to read people’s faces and emotions and hear what they’re not saying as well as what they are saying. There’s also a much more kind of deep psychology to interpreting those insights because, again, you’re reading between the lines. Quantitative, you need to know about driver’s analysis and cluster analysis. In fact, you need to know all of the statistical models that give you the derived insights that are so, so valuable from a survey. So they are usually kept separate. And I think that’s a shame because the best projects are the ones that do both of these things for a really strong final insight. 00:08:21.36 [Val Kroll]: Yeah, that was very well explained. And I remember in my market research days, a lot of times, if you think about if you were to do a survey, you have to, a quantitative survey, and you’re picking the list of options that someone is going to select that is the right answer to the question for them, sometimes that list isn’t always as clear, like what should belong in it, even if it’s a list of competitors. A lot of times, clients will think about in-category competitors, about who your competitors are for an alcohol brand. But that same dollar could be spent on other things that are out of category competition. And so you can use QAL to help explore to figure out what even is the list, because you could miss so much if you start just with a quant without going broad first, to be exploratory, like exactly, except Stefanie, to figure out what your hypothesis is. And the reason I want to dig into this a little bit is because this is one of the ways that I love helping illustrate or describe, especially to people describe, to people in the analytics, some of the value of bringing these worlds together. Because it’s not just using one single methodology or tool. It helps you illuminate a different part of your question or your process. And so how those two things come together very naturally inside of research is one of the ways you can kind of illustrate the coming together beyond just the research or the direct consumer or B2B context. 00:09:49.28 [Stefanie Zammit]: Exactly. And I think from a research-only perspective, I was already at a very early stage so powered up by the idea that if you put these two together, you’re getting better insights because you’re starting broad and then you’re getting specific with the quant. And I took that into as I gained more seniority in my career and started working, especially in-house, where you have colleagues in other disciplines, which is all data and insight. I took that into that as well to say, well, how can analytics be part of this? And why are we asking questions and surveys that we already have the answer to? That seems like a huge waste of time. How can we be using all these disciplines to get the best insight? And all of this ultimately comes down to a passion to chase the best insight, right? Methodology should be irrelevant to it’s not about the journey, it’s about where you end up. And I think having that crystallized in my mind from the very beginning really helped me see the world in the way that I see it now, that who cares what you’re trained in. At the end of the day, we use the best method to get the best insight. And it doesn’t matter whether that’s in this team or that team. 00:11:01.92 [Julie Hoyer]: And I feel like you are one of the more rare data leaders out there that recognize it has to be problem focused instead of leaders coming to their teams with ordering a solution. They have a problem, but they’re not always communicating that to the teams that are going to help service them. They’re kind of coming with, well, I want you to pull these numbers or ask these questions type thing instead of to your point just anchoring on, I don’t care how you do it, you guys are the experts in that part. But what I’m facing is problem X and I’m looking for ways to solve it. And then letting people get creative with how do they maybe partner together to get them the best solution? I feel like we just run into that at kind of like all levels of the business. We’ve all had different experiences with like trying to overcome that hurdle. So it’s really refreshing to hear you so eloquently like talk about it in the way you frame it. 00:12:06.64 [Stefanie Zammit]: And 100%, that’s so normal everywhere. I’ve experienced it everywhere that you have stakeholders coming to you. We want to do some quality. We want to run a survey. I need a dashboard. I need they they’re very prescriptive. And I think it’s part of our job as insights folks, irrespective of our training to say, whoa, whoa, hold on there. Let me understand what you’re trying to do. What decision are you trying to make? The answer might actually not be in a dashboard. It might be in a piece of custom analytics or it might be. So our consulting work is a big part of this job to find the best methodology for the best answer. We can’t expect our stakeholders to know what that is, although they’re very welcome to make suggestions, of course. But it’s a broad broad spectrum of tools that we could use. 00:12:54.40 [Julie Hoyer]: Yeah, absolutely. And one of the questions I’ve been dying to ask you is, why do you think, and Val, I’d be interested in your take, too, because you’ve kind of lived in both of these worlds as well. Like, historically, why don’t research teams and analytics teams always play together at all? Or if they’re playing together, they don’t always play nicely together? Good question. Get into it. 00:13:20.64 [Val Kroll]: Stefanie, you have to go first. 00:13:22.72 [Stefanie Zammit]: It’s a great question. And I think the answer is fear. I think that there is the fear of the unknown. And there is an assumption that the other world is so mysterious and so different to our world. We don’t understand each other. It’s literally a form of othering within teams, within departments. And it’s a fear I myself had. I could never keep up with these data scientists and, oh, they’re just so smart. And they understand all these things in a way that I never could. And then you start working together. And you see that you actually have more in common than you do have differences. And they see it, too. They understand, they can learn from the research process and understand, oh, hey, I thought research was just qual. I don’t know if you’ve ever heard that, any of you guys, especially Val, that the assumption that research equals qual, that’s the qual work. And so I just did a survey with 8,000 people in this hardcore conjoint statistical model. You can’t call that qual. But there isn’t an understanding. There isn’t enough knowledge. And everyone assumes that the other side of the coin is so different. It’s a whole different world. And I think it comes from the way that agencies are set up and departments are set up to separate these skills when actually they’re stronger together. 00:14:43.36 [Tim Wilson]: Michael, why does every quick question come with a 20-minute origin story? Well, that’s because our metrics have, I don’t know, lore. 00:14:53.76 [Michael Helbling]: Conversions might mean three different things depending on who’s presenting and how close we are to the next quarterly board meeting. 00:15:00.72 [Tim Wilson]: And I mean, every time you switch tools, you have to re-explain the lore like you’re residing ancient prophecy. 00:15:07.20 [Michael Helbling]: On the seventh day of Q3, the trekking broke and lo, that metric was doubled for July. 00:15:12.80 [Tim Wilson]: That’s why we’re excited about askwide.ai in Prism, because it has memory that actually remembers. You don’t have to repeat the lore. 00:15:21.76 [Michael Helbling]: Yeah, the jam system keeps context across sessions what your org means by revenue 00:15:27.84 [Announcer]: or conversions, which tables the source of truth. 00:15:31.36 [Michael Helbling]: And the weird exception, you’ll always forget until it’s too late. 00:15:34.56 [Tim Wilson]: Plus, you can save your best workflows as skills. Portable expertise you can actually reuse like a human. 00:15:42.88 [Michael Helbling]: So I don’t have to manually normalize UTMs or fix or tweak that GA4 channel grouping or deduplicate leads without breaking down into tears. 00:15:57.04 [Tim Wilson]: Yeah, so instead of rebuilding the same process like every week. 00:16:01.28 [Michael Helbling]: Yeah, I guess I just run the skill and go about the rest of my day. Kind of happy. 00:16:07.84 [Tim Wilson]: So one end, go to ask-wide.ai and join the waitlist. It’s in beta, but you can get in on the ground floor. 00:16:14.96 [Michael Helbling]: Yeah, and if you use code APH, you’ll get pushed to the top of the waitlist. That’s ask-the-letter-why.ai and use code APH. 00:16:26.16 [Announcer]: All right, let’s get back to the show. 00:16:28.56 [Val Kroll]: And where you landed on that, I think is one of the drivers that I see. The drivers that I see in my mind is if you think about how these two disciplines grew up in the world, research used to live within marketing. It used to be marketing research. And so we would sit within marketing teams. All of my clients back in the day were CMOs or reported up to the CMO. My first web analytics job, I was in IT and it was very technology heavy. It was about the tools and the way the data was collected. And like surveys, people think like, oh, like pen and paper, like mailing surveys or someone chasing you down at a mall with a clipboard. Like, would you like to take a survey? There’s so much more to that, like with panels and different ways that you can contact people nowadays. So I think- Limes have moved on. Yes, yes. There has been an evolution, yes. And so I think it’s just kind of been just how it grew up, 00:17:25.04 [Announcer]: was kind of thought of differently, budgeted for differently, like research, 00:17:29.52 [Val Kroll]: I think has always has this like, wrap a little bit. I’m interested in your thoughts on this too, Stefanie. That like a lot of times it can be like costly, like not just dollars, but time. So that it takes a long time to, you know, every, you know, we only do the brand tracking study once a year because it’s such a big, you know, piece of research or like it’s actually not valuable to keep track of that on a more frequent basis where a lot of the costs from some of the other analytics practices or areas are some more hidden costs because they’re in the technology or actually the human solutions. And so especially when we’re talking about like in-house, I think that it’s just budgeted for differently. And so people aren’t really like connecting the dots. 00:18:08.16 [Announcer]: But I think the organizations where you can break down those silos, 00:18:11.36 [Val Kroll]: because I’ve actually never worked for a client that’s had the setup that you’re talking about, Stefanie, which would be like, oh, Nirvana to have them like coming together. But I always found ourselves making the suggestion about like, did you talk to that team? And they’re like, who? And they’re like, well, I was scrolling through your active directory and I found this person with this title, like you should reach out to them to see if they can help us. But yeah, so that’s kind of what I think is like part of the rationale that I think that I do hope that this is like a movement though, like this evolution towards thinking more flexibly about the methodologies and what’s the right fit for the question at hand and what’s going to serve the business best. 00:18:50.80 [Stefanie Zammit]: Yes. And I should add that the Nirvana, I may be making it sound more Nirvana-like than it actually is. I mean, again, when you look at huge organizations like Starbucks, which it’s just such a math, there’s thousands of people at head office. Although we were all in a department together, the reality is that the silos were still very strong. At the time when I first joined Starbucks, I was in service to the loyalty team. So the rewards program and the app, you know, and how our customers use the app, et cetera. And I realized that I was trying to serve these stakeholders with insights about app usage and loyalty program behaviors and all the rest of it. Meanwhile, you had folks in analytics who were also answering questions for the same stakeholders. And there was just such a clear overlap in, we’re both talking about behavioral data, we’re both talking about what the client wants, the customer wants and needs. And so what we did was we formed a little, within the department, a little community of people who are in service to this stakeholder group, irrespective of where in the mega data analytics and insights department you are, we come together and we talk about all our projects. We formed a little, I don’t want to call it a Sierco because I hate the word Sierco, very anti-allergic. Oh my God. But just like a little forum of round robin, what’s everyone working on? And then we can say, oh, you’re doing that. I’ve got a survey coming up which directly overlaps with the objectives of what you’re looking at. But I can tell you why and you’re measuring what. So why don’t we put them together and, hey, our stakeholder will get something that’s more complete and less confusing rather than 10 different reports that all overlap, but the actionability is lost because we’re pointing in different directions on similar 00:20:43.12 [Michael Helbling]: and yet not quite the same topics. That’s crazy because I literally have built something almost identical but coming from the data side to the research side at a company I used to work at, which was forming this little team that we met and we’re saying, okay, let’s get together and get all of us together so we have a coherent story. And it’s always stuck with me that why is the organization having to be managed sort of bottom up in that regard when the reality is the structure or the layout of the org should be thought through to enable that kind of capability from the very top. And it’s just one of those things that sort of sticks out as a sore thumb. And I don’t know if I have prescriptions for that, but I will say it’s like Val, you mentioned we walk into a client, you’re kind of looking around like where’s your research team and why aren’t we talking to them too, which I think is really apt. But also like certain companies like you’re going to do analytics work and there is no research team or nothing named as such. And it’s sort of like a lost function or a missing function. And it might be kind of Julie to your earlier point, the leaders of the org just sort of think they’ve got it figured out so they don’t need someone to sort of like think about what the customer actually thinks because they’re thinking for the customer, if you will, not a great plan. But anyways, I’m just curious, Stefanie, because you’ve done some consulting in this space as well, like how do companies sort of jump the chasm first from just not even having a concept for doing research like this? Because everybody’s got analytics, like we’ve all got snowflake and data bricks or something running the back end of all of our data, but a lot of companies have zero going on in terms of like either customer or market research. 00:22:45.76 [Stefanie Zammit]: And if they do, they outsource it to agencies. So they would do a one-off project that an external company will run. And this is where you write it as complexity, because it’s rare that a company would have a full function in-house research team, because the manpower that you need on a per project level is huge, right? We’re talking thousands of people like, okay, not thousands, but at least 50 going out, interviewing in different countries and then your data processing and then your statistics. And then so the per project value for money of that much manpower is just not worth it. So they outsource to agencies who have economies of scale. Those agencies then in turn don’t think to ask, hey, do you have a data team? Do you have an analytics team? They sell analytics. Why would they say, hey, we’re going to build a segmentation for you. We’ll do all the research, but we’ll hand over to your in-house analytics and they can do the clusters. Like they’re never going to say that. They’ll be like, yeah, we do end to end, you know. But so I think that’s the challenge. And what I would advise organizations to do, to mitigate against that, is to have even just one person. At Bang & Offsen, we have one person. He’s a superhero. He’s a one-man research department. One person to coordinate all research projects and you can still use vendors, but you have an internal knowledge bank being built and internal consistency even. And then that person knows to work with analytics and to blend the two together while also getting the economies of scale from the agencies. 00:24:21.92 [Val Kroll]: Closing the chasm, I want to spend a little bit more time on this and your thought of making sure you have someone to represent that perspective versus like, does anyone want to add any new questions to this year’s tracker? Like it has to be so much deeper than that for it to be meaningful. But my first boss, when I was in market research, she grew up, Lynn Bartos, if you’re listening, she grew up at Burke. And so she was like hardcore, like you were saying, like having those skills. And one of the things that she always talked about was that she spent two weeks touring all the different departments, like someone who sat within, you know, data processing, someone who sat within coding for all the open-ended responses to get an appreciation for the operations and like how the sausage was made, because that makes you smarter when you request your banners or when you think about like, you know, how do I develop the questions to get me to a driver’s analysis or things like that. And so we did that ourselves on her team. And I really had an appreciation for that. Do you think that the closing the chasm is cross-training people so that they have like a better appreciation for each other’s skills, like when they do both exist in-house? Or is it more, you know, pizza lunches, like dog and pony show of like results? Or because I love what you talked about of having the little non-steerco-steerco, that’s aligned to a stakeholder group, like I would have never thought of that. But I’m just wondering if there’s some other things, unfortunately, like you were also saying, Michael bottoms up that people could do to kind of close this chasm between the team. Like what things would you recommend to listeners for who have an interest on the other side? Absolutely, 00:26:03.12 [Stefanie Zammit]: yes. So for anyone listening who’s not managing a team but is in one of these roles, whether you’re in data science, you know, data engineering, research, whatever it might be that touches on data, I’d really recommend the best that that person can do for their own career is to have a natural curiosity for the methodologies and the training that the others in the department have. And I always say to people in my team, if you find yourself in a conversation where you have no idea what people are talking about, or it feels like scary or just very different to what your expertise is, that is where you will learn that you should lean into those conversations. We should go out of our way to understand, not in an annoying way, like, hey, dude, what are you doing every day? But just to have a natural curiosity of how does your work fit with my work? And that’s not even just for analytics and insights or for data. That should be for everyone in a corporate job working for a brand where we serve our customers. We should all be understanding how do our worlds fit together for the customer. But especially within data, because data is the customer, we represent the customer, have that curiosity. And for anyone who’s listening who’s in a leadership role, then yes, I really recommend fostering that within your teams, 00:27:24.16 [Announcer]: that natural curiosity, and getting people to take a moment to question if anyone else in a data 00:27:32.24 [Stefanie Zammit]: related role can contribute to the project to add further insights. So would the data science team know anything here? Maybe they don’t have a deliverable, but maybe from their investigations or their work, they would have context that would add valuable insight to my work. Would the research team maybe have had a project about this? Maybe not, but again, the data folks are in, you know, they’re hands dirty in the data every day. The amount of information that they process and that they gain exposure to, which is not reported ever, is huge, right? We could never report all the facts, but it’s there, it’s in their heads and in their experience. So it might not be reported anywhere, but that’s a good person to talk to because they would have context. Same with the researchers, you know, they’re out conducting intercepts, ethnography focus groups, not everything makes it into the final report. But if you sit down and talk to each other, you realize, oh, yeah, I know something about that. I remember seeing something about that. I have a good quote that brings what you’re doing to life, whatever it might be. See, I really encourage curiosity. My first job, I started in QUAL, actually, which is like the furthest away from data science and analytics. And I was so scared to move toward QUANT. And I got reassigned to a tracker. So I went from like the QUAL team, you know, ultra open to a tracker, like 100% only ever working on this one tracker. And at the time, I was so grumpy about it. I was just like, this sucks. Like, 00:29:03.20 [Announcer]: where’s the creativity? You know, where’s the art? But actually, it was the best thing that 00:29:09.52 [Stefanie Zammit]: could have happened to me, because I didn’t know anything about tracking. And I, you know, I would have been pigeonholed if I’d have just followed my natural heart. It forced me to learn about a different world. And then I realized, actually, this is interesting. Actually, QUANT. QUANT is interesting. Look at that. Who would have thought? Tracking is actually interesting. There’s insights here. But unless you’re kind of forced into it, or at least when you’re young, you need to be forced a little bit, it’s difficult to just naturally expect that you would find these other worlds interesting. But I guarantee if you really, you know, scratch at that or, you know, peek into those boxes, you will find a lot of very interesting things that will help your own role. 00:29:50.88 [Julie Hoyer]: You touched on this a little bit earlier too, Stefanie. And I’m curious. So when you’re leading your team and you have this great point of view on how these things can work 00:30:01.84 [Announcer]: together. And obviously, we talked about trying to encourage the curiosity of everyone on your team. 00:30:08.00 [Julie Hoyer]: But are there some more like formal processes that you’ve also put in place for your team, or how you guys pick up projects or execute projects with your stakeholders that really help these come together in the best way possible? Because you mentioned earlier, like the, you know, starting with the, the qual and then you get a hypothesis, then you follow up with quant. So I wanted to like dive a little deeper in that area and hear what some actual harder, like boundaries or processes you utilize to help. 00:30:39.12 [Stefanie Zammit]: Yeah, 100%. And there’s so many examples, I’m going to try to stay focused here. So first rule of thumb, no survey asks a question that we already have the answer to. And the only way to know that is to go and talk to the data team and know what we have the answers to. So just as a rule of thumb, and even from our customer experiences, right, customers shouldn’t be telling us like their demographics, if they’re signed up to us and, you know, we should know who they are. Second rule of thumb, every research project, a sub sample, even if it’s not in your objectives to interview clients, even if you’re, say, you’re doing a new customer acquisition piece. And, you know, you want to go out to like a purely external sample, you don’t want any internal sample. Even if that’s the case, a subsection of the survey respondees should be from internal customer, known customer sample. And especially if you have a segmentation or you have certain key questions or certain key profile, you know, data points that you want, you want to continue building that knowledge internally, use your surveys in that way. So for example, we’re doing segmentation work now, we’ve designed, we’re starting, we actually started with data. So what do we know about our client? To what extent can we profile them before it becomes a mystery? Right. That’s the point at which now we take it to research, we fill in the blanks with research, but we, we interview as much of our own customers as we can. So that then once the survey is complete, we bring all that data enrichment back in house, it’s tagged to non customers. And we can use both worlds to create the segmentation. So you have your attitudinal stuff, you have your profiling that you would never be able to know just from data, and you have your behavioral data as well. And we have this amazing analytics team, they can do the segmentation. We don’t have to use an agency for the entire thing. So we’ve saved money, you know, we know where we can stop the agency to make use of our internal skills. Now we’ve got money left over for a different project. Great. Let’s go do some call with these segments and get to know them, bring them to life, do ethnography, take video. Now when we go out to our stakeholders, we’ve got our segmentation, it’s attitudinal and, and behavioral. We’ve got all this great qualitative bringing them to life. Right. So, and this goes for every project. So you’re doing a piece of work with drivers analysis. Okay, great. We might do a survey, you know, you do your usual like drivers analysis. But then let’s say, okay, how can we, some of that survey was internal customer sample, how do we bring that back to the analytics team and say, well, we want to grow. So let’s identify these people and do an internal business drivers analysis based on the learnings from the survey to see if we can replicate those drivers in our data. Lo and behold, you’ve got an internal business drivers analysis that you can now track because it’s in our data. So how if we move the needle, did we actually grow, we can actually say, yes, that was that insight was successful. And we did actually grow. There’s many more examples, but wherever we can put both worlds together in a single project, I absolutely recommend we do. The other thing is communities. If, if you’re a business that’s lucky enough to have a customer community, which is a qualitative research tool, it’s like a panel of customers that you can do quick polls and surveys, it’s like a social media for your customers. And they, you get so much qualitative insight. Those communities, the best versions of those are built on top of internal data lakes. So you can follow the strings down to who these people are and how they are transacting. Every project you run in your community now has a behavioral data trail to look at, okay, we’ve got insights, the business made a decision. Now you can measure the impact of that decision because we can track these people. Did they actually spend more? Did they actually convert? So it’s, the possibilities are endless. And it honestly all comes from recognizing that we’re all after the same goal here. And, and especially research quant, research quant have analytics teams. They’re doing, they have very similar backgrounds to in-house analytics teams. Like I said, there’s more similarities than there are differences, but the in-house analytics teams might not naturally be tasked with, we’re going to run a con joint or we’re going to do, you know, things like Max Diff, which is research analytics. It’s not really as well known methodology in in-house analytics, but they can learn why, you know, why shouldn’t we bring those tools to in-house analytics. And then it’s interesting for the analytics teams to learn these methodologies as well. Over time, you save money because you’re spending less on external agencies. That’s awesome. I love those. And you have better data. Yeah, I love the rule of thumb. Yeah, I feel 00:35:22.80 [Julie Hoyer]: like it’s an accelerator the way you’re talking about. I kind of hate the word flywheel, but it 00:35:27.60 [Stefanie Zammit]: makes me think of a flywheel. But the researchers also need discipline. Like they, they need to be really close to in-house data analytics reporting. Like it’s amazing how many researchers I’ve met that have never used like the dashboards, you know, they’re not in the BI at all. And like, why wouldn’t you be, again, as a rule of thumb, if you want to conduct good research, you need your sample to be representative of your customer base. How do you know what that looks like? Well, there’s BI reporting that shows, you know, you should be feeding that to the vendors to build the sample plan and the waiting plan. So it’s all connected. It’s all one thing. And I think when 00:36:07.12 [Val Kroll]: you’re talking about this connectivity too, you can be smarter about like, you know, even breaking it up like an example, especially if you have the panel or if it’s a known population, instead of asking like, amazing, like how likely are you to buy this again over the next six months or how often, you know, I remember working on advertising awareness research for a cruise line. And they would always ask like, how likely are you just to plan a cruise for you and your family over the next year? I’m like, over the next year, like these people, like they don’t know what they’re doing, they don’t know what they’re having for lunch. Like, why are you asking over the next year? Like, let’s look, there’s got to be other data for this. But in the same way, like there’s so many people who will be building out a fallout report in Adobe Analytics, like looking at them, like trying to discover the friction points and like coming up with the why like on their own, like, oh, they couldn’t make it to this next step because and it’s like, well, did you ask them if that was part of the friction because usability labs, like, you know, pop up surveys, there’s like so many different tools or ways that we can connect with the customer nowadays that, you know, not trying to fill in the blanks, like there’s a lot of different ways that we could just find out 00:37:18.08 [Stefanie Zammit]: directly. And that is another of my rules of thumbs that I didn’t mention earlier is the power of derived research versus stated, honestly, you’re wasting your money on stated surveys, they’re just no one knows humans do not know why we behave the way we do, right? It’s all like deep psyche. We’re super weird creatures. We have all these quirks that we don’t understand. So there, yeah, there’s you’re wasting your dollars on how likely are you to book a cruise? Like, whatever, it’s bullshit. It’s not sorry. Oh, we can swear on this. Yeah. Oh, we’re explicitly ready to send it. Let’s encourage. Yeah. Exactly. Absolutely. Like that, that needs to be the best quant research has analytics. If you’re running quant without analytics in your surveys, I don’t know what you’re spending your money on. Honestly, it’s yeah, it’s just not good value. And that again, 00:38:10.32 [Michael Helbling]: that ties then to internal analytics. This is so good. So I’m sitting here from a data practitioner standpoint and just loving the conversation. At the same time, I’m going to admit to you that like you’re throwing out certain terminology that I vaguely familiar with, but don’t necessarily know. Like what are there resources that could help someone kind of like level up and get better understanding of just sort of topics, structures, stuff like that, any like good overall books or 00:38:39.36 [Stefanie Zammit]: resources online, like anything you might recommend? Yes. And this is a really important point. I think another reason for the othering that happens in these fields is purely language. And so you’re right. I’m using, I’m trained in research terminology, which is fine. I just am like 00:38:56.08 [Michael Helbling]: admitting that I don’t know all the words you used. So yeah, but the, the, a lot of the words 00:39:00.88 [Stefanie Zammit]: I’m using, there’s, there’s a analytic or data equivalent of it. It’s, it’s just that different term terminologies used. So, you know, data might talk about addressable audience, which is like sample plans. The best advice I can give is research is grounded in academia. It came from, you know, like scientific studies or social studies. And so for anyone who, who was at university doing research projects as part of their university degree, that is the foundation of modern commercial research, but there absolutely are some great tools. There’s a really great book that I recommend. It’s, it’s one book. It’s the only one you need. What’s it called? I think it’s called, it’s the market research society’s main, main book. I think it’s called intro to market research. And that’s, yeah, that’s your one reference of just looking up these words. And you’ll be amazed how many of them you look up that you’ll recognize as not actually that unusual. So like an attribution model, how different is that from a customer journey? A research team would run a customer journey study, a data science team or analytics team would run an attribution model or call it customer journey, but you know, depending on what, where your journey 00:40:07.04 [Michael Helbling]: is. It’s a, it’s a customer journey has been used all over the place for all kinds of stuff. 00:40:14.48 [Julie Hoyer]: Really, whatever you want. Just like use case. Oh God. 00:40:20.64 [Michael Helbling]: The journey of customer journey is a little bit tricky. 00:40:24.64 [Val Kroll]: That’s, that’s, that’s a, that’s a cartoon strip right there. Well, so there actually is, so to your point, Michael, there is one, because you were just starting to talk about this about the difference between like stated versus like derived importance, which gets to max diff, which I, 00:40:40.88 [Announcer]: I literally couldn’t love anything more than studies where we get to do that. Because there’s 00:40:45.68 [Val Kroll]: so many, there’s so many different applications. But when I was, I worked on the telecom vertical is my first job out of college. And we would use that to figure out like back in the day, like cable internet TV, what should be involved in that packaging and at what price points. And people would say things like, Oh yeah, I need access to like 600 channels. When we asked like, what’s most important to you, but when we actually did like the force ranking, or there’s like different techniques, that that actually is one of the things that fell to the bottom, that it was about, you know, how long is it going to take for the technician to install the, the cable box. And there was like all these other things that were like, not something that someone would say necessarily, but it really, when it comes out in the wash. But anyways, that’s just like one example. But could you, especially if you have an example that you can pull upon to talk a little bit about state of versus drive importance or one of your favorite examples of that? It’s a really fun one. 00:41:40.40 [Stefanie Zammit]: I mean, in a nutshell, the difference is asking someone, yeah, how likely are you to do this? And with, whereas with derive, you basically give them an exercise and then you observe behavior. So for me, derived research is the same as what would be happening in a data team where you’re observing the behaviors, right? Because in data, there is no stated, like you’re not stating it, you’re just watching people behave and you’re tracking their data. So it’s the research version of that, that we give them different exercises or force response between many, like many, many, many choices, again, and again, and again, mixing them up again and again, so they could never remember. Yeah, like you could never remember the pattern. And through the continuous, like, it’s a choice between these four things or a choice between these and again and again, different scenarios, again and again, you can derive what the true behavior is or will be. So it could be used predictively to say, when this is true, this is the behavior that we want. Similar to building a predictive model based on behavioral data, that you can, you know, take all your data and map it over time and start to say, you know, just like run correlations to 00:42:54.56 [Announcer]: sort of say, when this is true, this is more likely to happen. So again, very similar outcomes, 00:42:59.60 [Michael Helbling]: but just completely different methodologies. All right, I’ve got another question that’s probably going to reveal how much I don’t know about this topic, but I want to ask it. Why are personas so bad most of the time? Segmentation makes my skin itch. If people are 00:43:18.64 [Julie Hoyer]: like, well, let’s look at the segments. I’m like, can we not? Because I honestly think, 00:43:23.68 [Michael Helbling]: I honestly think this is also a driver of the divide in a lot of ways. Like as a data guy, like I see people come up with these persona studies and stuff, and they’re dog shit. Like they’re really like terrible. And I’m like, scrap your personas. It’s all behavioral based at all. Like, you don’t because what I observe people do is they just make up who they like their customer to be. And then they’re like, this is this is Sally. And she’s a hip mother of three. And she drives a van. And but she’s got this cool thing that we like about our brand. And so that’s one of our personas. And it’s like, Sally doesn’t exist in our database anywhere. That’s not our customer. And the people who buy the products you’re talking about don’t look like her at all. Like it’s no correlation. Anyway, sorry, I’m now getting into my rants. But 00:44:18.16 [Stefanie Zammit]: what’s happening there? Like, why is it so bad? I love it. And I think the word segmentation or persona in themselves can mean so many things that these are words that are overused and not necessarily always used in the right way. And I don’t think there is actually a fixed definition. I think it just depends internally on what definition you choose. But why are they so bad? I have run uncountable number of segmentations, mostly from my market research background, where I have more years of experience. And I think that the win or lose of a segmentation is in how it is translated or brought to life for the business. So you cannot have a good segmentation without really solid underlying data, you know, hardcore, like, like a good, the factor analysis and the clustering, like, and all that has to be and you had the right variables and you had the right ingredients, all of that is really important. But that’s not actually matters or has impact. That’s just designing the output. It’s like a BI report. You can have like, you know, the BI report with 1000 million, like every possible data point, it’s amazing. But unless it’s, you know, user friendly, then it just doesn’t have impact. It’s the same with the segmentation. So without, if you only had a segment descriptively, this is Sally, you know, and Sally does this and Sally does that without knowing why or without finding Sally in the data and like saying, look, this is Sally, like specifically look, we’re going to, we know what Sally wants. So we’re going to, we’re going to send her a, we’re going to do a CRM strategy around Sally, we’re going to sell to her. And now we found Sally in her data, we can actually say, look at her changing her behavior. That’s when a segmentation is really powerful. And, and I think the best segmentation is to get to that level, you need both your behavioral data and your, your research, because research gets to how does Sally think, like what matters to Sally, you’re never going to get that from just observing her in data. You need to get into her psyche. So you put the two together. Now the marketing team have a strategy on like, who is Sally in terms of her psychology, what’s going to get Sally really freaking excited and get her to behave the way we want to do, but it’s underpinned by existing data. So you can actually see her, you can maybe test with her and, and over time see the impact of your segmentation and video, video, like I cannot overstate the power of a customer inside video, just like Sally walking down the street, like you can read, this is her life. It makes such a difference stakeholders in understanding who that person is 00:46:58.80 [Julie Hoyer]: beyond her being a data point. You make it sound like so, I mean, it is so ideal, but it makes it sound so like, duh, if you just did this, you know, you’d get everything you want because then we go where I go work with clients and it’s like, they’re just so far from that 00:47:13.36 [Announcer]: point. And it feels like such an uphill battle to try to help them fit together the two worlds that 00:47:19.20 [Julie Hoyer]: you’re talking about research and analytics, you know, segmentation is all based off of outcomes. And it’s like, but you want them to change behavior to drive outcomes, but now you’ve split them by outcome already. And then you ask them questions about outcome, it just feels like something’s been lost in a lot of the situation. And it’s, it’s sad to see because they would have to, I really think like start from the ground up to get it to where they’re utilizing analytics and research in the right way to get the benefits you’re talking about. Research hurts because every 00:47:52.16 [Stefanie Zammit]: time you do a study, you need to pay money, especially because like I said, no one has in house research teams, right? Not like fully intense. That’s not a thing. I think maybe Sky TV have one, but like most companies don’t. And so you’d have to make a business case to say, why should I spend money doing something that the analytics team can do internally using this data? Why is that not good enough for stakeholders to understand that without having ever seen what good looks like is really difficult. And it’s something I find really challenging in my job actually, just explaining the value of something without having it to hand. So it’s like hypothetical to a stakeholder, right? And this is where research teams are lucky if you have good relationships with agencies that will send you case studies and they feel sort of safe enough to send you examples that you can use to build your business cases. But when you’re talking in hypotheticals, it’s very difficult to get the budget and it’s not cheap. Market research is expensive. It’s slow. It’s a big investment for any company to make. But what I do find is once you start investing in it and putting the two together, showing the impact of that, the stakeholders will then understand like, wow, I get it so much more now because I’ve got my data, but I’ve also got my why and like my… I get how this person thinks. Put those two together and it suddenly you think, how did I ever make a decision without knowing this full picture? And then that’s where you will wet the appetite and it snowballs from there. But it is very difficult to do the very first one and hopefully agencies can help with those case studies. No, I have kind of a random question. So I’ll save it to Michael when he wants 00:49:30.72 [Val Kroll]: to wrap. Okay. I love this. So I actually had a very… The opposite experience of you, Stefanie. I started on trackers and I wanted to kind of branch out of that. And so I got thrown on the iHuts, which are in-home usage product testing, which is like could not be further from the tracking world. But I love that. Like when you said the power of video, Gillette was one of our clients and they sent out these new like beard trimmers to like men and they asked them to do videos of them shaving. And it was just so funny that like they were watching like, oh, like why in God’s name are they putting that clip? Don’t hold it like that. You’re going to cut their nose off. Like, oh my gosh, we need to change our directions. But it was so funny, including like testimonials, like the voice of customer, like in some of those reports that could be like leveraged 100 different places. We also had Haynes as a client and they were trying to… There was testing all the tag lists back in the day when everyone was switching. So it was like t-shirts and underwear. So we were sending out boxes and boxes of like whitey tighties and asking people like, tell us about… Did the tag scratch your butt? Like that was like the question. But the quote that we got, I’m like, I really wish I could see this used in like the 00:50:43.60 [Announcer]: internal decks, like where this went. But anyways, to your point about like it’s an investment in 00:50:51.12 [Val Kroll]: the first one, if you only think about it as like, I’m going to send out a survey and I wonder what they’re going to respond on like likelihood to agree to a certain attitude or statement versus like thinking more creatively about the different ways that you can interact with the customer, I think you can get people really excited about ways it can be injected in. So if you take one thing away, don’t think myopically about what research is and the ways it could be applied because it can actually be pretty fun and pretty enlightening. So… All right, Julie, 00:51:23.20 [Michael Helbling]: lightning round. Random question time. Lightning round. Because we have to start to wrap up here 00:51:27.68 [Julie Hoyer]: actually. Fine, Michael, we have to start to wrap. Okay, my question, because you were saying 00:51:35.20 [Announcer]: research is not fast and it is not cheap. But nowadays people like fast and people like cheap 00:51:41.52 [Julie Hoyer]: and people like AI because AI is… Oh, that was literally my question, Julie. So I have a slight spin. Let’s see if you went this far too, Michael, because maybe we were totally, totally parallel 00:51:51.60 [Announcer]: thinking, which I love. Because there were the two things, the faster and the cheaper. So 00:51:58.64 [Julie Hoyer]: we’ve had an episode in the past about synthetic data. I was curious like your thoughts on using synthetic data in this space, maybe some pros and cons. But then it immediately my brain jumped to, well AI in general is the fast and cheap option. And both of those things feel like people are very quickly going to grab for them to fill the gaps of classic research. But we’ve spent this episode saying that we’re weird creatures. And like to actually figure out the why, you can’t just ask them why, you got to take the time to observe. And those things are just at 00:52:29.68 [Stefanie Zammit]: such opposite ends of the spectrum. So it depends who you are as a company, how relevant or how useful synthetic data would be for you. So for example, at Bang and Alson, we work in small data, our transaction volumes are relatively low. You know, we’re lucky if we get a data set of, 00:52:52.24 [Announcer]: I don’t know, a couple of hundred thousand rows. So our client is so niche and so 00:53:02.88 [Stefanie Zammit]: under, I’m so misunderstood, or so not yet understood, because we are a luxury brand in the consumer electronic space. We can’t learn from other consumer electronics behaviors, but we can’t learn from other luxury behaviors. So synthetic data is just never going to be relevant for us as a brand. There isn’t enough volume and there isn’t enough lookalike profiles. And we’re still exploring the category, you know, being the pioneers of the category. If you’re a CPG company, and you sell in, you know, the typical supermarket or grocery store, 00:53:36.40 [Announcer]: then yes, absolutely. That makes sense. I would say though that there is a watch out that 00:53:43.44 [Stefanie Zammit]: we’re living in a changing world. So my rule of thumb when it comes to any kind of behavioral insights work is that they have a shelf life of around three to five years. But that’s been my rule of thumb since pre-COVID. And I do think that the world is changing more quickly now post-COVID than it was pre-COVID. So you have to think culturally, is the world the same enough for me to rely on synthetic data, which might go back, it depends where your cutoff is, right, where you start your data set from. So I would warn against caution to think about that. If there’s a huge world event, you’re probably going to need to go in with fresh questions or fresh exploration. And yeah, just the world we live in right now, it is so turbulent that, yeah, it’s not, the three to five year shelf life thing might not 00:54:31.60 [Michael Helbling]: be applicable anymore. Oh, that’s really good insight. And yeah, that was basically Julie, the question I was going to ask is about AI and its place in this, because I’ve seen startups going around, you know, being like, we can create a 100 personal digital twin research panel for you on the fly with AI and you can do your pre-research research with it and stuff like that. And I think there might be a place for it. But like, like you said, Stefanie, you have to kind of like think through the applicability. And I like the way you specified, like, hey, for our brand, we understand how unique we are. So a group of averages is not going to get us to an insight that we could use, which is, I think, very, very relevant. That’s really good. All right, we’ve got to start to wrap up. This is so fascinating. So thank you so much, Stefanie, for joining us. And it’s very good, educational, and really fun to talk about. And I know Val, you probably also are loving this episode. So, okay, what we’ve got to do last calls, something we do every show, we just go around the horn, share something that might be of interest to our listeners. Stefanie, you’re our guest. Do you have a last call or a couple you’d 00:55:45.68 [Stefanie Zammit]: like to share? I do. I have two last calls. And you know what, of all the prep I did for this episode, this was the thing that stressed me out the most, because you guys’s last calls are so good, as I got it coming with something good. It can’t just be any old thing. I did lose some sleep over these. But I think I got two good ones for you. So the first one is it’s actually a game. I’m a gamer. I love any type of game, board game, video game. And I attended a leadership training that was organized by our amazing HR team at Bang and Olson. And we played, it’s essentially a simulation game. You’re given a group of employees and they have to deliver a project together. And you know, it’s a bit like Moenopoly, you get chance cards and things go wrong. And it’s like, oh, the project, you know, somebody went on stress leave, what are you going to do? How are you going to keep to the time and the budget? Uh-oh, your main stakeholder has suddenly decided that they forgot what this was all about. What are you going to do? 00:56:42.72 [Julie Hoyer]: That sounds traumatic. I was like, this is giving me like stress. 00:56:51.44 [Michael Helbling]: It was so fun. I play that game every day, Stefanie. What are you talking about? 00:56:58.72 [Stefanie Zammit]: Sorry. No, but you’re, you’re sorry. We do play that game every day, but the fact of having the safe space where you could have these, oh shit, like everything’s going wrong in my project moments, but you’re learning how to deal with those in the safe space so that when it comes to your real life game, you’re prepared. I thought it was a great idea. It’s the game that we played was called the Playmakers game. It’s made by a company called the Works, Works with a Z. And I think they have a bunch of other sort of professional world simulation games as well. Super recommended. 00:57:31.12 [Announcer]: Yeah, next onsite. And it was a great way to like build rapport with your stakeholders as well, 00:57:37.44 [Stefanie Zammit]: right? Cause safe space and you play the game with a team of actual colleagues. So it was good 00:57:42.88 [Val Kroll]: for the bonding. Oh my gosh. You should like reverse roles. Like I get to be the stakeholder this time. Yeah. That would actually be hilarious. What should I do with all this power? 00:57:53.60 [Michael Helbling]: My problem would be like, really? You’re going to do that? Like no. 00:57:59.28 [Stefanie Zammit]: You have to all agree on the decision. That’s the game. Like how are we all happy? All going to do it. Oh no, we lost the team to stress leave. Damn it. Like we, so yeah, it was 00:58:11.36 [Announcer]: that’s awesome. That’s very cool. What else? Well, I have measured my second one just because I 00:58:17.20 [Stefanie Zammit]: am really excited that this is like hot off the press. It’s literally just gone live. I think two weeks ago, as you know, I worked for Bang and Offsend and we’ve just opened the factories up for anybody who’s interested in audio to go experience the manufacturing of our products. And honestly, if you’re a sound nerd or an audiophile, it’s a incredible experience. Like a really just sort of once in a lifetime immersion into the world of audio in a very beautiful part of the world. So yeah, that was my second one. Fun. Wait, Stefanie, I did see your note. You have to say the freaky part. Oh, the freaky part. Yes. So it’s a tour all through the, you know, like how the tonmeisters are called, how they find the perfect sound. And there’s a lot of different rooms that are created in a way for you to experience sound in different ways, which is how the products are developed. And one of the rooms is the 100% noise-proofed room. And it is the scariest place. Like honestly, I couldn’t stay in there with the door closed. You wouldn’t believe how scary 100% soundproof is. You can hear your blood flowing through your 00:59:27.44 [Julie Hoyer]: veins. It’s terrifying. Insane. I was like trying to imagine that. And I was like, 00:59:34.96 [Stefanie Zammit]: that’s just like breaking my brain. Honestly, 10 minutes and you’re like, get me out of here. Like I’m going crazy. Yeah, I bet. Okay, now I need to go do this. 00:59:45.68 [Michael Helbling]: I know. Stefanie, great job. You’ve upheld your end to the Les calls by far. Yeah, 10 out of 10. Absolutely. Yeah. Who wants to follow that? Val, what’s your Les call? 00:59:59.28 [Val Kroll]: So mine is going to be a little research-related. So I thought that one of the things we might discuss, and I do think we spent a good amount of time on it, is how to be curious in how to get broader in your understanding of these different methodologies or teams that might exist inside your own organization. And so my recommendation is to look out for some different communities that you could be a part of or join. The one that I’m still a part of today is the women in research, the wire group actually started in Chicago a long time ago. But I have benefited so much from my different mentorship conversations. I still stay in touch with the mentor I was assigned with, I think 13 years ago now, 14 years ago, Sheri Binky, shout out, I know she’s a listener. But there’s like, I’m a part of like women in product groups, and it doesn’t have to be like women only groups too, but they have so many different great events where you can get out and talk to people. So get out, touch some grass, talk to people, learn about how people are putting some of these different ideas to use inside of organizations, because that can totally be an inspiration for the way that you bring it to your own work. 01:01:12.96 [Michael Helbling]: Outstanding. All right, Julie, what about you? What’s your last call? 01:01:17.04 [Julie Hoyer]: Fine, it’s a little bit random. But honestly, this is something I’ve definitely heard mentioned before, quantum computing. And I do think this is like the next leap probably after AI, if my naive take on it is anywhere close to true. I was reading a newsletter that I always get, and there was this one mention of like the next big leap, like leap, I think it was called. And so I clicked on it, and it was all about quantum computing. It was an infographic about I’m like, well, I’ve heard it mentioned, I don’t know what it is, like, sure, I’ll take a look at infographic. And it, it was really good. And the way they broke it down within like not a very long read, I totally have a new appreciation of what this means and why so many companies are going after it. Pretty much they’re saying like, instead of using electrons for zeros and ones, like, we’re going to use a subatomic particle that is like super finicky to keep stable. But pretty much we go from being able to compute things one at a time linearly to doing things what’s it called simultaneously. So all these computations simultaneously that you could do and you think about how much computing like AI is doing or different industries like finance or supply chain even, and they, they walked through like a supply chain example. And it was amazing to think that you could go from something that would take a normal supercomputer, like they even said like a trillion years in their example to doing it so much quicker with quantum computing. And they said that some of this quantum computing power could actually happen in the next two to five years. And some of these people working at companies were like, I was told I wouldn’t see some of the milestones we have hit recently in my lifetime and like they have hit them. So it was one super interesting. I finally feel like I kind of understand what it is. I was a really quick read too. So it got me kind of freaked out and excited. And I just felt like, Oh, I learned something. 01:03:16.32 [Announcer]: Sounds good. I like it. Yeah. What about you, Helbs? 01:03:20.56 [Michael Helbling]: Well, I, as per usual, love everything I read on CommonCog.com, Cedric Chin, and he wrote an article recently about how to make sense of everything that’s happening at AI, because it just sort of feels overwhelming most of the time. And it actually sort of ties back to the episode in a way, because one of the points he was making was sort of like, don’t listen to what people say about AI, watch what they’re doing with it in the real world, to use that as a guidepost for how you should be responding to AI, which kind of goes back to sort of like user research. So anyways, the article is really good, but it’s very practical in terms of just better sense making around, like, okay, there’s sort of this hype and concern and all these other things. But like, look at actual detailed examples of how people are actually using it, and then ask some questions from there, like, what other outcomes are possible? What actions could I take? What matters most in my context? Those kinds of things. So anyways, really good article, just to like, take some of the pressure out of what I think a lot of us are feeling about AI, like, half the time it’s like, is it going to take my job? And the other half of time, this is so cool. I can’t believe I don’t, you know, I’m just doing everything with AI now. So there’s some balance where you have to find a balance. Otherwise, we’re going to blow up. Anyway, so that’s my best call. Blow up. All right. Yeah. Tune in. What’s the old, whatever, I won’t try to remember what the hippies used to say. Stefanie, thank you so much for coming on the show. This has been so fun. 01:05:01.12 [Stefanie Zammit]: Thank you. Yeah. Thank you for having me. It’s awesome to get to talk to you guys and talk about 01:05:06.40 [Michael Helbling]: nerdy topics that I love. So thank you. Yeah. No, it’s been great. And I’m sure as you’ve been listening to the show, you might have questions or you might have ideas. We’d love to hear from you. The best way to reach out to us is through the major Slack chat group or LinkedIn or via email at contact at analyticshour.io. And please feel free to reach out and leave us a review on the platform that you listen to us on, whether that’s Apple or Spotify or whatever, you know, whatever one we’d love to hear from you. We love getting feedback on the show. So definitely do that. And we’re still asking you to give us some questions just a couple of weeks to go until we’re going to be recording a show live at Marketing Analytics Summit on April 29th in Santa Barbara, sunny California. And we’ve got a survey which is out on the show notes page. Go fill it out. I mean, perfect example. Hopefully we did a good survey. I’m pretty sure we probably did. I know, right? Although again, I just want to caveat each time I had nothing to do with the art at the end of that survey. You’ll have to fill the survey out to see what I’m talking about. But it was I had no editorial control whatsoever. But if you have a question, we want to gather lots of great questions from listeners, either if you’re be there or not, we’re going to let answer them live on the show when we record it there at Marketing Analytics Summit. So 01:06:29.92 [Announcer]: looking forward to that is just a couple of weeks away. All right. Great show. Very fun. 01:06:37.76 [Michael Helbling]: And I know I speak for both of my co-hosts, Val and Julie. And I say, no matter what your 01:06:42.96 [Announcer]: market research says, keep analyzing. Thanks for listening. Let’s keep the conversation going with your comments, suggestions and questions on Twitter at At Analytics Hour, on the web at analyticshour.io, our LinkedIn group, and the Measure Chat Slack group. Music for the podcast by Josh Crowhurst. Those smart guys wanted to fit in. So they made up a term called analytics. Analytics don’t work. Do the analytics say go for it, no matter who’s going for it? So if you and I were on the field, the analytics say go for it. It’s the stupidest, laziest, lamest thing I’ve ever heard for reasoning in competition. 01:07:24.72 [Val Kroll]: Julie has this mic that like, she could be across the room and she’d be like, 01:07:28.88 [Announcer]: it’s like so loud. 01:07:35.28 [Michael Helbling]: Julie inadvertently must have bought one of those ASMR mics or something. 01:07:39.92 [Julie Hoyer]: Literally, I have like the game turned as far down. It’s like more than on arms length away for me. I’m like speaking, you know, I’m trying to speak on the softer side and I’ve turned on my volume in reverse side. No, I think it sounds good. And I’ve made sure my humidifiers are off. So hopefully no background humming. I unplugged my wine fridge, like all the things. Yeah. 01:08:03.28 [Michael Helbling]: And the worst part though, Stefanie, is we have this really great engineer, Tony, who goes through and does like all the audio editing and he gives very specific feedback about who’s audio quality was terrible. And so we’re going to use notes back from Tim of like, oh, Michael’s awful this episode. And we’re like, oh, thanks. That’s so great. So that was like so cautious. You sound a little soft, but you sound okay. Yeah. Anyways, no, it’s just one of those things where you’re like, you think we’d have it nailed down after so many years, but we’re still like 01:08:38.00 [Val Kroll]: every episode we’re sort of like fine tuning. You know what y’all need? 01:08:42.32 [Julie Hoyer]: Carabangan Olsson’s just saying. Yeah. That’s right. Yeah, you’re the person to ask about that. I 01:08:48.80 [Announcer]: should go look. Well, I’ll wait for Val to stop typing. I know this. Usually it’s Moee and she’s 01:08:58.16 [Val Kroll]: like keyboard cat, like you’re like Moee. Yeah. We’re a very serious professional podcast. 01:09:06.48 [Michael Helbling]: That’s right. Bringing it all together. Here we go. All right, I’ll give us a five count and 01:09:10.80 [Announcer]: we’ll get started. We’ll go in five, four, three, rock flag and two worlds, one family. The post #295: Research and Analytics: the Peanut Butter and Chocolate of Data? appeared first on The Analytics Power Hour: Data and Analytics Podcast.

AI is moving fast. But so is life. AI is widely recognized as a must-adopt technology, but how and where are data workers expected to find the time for that?! Organizations are struggling to find effective ways to productively drive healthy adoption of AI: What is it they expect their workers to do with AI? Is it purely an efficiency driver, or should they expect other avenues of value creation to be pursued? What guardrails need to be in place? What incentive structures are (and are not) effective when it comes encouraging team members to take the AI plunge? One tactic that is definitely effective is to have leaders who are excited, engaged, and transparent as they get their hands dirty. And, boy, did the algorithm deliver one of those to us in the form of John Lovett, VP of Analytics at SEER Interactive, for this discussion! This episode is brought to you by Prism from Ask-Y—your agentic analytics platform for automating analytics, exploring data, creating repeatable workflows, and delivering accurate insights—all without the need for manual query writing. Links to Resources Mentioned in the Show (Book) The *NEW* Big Book of KPIs: (Key Performance Indicators) by John Lovett Wil Reynolds AI Operating Manual: A Step-by-Step Guide to Teaching AI Systems How You Actually Think Twyman’s Law (Markdown file) Twyman’s Law Data Quality Module For LLM Analytics Integration & MCP Servers (Blog) Olympics analysis w/ AI by SEER (not published at time of recording, so we intended to track it down before this went live; if you’re reading this, Tim’s system failed and that did not happen!) (Blog) Live GEO Olympics Winter 2026 Results (Book) The AI Marketing Canvas, Second Edition: A Five-Step AI Plan for Marketers by Rajkumar Venkatsen and Jim Lecinski The diagram that Moe referenced from the above book (also called out two episodes back!): (Podcast) The Artificial Intelligence Show – Paul Ritzer and Mike Kaput (Conference) MAICON: The Marketing Artificial Intelligence Conference – Oct. 13-15, 2026 in Cleveland, OH LinkedIn GEO Community (Substack) From Data to Product by Eric Weber (Book)  Code Name Hélène: A Novel by Ariel Lawhon Data Kids Visualization Contest for Children (Conference) Marketing Analytics Summit – April 28-29, 2026 in Santa Barbara, CA Go to analyticshour.io/listener to submit a question for us to (potentially) answer when we record at Marketing Analytics Summit! Photo by Maximalfocus on Unsplash Episode Transcript00:00:05.76 [Announcer]: Welcome to the Analytics Power Hour. Analytics topics covered conversationally and sometimes with explicit language. 00:00:15.87 [Michael Helbling]: Hi everybody, welcome. It’s the Analytics Power Hour. This is episode 294. Yeah, probably right after the old RTO policies got rolled out, you probably got another executive communication about going AI first or whatever the hell that means. And let’s be honest, I think we’re all making pretty heavy use of AI now in some capacity. But what does that even mean exactly? And specifically in our world of data? I mean, should I just be uploading all my data to Claude and letting it come up with my analysis? Or is it in planning where you are constantly having to tell chat GPT not to jump the gun and start writing SQL when you’re just exploring some concepts or ideas? And I don’t know. So welcome to the AI Analytics Power Hour, I guess, this time. We’re going to talk about it, not just using it yourself, but how to think about rolling AI out across your team or your organization. As always, I’m joined by my co-hosts, Julie Hoyer. Welcome. 00:01:16.73 [Julie Hoyer]: I’m back. 00:01:17.39 [Michael Helbling]: Actually, welcome back. Glad to be here. Oh my gosh, yeah, of course. It’s like you’ve never been here. It’s been a while. It has been a while. And Moee Kiss, how you going? I’m going pretty good. Outstanding. And I’m Michael Hellblink, and I’m excited for our returning guest. John Lovett is the VP of Analytics at SEER Interactive. He previously held leadership positions at Further and Web Analytics to Mystified as well as many other companies. He’s the author of at least a couple of books, the most recent one being The New Big Book of KPIs, Key Performance Indicators. And today he is our guest. Welcome back to the show, John. Thank you so much. It’s great to be here. Awesome. Well, it’s great to have you back. Finally, it’s been a long time. So it’s a long time coming. But I think what drove us to this was specifically how it seems like you and all the team at Seer are really diving headfirst into using AI in significant ways across your organization and across the teams to do your work, to bring new ideas to life. Just talk a little bit about what it’s like inside your four walls and how AI is impacting your work. 00:02:27.94 [John Lovett]: Yeah, yeah. Well, I’ll start by saying, so I’ve been at SEER now for a little over three and a half years. And the last 18 to 24 months of that, I’ve just been immersed in it. And I’m not like talking as an AI enthusiast. I leave my analytics division, I hold a P&L, and basically I’m making hard bets. And as the kids say, I’ve got some receipts to show for it. So I do want to say, though, that At our company, I’m obviously going to talk a lot about my experiences and how I rolled it out to my team. But the first thing I really want to say is that getting your hands dirty with AI isn’t optional anymore. You said this in your preference, Michael. But doing it without accountability is really how you lose credibility. Every story I’m going to tell today is about building infrastructure that makes AI trustworthy. And not just fast and not just do more, but this is a little trite, but I developed this when I first got to SEER. The tagline for my division is we build trust in data and we empower people to use it, to use their data. And that hasn’t changed with AI. AIs come along, but that’s really fundamental is trust in data and really being able to use it. So that’s kind of the first thing that I’ll say about that. And if you indulge me and let me ramble a little more, just talking about SEER as a whole, like I couldn’t, well, what I’ve been able to accomplish, starts at the top and shout out to Will Reynolds, the man, the myth, the legend. He’s the CEO. In fact, he doesn’t like to call himself CEO of Sear, but he founded Sear and he basically said, guys, listen, we are all in on AI and everybody here needs to get on board or. You don’t need to, but there’s the door. And made it such that we had a big pivot. I want to go back to December 2025. It was a mandate for every single person in our company. I think we’re about 175. We’re closer to 200 people now. Had to take an AI training course and get certified in AI. So it was like mandatory requisite. Everybody gets trained. And then we rolled around to summer of 2025 and Will said, listen, everyone at this organization, from associate all the way up to executive, senior VP has agency to be able to take license to develop with AI, build the prototypes, show them to your clients as prototypes, get feedback, and let’s get them into production if people like them. So the company as a whole basically said, we’re all in on this. We’re going to give you the tools. We’re going to give you some training and let you loose on this and really make it a part of our culture at the company and make it such that it was a part of everybody’s job. I’ll pause there. I’ve got more on that, but I’ll pause there for a minute. 00:05:36.80 [Julie Hoyer]: Even with the leadership mandates, that is pretty big to hear such big pivots and saying they wanted it company-wide. That’s awesome to hear because it sounds like it gives you and your team a lot of opportunities and the invitation to go try things out with AI, see what works, get ahead of some of these trends or figure out what it could be best used for. But I am curious, even smaller, before the leadership mandates or right afterwards, what was your first thought around it, John? Or what was your first even small step? Because you were running a whole team. How did you then as a leader of an entire delivery team of analysts, go about that. 00:06:22.67 [John Lovett]: Yeah. You know, honestly, it was kind of like staring at a blank piece of paper like, what do I do with this? I was just like everybody else. And I think my first experiences, which I do encourage people to try, are just use it for your personal life. Like I would literally open my refrigerator door, take a picture with chat GPT and say, what can I make for dinner? And it would look at it and say like, oh, you’ve got pasta sauce there. I see some tomatoes and I see a bell pepper. Like here’s a recipe for you. And then I would just start using it and talking to it. My family and I took a trip to Ireland last Thanksgiving, back in 2000. 24. And basically, I had to build our itinerary. And so we’re driving around Ireland. And of course, I had to put the chat to PT Irish accent on my on the talker. And so my wife called my girlfriend. It was like, that was like, where should we go today? What restaurant should we go to in Galway? And it was like, it was giving me recommendations. And they were great. It told me where to stay. The funny things that started happening when I would talk to it and it tell the things You know, I said like, hey, I’ve got three teenage boys. Here’s the things we like to do. This is what we want to try to do on our own. It was like a month later and I’m in the kitchen and I’m asking it something probably what to cook because I rely on that a lot. It thinks I thought I liked to cook and I say, I only ask you because I hate to cook. I just need the ideas. And it said to me, how are the boys doing? What would they like for dinner? And that for me was like, Like, and of course, like, this was my… Oh, see that? 00:08:00.48 [Julie Hoyer]: I feel like that would scare the shit out of me. 00:08:02.62 [John Lovett]: It totally scared me because I was like, what? And I almost dropped the phone because I was like, how do you even know that? And then the voice said back, well, you told me that you, I know that you went to Ireland with your boys and you talk about them a lot. And it brought that up in conversation. And that for me was like… like blew my mind in terms of what the possibilities were. And that was all just like my personal life before I really even started getting into it with work. 00:08:28.23 [Moe Kiss]: Can I ask a little bit? I’ve had a similar experience where there’s been a lot of support leadership down in terms of everyone at the company has been given access to every single tool that they could possibly want, enterprise grade. I do personally think that is an absolute game changer. I’m not going to tell you which tool to use, you can use any tool. But talk to me about the team experience, because I do feel there are those that are like, yeah, I’m going to roll my sleeves up. I’m going to get my hands dirty, that sort of stuff. And there are folks that are really scared. And they have that, it’s going to take my job. How did you navigate that team dynamic of the folks that are super keen versus the ones that are very apprehensive? 00:09:15.83 [John Lovett]: Yeah. Yeah. associated with that. There were people on my team that were like, let me add it, I’m all in. Others that are like, I don’t think so. I’m skeptical. One of the things that we did as an organization that I think helped open the door a lot, and this was early last year. So actually, many of you have even been early 2025. We, as a company, every division looked at every single deliverable that we had. What do we regularly produce for clients? What are our workflows? How do we work? And then built this huge list of, if we do all these things, where can we start AI that would help us to do these things faster, more efficiently, better? And we came up with a list at that time of 15 different workflows, deliverables, processes that we did on a regular basis. Then we said, okay, we’re going to prioritize this subset in Horizon 1. We call them, we have horizons of work. Horizon 1 is going to be this first build. We had a series of them in Horizon 1, then we go to Horizon 2, and Horizon 3 is bigger thinking, stacking them all on top of each other. But that really gave us the opportunity to think about the day-to-day work that we do, and how AI can be a part of that. I reiterate it at every team meeting, at every dead meeting. I’m going to hold that for you guys, I don’t know if you’ll see it, but I keep a sticky note. You probably can’t read this, but on my monitor that says, how can AI help me do this? And I genuinely think about that. It’s right here on my monitor. I have to look at it every minute because I encourage my team, like, you’re going to do something. Think about if AI can help. And there’s plenty of things that AI cannot do for you, but there’s a lot that it can do. So that was sort of a door opener for people to say, oh, I do this every day. One quick example I’ll give you is like, we’re an agency, right? And despite me trying to squash this as much as I can, we live and die by the billable hour. So we have to track time cards, right? And I’ve got team members that do it religiously and team members that just don’t. And they’re not part of their workflow, not part of their habit. We built an agent that said like, hey, connect. In our case, it’s a Claude instance to my calendar, to my rake, which is our project management. We’ve got connections to all of our platforms, but I can say, hey, look at my calendar and see everything that’s on there and populate my timecard for today or this week. Just that alone, I spent maybe less than an hour on that every day, but now I’m like, boom, I got it and I can do it. That same tool that I built to do that, Now, every morning I get in and I say, what’s my morning briefing? What is the most important thing I have to do this day? And it goes to my inbox. It goes to my Slack. It goes to all these different tools that I use every day. And it can say like, oh, you’ve got a podcast tonight with the with the greatest podcasters on the planet, you need a prep for that. And so it will tell me kind of the most important things to do. And that for me has been like such an eye-opener to say, like, you know, I would sit at my desk on Moenday, it’s like, oh, crap, what do I have to do this week? And I would write it out with a pencil on a piece of paper. And now I can just ask my AI partners, like, how do I, you know, what’s important to me? And that’s been a big change as well. 00:12:40.11 [Julie Hoyer]: Did you? Okay. This is a little nitty gritty, but I’m curious. Some people still love like, I still love to write like a to-do list. Obviously that’s not AI friendly by any means, unless I put it somewhere digitally. Um, you do. Okay. So I’m, I’m curious, like, do you then to help with that workflow of utilizing AI to surface things to you? Are you actually like snapping a picture of your physical to-do lists? Are you retyping them? Are you dropping slacks to yourself so that your AI will read it? Like, 00:13:09.07 [John Lovett]: A lot of times, and maybe this is me, old guy, old school, I take notes when I’m on a call with a client and they’re just riffing and talking about stuff and I’m either trying to capture requirements or figure out what’s important to them. I want to be in that moment. And for me, even though Whenever I can, I have a inscriber that’s recording the call so I can have the whole transcript and we can talk about that later. But that’s critical for me going back to, but I’m still taking those notes because when I go to write a scope of work or proposal or just update a report for a client or whatever I’m trying to do. That to me is like, okay, my brain interpreted this, I wrote it down, I know I need to get this done. I generally don’t take pictures of that and put them into Slack. A lot of times I will send myself Slack messages for reminders, but more often than not, those are links and different things that I want to come back to. But I do rely on head and paper for my thinking process to help me do things. And I still haven’t gotten away from the satisfaction of checking something off the list. I give that to myself. That’s so good. 00:14:21.81 [Tim Wilson]: Michael, quick question. How many times have you solved this same analytics problem this month? 00:14:29.56 [Michael Helbling]: Oh, probably enough times that I’m considering invoicing myself. Step one, fix GA4 source medium. Step two, lose the will to live. 00:14:39.21 [Tim Wilson]: Cool, so let’s stop living like that. Prism by Ask Why lets you save your best workflows as skills, portable expertise you can reuse across datasets and tools. 00:14:50.95 [Michael Helbling]: Okay, so like normalize UTMs, dedubleeds, merge Facebook and Google spend, maybe rename 37 versions of newsletter into one civilized channel. 00:15:01.69 [Tim Wilson]: Exactly. You build it once, then you run it again, instead of recreating it like it’s Groundhog Day, but with more spreadsheets and less Bill Murray. I mean, I like Bill Murray, but I do like fewer tabs. Plus, you know, there was Andy McDowell as well, but really plus jam, jam memory. It remembers context across sessions like your org’s definition of active user, which table is the source of truth, and that one cursed date range where tracking alas broke. 00:15:31.61 [Michael Helbling]: So I don’t have to start every meeting with before we begin. 00:15:35.52 [Tim Wilson]: Here’s the lore of our metrics Yeah, or or we explain that revenue means net of refunds here not whatever looks best on the slide Okay, well my dashboard has now been personally attacked Well, that’s good. My mission, my mission is complete. But skills plus memory means prism gets smarter about your world over time, your processes, your definitions, your shortcuts. 00:16:03.31 [Michael Helbling]: So it’s like an assistant that actually remembers my preferences, like I’m not getting from most of my streaming apps. 00:16:10.86 [Tim Wilson]: Exactly. So if you want early access, you can go to ask.y and join the waitlist. Speaking of waitlist, use code APH when you sign up and you will be bumped right to the top of that list. 00:16:22.93 [Michael Helbling]: OK, good deal. 00:16:23.93 [Tim Wilson]: I’m already on the website. Well, we’ve been doing this ad spot for a while. I hope you, Michael, have already gone to the website and signed up. But for anyone else, that’s Ask the letter “y” dot A I and use code APH. Yeah, I’m putting myself in the place of the user. 00:16:40.47 [Michael Helbling]: It’s called Empathy Tim. Oh, well, I don’t understand that. And I’m already over here saving time and my sanity using these skills. Yeah. It’s too good. I take my notes online, but, but same thing where I’ve got the transcript plus the notes. And sometimes I can push them together into the prompt and use both. But yeah. 00:17:01.92 [Julie Hoyer]: Okay. Small tangent. Well, I’m just curious. So John, you have your three boys. Um, and I have a few wonderful people in my life that are, you know, their teenage years getting ready to go to college. And I fell into a very interesting conversation. And so I’m curious if like what you tell your boys, like going into college, maybe they’re already in college, out of college, but like in the AI times, I was talking to someone and they were telling me that like they aren’t great at taking notes. And I kind of panicked for them thinking, you’re about to go to college. Like you have to get really good at taking notes. And they said, yeah, but there’s AI. But as we just talked about, like there still is this analyst skill of hearing certain things from a stakeholder or still having your own like mental filter right of like what you think is important or you really want to reiterate on or you want to build a story later so to be able to jot it down and it is a skill I think to actively listen. you know, take your notes, whether you’re typing or writing. And I suddenly got a little worried and I didn’t want to like harp on them saying like, well, you really need to like learn how to do it. But it had me thinking. What are your thoughts? 00:18:13.43 [Moe Kiss]: I still take notes as well, even though there’s like a transcribe function. For me, I wouldn’t say it’s necessarily, sometimes it is like, what is the key points that I’m taking away that I really want to follow up, but I actually think it’s how I listen. Like for me, how I absorb the information, if I’m not taking notes, I will, my brain will probably go off on five different things. So I wonder if it’s like more a style thing than a like AI, not AI thing. 00:18:42.26 [Michael Helbling]: I have noticed in meetings, I will summarize and repeat in the meeting sometimes so that the AI picks up on it better. And that’s a change I’ve noticed just for the transcript. I’ll be like, OK, so to summarize, we’re probably going to make sure we do this, this, and this. And then I know that the AI then will come through the meeting transcript and be like, oh, I’ll pull that out as it to do. So even my style of conducting the meeting is shifting a little bit behind it because I know that then I don’t have to write that down. I get the AI will surface that as my to-do. But yeah, it’s crazy how we’re adapting. 00:19:14.79 [John Lovett]: We joke about that. I do that too where we’re like, hey, transcribe or remember this piece and we kind of joke. I did literally just take out a pen and paper because I don’t want to forget the questions here. So Julie, starting with the question, I definitely worry about the future for kids. My oldest is about to turn 21, which is a frightening thing in and of itself. He’s off at college in North Carolina. was a very, he helped me may listen to this because such a proud dad, he changed his major. He was a finance major. He changed his major and I said, which change to? And he’s like, dad, I entered the business analytics program in the business school. And I said, What? Shut the front door. And I said, you do know that’s what I do, right? Because I never got a notice. I mean, he’s like, yeah, dad, I know that’s what you do. So, like, such a proud dad moment. But he’s never been, I actually wrote a blog post about this. He’s had dyslexia. He’s had learning challenges. He switched high schools because he wasn’t getting the support that he needed. To your point, His handwriting, even as an adult, was horrific. He just didn’t read like anybody else. He didn’t do things like everybody else. When he did math, he did it all in his head. He didn’t write out the problems. And so he just thinks differently. And so he has been a very early adopter of AI. And for good or for bad, he’s also as a high school student and now a college student figured out how do I use tools like chat tpt and what have you and then not have my teachers think that I wrote it with AI. So he’s got the anti-AI tools to figure that out, which honestly, like I’m like, buddy, I’m going to pay for that subscription. I’m going to pay for your chat tpt subscriptions or whatever he needs because I want my kids, when they get out of college, to have this as part of their skill set. My middle son is a junior right now in high school, and we’re looking at colleges. We went to Syracuse a couple of weekends ago, and those are my questions. I want to know from universities, how are you guys going to teach AI? Because the university that says to me, oh, it’s off limits, they can’t use it, I’m going to be like, you know what, you’re not going to prepare McTin for the workforce. that might not be the way I want to go. And I just think it’s a matter of, like, we took notes, we listen and think with our brains and our hands and record things. My kids, like, they’re on their phones. They’re, you know, they don’t actually know how to talk on a phone. They text and I don’t know if you guys have younger kids, but like, when you call a kid on the phone these days, they’re like, they don’t even say hello. They don’t even say what. It’s weird. They know how to. 00:22:00.98 [Moe Kiss]: No, but they know it. I mean, my three-year-old knows how to like pinch and zoom and swipe and you’re like, what the? 00:22:07.63 [John Lovett]: It’s wild. So I just think about learning style. I’m seeing with my own children shifts and I wanna, and I tell them like, you may not plagiarize, don’t just take it and copy and paste it. You have to put your brain into this. It’s gonna give you an output, but the output it gives you is generic stop. And until you put your voice into it and teach it and put your brain into it, that’s when it becomes a partnership, not just a, dictation machine or an answer engine, it’s really when you start to leverage the value of it. And there’s little tricks that I do, we can talk about later about like, like I teach AI my voice. I said like, I uploaded my books, I uploaded my blog posts. I said, I uploaded like email examples. I said, this is how I write, this is how I talk. And I want you when you’re writing on my behalf to mirror this, to use this, to add this to your knowledge so that when you’re generating something, For me, it sounds like me, and it’s legitimate like me. And all of my agents and tools, no M-dashes, I sign off cheers, so my email messages, I have all these little quirky things that I wanna say. I’m like, no jargon, don’t use buzzwords. I put things that I am like, no pie charts, stuff like that. I’ll put those into my instructions. And in fact, if anybody wants, this. I can give this as a resource, but I built, we were first to choose at our company. I haven’t made the choice yet, but it’s like, do you want to go with chat GPT or Claude? And I was like, well, if I give up one, how do I take all that teaching and learning that I taught it and bring it to the other. And so I developed a guide to be able to take out of the model and say, what is my personality? What do you know about me? How do I talk? How do I think? How do I act? And then give me those instructions that I can upload to my next agent so that it teaches it what I am like. And I found that to be like a transferable thing that I could say like, oh, if I suddenly lose access to one of these tools, how do I not lose all that history with what I’ve told it, darling? 00:24:20.10 [Moe Kiss]: Okay, John, you’re brilliant. And yes, we want all the resources, absolutely, because I’m literally doing that at the moment. Like no comment on politics at the moment, but yes, migrating from one tool to another. But okay, what you’re talking about is like fundamentally leading from example, putting in the time and effort to do it well. Back to your team. I am sure there are people that are dabbling and just producing utter shit. And as someone on the receiving end of reading lots of shit, it’s like that exploration period. I guess I just like I want to better understand is like you have gotten to the value point. I’m I think probably quicker than most. How do you how do you help your team get to that value point more quickly? Because The feedback is always, I’m busy, I don’t have time, and I’m the first to say all of those things. How did you create the time for both yourself and then your team to get to value faster? 00:25:17.97 [John Lovett]: Yeah, yeah. So time is, you still have to do your day job. So for me, I do end up working more. I try to tell my team not to do that. But sometimes it happens. But putting that aside for a second, One of the things I did early on last year, I’m a huge believer in conversational analytics. I think it’s coming. I think it’s coming for us all. And so I built a conversational analytics, which is how do you get an MCP to communicate with whatever LLM you choose, chat, GPT, quad, whatever you want. I started with GA4 because that’s what we had most access to, and I expanded it to BigQuery. And I said, everybody on the team has to make these connections. Use the MCP, follow the instructions, set it up so that you have the ability to talk to your data with these tools. And so, some begrudgingly, as we started, some jumped right into it, some were slower to act. I think it was not even a week, it was the first few days of doing this exercise. One of the analysts on my team sent out on our company-wide Slack, oh my gosh, our blog post just went viral. We had a huge spike in traffic. It’s amazing. This blog has never seen this much traffic. Will, our CEO, he’s got all the agents and the MCP connected as well. He’s a big runner, right? He’s a marathoner. He talks to it when he’s running through his headphones and using Claude, as he’s running, and he’s like, hey, so-and-so just posted this post, so we had this big viral spike, and he started asking questions. He’s like, I’m just curious, where did this traffic come from? And the agent responded, and it’s talking to him, oh, it looks like 99% is from China. And then he goes deeper and he goes, oh, really? Like, what kind of traffic is this? Did they bounce right away? What did they engage? And they were like, no, all the visits were sub one second or whatever it was. Come to find out it was a bot. You know, so a bot was hammering our site. My analyst team was like, hey, we made this great discovery. Look at me. I got this new conversational analytics thing to work. And that was a like screech hit the brakes, like needle off the record moment for me. I was like, wait a minute, I got to put some controls on this. So it was that same week, I had seen, actually, I think Tim Wilson reposted, oh, and I’m gonna forget his name, Twyman’s Law. I’ll come back, we’ll get this in the show notes, who gave me the reference. I wrote a lengthy post about it. But Twyman’s law, for those who don’t know, is essentially, if any number looks too good to be true, it probably is. And Twyman never really published this. It was like word of mouth, and it got around. It has become this marketing staple. And so I was like, hey, that would really work for my conversational agents. Why don’t I build that in to say, if you’re seeing this huge spike in traffic and it’s anomalous and it doesn’t match any of the other patterns and the data doesn’t match, question it and dig in. And that was really a groundbreaking moment for me to say these things a lot of you. You know, they’re gonna tell you, if anybody’s used chat GPT, I definitely call that the yes man in my arsenal or my toolkit, if you will, because it’s always like, John, you look great today. That shirt looks awesome on you. You cooked the best dinner ever. Like it just, it always gives me props and like, yes. And that’s what he was doing. It was like, you found an amazing insight. Look at this. And then you put something like the guardrails on it, like Twyman’s Law, to be able to say, you can’t just throw out a number like that. You need to verify it. And since then, I’ve actually adapted it to This isn’t too technical, but all of these models that we use, or most of them, are all probabilistic. You ask GPT question, Claude, Gemini, whomever you want, and it’s going to give you probably the answer. It’s doing word by word, and what’s the next logical word, and how does it go? If you ask it, what’s two plus two, they say, I’ve seen this enough, that’s probably four. But when you’re asking it to do analysis and the metrics and dimensions and all these different things, it sometimes with a thousand percent confidence tells you, you had this massive traffic spike, you got a viral sensation on your hands, and it thinks it’s true. And so I figured out a way to be able to take those probabilistic models and built into my instructions, deterministic instructions. So I say, never give me a number. Every number that you generate, I want to see the SQL query, I want to see the math behind it, and I want to know the logic. I want to know what’s missing from the data, and I want to know what you can’t show me reliably. And that has also helped me to provide those guardrails. So I started with Twyman’s Law, but I’ve evolved it to have this deterministic layer to say, like, don’t just give me a number. I want you to perform the calculation. And I actually have SQL built into my instructions that says, like, how do you do this? And how do you get at it? And that for me is really up the reliability of the answers that I get, because it will give you garbage and junk and mislead you out of the gate unless you put those filters on it. 00:30:33.63 [Moe Kiss]: So what’s your perspective then on, you’ve applied that to your own instances and Will is a phenomenal leader who gets the data side, right? But there are lots of stakeholders who don’t, who won’t build that. So is your thinking then that you need to find a way to apply those guardrails for everyone in your company? Or how do you scale that? 00:30:57.05 [John Lovett]: Yeah, so great question. Remember, I mentioned that we’ve all had agency to build our own agents and experiment and do things. I am the vibe coding master. In fact, I was up till 2 AM last night just vibe coding because I was having so much fun. And so I just build these prototypes, and I build them for me, and I build them, and I play with them, and I see what works. I iterate on them. We just had our first release day. So release day was, we had, I believe it was, 11 agents that got released to the company. So these are somebody’s vibe coded that went to our engineering team, they stress test them, they productionalized them, they made them ready for the whole company to use. And that is where I get things like Twyman’s Law, the data guardrails into production for everybody. As a company, the engineering team has built out, we call it the CRMCP. The CRMCP has connections to our data sets. It’s got connections to all of our sales transcripts. It’s got connections to every one of Will’s presentations. It’s got connections to every one of our town halls. You can get so much information from this one. CR MCP, that that is how we productionalize things. We bring them as an organization to once they’ve been viped and tested and tried, we productionalize them by having it go through a relatively specific process. But everybody’s encouraged to bring the ideas kind of back to the original question is like, for my team, I built an AI innovation lab. And we’ve got weekly meetings, they’re optional. I usually have them on Fridays. And I’ve got a core team that prioritizes ideas and puts things on our roadmap for delivery. But everybody is allowed to bring ideas. Everybody brings challenges. And this is something I’m playing with, or this is how I’m trying to work through this issue. And so that just opens up the the door to possibilities and everybody trying things and everybody getting excited about what they can possibly do with AI that can help them. So that’s been a big lift for us as well. 00:33:01.01 [Julie Hoyer]: I’m curious because I’ve seen it so many times. I’m sure all of us have a new technology. It’s very exciting to use it. John, I’m curious how did you, because it sounds like you guys have gotten to the point where you’re very focused on problem solving, but like what you’re saying with your Friday meetings, right? Of like bringing ideas or problems, things they’re trying to work on. Like how do you keep your team or in the beginning, how did you get your team to really think of using AI as a tool to solve specific problems? Like is that a fight you’re still fighting? Do you feel like you guys are pretty mature in that thinking? I’m curious how you got there, if you are. 00:33:34.26 [John Lovett]: Yeah. So it is definitely a tough one. I’ll give you another real-world example. So it was three weeks before the Olympics were about to start. I’m a huge Olympics fan. And I think it was a Saturday morning, whatever, scrolling on my phone. And I said, hey, what if I ran the world’s largest geotest to test a bunch of prompts and see what kind of data I can get back about the Olympics, that LLM model. So this is you typing in a prompt to chat GPT, perplexity, like all the different models and seeing what responses come back. And I developed five hypotheses. So like narrative persistence is one, like how long does a narrative stick before it changes? Temporal velocity, how soon when an event happens or let’s say somebody is awarded a medal, do the AIs pick up on that and see, I looked at social proof, does social presence make a difference with which how frequently athletes showed up in LLM responses. So I had these five hypotheses. And I basically, I wrote a blog post about it. I was so excited at all the state of collecting. And I put out a general thing to the team. I was like, hey guys, I’m the only person working on this project right now. Everybody’s invited. Come on, jump in. Like, give me some help. And it was crickets. Like everybody. Oh, I was going to be like, no one responded, right? Because like, nobody, everybody’s like, I got my day job. I got all this stuff to do. Like, and, and again, like this is sort of that, uh, I don’t want to call it apathy, but like, uh, I’m afraid of it. I don’t know how to do this. I don’t know what you’re asking me to do. I’ve never done that before. How do I do this? And so the Olympics started, I had three ways, pre-Olympics, during Olympics and post-Olympics. And obviously we’re in that post-Olympics phase right now. During the Olympics phase, Nobody responded to me. It had been three weeks, almost a month. And I said, you know what? I can’t have this. I went to four of my people on my team and I said, I need you to be a leader here. I need you to, here’s the hypothesis, it’s framed. All you have to do, I built an agent to be able to analyze the data. It had all my guardrails in it. It had the hypotheses. It had what we were testing and it was showing data and I was doing preliminary results. I said, I need you guys to log in here and either prove or disprove these hypotheses with the data. And every single one of the four people I asked was like, I will do that. And again, I had to think about how I was going to write it. I used my agents to help me, like what’s a persuasive message that people with not a lot of time are going to want to do this and adapt to this. And, you know, so I was thoughtful about it. Do you think part of it was like the just a bystander effect like you kind of ask everyone and everyone’s like and soon as you went to people individually prop could very well be in and again that’s part of you know I’ve I’ve got a big team out see everybody every day and so I’m reaching out to people that I see on zoom maybe once a week or just at big meetings and me reaching out and saying like, hey, I need your help on this. This is an important project for us. This is going to help us know what to test with regard to GEO. It’s going to tell us about how the models think and how we can use that to build tests and experiments and really understand what’s going on. Everybody’s on board. We’ve got to give all hands tomorrow and we’re not even done the analysis. I’m the fifth person. There’s five hypotheses and We’re all going to do five minutes on what are we learning so far? What have we found? Like, how is this going? And it was that little nudge, that personal touch, that reaching out directly that helped me get my team on board. Because everybody, you know, when you do ask that bigger question, there’s a lot of like, that’s, he’s not asking me. I can kind of shirk off into the shadows. and see who else will step up first. So that was a big one for me. And I’m super excited about the results. Like I’ve already found so many fascinating things just through this research already. So that’s something that maybe by the time this actually probably will be by the time this is published, you can check out the GEO results. 00:37:44.14 [Michael Helbling]: Yeah, I like that. I find that for certain people, they just jump in and start doing their own AI process. And other people need AI defined into the process for them. And I think that’s sort of where I’ve seen, like I was talking to somebody recently, and they were like, I need my team to be doing this. And I was like, well, why don’t we set up a process where you take them through these steps? And one of the steps is you go do this thing with AI. And now you’re putting an AI enablement step into the process, just making it part of the standard process for them. And it was like, oh, yeah, that’ll totally work. And so some people just need you to give them delay out the how do I do these steps even though like a lot of us because we’re the way we are with analytics and in curiosity and asking why we’ll go into the AI and be like I want to set up a series this I want to set up a process or so we’ll just start with the AI and work through. and learn as we go. And then other people need like, I need you to tell me exactly how to do it, but AI can be part of it as part of that. So it’s very interesting because I’m watching adoption like this too. And I’m sort of like, yeah, not everybody is just going to jump in with both feet. So how do I get them active? And that was one of the ways that we, we kind of thought through about, about that was just sort of like, okay, well, one of the steps is you go to the AI and you do this, this, this and this with it. And that’s how you get through the process. 00:39:09.28 [John Lovett]: I love that example, Michael. One of the things that sort of runs in parallel with that, when I use AI to do something, if I like, let’s say it’s an analysis, I’m just like, hey, I’m trying to understand why does this brand get a bunch of citations, but not a bunch of mentions? And I do an analysis, I get a good output that I like, I usually go back and forth with the AI, and I’m a, Maybe I’m a beneficiary of having lots of tools, but I’ll take a Claude output and throw it over to ChatGPT and I’ll say, what do you think of this? And it says like, oh, that’s pretty good, but you forgot these three things. And then I’ll read it and edit it and I’ll put it back in Claude. And I was like, hey, how about this? And I was like, wow, that’s really smart. Those are good ads. So I play them off one another. But the one thing that I always do, and this is maybe a limitation of tools in 2026s, I use Ninja Cat, I use Claude, I use Gemini, I use basically use everything, but I am so worried about losing my work. And so it’s like that whole thing like I didn’t even save your file and your computer stopped and you lost it. I tell it. So I hit context windows. We have like you hit your limit for the company this month. You can’t log in for 24 hours. As soon as I think that’s going to happen or as soon as I get an output that I like, I tell the agent, write the instructions so that I can replicate this and give those instructions to another agent and teach another agent how to do what we just did. That for me, I can then pick up and say, hey guys, I show my team, I built this analysis, here’s something that we do every day. Here’s an agent that will help you do this. you can ask it any question and it’s gonna guide you down this path of using the right information, asking you questions to be able to get to the right outputs that are gonna produce something that’s relatively consistent. And for me, that’s been a huge unlock because those people who are like, I don’t know what to do with this thing. I don’t know how to build an agent. They can certainly use an agent and they can certainly use something like that to help guide them through an analysis or really any type of workflow. 00:41:12.45 [Julie Hoyer]: How have your analysts felt? Because Moe, you and I have talked about this on previous episodes of the shift in work of looking at a blank page and you’re creating your own thing, your own work, you’re on thinking your own analysis, right? Compared to if you are using AI to give you something to react to, it’s a completely different process in your brain. So I’m curious, John, along those lines, What has been the reaction or the feedback from your team? If they’re using AI in these tools for an analysis, how have they liked, disliked, you know, pros and cons of using AI to start an analysis and they’re like checking the work. They are confirming what AI has found, things like that. 00:42:00.86 [John Lovett]: Yeah, I think it definitely happens. I would say that the first reaction of the team is, this is just slowing me down. I have to ask it all the questions, do all the things I would do the analysis for, and I have to go make sure it didn’t hallucinate or give me bogus answers. it does slow you down at first. And it does make you go a little bit slower to say, I’m going to question that. I’m going to be curious about it. The whole part of AI, is it going to do the job of the analyst? It can help to surface insights and get things. But if you don’t have the curiosity, if you don’t have a spidey sense for that number ain’t right, it will just give you junk. I would say that initially it does take longer to do things. And this might take us on a tangent, but what we’re doing for that today is My team still builds dashboards, right? We still have reports, and that is our source of truth. We get our data, we pump it through our tools, we use tools like Funnel, and we pump it to BigQuery, and we can do queries out of there. I find out the specific regex for the queries, and I replicate that in my tools in my agents so that when I’m doing an analysis, I can look at my dashboard and say, okay, LLM visibility rate is 42%, share voice is whatever. What does it say in the agent? And if they match, then I feel good about that. And I’m like, okay, this matches my source of truth. If it’s way off, I’m like, okay, why was it off? What was going on here? What was happening? We’ve tried to build in those things where we can say, let’s have a source of truth. It used to be for me, I would ask it a question and then I would go to GA4 or Adobe Analytics and like, all right, let me dig up this number. Honestly, like I’m so far out of those tools from the day-to-day perspective, I’d be like clunking around and be like, oh, how do I find, I don’t even know what explorer to build to get through this number versus having the conversational agent when I could just ask it things. My team was great at that. They would do an analysis and then they would verify a GA or whatever platform so we could see those two things. But having that source of truth and having that dashboard, I’m still gonna, we will still rely on those. AI isn’t going to kill the dashboard just yet, but I think it’s an important resource to have for that validity, for the data quality, for the ability to make sure it’s not, you know, you’re not getting AI slop. 00:44:37.28 [Julie Hoyer]: Have you guys then turned the corner where you’re seeing efficiency gains from your new process of using AI like in your day to day work. And then second part of the question, I’m going to hit you with two before I forget my second part of my question. On our International Women’s Day episode, Moe, you guys were talking about how maybe efficiency gains is like not the only outcome or great part that could come from AI. But right now, that’s what people are most focused on. So I’m curious, John, have you guys found the efficiency gains? And are efficiency gains the only positive that have come out of you guys integrating AI into what you do, or have you found other great things coming from? 00:45:20.29 [John Lovett]: Definitely, the efficiency gains are a big thing. For us, it’s been a lot about, hey, we’ve got this process that we do. It’s part of the workflow. Now when we do it, we can repeat it across client to client to client. That’s agency life, right? It’s like we’re repeating these things. We’ve got similar analysis, different data sets. That has definitely helped us move faster through these things. The structured prompts, the way that we build methodologies and the way I tend to take, hey, I built this once, give me the instructions to build it over and over again. That has gained us a lot of efficiency. And then the ability to upscale employees, right? So it’s like, I get a new team member, I get a contractor on my team. And I can say like, Hey, here’s an agent that’s already built, you can get up to speed much more quickly using this. So those have definitely been the case for efficiency. I think, I think the other thing, the second question, if I’m right, It was like, what else besides just the efficiency is that? 00:46:29.17 [Moe Kiss]: So, John, there’s a concept we talked about a couple of episodes and I keep it. It’s funny, Julie, that you mentioned it because I was going to bring up the exact same point. So, Jim Lysinki, I think is how you pronounce his name. He wrote a book, The AI Marketing Canvas, and he has this like quadrant thing and, you know, talks about internal productivity and that’s really where everyone’s focused. But the other quadrants are like internal growth. So, like, using tools to accelerate your workforce. Then there’s external productivity, which is a lot more of that customer service, how you can use it to have productivity gains that are for your users. Then there’s the fourth quadrant, which is really external growth, using AI to completely unlock new revenue streams. My observation is that everyone’s really stuck in that internal productivity quadrant. From what you’ve shared, it sounds like you’re also using it for internal growth. Is that a thing that you’re seeing play out where the productivity gains just seem to be the thing everyone’s so anchored on? The thing I also then want to understand is if you are having productivity gains, how are you measuring that? 00:47:39.66 [John Lovett]: So with, I mentioned earlier the horizon builds that we’re doing, every horizon build has assigned productivity metrics. Like how much time did it save? How much money did it generate? We have KPIs that we build. You guys know I like KPIs. So we got KPIs that we build around each one of those things. So we are measuring productivity in a number of different ways. I think with the growth, this is an interesting one because it is sort of a creeper. It moves more slowly than just the productivity gains. But one example I’ll give to you. So I mentioned that we record all our calls. We ask our clients, can we record these calls when they allow us to? We do. So we’ve got all these transcripts. And so my My head of BD comes to me and says, hey, I’ve been talking to this prospect actually since October. And we’ve had a dozen conversations. I built a notebook LM that contains all the transcripts, contains what we talked about. And now, here we are in January or February, and they just asked for, we think we need to include analytics in the scope. And so everybody’s been talking about this project. We’ve got pricing calculators. We’ve got scopes of work. We’ve got all these things. And he basically said, I need you to get up to speed on this. And so I was able to use all of the resources, the transcripts, what the client wanted. I did get on one call with the client and talked to them and got to ask my very specific questions. But immediately after that call with having no prior knowledge, I was able to write a scope of work and my BD guy came back to me and he’s like, holy shit, John, you nailed it. Like, I can’t believe that you got that figured out in such a short amount of time. And we didn’t even talk about it. Like he just gave me the resources, but I was able to plug in and use my tools to be able to say, what does the client need? How does that match up, match up with my products and services? And then what can we offer them that’s going to fit what I heard in our conversation? And so that for me was a growth moment where I could say like, that really not only did it save me time, like it would have taken me months to get up to speed, but I was able to turn that around in like 24 hours and get something that was so spot on that my BDO was like, that’s amazing. And hopefully fingers crossed that, you know, that deal comes through, but it was just a good growth moment. That’s one example that I can think of there. I think the other thing I’ll just mention, you know, productivity is, Obviously where you wanna go, the part of this as being analysts, there is a whole new discipline and we call it geo, but it’s AI search, right? So it’s like, hey, we got all these models, people have questions, we’re moving toward this zero click world where it used to be, you’re ranked on a, on Google or Bing or wherever, somebody saw your link at the top and they clicked you, and they got a visit to your website. Now, your brand is getting surfaced via these LLM models. They see your brand, and they may not get cited. And they’re like, OK, I’m narrowing my list down. I’m seeing these things, but I’m not even clicking through. And I’ll just type in direct to get to that brand’s website. And so for me, part of this being that curious analyst and I’m collecting all this data and writing prompts and developing prompt methodologies, I’m like finding wild stuff. One example was, hey, in Claude, we see mentions, which is like your brand is mentioned, and then citations, which is the link to, you know, whether it’s a podcast episode or your resources, whatever it is, like all of a sudden in Claude, all the citations dropped off December 1st. And I’m like, what happened? And just because I’m looking at all this data across all my clients, I was like, oh, Claude really did stop using citations at that point in time and just cut it off. And then the other example, I’m analyzing data. I’m trying to build a report for a client. And I see the data went back to December 15th, 2025. It was the week that ChatGPT announced they were gonna start having ads in their free accounts, right? So you won’t have them on enterprise or paid accounts, but the free accounts are gonna start getting served ads. And I actually looked at the data and I was like, what is going on here? I started to see the nature of the responses change to purchase intent-driven responses. they were actually preceding the way that chatGPT responses to same questions we were asking like months ago, they were changing this dynamic. And I saw all these changes in the responses. And I was like, something’s going on here. And I connected it to the fact that they just introduced ads. They are prepping, they have been prepping for like a month and a half, you get ready for this. So I mentioned that because There is so much to learn about how these models operate. And it’s kind of like going back to like search days when you’re trying to understand the algorithm, like we’ve got this whole new field of who knows what the hell’s going on within these models is up to us to, and if anybody tells you they do, like I’m calling bullshit on that because All we can do right now is experiment, test things, try things and see what works. Honestly, in my 20 years in analytics, this is the most fun I’ve had because I’m learning new stuff, I’m playing with new tools, I’m getting to see all these things. For me, that’s growth. I am growing as a professional because my tool gets expanding, I’m learning all these new things. I can tell clients, if I can surprise and delight a client by saying, look at something I found about you. I’ll just give you one example. My client said, hey, this blog post just popped. Over the summer, it started ramping up and ramping up. We get all this traffic to it. It’s amazing. We’re so stoked on this. And on the call with the client, this was like a regular status call, I looked up their prompts and their geo-reporting, and I found the URL that they had referenced. I said, wow, this is amazing. The last two weeks, you guys have gotten 102 citations on this particular blog post. But yet all of those responses, nobody mentions you. You’re never mentioned in this set. And the category was like risk management. They had become this authority on risk management that enterprises were citing, brands were citing, you know, forums were citing, all these people were citing, but their name never got associated with that because it just wasn’t in there. And we developed a simple task for like, hey, just don’t do it in a pitchy, salesy way, but just insert your brand name into You know, here’s a description of what this is, and by the way, our products solve for this. In your FAQs, I had a big FAQ section in the blog post, I said, hey, just enter your brand name as, you know, when you’re closing it out, say like, we do this, and here’s the product that delivers this. Within two days of that test, mentioned started showing up in AI Overviews, which is one of the fastest to pick up on changes to websites. And I was like, boom, proof point right there. I got the first signal where it was like that change that they made to their content pages produced a mention which had never been seen before out of like six months of testing. And so, you know, that was only like a couple of days into the process. So when I can show a client like that, and then I did another analysis yesterday morning and I’m like, okay, signal strong. And it led me down this whole other rabbit hole of like understanding mentions and citations. And I learned about ghost citations, which I won’t get into. But like, this is the fun stuff where it’s like, we’re curious analysts, we’re trying to figure stuff out. And never before has there been this playground of like so much data, so much information. that we can just dive into and show clients things they’ve never seen before. And that for me is, I think that’s the growth that, you know, it’s gratifying, it’s fun, it’s exciting, and it’s definitely keeping me going. 00:55:25.19 [Moe Kiss]: Oh my God, John, this is like positively infectious. I love it. 00:55:28.76 [Michael Helbling]: Yeah, I know. I’m trying to remember a time when I’ve seen you like this fired up, John, honestly, like I’ve known you for a long time. This is great. 00:55:37.35 [Moe Kiss]: But so like you’ve had, I guess, I’m going to say the privilege of approaching this as a leader who then is trying to like bring your team along. There are lots of folks, I would say, who probably have the same level of enthusiasm as you, but might not be in a leadership role. What advice would you give that person, that mid-level analyst who’s doing all this playing, they’re having lots of fun, but how can they have a ripple effect on their business if they’re not in a leadership role? 00:56:05.44 [John Lovett]: Build something cool, show it to somebody. Show it to your manager, show it to your boss’s boss. If nobody picks up on it and you think it’s brilliant, share it on LinkedIn, share it on Measure Slack, share it somewhere, get some feedback on it. And if you get that positive feedback where people are like, this is cool, we’ve actually done this with our blog posts. at SEER, we’re almost not allowed to write a blog post anymore until we’ve seen something on LinkedIn, like see any of your reacts, see if you get any comments, see if you get any mentions on it. So test it, like play with it, put it out there, see what you get as a response. You know, and this may be harsh, but if you’re an organization and you built something that you know is productive, adds growth, is clever, is adding to what you do, and your leadership doesn’t recognize that. I’d be time to look for new leadership, but it is hard. I would just encourage people to experiment with things, build things within your boundaries that you’re allowed to do, and then share. put them out to the world. And if your leadership won’t listen, take it to LinkedIn, take it to Measure Slack, take it somewhere that you can find an audience that thinks that’s cool, and you’ll grow your brand that way, you’ll be able to find your people, I guess. 00:57:21.29 [Michael Helbling]: Yeah, that’s great. All right, we do have to start to wrap up, unfortunately. This is so good though. All right, well, one thing we love to do is go around, share a last call. AI is never going to change that. Well, maybe it will, I don’t know. But John, you’re our guest. Do you have a last call you want to share? 00:57:39.20 [John Lovett]: Well, I have two quick ones, but I guess I need to ask permission. Am I allowed to offer another podcast? Yeah, of course. 00:57:45.93 [Michael Helbling]: Yeah, come on. Do you think we follow rules around here? 00:57:49.53 [John Lovett]: I would bring it anyway, but the artificial intelligence show is a podcast. It is run by, I want to make sure I get their names right. Paul. Paul Ritzer and Mike Kaput. And every Tuesday they put out a podcast and they aggregate all the most recent AI news. And it’s brilliant. My wife actually loves listening to it with me in the car. We listen to it a lot. And she’ll talk, she’ll be like, oh, their voices are so soothing. But just great intel, great information. This is also the company. I think their company is changing brands, SmarterX. They were on the MACON conference in Cleveland, I wanna say. Cleveland, yeah. Yeah. Yeah. And Julie, you need to get there because it’s all in Cleveland. Yeah, it’s a great conference. But that is also the training that everybody at SEER was required to take was piloting AI. So great resource they’ve got. They do a free one-on-one training on a bunch of different things once a week where you can tune in and ask questions. But just a fabulous podcast, a fabulous resource, definitely worth checking out. And then my second one, a very quick hit. I encourage everybody on LinkedIn. There was a community that started, and I just happened to see it that was called the Geo Community. And Geo stands for Generative Engine Optimization. Some people call it AI Search. Some people call it all sorts of different stuff. I just happened to be, I saw it. I was like, that seems cool. And the first couple posts, were very intriguing to me, and I started commenting on it, and all of a sudden, I want to get a Rohit thing as the founder, and he’s like, hey, would you want to be an admin on this and join me in kind of managing this community? So I think we’re only a couple of hundred people strong, but if you’re curious about Geo and all that stuff, I got super excited about learning. Check out the LinkedIn Geo community, the Geo community, good resource to get up to speed. 00:59:48.66 [Michael Helbling]: Nice. Excellent. Okay, Moe, what about you? What’s your last call? 00:59:53.42 [Moe Kiss]: Well, I am quite excited. Probably not as excited as John, but my good friend Eric Weber is… back writing, and I’m super, super pumped about it. So he has a great blog from Data to Product On Substack, and I get it via email. The latest one was the Conundrum on Buy versus Build, which is something I always am super interested to read about. 01:00:18.51 [Michael Helbling]: Awesome. All right. Julie, what about you? What’s your last call? 01:00:22.57 [Julie Hoyer]: Okay, my last call is totally not AI industry related at all. My life the past few months, you know, I’m just trying to keep my eyes open in the middle of the night with a little baby. So I’ve been doing a lot of reading. So my last call is I went down a path of reading some historical fiction books. And I read one that was really good. So if anyone’s looking for some new reading material, a little break from AI news, maybe, you know, switch it up. It was, and I know this is popular, but it was codename Helene. And it’s by Ariel Lohan, if I’m saying her last name right, but either way, really great book. It is about a British spy going to France near the end of World War. So it was a really like interesting take, a different storyline that I had not really read about. And it was just an awesome breed. 01:01:16.18 [Moe Kiss]: Julia, I’m going to sidebar you after this and send you the name of an author who’s written like six books very similar to this. I’m going to read this one, but I’ll send you mine too. 01:01:22.79 [Michael Helbling]: We got a whole other podcast going here. Yes, I have a last call. So we heard about this and we’re kind of think it’s really cool. There’s a new visual data visualization contest, but it’s for children. So if you have a kid between the ages of seven and 12, there’s two different age groups. I know. So not everybody’s kids fit into that category. 01:01:47.26 [Moe Kiss]: I can fake, he’s tall, I can fake his age. 01:01:49.93 [Michael Helbling]: Yeah, whatever you wanna do. It’s like Aussie age, you know, it’s different. It’s, yeah, the conversion. All right, anyways, we think it’s a really cool idea. There’s some really great advisors behind it, but they’re doing a data for kids visualization contest. And it opens, the contest opened literally yesterday before the show comes out. So there’s still time right now. You can go jump on their site, we’ll put it in the show notes and you can check it out. But if you have a kid in that age group that’s really different age brackets, I think between seven and nine and 10 and 12. And so you can kind of work with your son or daughter and just come up with a cool data viz together and might be a fun little project. So anyway, that was my last call. All right, John, what a pleasure. Thank you so much for coming back on the show. It’s so good to talk to you. It’s been fun. It’s been great talking to you all. It’s, yeah, and I know we’re gonna see you at Marketing Analytics Summit, right, in April, so. Can I do a team look forward? Am I allowed to do that? Yes, of course, yes. 01:02:53.57 [John Lovett]: Absolutely. The extra day till Thursday, I am doing a half-day workshop on conversational analytics and how you can connect your GapGPT LLM of choice with BigQuery or Google Analytics, and so you’ll see it live there. Nice. At the Marketing Analytics Summit in Santa Barbara. 01:03:10.15 [Michael Helbling]: It’s funny, John, your blog post inspired me to create my own conversational analytics integration with Google that I built myself. Because I was like, hey, I should try to build something like this because, you know, I read your blog post and I was like, yeah, this was pretty, pretty cool. And I made some cool things out of it. Anyways, so I want to kill you. And it didn’t kill me. And I’m okay. I did stay up until two o’clock in the morning one time working on it, but that’s the fun part, I guess, you know. No, but you don’t have to stay up until two o’clock in the morning to come to Marketing Analytics Summit. And there’s a couple of really important things about that. One is it’ll be April 28th and 29th. So John will be there, we’ll be there. And we want your questions. We actually have a survey live right now. You can go to analyticshour.com. IO slash listener. Did somebody get that right? Yes, listener. And take our survey. And then you can submit questions that we will answer on the show, hopefully. So that’s kind of out there right now. We’d love to hear from you what questions you have to answer live at Marketing Analytics Summit. So that’s coming up. And so don’t miss that. We also love to hear from you every other witch away too. So please reach out to us. If you’re doing cool things with AI, if you’re inspired by some of the stuff you’re hearing, of course we’d like to hear from you. Obviously, when talking to John, it sounds like John, you’re pretty active on LinkedIn. So that’s a great place to find you and follow what you’re doing and interact with you there. And then also in the Measure Slack chat group. And we also love to hear from you via email contact at analyticshour.io. So please reach out. And we have stickers and Tim loves sending them out. So you can ask for stickers too. So just send us a little note. All right, I know that I speak for both of my co-hosts when I say, no matter how AI is changing your work and no matter how you’re getting your processes rolled up, hopefully it’s being both efficient, driving efficiency and increasing productivity. But remember, keep analyzing. 01:05:20.12 [Announcer]: Thanks for listening. Let’s keep the conversation going with your comments, suggestions, and questions on Twitter at @analyticshour on the web at analyticshour.io, our LinkedIn group, and the Measure Chat Slack group. Music for the podcast by Josh Crowhurst. Those smart guys wanted to fit in so they made up a term called analytics. Analytics don’t work. 01:05:44.69 [Charles Barkley]: Do the analytics say go for it, no matter who’s going for it? So if you and I were on the field, the analytics say go for it. It’s the stupidest, laziest, lamest thing I’ve ever heard for reasoning in competition. 01:05:57.91 [Tim Wilson]: Tony? No. None of this in the outtakes. None of this. None of this. 01:06:01.92 [Michael Helbling]: None of this. Yeah, that’s fine. It’s yeah, that’s fine. 01:06:07.96 [Moe Kiss]: That’s my hopes. That’s my thinking face. Like, what do you want me to do with that? 01:06:12.49 [Michael Helbling]: Moee, I’m just it’s fine. And people know who we are now. If our listeners are like, I can’t believe they didn’t look engaged enough in this short video that they put on their website. I’ll be like, you know what? That’s why we can’t have lights things. 01:06:28.68 [John Lovett]: I love the images you guys are putting out there. They’ve been fun to watch. 01:06:34.74 [Michael Helbling]: Oh yeah, thanks AI Studio, Google AI Studio, Nano Banana Pro. I just, it’s, what’s hilarious is like, I don’t even have good pictures of all of us. I just grab random headshots and throw them in there and be like, make a picture of this. That’s pretty good. I don’t know if John, if Tim shared the video I created with Vio of him and I, Crip Walking, but we’re not going to put that on social media. 01:07:01.19 [Tim Wilson]: That’s in the, that’s in the slide channel. 01:07:03.87 [Michael Helbling]: Yeah, that’s in the slide channel. 01:07:05.41 [Julie Hoyer]: That’s the only reason I joined the, the Mass Life channel. Cause I agreed that there was some fun happening. 01:07:10.52 [Michael Helbling]: And I created an analytics power hour brain at 40 else that I’m holding while we’re doing it. So nice. 01:07:18.87 [John Lovett]: Nice. Oh my God. 01:07:19.29 [Michael Helbling]: It’s you, it’s like imagery wizard. Oh, oh no, John, you don’t understand like, I’m fully AI enabled at this point. Like, it’s a problem. 01:07:32.12 [John Lovett]: AI enabled the dangers. 01:07:33.34 [Michael Helbling]: It’s not good. 01:07:42.54 [Moe Kiss]: Rock flag and review your workflows first. The post #294: Adapting an Analytics Team to an AI World appeared first on The Analytics Power Hour: Data and Analytics Podcast.

The one rule about the Analytics Power Hour is that we don’t talk about specific tools. But that doesn’t mean we won’t talk about tool SELECTION! Jason Packer recently released the second edition of Google Analytics Alternatives, (also available on Amazon) and his approach in the book is very much not an RFP-like “check which features your tool offers” system. And his rationale for that seems just as applicable (to us, at least!) for any data platform selection, be it a digital/product analytics platform, a BI tool, database or storage infrastructure, or, well, you name it! Ultimately, the challenge is how to go about getting a reasonably strong understanding of the philosophy and historical roots of each platform being considered and then marrying that up with the foundational priorities and needs of the organization. Is that a lot harder than a feature checklist? Yes. But them’s the breaks. This episode is brought to you by Prism from Ask-Y—your agentic analytics platform for automating analytics, exploring data, creating repeatable workflows, and delivering accurate insights—all without the need for manual query writing. Links to Resources Mentioned in the Show (eBook) Google Analytics Alternatives, Second Edition by Jason Packer (use “APH” as a discount code!) (Paperback / audio book) Google Analytics Alternatives, Second Edition by Jason Packer The music league: join the Measure Chat Slack (join.measure.chat) and then join the #measure-music channel (Rock opera) User Journey – The Rock Opera (aka, “Universal Sunset”) (Podcast) Pivot (Article) The AI Analyst Hype Cycle by Marc Dupuis (Video) Why You Should Fail 15% of the Time by David Epstein (Article) What If We Don’t Need the Semantic Layer? by Jacob Matson MeasureCamp NYC Go to analyticshour.io/listener to submit a question for us to (potentially) answer when we record at Marketing Analytics Summit! Photo by Alexander Schimmeck on Unsplash Episode Transcript00:00:05.76 [Announcer]: Welcome to the Analytics Power Hour. Analytics topics covered conversationally and sometimes with explicit language. 00:00:17.05 [Michael Helbling]: Hi everybody, welcome. It’s the Analytics Power Hour. This is episode 293. Okay, listen, we draw a hard line of this show. We don’t talk about tools, but we never said anything about tool selection. And let’s be honest, we have all been there trying to figure out which vendor to go with after putting in tons of effort into our carefully crafted spreadsheet with all the selection criteria, which somehow every vendor says, yes, they can absolutely do all the stuff on there. It’s enough to make a person cynical. And we analysts don’t need help with that. So take a pause from reading the cold sales emails from the latest analytics AI SAS vendor. And let’s talk about the ins and outs of selecting a tool. But first, let me introduce my co-hosts, Tim Wilson. Or as I like to call you, Tim, tool selection, Wilson. No. How are you doing, Tim? 00:01:11.97 [Tim Wilson]: I’m just about ready to select a new podcast recording platform. Oh, perfect. 00:01:18.30 [Michael Helbling]: That’s going to probably trigger a bunch of inbound emails. All right. Moee Kiss, how are you going? I know you do a lot of vendor evaluation and selection in your role. 00:01:31.25 [Moe Kiss]: I certainly do. I’m very pumped to talk about this. I think the only thing you missed is the like, oh, don’t worry. If we can’t do it yet, it’s on our roadmap. Oh, yeah. 00:01:40.86 [Tim Wilson]: That happened yesterday to a client. They turned to the vendor and they’re like, yeah, we can’t do that. And the response from the client, which was a very large company was like, well, is it on your roadmap? It’s on our QN plus one roadmap. 00:01:57.02 [Moe Kiss]: Yeah. 00:01:59.38 [Michael Helbling]: All right, and I’m Michael Helbling, and we wanted to bring on a guest, and we found a great one. Jason Packer is the founder of Quantable Analytics. It’s an analytics consultancy focused on analytics engineering and implementation. He’s also the author of the book, Google Analytics Alternatives, now in its second edition. And the genius behind the Measure Music channel on the Measure Chat Slack group. And now he is our guest. Welcome to the show, Jason. 00:02:24.65 [Jason Packer]: Thanks, Michael. I’m really happy to be here. It’s a bucket list item to finally make it on the podcast. 00:02:30.44 [Michael Helbling]: Well, it’s awesome to have you. All right, so maybe to kick things off, Jason, maybe just walk us through sort of what brought up the idea and was behind the idea of writing the book in the first place. 00:02:42.55 [Jason Packer]: Yeah, so I’ve always been really interested in evaluating software and knowing what’s out there, even back to my early days as a Unix administrator and software developer. I liked looking at all the different tools and back in the era when the Google Universal Analytics Universal Sunset was coming up. There was a lot of people that were asking these questions. There were a lot of people asking me these questions. And so I thought, well, I may as well start doing this research. Seems like a fun thing to do. And I started out thinking, well, maybe I’ll write a series of blog posts. And then someone at Columbus at the time, Web Analytics Wednesday, said, well, why don’t you just write a book, Jason? And that seemed like a good idea to me. And so I did it. And now a few years later, there’s some, you know, things have changed. There’s some new tools I wanted to look at. And I thought I would just, you know, make the same mistake again. So that’s here. Here we are. 00:03:47.46 [Tim Wilson]: Wait, who was it? Who was it? It didn’t last Wednesday. Who said that? 00:03:51.94 [Jason Packer]: It was Ahmad. Ahmad. Oh, OK. Nice. Which I think I credit him for in the first book, at least for the for the idea. 00:04:02.15 [Tim Wilson]: Look, I read it. I didn’t memorize the acknowledgments. Jeez. Come on, Tim. 00:04:08.73 [Moe Kiss]: But it sounds Jason like a big part of your process and like understanding the capabilities of the tool is like really playing with it, right? And I think one of the things that I’m often thinking about is like, I see folks trying to evaluate tools without getting their hands dirty. And so like, Do you think that’s what everyone should be doing, or is that just the thing that’s always worked for you? 00:04:32.16 [Jason Packer]: Well, I think everybody loves to have an opinion about a tool, and it’s very easy to form an opinion. You get in there, you see how it looks and how it feels, and that’s fine. I have opinions about that too, but you really have to balance that against really learning what the tool is about. And for me, the way to do that is to use it and to use it with real data, not to use it, not to watch videos about it, not to be walked through a demo by somebody, but to install it on a website, even if it’s just a trivial website. install it and use it. And that’s how I learn best. That’s how I learn most quickly. And do you think that using it with real data? 00:05:17.98 [Moe Kiss]: The bit that I’m taking away from that is it helps you understand it. But how do you think it changes the evaluation process itself? 00:05:28.04 [Jason Packer]: I think using real data will show you a lot more about where the issues are. For example, if you’re working with a vendor and they walk you through it, they’re going to show you the highlights. They’re going to show you the things that work well. They’re going to show you a tool that’s completely, perfectly set up. and we all know. That’s not how it is. In the book, everything that I evaluate, I used on real websites with real user data. For example, one of the issues with those real websites is one of them had a terrible bot problem. It was a site that I bought on the secondary market. I didn’t make the website, I just bought it. you know, it had some real traffic, but it was just like, you know, littered with bots. And so the traffic looked really weird. And like, there was all kinds of strange hits to pages that weren’t there. But that led me to learn a lot about how, you know, these different tools worked in the cases where there’s a bunch of 404s or there’s huge amounts of bot traffic. So like, that’s the difference between no vendor, whatever, like, show you a demo where 90% of the traffic was bots. That’d be crazy. And in some ways, it can be challenging to do that because that might not be your use case. So a lot of the things I talk about in the book is use case match. That’s the challenge as a tool evaluator is to match your constraints of your use case to the best match of a tool. Like I said, opinions, everybody’s got them. And there are in some ways in which some tools are more technically advanced than others, or some tools are faster than others or whatever. But it’s really about matching use case to tool through the lens of those constraints. 00:07:30.86 [Tim Wilson]: Back to the using the actual data, so the book was kind of digital analytics, product analytics stuff. I would put BI platforms in there, put data warehouse platforms. All of those when it’s like, you want to try it with your data. I mean, a really high bar or a real challenge seems to be, we want to do a bake-off or we want to do a proof of concept. We want to try it out. I’ve gone through processes where it’s like, we’re going to do the RFPs, we’re going to select some finalists, we’re then going to do a bake-off. And that does mean you’re fundamentally doing some sort of mini implementation and trying to draw the line of, you know, and that can include getting through some compliance hurdles to say, yeah, we’re using our real data, or do you say, well, we’re going to dummy up or we’re going to do an effort to make it’s kind of like our data, but it’s been anonymized to the point that it’s not our data, but it’s still mimics our data enough that we could actually try it in this platform. It does seem like companies To me, that’s what motivates a lot of the not wanting to go through that process. 00:08:48.93 [Jason Packer]: Ideally, it would be great to use your actual data and to do, like you say, a real mini implementation, but that’s just not feasible in a lot of cases. 00:08:56.60 [Tim Wilson]: I mean, Moe, have you done that? 00:08:59.32 [Moe Kiss]: Yeah. I’m not going to bait around the bush. I do a lot of… like analysis of different vendors and different tools and that sort of stuff. I would say I definitely lean towards the, we should do multiple POCs. Like the last major tool selection we did, I think I wanted to do maybe four POCs. And obviously like that’s a negotiation with the business and capacity and things like that. We ended up agreeing on two. But I think the thing that I found really hard is like, often the folks doing the evaluation and the assessment and those sort of things. I don’t know if the incentives are always there to do multiple POCs. I find that hard to reconcile with because it is. It’s really hard to understand how good a feature is or a particular capability that you’re looking for without stress testing it. And yeah, I don’t know if I just air too far on the POC side maybe. I think folks internally would probably say I do. 00:10:04.27 [Jason Packer]: I think that’s really challenging, right? Because a POC is great, but even before you want to get to that POC, you want to feel like you’ve narrowed it down to something that’s worth the effort there. And for me, part of that can be not even doing a real POC, but doing a toy test. Oh, let me do it with my podcast website. Let me do it with my… a personal website or whatever. That’s part of the reason also why I’m a big proponent of free tier, even on enterprise tools. That can be a challenge, right? Not everybody can offer that. Sometimes, if you’re talking about a huge BI platform or something, what would a free tier even mean if it’s even doing a simple example, implementation means putting in 100 hours of work or something. But the ability to get a little bit into the product before you really start talking about committing company resources to it, I think, because I do love the POC approach and the more, the better. But it can be hard to get those resources for sure. 00:11:23.11 [Moe Kiss]: Also, just getting it through security is a really big step. You’re basically doing a procurement process for something that you’re running a PRC on. It takes a lot of time and energy, but I obviously am very biased here because I lean strongly on the side that that’s worth it. Yeah, that’s my lived experience, but yeah. 00:11:46.77 [Tim Wilson]: Well, Bill, have you run into, because I can see the doubt. Say it’s only two, you get down to two tools and you get in and you’ve got multiple people who are all trying it and they all have different things they most care about. And then you get to the end of that, and you’re like, all we’ve done is allowed people to dig their heels in further on their preferred tools, because now they have hard evidence that that other tool doesn’t do this thing that I think is really important, and it does this thing. Like, do you wind up saying, well, this is supposed, we’re hoping that we arrive at a clear winner, but even if you do a POC of four tools, They’re still not the one clear winner and you’re still in kind of a negotiating phase. And you’re also setting up the people who didn’t back the ultimate winner to be able to say, see, we did the POC and I told you we shouldn’t have that one. Sorry, that’s just depressing me. No, no, no. 00:12:45.80 [Moe Kiss]: I can still remember like a few years ago, we were doing a BI tool selection. It must have been like five years ago and all the data analysts got in a room and we like this, this was the absolute worst way to do it. I would never ever do this. But we were like, how important is this thing to you when everyone would go to one side of the room or the other side of the room? And almost every time I was on the side of the room on my own. And I think, so it’s suffice to say we did not pick the tool that I wanted to get, but it is what it is. I think the thing that I find so difficult about data tools in particularly, and I know we had Colin on previously, from Omni talking about how especially BI tools, you’re trying to be many things to many different people. And I think what’s so challenging about data tools is data folks have very strong opinions about the things that they do and don’t want to work with. But also their opinions are normally representing what is best for them and not always what is best for the business. And that’s human nature, right? You think about what’s going to make your own job easier. And so I think I I often come with this perspective of a data tool is actually for our stakeholders. So even if it’s a little bit trickier or a little bit harder for us in our day-to-day, is it going to help our stakeholders in their relationship with data be better? Because I will up wait that. But I don’t think that’s the common. I’m not sure that’s necessarily a common view. 00:14:13.11 [Tim Wilson]: Michael, what’s your relationship status with SQL? 00:14:20.42 [Michael Helbling]: Oh, I think you know it’s complicated. It keeps gaslighting me with a syntax error near from, like, I don’t know where from lives. 00:14:30.46 [Tim Wilson]: Well, here’s a healthier relationship. Prism by Ask Why. You ask in plain English? Prism writes the sequel. 00:14:37.02 [Michael Helbling]: Ooh, like Revenue by Channel week over week, excluding refunds. And instead of me crafting a 47-line query and a three-line apology, Prism just does it? 00:14:46.36 [Tim Wilson]: That’s right. The best part? It doesn’t forget everything the moment you close the tab. Prism’s jam of memory remembers your reality, your definitions, your quirks. I mean, not your personality ones, but, you know, your coding quirks. 00:15:00.86 [Michael Helbling]: Well, but like the BigQuery table is the source of truth and conversion means this and not whatever gets decided by somebody like mid-meeting somewhere. 00:15:11.01 [Tim Wilson]: Exactly. So you don’t have to re-explain your business context like it’s a bedtime story for robots. 00:15:17.69 [Michael Helbling]: Yeah, I have to admit I’m a little tired of starting every session with previously on analytics. 00:15:24.28 [Tim Wilson]: And when Prism generates SQL, you get traceability. You can track changes, see what was created, and follow the logic. 00:15:31.25 [Michael Helbling]: I like that, because when somebody asks me where this number come from, I can stop saying, well, from the number tree. 00:15:38.33 [Tim Wilson]: It’s like version control for your analytics brain. 00:15:41.34 [Michael Helbling]: I like it. A little bit of accountability, but it’s convenient. 00:15:45.58 [Tim Wilson]: That’s right, so do you want in? Go to asky.ai and join the waitlist. That’s ask-the-letter-y.ai and use code APH to go to the top of that waitlist. 00:15:58.78 [Michael Helbling]: I like the idea of letting AI write some of the SQL. 00:16:01.56 [Tim Wilson]: And let your memory do literally anything else. 00:16:06.29 [Jason Packer]: No, I think it’s not. And I think, you know, everybody also wants to work with the cool that’s good for them. Like personally, well, like, right, like this idea of sort of like you’re implying that like, hey, I want to work with the new tool. I want to work with the cool tool. I want to work with a tool that’s good for my career. I want to work with a tool that my LinkedIn posts are going to be, you know, go with. And, you know, that, that. A lot of times that’s not the right fit. It’s really about the whole organization, not just the analysts, but a lot of times the analysts isn’t even really the one. 00:16:44.96 [Michael Helbling]: Flip it around and people want to work with a tool they’re familiar with. I used this in my last job, so I want to use it here. 00:16:51.57 [Tim Wilson]: Which was good when GA4 came out and Universal Analytics got sunset, then it was like, well, nobody’s familiar with it. So reset, yeah. 00:16:59.78 [Jason Packer]: Yeah, that’s what I was going to say, too, is that a lot of times a tool switch is not the right answer. We all like to think, hey, there’s a tool out there. The perfect tool out there that’s going to fix my problems is going to make my personal life better, my company do better, et cetera, et cetera. But there’s no perfect tool. There’s no Grasses looks greener, but a lot of times the tool you have now just isn’t implemented correctly. The new one you get isn’t going to be implemented correctly either. That can be a real challenge too, especially if you’re like, hey, I want to do these Hey, we’re going to do two POCs and put in all these resources. In the end, we’re going to say, oh, well, actually, I think the answer is that we stick with what we got and we just spend a little more time trying to improve our reputation. Nobody wants that answer. 00:17:52.95 [Moe Kiss]: In your experience, talk me through when There are trade-offs, right? We’ve all said no tool is going to meet the brief perfectly. How have you approached balancing those trade-offs? What’s your thinking? And how do you, when you’re working with businesses, convince them of the trade-offs they should make versus shouldn’t? 00:18:11.59 [Jason Packer]: Yeah, it’s really difficult because how I evaluate the tools from the book is a totally different mindset than how I think when I’m talking to an organization. A lot of times, I won’t even really be talking about the same things. In the book, I talk about the underlying tracking structure of different tools, the databases that different tools use, how they work with consent, things like that. And when I’m talking to a particular business, I listen for what their real pain points are. Is this an organization that they just need to get off of GA because of compliance issues? And that’s like, Then I focused their selection on solving those pain points as directly as possible, but also trying to not get into the weeds with them about the details of the tools that the people listening to this might find interesting because they’re not going to find that interesting. 00:19:20.08 [Tim Wilson]: I think you just kind of mixed it because part of what you did and maybe it’s worth having you What I loved about both editions, because the structure stayed the same, is that the tool by tool, blow by blow, and it’s not a feature by feature, but the tool by tool kind of write ups are the second half of the book. The first half of the book is you got to have kind of a framework of what matters to you. You admitted throughout, you’re like, there is no perfect categorization, but you just talked about one of those was the tracking methods. I could see for the right company, they would say, we’ve been getting burned by our current tracking method and we have got to find something. You’re like, cool, well, let’s then think about the philosophical difference from the different tools. If somebody else says, we just need something super cheap, it’s like, okay, well, then let’s talk about the nature of your digital experience in the different pricing models. If somebody says, we just got to get off a GA because it’s compliance. We actually love everything about it. Our compliance team has said we have to get off of it. I would say in the example you just gave, it was how you approach the book. It’s just where you’re going deeper in that understanding what attributes truly matter and then going deeper, right? 00:20:51.82 [Jason Packer]: Yeah, I think actually that’s fair. One of the things I’ve talked about is how things like that’s all about constraints and how price is a constraint. Price is a real important thing for organizations. It’s not the coolest thing to talk about when it comes to tooling. Similarly, it’s just a question of how you’re engaging with the decision makers, I guess. are in that first half of the book are just a long list of the things that I think about. I might think about a bunch of those when talking to a particular organization about a tool. I might not be talking to them about all those things, but I’m certainly thinking about a lot of them. I think it’s important to understand them to a certain degree. For example, in the new edition, there’s a chapter on server side. Obviously, I’m not going to teach someone everything about server side analytics in a chapter, a 3,000-word chapter of my book that’s not primarily about that. understanding at least enough about that to know if you’re talking to a vendor when they say, oh, yeah, we support server side. It’s easy. This is what you do to be able to understand, interpret what they’re saying, to know like, oh, well, really kind of like anybody could do server side. It’s not really about the tool, it’s more about the deployment about, oh, are you using server-side GTM to deploy that? And if you are, then this, and perhaps the real underlying problem is tracker blockers or something like that. And then your lens for viewing that is different. So that’s why I think that the The first half of the book, the guide part of it, rather than the product evaluations, is the lens in which I look at all product evaluations, and I’m trying to share that viewpoint in the first half. Tim liked it at least. 00:23:08.83 [Tim Wilson]: Can I ask, and this is probably also a question for multiple people like you, and you said it in kind of some of the earlier discussion that you explicitly did not talk to the vendors, even though they were, especially after the first edition, they knew you were doing the second edition. And they’re like, come on, just let our sales engineer help you out. You know, once you just understand, and I think you did that to say, I want a level playing field and I need to finish this book at some point. And if doing 15 POCs is tough, letting their sales teams get their hooks into you would be absolutely impossible. Whereas, yeah, and so whereas Moe, I feel like if you’re down to a couple, the where does sales play? So I don’t know, maybe you can talk through that. 00:23:56.68 [Jason Packer]: I mean, yeah, that’s sort of an unusual choice that I make in the book is to like, I mean, I definitely have talked and I know a lot of really great people at a lot of these vendors, like, especially after the first edition, you know, I’ve talked to a lot of these, these people and there’s a lot of them told you what you got wrong. Not as many as some, some, yeah. But it’s important to me that I was really, really fair more than I was particularly making any value judgments or anything like that. But the not engaging with them is about loving the playing field to some degree. It also fits in well with how I learn, like describing the learning from doing. Again, getting a demo account or some kind of account where I can use the product is the fastest way for me to learn rather than being on sales and engineering calls. But I think that was my case for writing the book. That’s different than most case with engaging with vendors from a large org that has specific needs. I think that a lot like engaging with vendor reps can be really, really helpful, but it also gives you an idea too of the culture fit between the product and your organization, which is a real thing. Something that when I started the first edition of the book, I didn’t expect to be so important, but is, I think, quite important. 00:25:36.33 [Michael Helbling]: Do you hear that, Tim? Culture is very important. I just wanted to reiterate that point really quickly. Sorry, you’re going to mo. 00:25:45.99 [Moe Kiss]: Just to add to that, I have personally found that engaging with sales, engineering support, whatever, is like a really big part of the process because I want to make sure that we can learn from their expertise that we’re not facing challenges that are very easily fixed. And I think part of then, even in the playing field, is making sure that you get that with all the companies that you’re PGO seeing. It’s not a favorites game. And you’re so right, Jason. Such a big part of it is about the culture or the ways of working that you then get to explore with that other company. And very transparently, I’ve talked about our relationship with Snowflake quite a bit and a big, big part of our success. I will rail on about implementation for years to come. But a big part of it is we’ve had really close relationships with their product teams, with their product managers, their tech leads, we will have calls like testing out new features and new functionality and being able to influence a roadmap. That is a huge, hugely important thing for us when we’re doing vendor selection because we want to make sure that in a year’s time, we have the kind of relationship where we can push their product if we need to. And so I think that letting those folks in the room so that we can stress test each other is a big part of the evaluation for me. 00:27:12.97 [Jason Packer]: Yeah, I agree with that. Again, it depends on your organization and why you’re buying the thing to start with. If you’re a tiny startup and you’re not really going to if you’re the thing that I hate is like you’re you’re a tiny startup and you’re you’re talking to an enterprise software provider and you know you get to the point where okay we’re ready to actually like talk some real prices and like okay well you know start for for your volume data we’re starting out with $65,000 a month and you’re like that’s my what are you talking about that’s like my entire yearly budget for all of my analytics so you know it I love transparency. I make that pretty clear book. And I think that like, you know, that’s just a great thing to get people on the same page as quickly as possible, because that’s super important. And I think that like, When you are engaging with the vendors, being transparent with them helps everybody. Nobody wants to seem like a dummy when they’re talking to a vendor, but if it’s a new tool, I don’t know the tool. They know the tool. They know they’re immediate competitors far better than I will. I try to be very direct about, hey, the budget is this and here’s my seemingly very stupid question. When you gave me answer, I didn’t understand, I’m going to just ask that stupid question again because it’s important to everybody that we find the best fit in the most direct way possible. 00:29:03.33 [Moe Kiss]: And I do think, obviously, I come from a place of absolute tech privilege. I think a lot about what are the hills. We say that all the time. We’re like, well. Well, anyway, I just want to be conscious of other folks have very different budget constraints when it comes to tool selection and things like that. But there are things I will die on a hill for. And one of them is, I do think, obviously, budget is incredibly important. But if there is a very good tool and it is not 10x, like other options, but it is a good fit. I personally think that is a fight worth having with the business, like getting support for that extra budget to make the right tool decision versus being so constrained by it that you make a really, really shitty choice. And again, everyone’s not in that position, but that situation you just described, Jason. Knowing the prices much earlier in the process is absolutely something that folks should be doing. You can’t wait till you’ve done a POC to start getting an idea of their pricing because if it is way out of the realm of possibility, you don’t want to waste your time and energy on it. 00:30:12.54 [Jason Packer]: I guess I haven’t thrown Google under the bus yet. There we go. Here we go. Yeah. One of the things that I think Google really made hard for is that they made a, with Universal, they made a pretty darn good product, and they made it free to just an incredible degree. It was technically free to 10 million hits a month, and in reality, it was quite a bit higher than that. And with GA4, of course, there’s no hard event limit. the one million per day export limit to BigQuery is probably the thing that people hit first. But they’re giving away so much for free. And that’s really caused people in the industry to think that analytics should be basically free, that the software should be free. And it’s very distorting. things really hard for new tools to come out there and to get a foothold in the market. It makes what Moee you’re saying as far as, hey, we need to understand that even if this tool is a little bit more money, think of the cost in people, the cost in data decisions in the organization. you’re really undervaluing analytics, and part of undervaluing analytics started with VA being free. And that’s still happening. 00:31:53.35 [Moe Kiss]: Oh, Jason, I feel like we could sit around and like… chat for hours because I think fundamentally one of the biggest mistakes I see is, yes, folks want to work on cool shit to put on their resume or LinkedIn or whatever, but it’s also the open source fallacy or the free fallacy, which is like, oh, this is open source or it’s free, it’s not going to cost us anything. I will push pretty heavily on like, that does not mean it’s free. We need to actually think like, we’re talking about a solution here that has five full-time engineers supporting it. That is not free to me. That is actually a huge cost to the business. And if we want to do that because we think that’s the right decision, that’s okay. But that needs to be a line item in our decision as well, not just the like on paper cost of the tool. 00:32:39.25 [Tim Wilson]: That also then extended if we’re going to have to support and we have those five engineers and one of those engineers leaves, what’s the size of the pool of candidates that we’re going to have to replace it, which is one of those where Market leaders and whatever tend to have a leg up, and it’s a legitimate leg up. They’ve achieved some critical mass. Nobody got fired for buying Tableau or Power BI. Part of that is because everybody’s been exposed and is familiar, but it’s also legitimate saying, well, if I need a Power BI developer, That’s a much larger pool to draw from, right? 00:33:20.45 [Michael Helbling]: Yeah. When Moee excitingly get ready for a new wave of that with AI, because now people are going to be like, it’s free. We can just build it with AI. It’s a question to Jason, sort of from your perspective, because obviously, we’ve all kind of been through vendor selection processes and we kind of touched on how Google Analytics is free. So obviously, as people are sort of jumping on the AI bandwagon and seeing how easy it is to prototype things, not necessarily build full-on products yet, but we’re moving in that direction, I would say. Do you think that’s going to be something that will enter the process of the build versus buy debate certainly changes a lot in the future? 00:34:08.82 [Jason Packer]: Yeah, I think so. I think that’s already happening. I think that it’s happened not exactly with AI, but the simplified realm of these tools like the In the book, I call them simplified web analytics tools, including things like plausible and fathom. Um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um, um And there are so many of those tools out there now. Every few months, a new one comes out. Some of them are quite good. It’s not a hard thing to prototype, and it’s even easier with AI. I could go in and especially if you start with the ones that are open source, you can be like, hey, build me an Umami clone. Here’s the Umami GitHub repo about it. And you could build yourself something like that pretty quickly. And some people are doing that. I think you run into some of the problems that Tim was talking about as far as having expertise with the tool. If it’s some internal tool, then the internal people are going to be the only ones that have the experience with it. And also, when it comes to some of the more complicated underlying database things, that’s I think far beyond the complexity of what AI can do a good job with. And so, you know, like, and also things like schema AI doesn’t do great job with, I mean, it can do okay, but it needs a lot of like, you know, human hand holding. So I think that like, ultimately, it’s like, a mistake for most organizations to try to think that they can build their own when there are so many really great platforms already out there. I think it’s maybe fine to think like, oh, we’re going to extend. We’re using whatever. We’re using Postog. And Postog is an open source tool. It’s one of the widest tools in the market. They have 34 different apps built into the tool. of whether you want session recording or feature flags or whatever, or LLM analytics, they’ve got it probably. And if there was something in there that you need to add to that, then using AI on top of that, I think to extend it, it makes sense. But trying to build a foundation to your analytics platform, your analytics practice, without really having that real strong attention to detail that these platforms that have been out there and tested have. It doesn’t make sense whether it’s, you know, humans building it. or it’s even worse if AI is building it. But I think it makes a lot more sense to build on some of the great tools that are already out there. 00:37:26.55 [Michael Helbling]: That gave me a cool weekend idea though. The mommy club coming out. 00:37:30.72 [Jason Packer]: Yeah. 00:37:31.04 [Michael Helbling]: Yeah. Just because you can. I don’t know. 00:37:34.58 [Tim Wilson]: But as you brought up post-hoc, because that’s one that I’m not familiar with, but you mentioned them because they kind of, well, actually, I think I sat and watched you talking to a different vendor and you brought them up a lot as being kind of like, who was the tool built by and for? Posthog is like, of by and for the developer. Google Analytics, Universal Analytics was kind of intended to be for the more casual initially, and they tried to stick with it. They’re like, this is easy. This is for the marketer. BI platforms seem like they’re similar. If you’re in the Power BI, you’re in the Microsoft stack, you’re like, this is built for the enterprise that wants to have the complete ecosystem and progression of tools. What do you think you’ve said here? You definitely said it in the book. What is the philosophy? What is at the What’s in the DNA of the company that’s building it? Who do they feel is the user that has primacy? Is that a fair… Absolutely. 00:39:01.22 [Jason Packer]: You and I have talked about that. Product philosophy and outlook is more important than any sort of feature comparison. Probably some people have heard me complain about this before, but future comparisons are helpful in some ways, but they also are already outdated. By the time you posted your future comparison list, it’s already out of date. The one checklist item that it says it does X, There’s so much to unpack underneath that, that their version of X might not be what you really think that you’re getting when you get that feature. And people want more features, like I just talked about postdoc, they’ve got every feature under the sun, they’ve got, you know, They’ve got session capture, but maybe you already have session capture. You’re already running Microsoft Clarity or HotJar or something like that. So while that looks like a great thing on a future comparison list, that’s not something you need, or that’s not something that’s going to help you. It’s going to add confusion to the product. The more features you add onto a product, the harder it can be to use. That’s just the way these suites work. But it can be really hard at the same time to understand, to peel back that layer of marketing a little bit and be like, well, what is really this product philosophy? Who is this for? you know, when you’re in that job, when you’re, you know, when you’re an analyst and you get, I don’t know, whether you get in front of like Adobe Analytics workspace, you’re like, oh, okay, I get it. This is, for me, this was written by people that have been listening to people like me. This makes sense to me and is useful to my use case. That It’s clear and a lot of times once you get in and use the tool like I was saying before, but when you’re just looking at the marketing, it can be like, well, both platforms say they have the ability to customize reports like done, like they’re the same when they can be wildly different. 00:41:10.19 [Moe Kiss]: Talk to me about this philosophy piece because I find this really interesting, the philosophy of the platforms. I think maybe I was alluding to a similar idea before, but how do people figure that out? Is the philosophy who the user is or the direction they want to take it? 00:41:28.25 [Jason Packer]: I think it’s not easy because, in a lot of ways, I think the vendors themselves don’t know a lot of times. It’s a product of the history of the company. It’s a product of what the target market of that tool is, and it’s a product of the people that built it and who they were listening to when they built it. Let’s take, we’re talking about possible on some of the simplified tools, they have a clear philosophy of this is a simple tool. There’s not going to be vast ability to customize reporting. Everything is going to be on one screen or pretty much everything is going to be on one screen. We’re not going to have a drown you in configuration options. This is designed to be simple and part of that is in that as privacy as well, is that it’s you’re giving up some amount of complexity to make it easier and to perhaps make it more private as well. That’s a philosophy. And I think that philosophy for them is, can be pretty clear, right? When you look at their marketing, you look at the sample, you know, you try out a similar product. It can be pretty easy to understand their philosophy when they communicate it well. And it’s not, you know, a sprawling platform with 27 different components or whatever. versus some of the more complicated tools like I talk about piano in my comprehensive category along with tools like Adobe. If you’re looking at either of those tools, they offer so many different features and functionality. And there’s a much more complicated onboarding process that it can be really hard to understand what that philosophy is until you get much further along in the process. I do think that talking to the vendor and engaging with them before you get too far along can help you understand that, but it also can confuse the process too. I don’t know that I have a real great answer. 00:43:37.60 [Tim Wilson]: I like that because you said you have to sort of where the company like looking at the roots of the tool and I don’t have a million examples, but I look at like in the BI space, you had Tableau, which was like one of the second generation of tools that clearly was like The shit should be drag and drop, and we should be able to customize it to conform to things that Steven Fugh would give a 10 out of 10 to. They were coming at it saying, it’s got to be a drag and drop, WYSIWYG interface that you can have highly customized to be very, very clean visuals. So there were a BI tool philosophically, I think it was forward on the quality of the visualization. Contrast that with DOMO comes along a number of years later. And I would say DOMO was saying, no, no, no, it’s all about the ease of connecting to all of your data sources. And they kind of led with the connectors. Now, they’re competing with each other, so over time, their sales teams are saying, we’re losing Domo, we’re losing deals because our visualizations are shitty, and Tableau’s getting pushback of saying, we’re losing deals because we’re not easy to connect to all these different things. not necessarily permanently handicapped, but it is one that I would say both of those tools. That’s where their various strengths versus weaknesses are, which means if I’m looking at a BI platform and those two are in the consideration set, I may be thinking, do I have stuff going pretty tightly into most of my stuff is going to go into a data warehouse and occasionally it’ll be pretty normalized and I want to hook into it and occasionally I might want to hook into something else or are we going to just live in a chaotic world where I’m always going to be needing to hook into a gazillion different data sources that are all going to be messy and I’m going to need to be able to do transformation within it. I think it does take a lot of and maturity or wisdom or thought to try to map where is my company’s kind of, which philosophical or historical underpinnings are most aligned with my needs and then stand up and say, and guess what? That means our visualizations will never be as good as what the perfect idea would have because that’s a lower, 00:46:09.08 [Jason Packer]: Yeah, that’s where I think I talk a lot about understanding fundamentals, how that can be really helpful to close. Some of that you’re talking about is the gap between the marketing that you see from the vendor and the reality. Part of closing that gap and understanding really what a tool is all about can be understanding the fundamentals of how particular things work. If we’re talking about you know, databases, right? If we’re talking about this product uses MySQL and this product uses Snowflake and this product uses Postgres and this product uses Clickhouse. Knowing just a little bit about the differences between those two tools is going to tell you a lot about the product, you know, like if we’re talking about We’re talking about, say, we’re comparing PivotPro and Matomo. PivotPro uses Clickhouse as the database underlying their product, and Matomo uses MySQL. On the surface, they’re pretty similar products, but they end up working quite differently because of that difference in the underlying database. MySQL is a simpler database. It’s something that’s easy to self-host. It’s something that’s easy to see the raw data from. It’s something that’s not super performant in a lot of more complicated analytical queries. And all those things surface in the products. And if you know that background and you know that It’s not like you need to know how to use those tools, but just knowing a little bit. The same is true for like tracking methods like cookies, tracking with cookies versus tracking with this IP plus user agent method or tracking with browser fingerprinting or whatever. Just knowing a little allows you to sort of see like, oh, the vendor says X. Oh, I think what they mean is this. There’s not as much as the vendor might say, it’s not like there’s a million new things and a million new ways to do things. There’s a limited number of ways. 00:48:12.40 [Michael Helbling]: Before I wrap up, I’m going to give Moe the opportunity to jump in one last time, but we do have to start to wrap up soon. But yeah, go ahead, Moe. I know you want to ask one more. 00:48:23.25 [Moe Kiss]: Jason, we’ve talked about a lot of different concepts and things you need to think about in this whole tooling decision space. If I’m sitting at my desk and I just take like your one like absolute, this is the thing that should be most top of mind from all the things we’ve chatted about today. What would be like the one thing that you would say, just if you pay attention to this, then you’ll probably make a slightly better decision. 00:48:49.96 [Jason Packer]: Oh, that’s a tough question. I might actually say price. So I’m disappointed. It is. I’m disappointed. I’d like to say something cool like the fundamental database schema or something like that. It’s a shortcut to a lot of putting you in the right area. I don’t want to do, but that’s where I would go. 00:49:27.05 [Tim Wilson]: Price is one input to a total cost of ownership. I mean, that’s, again, maybe another one. Have you ever come at it that way, Moee, with any of your… That’s a better, you know, total cost of ownership. 00:49:38.75 [Jason Packer]: Let’s just say that. Say I said total cost of ownership of price. That’s what I meant. 00:49:42.51 [Moe Kiss]: There you go. That needs to be in version three of your book, because I like that framing. Total cost of ownership sounds way better than… I think I do use total cost of ownership. 00:49:52.13 [Tim Wilson]: I don’t know if… Maybe it’s 10… I mean, I think it makes sense if you’re going through it differently. Philosophically, how much am I going to have to invest in added tooling to work around a limitation in their tracking or something? It could be, but yeah. 00:50:07.46 [Michael Helbling]: All right. Well, we do have to start to wrap up. This is awesome conversation and honestly so It’s a good conversation, because I think everybody deals with this in some capacity in their analyst career. So Jason, thank you so much for coming on the show and being our guest today. One thing we like to do is go around the horn, share last call. It could be any topic, anything at all, just something that might be of interest to our listeners. Jason, you’re our guest. Do you have a last call you’d like to share? 00:50:38.79 [Jason Packer]: So my last call is something that you already mentioned, Michael, which is Music League. Nice. Michael and I, and I’m how I believe your sister is a part of this as well. Music League is a, like, It’s a competition sort of, it’s a friendly competition where every week somebody, like there’s a theme, like this week’s theme in the music league that I’m part of is Beatles covers. So everybody picks a Beatles cover that they like, then a playlist is made automatically from that, whatever, 20 songs. and everybody votes in the ones that they like and fun as hell. It’s not complicated. It’s fun to do with your peers, your friend group, your work. We’ve been doing it on the measure slack for what, three years now or something like that? It’s, I mean, it’s a lot of fun. 00:51:40.10 [Tim Wilson]: Is it in the measure? What? For somebody who’s interested, they have to be in the measure slack and then in the measure music channel, they can find it. 00:51:47.46 [Jason Packer]: Yeah, that’s where the conversation happens. You don’t technically have to be part of that. But anybody can start musically too. And there’s also like free 00:51:57.09 [Tim Wilson]: You know, like, yeah, but we like to do stuff around, you know, us. Don’t just get people to go out and do their own thing. 00:52:05.14 [Michael Helbling]: They got to be part of the measure slack to do this. So join that first. Obviously, top tier. Yeah. Yeah. And the group is amazing. Like, that’s also great. We have tons of cool, fun, music-based conversations with all your peers in analytics and, um, It’s a lot of fun. So for my own personal experience, it’s it’s a great time. And I’ve got a great idea, Jason, because, you know, we’ve been growing as we grow and then we get the big power hour bump on this now. We can start like different levels of leagues. So there could be like a Premier League with relegation and a championship league like like British soccer, you know. 00:52:44.59 [Jason Packer]: I think I would be relegated. I’m not sure I would like that. I do not. 00:52:47.60 [Michael Helbling]: Well, I probably would be too. I don’t often score very well, but I have a lot of fun. Anyways, it’s also really cool to get a new playlist every couple weeks or so of songs you might not have ever heard or genres you’re not that into. So it’s nice. I like it. 00:53:02.80 [Tim Wilson]: So we do occasionally get comments from people who are like, you guys mentioned the measure slack where it’s like, if you literally go to measure.chat and then you join.measure.chat. And we’ll also have it on the show notes page. So if anybody’s like, you guys keep mentioning it and you, and it’s in our outro and we don’t have instructions for how to find it. 00:53:22.46 [Michael Helbling]: So listen, if you’re committed, you’ll find your way in. All right. No, thank you. Yeah, that’s awesome. And Jason, thank you for kind of being the oomph behind that, as I know it’s a ton of work on the back end to make it work. 00:53:36.24 [Jason Packer]: The official commissioner. 00:53:38.54 [Michael Helbling]: Yeah, the commissioner. The ska loving commissioner of the Measure Music channel. All right, Moe, what about you? What’s your last call? 00:53:50.95 [Moe Kiss]: Okay, so my husband has been listening to a podcast for a long time that folks will probably be familiar with. I have noticed it indexes highly to men. I know a lot of men that listen to it. I don’t know a lot of women. 00:54:03.93 [Tim Wilson]: Joe, you’re wrong. 00:54:08.85 [Moe Kiss]: Sorry. the Pivot podcast. One of the, my husband listens to it on like loud speaker around the house and it like really like drives me nuts. And I have not been the biggest fan of Scott Galloway. However, I have had my opinion changed very significantly. I am now a listener of Pivot. I have been incredibly impressed with how they’ve talked about I mean, AI and like tech over the last few months, but particularly the coverage on the Epstein files is something that I just really, like it really impressed me and that’s why I’ve become a really big listener. Scott also last month did this like resist and unsubscribe initiative, which folks might have seen in the media, which was really cool, which was like encouraging folks to basically use our economic power to let tech companies know that we’re not happy with how they’re supporting the administration. I felt like they were using their voice to share their perspective on something in a really meaningful way. Also, just for everyone out there, checking on the women in your life, the last few months have been like shaken us to the core. And so just to just check in on your, your wives, your mums, your daughters, all the women. 00:55:38.89 [Michael Helbling]: Nice. Yeah. Great. 00:55:40.63 [Moe Kiss]: All right. 00:55:41.50 [Tim Wilson]: Yeah, great. Yeah, Tim, what’s your last call? What have you got? Well, there was this episode of the Rogan. No. So, I’m going to do two. They’ll be quick. One, David Epstein, who I’m a big fan of like his books, like his videos, but he did a 15 minute video called why you should fail 15% of the time. And he talks about desirable difficulties, which is a phrase I don’t think I knew, but he kind of breaks down the value of doing hard things the hard way and specifically what that does for you, which in the world of vibe being shit, there are a lot of people grappling with it, but he’s just a well done video and he’s delightful to listen to. And then maybe kind of adjacent to that, there was just an article, I don’t know, it was the metadata weekly, Mark Dupuis, the AI analyst hype cycle. And I just, there were some quotes in it that were just, I thought were gems, like quote, if AI can only answer questions that have been preconfigured by the data team in a semantic layer, what have we actually built? An expensive natural language interface to existing dashboards, which And he kind of makes the case of where is this all going? It’s narrowing down to where what you actually get is maybe not that great. But they also had the analysts who thrive will be those who can translate business problems into the right questions, validate AI output, build the context systems that make AI useful and provide the judgment. and recommendations that AI cannot, which I think a lot of people are saying, but that’s kind of like a cheap throwaway thing to say when I look what people are then also saying, I did this thing. It often kind of skips those components of it. The AI analyst hype cycle by Mark Dupuis is my second one. Michael, what’s your last call? 00:57:43.19 [Michael Helbling]: Well, we did an episode a while back talking about semantic layers with Cindy Hausen from Thought Spot, which was awesome. And we also did an episode about AI that I remembered something Moee said about how So letting AIs leverage how the queries are being used in the organization is also a way of training the AI to do that. And I read an article recently from Jacob Mattson at Moether Duck about rethinking the semantic layer and kind of challenging the idea that a semantic layer is kind of the only way to go. And I just thought it was a cool counterpoint. I don’t know that I’ve got a strong opinion one way or the other. I very much respect the conversation we had with Cindy and I really thought it was really powerful. But there’s some interesting research and discovery going on as well on sort of like letting the AI consume all your SQL queries and using that to help it understand some of the context behind where and how your data is getting pulled together. So anyway, it’s a good read, good to kind of think through those things. I don’t think we’ve solved it for our industry. So I think it’s early days on all this. So yeah. Oh, and what’s this breaking news? I’m getting word now straight from our correspondent. There is a book out there that Jason Packer has written called. What’s the name of the book again? Hold on. I haven’t written down Google Analytics alternatives. And for listeners of the analytics power hour, he’s going to give you a 20% discount. So that’s pretty sweet. If you haven’t already bought the book, that is the incentive to do so. Discount code APH. So there you go. We’ll put the link to that in the show notes as well. All right. Well, Jason, once again, thank you so much for coming on the show. This has been a lot of fun and a really good conversation. Appreciate all the work you’ve done. It’s a labor of love, I’m sure, just to do all this. And so very much appreciate it on behalf of a vendor weary industry, I think you’re doing us all a big service. So thank you. 00:59:50.85 [Jason Packer]: Thank you. You’re welcome. Yes. 00:59:52.19 [Michael Helbling]: Great time. All right. Well, we’d love to hear from you too, because you’ve been listening and you probably have questions or you’ve got thoughts. And so reach out to us and you can do that on the Measure Slack chat group, which we’ve spoken about on the show. as well as our LinkedIn page or via email at contact at analyticshour.io. And we also love to get your comments and ratings on whatever podcast platform you listen to. Please feel free to do that as well. And I think I speak for both of my co-hosts. 01:00:23.72 [Tim Wilson]: Boy, have you listened to a few things that are a little important. If only our show prep had it. So one, just know that Michael, you and Jason and I will all be at Measure Camp New York on the 28th of March, so if you want to see us. 01:00:39.95 [Michael Helbling]: That is true, we will. 01:00:41.09 [Tim Wilson]: But even more important. 01:00:41.97 [Michael Helbling]: I didn’t expect by now there’d be tickets left, so I was leaving that out because it’s too late, you probably can’t make it. Well, there’s something that’s available. 01:00:50.97 [Tim Wilson]: If you can get a ticket. Moere important from an operational perspective, the marketing analytics summit that we’ll be at on April 29th, Yeah. Okay, now you’re… Yeah, I did skip that. 01:01:04.63 [Michael Helbling]: I did skip that, yeah. Moere breaking news, I’m getting… Yeah, we’re going to be a marketing analytics summit and we need your help. We want your questions. We’ve got a very cool survey of which there’s an Easter egg at the end that I had no part of. And we’ll have to take the survey and ask a question to see it. But yeah, go to analyticshour.io slash listener and submit a question. We’ll be recording at the marketing analytics summit at on April 29th in Santa Barbara, California. And we hope to see you there. But if you can’t make it there, we can still ask a question and we may answer it on the podcast. So please do that if you want to ask us a question. And even if you don’t want to, push yourself a little bit and ask what anyway. I highly preference questions that make Tim feel uncomfortable. So like, you know, asking emotional questions about, you know, the best manager he ever had or 01:02:09.06 [Tim Wilson]: Yeah, luckily, we have not figured out how we’re sharing access to all the questions with all the co-hosts. 01:02:14.85 [Michael Helbling]: Oh, yeah, that’s the little tricky part of that. All right, well, before I forget anything else about the show wrap-up, let me just say thanks once again, Jason. And I think I speak for both of my co-hosts, Moe and Tim, when I say, no matter what vendor you need to pick, just keep analyzing. 01:02:37.26 [Announcer]: listening. Let’s keep the conversation going with your comments, suggestions and questions on Twitter at @analyticshour on the web at analyticshour.io, our LinkedIn group, and the Measure Chat Slack group. Music for the podcast by Josh Crowhurst. Those smart guys wanted to fit in so they made up a term called analytics. Analytics don’t work. 01:03:01.49 [Charles Barkley]: Do the analytics say go for it, no matter who’s going for it? So if you and I were on the field, the analytics say go for it. It’s the stupidest, laziest, lamest thing I’ve ever heard for reasoning in competition. 01:03:15.95 [Michael Helbling]: All right. Well, we do have an editor who we’ve been talking so fondly about. So we can stop and start as needed. 01:03:31.43 [Tim Wilson]: Well, without further ado. Well, actually before, so just like Moee, are there any, because I mean, you’re kind of often in the midst of vendor selection stuff. So you’re comfortable. There’ll be anything you talk about. You can yourself edit for whatever, named and unnamed. 01:03:49.58 [Moe Kiss]: Yeah. So like we just signed a new BI tool, which I probably can’t. say, but I will just say I’ve been involved in multiple BI tool selections and stuff like that. Okay. 01:04:02.06 [Michael Helbling]: Yeah. All right. All right. Let’s start clackin’ the keyboard and record this thing. 01:04:14.42 [Moe Kiss]: I think I got it. You got it? I think so. I was like, I better do it before you start, because if I do it half a year, it’ll be like… 01:04:23.24 [Michael Helbling]: That was great timing actually. 01:04:26.59 [Moe Kiss]: Pretty sure I got it right. 01:04:33.76 [Michael Helbling]: Here we go in five four 01:04:47.01 [Tim Wilson]: Rock flag and an instrumental rock flag rendition by our guest. 01:05:13.64 [Michael Helbling]: Oh my gosh. That’s the permanent one at the end of every show now. That’s incredible. 01:05:24.53 [Tim Wilson]: I don’t know why I was showing that it was going to play that one, and instead it just played like Transition 2, so it’s good. 01:05:32.12 [Moe Kiss]: Fucking Rostat. The post #293: Tool Selection and the Unhelpfulness of Feature Comparisons appeared first on The Analytics Power Hour: Data and Analytics Podcast.

As Kevin McCallister once taught us: just because the house is still standing doesn’t mean everything’s under control. Everyone’s racing to adopt AI, but has anyone actually read the fine print? For this year’s International Women’s Day episode, we are joined by Aubrey Blanche to unpack the hype, the hidden tradeoffs, and the quiet ways teams are giving up agency in the name of “productivity.” We explore how data and tech teams are uniquely prepared and positioned to ask better questions, measure what really matters, and avoid letting the AI teenager run the house. Learn more about “phantom value” and why faster isn’t always better… or even cheaper! This episode is brought to you by Prism from Ask-Y—your agentic analytics platform for automating analytics, exploring data, creating repeatable workflows, and delivering accurate insights—all without the need for manual query writing. Links to Resources Mentioned in the Show The Adolescence of Technology: Confronting and Overcoming the Risks of Powerful AI Exposed Moltbook Database Let Anyone Take Control of Any AI Agent on the Site Disempowerment patterns in real-world AI usage AI safety is not a model property: Trying to make an AI model that can’t be misused is like trying to make a computer that can’t be used for bad things The AI Marketing Canvas, Second Edition: A Five-Step AI Plan for Marketers Should your AI notetaker be in the room? Heated Rivalry I Don’t Care What You Build (And Neither Should You) And the diagram Moe referenced: Photo by Johanneke Kroesbergen-Kamps on Unsplash Episode Transcript00:00:05.75 [Moe Kiss]: Welcome to the Analytics Power Hour. Analytics topics covered conversationally and sometimes with explicit language. 00:00:14.99 [Moe Kiss]: Hey, everyone. Welcome to the Analytics Power Hour. This is Episode 292. The world of AI is moving lightning fast, and I think it’s fair to say that most of us are struggling to keep up. I know I am. There’s new tools, new capabilities, new risks, new headlines, and they seem to land pretty much every week. It’s getting harder to separate what actually matters from the hype. So for this episode, we want to chat deeply about what all of this means, especially when it comes to ethical AI and the real world conundrums that we’re all facing in tech right now. If we’re honest, it feels a little bit like we’ve left the teenager home alone and that teenager is AI. The house is still standing right now, but maybe it needs a bit more supervision. But before we get into it, let me introduce my co-host, Val Kroll. Great to have you here. Hey, Moe, excited to be here. I know. And this is actually our special International Women’s Day episode. So we have an all women class today, which is awesome. But it’s also really fitting because today we’re going to welcome back a guest who joined us many years ago for another important conversation about creating balanced teams and avoiding group think. So we’re really thrilled to have Aubrey back on the show. Since her last appearance, Aubrey has remained a sharp and influential voice at the intersection of technology, power, incentives, and human impact. She’s held senior leadership roles, including as a director at the Ethics Centre, a VP for equitable operations at Culture Amp, and as the global head of diversity and belonging at Atlassian. She’s also served on lots of influential boards and advisor roles in the tech community. She’s currently completing a master’s of AI ethics in society at the University of Cambridge. So, with that amazing wrap, so far to say, she’s been spending a lot of time thinking about AI, about agency, about risk, about what responsibility actually looks like for the people building and deploying these systems. And I’m so excited to dig into what’s been on her mind, what she’s been reading and what she’s been thinking about. So, Aubrey, massive welcome back to the Analytics Power Hour. To kick us off, what the current state of AI. Let’s just open it up. What feels like the most important bit to get stuck into? 00:02:34.48 [Aubrey Blanche]: Oh, my gosh. I love the metaphor you’ve used about the teenager. I have to say, there is no greater joy for a neurodivergent person than someone asking about our special interests. And I think it’s mine, so I’m so happy to be here today. But for me, I think the headline of what I’m trying to convey is actually about working with more intention in AI. So right now, there’s an incredible amount of hype, and in that hype, there are narratives that are just fundamentally untrue. So, there’s this narrative about inevitability. Now, the reality is the teenager exists. Now is not the time to debate whether we should have a teenager because the teenagers are in the house. But I think that the idea that AI is going to do XYZQ is actually not written yet. Yes, there are structural incentives that make certain things more likely. But that doesn’t mean inevitable. And so I think if we all embrace this idea that the way things have gone is not the way they have to go is a really powerful counter narrative to what folks who quite frankly, you know, the commas in their bank accounts depend on you believing that AI is inevitable and going to create value and things like that. And I’m not sitting here saying that can’t happen, but for my professional expertise, having scaled some of the most influential tech companies in the world, I actually think the way people at multiple levels running tech companies, certainly legislators in Australia are thinking about this, actually decreases the likelihood that we get to the good stuff and increases the likelihood that we get to the bad stuff. But again, we can choose differently. 00:04:12.80 [Moe Kiss]: Okay, so what you’re saying is that at the moment, people think it’s inevitable that AI is going to be very influential, have all these amazing outcomes. But the way we’re approaching it right now actually increases the likelihood of the less good outcomes, but not necessarily the good stuff. Is that what you’re saying? 00:04:32.20 [Aubrey Blanche]: Yeah, so let’s take like there’s this idea around that AI makes us more efficient and productive. Okay, so that’s not like on its face completely false, but it’s actually incredibly dumb way to conceptualize the goal of AI. So if the goal is efficiency, what you’re saying is, I’m wedded to the status quo, I just want to do it faster. And it takes a very particular person to think the status quo is sufficient is the way we want to run the world, right? Like you have to be having a pretty good time. Lots of those comments. Lots of those commas or you just like aren’t likely to get killed by police walking down the street. And so, but I think what if we flipped it on Ted and we said we actually believe that AI can be used for increased innovation and value. Right. And efficiency might be one of the tactics that we use in certain scenarios to achieve that, but efficiency itself is a bullshit objective. And so when I see, you know, in the case of like a national plan around AI that’s specifically focused around productivity, what I don’t see is consideration around the questions of what are we producing? Can we produce new things that actually are of higher net benefit to a broader set of society? Like we should be having those questions, but because I think there is such a gap in understanding how this technology works and the implications between folks who are sort of running companies that are building it and those that are trying to regulate and govern it, we’re not having the productive conversation we could have. And at the end of the day, I think it means we’re not going to get access to a lot of the benefits that are possible. And so part of my prescription for solving that, because I’m not seeing leaders, quite frankly, act in a way I’d like to, is that we each take on our own little sphere of influence and say, how do I make principled decisions about the use of this technology in the sphere that I operate in every day? Because that actually does make a difference. 00:06:42.23 [Val Kroll]: So I’m curious if the reset on the objective, is that one of the ways that we can be more intentional upfront? Is that one of the things you’re thinking about in that space? 00:06:52.26 [Aubrey Blanche]: Yeah, I think so. Because I think if we agree that the objective is innovation or human flourishing, then we can then say, oh, it’s not about I’m going to throw AI on everything. It’s about saying, what’s the class of problems for which AI is an excellent tool? and what is the best way to use that technology in that use case to achieve that objective. It completely changes the analytical frame. It doesn’t mean we run away from the technology, but it does mean that we probably aren’t yet chasing phantom value that there isn’t a lot of empirical rigor to suggest that it actually exists. That’s the thing that shits me a little bit is everyone’s like, oh, we can reduce our workforce. Okay, maybe that technology will work, but it lies to you a lot. And so like maybe you don’t want to get rid of the humans quite yet, even if it looks good for your P&L and your ASX quarterly results. 00:07:52.95 [Moe Kiss]: One of the things I read recently, and I can pop something about it in the show notes, it was by Jim Lysinki. And he had kind of like a framework for thinking about this, which is like a quadrant. And it’s like basically like in the bottom left quadrant, he’s got like internal productivity. So quick wins, repetitive tasks. And like in the top quadrant, is really about external growth. It’s about entirely new revenue streams, addressing new customer problems. Really innovation is the way that I would think about it, that bucket. And one of the things, it was a very marketable way of framing it. But I really like the way that he thinks about, we need to get from that bottom quadrant to that top quadrant. And I feel like there’s a lot of commonality with what you’re saying here as well about, We’re doing the dumb shit with AI right now. 00:08:49.19 [Aubrey Blanche]: Yeah, and I’m not saying that we can’t, because I’m happy to share this, but I just wrote a piece about the ethics of using AI note takers, because I tortured myself about it for a while. And then I started using it, and I was like, wow, this is really great for my brain. And so that is an example of where something that’s basically become automated is actually a value add for me, because I’m doing more interesting stuff with it. Yeah, like that’s not an innovative, like I’m not creating something new with that. And so yeah, but I haven’t seen that, but I would agree that I think there is just more that we can do. Now, I have a classmate at Cambridge. She’s incredible. Her name’s Oya. And she talks about this, like the idea of co-intelligence. is that so many people think about AI as replacing a human, which is this very capitalistic, I’m just trying to reduce my operating costs question. But she does the most interesting, amazing stuff, but her contribution is that she talks about co-intelligence, is that looking at the way that humans think and the way that machines operate and working together actually creates more value for organizations, So in that way, these ideas I think are just not being talked about because people are so focused on the short-term returns that they’re getting. But I think if we start to optimize over longer time horizons, these ideas around experimentation and innovation and value creation potential actually expands those possibilities. 00:10:26.53 [Val Kroll]: Does it seem like, I guess my perception, I shouldn’t say it doesn’t seem like, my perception is that lots of organizations focus on some more of those productivity or replacing the headcount. not only for the P&L purposes because it feels safer because it’s like, oh, it’s behind closed doors. It’s not like I’m throwing a chat bot out there that’s going to do something dangerous to my customers or make promises or hurt my business in any way. It feels like a safer way to test it out to expand into those areas. First of all, I question if it’s actually air quotes safer to be doing any of that because it’s still playing with resources and people, but is that why organizations start there or what’s the common thread between being that bottom quadrant before they could start to enable more of the air quotes good stuff? 00:11:16.90 [Aubrey Blanche]: Yeah, I think part of it is a lack of appetite for risk taking, or risk taking of a particular type, I should say, because the way that risk is thought of. So there’s some research, and I’m sorry, I can’t remember the citation, but they were talking about how 80% of corporate leaders felt that they were behind on AI. But when you looked at the data about where their companies were on AI adoption, they were either averaged to slightly ahead. And so people’s beliefs about what’s happening and what’s actually happening are quite divergent when it comes to the use of AI and organizations. And so I think there’s a particular kind of organizational risk management where, yes, of course, no one wants to put in a chatbot that starts identifying as Mecca Hitler, which you can Google, and the answer is Grock. But yes, so I think there’s a particular corporate risk of saying, because the reality is once you get into that innovation quadrant, like the percentage of things that fail goes up and they fail in unpredictable ways. So this is something when we’re talking about generative AI in particular is a known property of these models. like large AI models fail in unpredictable ways. And so the level of risk that an organization is taking is high. Now, I would say there are certain types of risks that aren’t necessarily being managed in the same way. So I rarely see corporate leaders, unless they’ve specifically engaged me to talk about this, considering the risk of Earth’s dwindling, you know, freshwater supplies, like when they’re thinking about AI adoption in their organizations. And so that’s something I would say is, again, I just think there’s a conservatism in organizations as holding them back from achieving the benefits and also having principled and hard boundaries about places we shouldn’t be using this technology or shouldn’t yet be using this technology. 00:13:15.09 [Moe Kiss]: How like, I don’t know, I think one of the things is like I was prepping for this episode. A lot of what was rolling through my mind is like it felt a lot like the privacy debate of a few years ago, where like individuals would give up their their privacy for like little personal wins. But if you’re a big corporate, maybe you have to be more stringent. And this feels like a similar space where people are willing to accept a shittier output for something like low value, but high value. Actually, the stakes are higher. I guess what I’m trying to really conceptualize. When you’re in a technology business, how do you think about those higher fidelity problems and what the guideline should be of where it’s acceptable to use it here? How do businesses do that other than just paying you lots of money, which I highly endorse is a good decision? 00:14:10.12 [Aubrey Blanche]: Oh, thank you. I mean, one of the things, I was just chatting to a pro bono client yesterday, and they’re a particular organization in that kind of an off the shelf like AI governance framework and like decision making principles like is not appropriate, like we have to do something fully custom. But in my mind, the way to like get to this is first to like craft an organizational perspective on AI use. So whether you call that acceptable AI use and AI policy, But that should detail kind of the vision and the general beliefs that you have about how this technology is used, probably have a set of principles that guide particular decisions. Then I think you should have pre-worked through at least a handful of anticipated scenarios that are going to come up. But then you also need to do enablement for employees, not just on how to technically use the tools, which I think is important, which has to include safety and responsibility behaviors, But also, actually, most importantly, teaching the individuals within the organization how to apply the decision-making framework that you’ve made. It should be grounded in your values, the particular positioning of your organization. And that’s something that I love. I obviously do this in my consulting with AI, but at the Ethics Center, one of the reasons that I joined was because when I was chatting to Simon, the executive director, one of the things that really struck me was he emphasized that at the ethics center, our mission is to bring ethics to the center of everyday life, but we teach people how to think, not what to think. And I think that we need to take that principle into AI because the reality is so many of the problems and the challenges that people face around this technology is because they actually haven’t been given a framework of how to make decisions within their scope. And I think there’s a special risk because most of us have not grown up actually being caught ethical decision making in particular. And so there’s a skills gap in the workforce to actually be able to, and there are ways to Think about ethics and responsibility in a structured way, but most people haven’t been exposed to a framework or a process to be able to do that for themselves. 00:16:22.21 [Val Kroll]: Michael, how many tabs do you have open right now? 00:16:25.62 [Michael Helbling]: Oh, I’d say enough to qualify as a distributed system and probably a cry for help. 00:16:31.69 [Val Kroll]: Well, same. If you’re an analyst, you’re basically full-stack now. Excel, BigQuery, SQL, dashboards, plus explaining conversion like it’s a bedtime story. 00:16:44.14 [Michael Helbling]: Yeah, and every tool wants the same context over again. Which table? What’s revenue? Why is July doubled? Okay, sure. Whatever. I guess I just live here forever now. 00:16:57.34 [Val Kroll]: Well, that’s why we’re hyped about Prism by Ask Why, the AI analyst moment. You ask in plain English and Prism orchestrates across your stack, queries, views, charts, all without the constant tool hopping. 00:17:10.04 [Michael Helbling]: Yeah, it’s context-focused too. It remembers your definitions with the jam memory. I mean, it will literally hang onto what does conversion mean in your world. 00:17:19.36 [Val Kroll]: And you can save your best workflows as skills, portable expertise you reuse anywhere, like clean GA4 medium field variations so you don’t reinvent the same duct tape logic weekly. 00:17:31.39 [Michael Helbling]: Yeah, and there’s community skills. Stuff other analysts have already proven works. So you don’t end up debugging a formula. Sometimes it looks like some kind of ancient ritual or something. 00:17:43.37 [Val Kroll]: Prism also does SQL views with version control, so you can save, commit, and rollback changes like a responsible adult. 00:17:51.69 [Michael Helbling]: It sounds amazing. It’s built by analytics practitioners, and there’s free early access while they continue to refine and build the product. 00:18:00.60 [Val Kroll]: So to go see for yourself, go to ask-y.ai and join the waitlist. 00:18:06.12 [Michael Helbling]: And the best thing about it is use code APH and that will jump you to the top of the waitlist. That’s ask-y.ai with code APH. Just think about it. Fewer tabs, more answers, same chaos, but this time more organized. 00:18:25.10 [Val Kroll]: But how does an org, I guess like, you know, outside of bringing in people from the outside, I’m just curious about who inside of an organization is best poised or could mobilize to think about the valuation of those types of risks. I like how you said in scope. Everyone’s role probably has a different amount of risk that they should be allowed to take or comfortable taking, but how do you start to think about assessing that risk? 00:18:52.71 [Aubrey Blanche]: So I think it’s, and there’s so much debate in kind of academic circles about like where accountability sits and how governance structures work. And so I think it really, but for my, I hate to say the answer is a committee, like in general. But I kind of think that in that you need a set of people to think about these risks. You need folks who are actual risk management professionals who understand those processes, but you also need an ethicist who understands the use cases and market that you’re in. because the risks of AI are so unique to how it’s being used. There’s a side note that AI doesn’t mean anything. It’s like a giant bundle of technologies doing a bunch of stuff. You need an ethicist who is qualified to speak to you about those particular issues. You also need someone to represent the customer or the external face of the company because there are major reputational risks and considerations in this. as well. And then you probably need some technical folks who can reign us in when they get a line about what’s actually achievable. So if we’ve decided philosophically this, we’ve made an operational decision, but what does that actually look like in terms of developing or delivering a product or in putting a tool in the hands of our employees? I recently learned about a company that’s based in the UK, that they are incredibly rigid about their responsible AI approach to the point where any employee at any time can raise an ethical issue about something they’re doing with AI that can actually be deferred to an ethics council for debate and dissolution. Wow. So really, really cool. And so I offer that as an example that like If they wanted to, they would in the sense that like, yes, we can’t mitigate our risk. I’m not trying to say that, but we could do much better than we currently are if companies had the will. And as someone who spent a lot of my career in DEI, so diversity, equity, inclusion, doing kind of anti-discrimination and social justice work across tech, like the number one factor that I have seen in whether programs that are about responsibility and ethics and social justice, et cetera, does the CEO care enough to keep it funded when every other incentive in the company is to shut it down? 00:21:17.94 [Moe Kiss]: I want to push you a little bit because I feel like folks will be like, listen, this episode will be like, yes, I want to do this. I want to go into my organization. And someone is going to say this. And I’ve sat around with you and debated many a time. And I know I’m going to say Aubrey is probably like the best, best person ever to have a discussion with because she’s so good at like reframing things. So sorry, I’ll turn down the fan girl right about now. But so someone in the organization will be like, yeah, cool. We can build some frameworks and guidelines. But AI is moving so fast. By the time we build the guidelines, they won’t be relevant anymore. So how would you handle that conversation? 00:21:58.42 [Aubrey Blanche]: Okay, I kind of think it’s silly. I wouldn’t say that if I was in an actual debate because I care about influence and changing people’s minds. But no, so I think that’s true. But that’s why I think for me, and there is debate about this, so I don’t want to act like it. But like, for me, principle-based frameworks actually solve some of that problem. because the idea is if you get into a framework where it’s like this is in this is out and you have a laundry list. Yes, that’s going to get stale really quickly because the way the technology works is going to change fundamentally or or the way it’s being deployed is going to change really quickly. But the idea of For example, a company could make a decision that says, we don’t deploy technology that makes decisions about humans into the market without having done thorough impact assessments measured for the potential of bias. and also developed a process for someone to alert us if something has gone wrong. You can decide that, and the underlying function of the technology changing actually doesn’t change that as a governance structure. The way you achieve those things may change, and so you need to be flexible and always willing to update your processes. But so yes, I do think it moves fast, but the idea that like, oh, it moves so fast, we just can’t do the right thing is like the bullshit that the tech elite has been selling us for decades, because it’s more convenient for them and because it maximizes their profits. And I want to say something really specifically. There is a difference between believing Profits should always be maximized as the primary goal and like we can maybe give up a little bit of that to not destroy civilization So there’s often this binary of like oh you like you hate money or like you want to make all the money in the world like no We could make principal decisions that yes may actually have some potential like marginal impact on profit but like I sometimes push leaders to say, are you standing behind the behavior that maximizing your profit is more important than the welfare of your employees or customers? And would you be willing to say that to the media? Because that’s the implication of your decision. And so I’d put that to folks to say, if you believe that, there’s probably nothing I can do to help you. But if that’s not what you mean, we can actually take different actions to align those values and beliefs. in a way that supports business, supports growth, but also balances the kind of risks that come off. So like the middle way is possible. And so I just want to call that out is like, it’s not one or the other. There’s a giant spectrum in the middle. 00:24:33.72 [Val Kroll]: I think that a lot of, especially thinking about analysts working inside of organizations are feeling disconnected from those larger implications when they’re deciding which note picker am I going to send to the meeting to pull on that strain from earlier. But I guess, is there anything that you would offer or suggest for someone inside of an organization that has access to use those tools internally, but maybe hasn’t been given a lot of guidance, but wants to be a good actor in all this, that maybe they’re not going to be the one to run up the flagpole to the CEO, that we need to be doing all these things, but is there anything in the middle for them that you would suggest they keep in mind? 00:25:16.80 [Aubrey Blanche]: Yeah, I think it’s actually the same kind of advice I would give to anyone who wanted to be kind of an advocate or an activist within an organization is look at who you are. So what’s your position in the world? And then what power do you have in the organization? So people often think of power as like formal power, like I can hire you, fire you, promote you. But things like, do you have relationships? Do people trust your judgment? That’s the type of influential power. And to say like, number one, make your decisions for you. So we’ll use the note-taker example, because I’m like all about it. 00:25:53.79 [Moe Kiss]: I’m going to ask you to like walk us through how you made those ethical decisions. 00:25:58.99 [Aubrey Blanche]: Sorry to distract, but. No, and I have a whole article that you can put in the show notes, but basically the like, Let’s walk through how someone says, okay, I can’t control this, but X tool has been white listed for or allow listed for note taking in my organization. I’m going to decide how I’m going to use it. So I’ve decided that there’s utility benefit to me, like there’s an obvious benefit, but there are harms in terms of potential privacy of data leakage issues if they’re training on my data or depending on where that data is stored. And so for me, when I walked through that, I said first, like, one, are they using my data to train? I don’t ethically stand behind companies using my data to train their models. My economic argument is that’s a resource they’re not paying for. I’m paying them for the service, and then they’re extracting value from me, like that doesn’t feel like equitable value exchange. It also exposes myself and the people that I have meetings with to potential security issues. So a lot of these companies are newer. They don’t have the robust security architectures that you would expect of enterprise tools. And also the use of AI creates security vulnerabilities that are often unanticipated and there’s a huge rise in like AI assisted cyber attacks. So data leakage risks are just higher. So for me, that meant, okay, I’m turning off data training in the tool that I use. It’s important to note that companies have an incentive to keep that turned on by default. So you have to go and turn it off. I think that’s an unethical design choice. I think the default should be off and people can opt in if they want to. It’s a dark pattern. Also thinking about, have I done my due diligence about the security practices of this organization? So I want to look to see if they have a SOC2 type one or type two certification, if they have ISO 27001. And then ideally, so those are like standard security control certifications. There’s also a new standard called ISO 42001, which is the AI management system standard. So this is something I would want to see, but recognizing that I think somewhere slightly north of 50 organizations in the world, it is believed have the certification at this time. I do want to give a shout out to Culture Amp. because they are one of the companies that has that certification. And I love that about them. But so those are the things I would look at. And then I’m thinking about how I’m gathering consent to record. So depending on where people are in the meeting, that might be a legal issue. But it’s also an ethical one that people need to be able to opt in. And so for me, the qualities that generate fully informed consent are one, everyone knows they’re being recorded. They understand the risks that they’re taking with their data and how it’s being stored. And also they feel full agency to opt out. And because the note taker I use doesn’t call into the meeting, so no one can see that it’s there, I explicitly start the meeting by saying, hey, I really like to use an AI note taker so I can be more present in meetings. I want to be thoughtful that this does not train on your data, but the data is stored in the cloud on AWS. So recognizing that if you’re boycotting Amazon, that might not work for you. If you have any problem with this, I’m happy to take notes by hand. So I start my meetings where I choose to use that, but I also only use note takers in meetings where there isn’t sensitive or confidential information being shared, so I don’t take any risk with that data leaking. But that process, yes, I’m literally an ethicist, so I do that, but you can follow people who do that pre-work for you. So you have control over whether you use that note taker and the costs and benefits but I would say like for me I curate my social media ecosystem with a lot of people smarter than me so that I for a lot of things can. kind of skip the deliberation process because they’ll walk through their thinking and I can go oh you shortcutted me and I figure out the ethical decision I wanted to make and that’s something that a lot of my online content does is I try to talk through how I’ve thought through a problem so that people can decide if like that’s the lane they want to go in also. 00:30:10.88 [Moe Kiss]: I love this. I love also how much personal responsibility you’re demonstrating because I think sometimes it can be really easy to be like, well, someone should give me guidelines and someone should sort it out. And like, I can’t do anything within my sphere of influence. And I really love to like stand some personal responsibility. I’m curious to hear your thoughts specifically as it pertains to data practitioners. Like, I think there’s another layer on top of that, which is we’re often privy to user data, first-party data. There’s a lot of ways that it’s unlocking incredible value for data folks. But do you think there’s another layer of dimensionality on top of that we should be thinking about, particularly in the data space? 00:30:56.04 [Aubrey Blanche]: Yeah, so I would say like the closer you are to sensitive or confidential information, the closer you are to harms. And so I think the level of personal responsibility goes up. But one thing that I’m really encouraged about by especially folks in the in the data space is that We’ve practiced for this before. When GDPR came into effect, the hygiene behaviors around privacy, the bar got raised in major ways. Obviously, if you’re not operating in Europe or on European citizens, but I think that the norms and practices around privacy, this isn’t actually fundamentally different. I hope that people would take a bit of hope in that and that they actually already have a lot of the skills needed to do this well. So this isn’t some, it’s easy to be like, oh, it’s an alien species. Sure, it’s kind of weird, but it’s not fundamentally different. And there’s a whole academic literature debate about whether AI is fundamentally special or it’s actually just a normal technology that mostly works faster. I tend to believe it’s more of a normal technology. And so to me, what that says is the skills and frameworks that we already have are useful. for governing and managing the risks associated with this technology. But I do think from a data professional perspective, the most powerful thing you can do is be open about asking what could go wrong and what do we need to do to prevent that. And I think if we just got in the habit of before we do take a moment of consideration to say, like, what’s the worst case scenario? Now, one thing that I will say that concerns me is that there’s kind of two specific issues that make answering that question correctly really difficult. One is that there’s a huge number of people who do not understand how AI works. I think data professionals, that is less of a risk. They just tend to understand the technology more. But the second piece, and I know this is now going on a half a decade of talking about this, but a lot of people in the data and text space don’t have the lived experience to accurately answer that question because the worst case scenario doesn’t happen to them. Wait, say more. So like, and this is an oversimplification, but like how many like Rooms full of data people are like a bunch of white dudes with no disabilities who like make over six figures. And I’m not critiquing them for those qualities, although I could. It’s that the likelihood that they have, for example, read a bunch of black feminist theory is quite low. And we know that black women are uniquely at risk of being harmed by poor deployment of these technologies. So it’s great that you build the muscle. to ask what could go wrong. But you also need to critically question your own ability to answer that question in a way that’s universal. So Lucy Suchman, who’s a feminist science and technology studies scholar, talks about the idea that people with a lot of power or privilege build things in their own image and assume that their experiences are universal. And so that’s something I want to talk about is we still need to talk about who’s in the room and what qualifications they have. to make those decisions. There’s also some interesting research being done by someone a year ahead of me and my master is looking at the demographic distribution of people in the AI ethics versus like more technical spaces, because AI ethics is actually much more female, much queer, much browner than like the technical things. And so there’s, again, this is why I say It sounds a little self-serving, but you need an ethicist in the room because the likelihood that the room is qualified to answer the ethical questions is quite low without them. 00:34:58.64 [Val Kroll]: I like that a lot. Can I ask another data, bring it to the data crew specific one? Cause that was, that was awesome. You mentioned something earlier about, um, phantom value. And I wonder if you could expound upon that a little bit is I think I know what you mean by that, but I would love to make sure that I, I understand. Is it just like a perception versus a reality or a lack of measurement to objectively say whether or not, you know, use case was valuable to the organization or yeah. 00:35:29.09 [Aubrey Blanche]: D, all of the above. So all of the above. So there is a couple of particular threads that are all contributing to that belief that I have, which is one, there’s not a ton of research. And part of it is because we only have a couple of years of LLMs rewriting the world, although AI as a concept has existed for a long time. depending on how you define that, which is, again, a very specific thing. That’s another episode. Yeah, like, people with PhDs debating what artificial intelligence means. And so I think there’s, number one, like, there isn’t a lot of hard evidence that, like, the primary benefit of using AI is, like, increased financial return specifically. Like, the data is just pretty thin that you can draw a direct line between, like, throwing AI at a problem and I’m suddenly making more money. Again, if you lay off 20% of your workforce, that’s probably going to happen, but you’re going to likely incur a bunch of other problems that are more expensive than whatever game. And not to call out specific companies because they’re not the only ones doing it. So I think it’s really important that this is a broad issue. But like Klarna got rid of a bunch of their customer success staff, and then a year later hired the team back because they realized that the technology couldn’t do what they somehow decided it could do without any proof. So I think there’s that. And then I think there’s also just the Yeah, the reality that this tool may or may not return the kind of returns that we’re thinking about. And so I don’t think we should be going all in on something that’s so untested because right now like entire markets are responding to like PR talking points written by people who have an incentive for you to believe that and don’t really have any accountability structures to tell the truth. 00:37:23.30 [Moe Kiss]: I think one of the things that I’ve been chatting to a few friends about who own small businesses and whatnot is a lot of the pressure that’s on them is that investors or clients are basically expecting the returns from AI to reduce prices or increase revenue streams, but it’s not actually performing at that level yet. Some of the smaller businesses are really in a crunch position because they’ve got these clients who are like, well, I expect that you’re going to charge me less. But it’s not actually providing that kind of value to our business yet. Now you’re just asking. That’s a really difficult position to be in. 00:38:02.47 [Aubrey Blanche]: Yeah, I mean, you know, perhaps a little bit more radical philosophy than like the average listener of this pod is on. But like, yes, so in general, there’s a ton of academic discussion about the fact that like the logic of AI as it is currently being built and deployed is like extractive and capitalistic in that it is inherently being used to devalue labor and expertise. I cannot remember who said it, so I feel really bad about this, but AI is kind of at a point right now where it can write a really good facsimile of a PhD level paper, but you wouldn’t trust it to make decisions about your kids. And so I think that Again, we just need to be a bit more deliberative about this. I think we are a bit as a society drunk on the marketing hype and we’re not making principal decisions. I think there’s also a question in that with a small business. So I’m thinking like professional services, right? Like a consulting business. It’s like, oh, well that took you less time to do. And it’s like, okay, so you are assuming that the cost to you is based on the time it took me to execute that as opposed to the quality of the work, which speaks to an underlying belief about how we value expertise and labor, which doesn’t make sense. The story that I think illustrates this, well, and I don’t know if it’s actually real, but it’s like floating on the internet, so Pablo Picasso is sitting at a bar, and some dude six months later realizes he’s Pablo Picasso. and says, oh my God, could you draw me a thing and doodles on a napkin? And he says, that’ll be $30,000. And the guy says, but that took you five seconds. He said, it took me 30 years to be able to do that in five seconds. 00:39:48.45 [Moe Kiss]: Oh, I love that analogy. 00:39:52.40 [Aubrey Blanche]: And so I think that part of getting away from that is actually equipping small businesses to explain the source of the value that they’re providing to a client. And then also recognizing that some clients just only care about the bottom line and that sucks. But I think we need to equip them to say, but also move to more fixed fee. project structures. So there are ways to structure your pricing that can deal with that in a way that avoids those conversations. But again, I don’t know who is enabling SMEs who are already stretched them to figure out how to cope with that. That’s definitely something I worry about is corporate consolidation, noting that in Australia in particular, I saw a stat that something like over 99% of businesses in Australia are SMEs. like the corporates we’re talking about actually make up a vanishingly small amount of the overall economic ecosystem here. So yeah, just something to think about. Interesting. 00:40:54.44 [Val Kroll]: So is the measurement piece, because it feels to me like analysts are uniquely poised to measure cause and effect. And so if this could be one of the other areas that we could have a little rallying cry to the analytics community to say, hey, if you’re going to fire the CS team, a customer success team at Klarna, we’ll pick on them again for this example. Let’s, oh, I don’t know, think about what we intend that to achieve and let’s measure that. And if it doesn’t, then let’s figure out what the next plan is or whatever. But is that like another area where you think analysts could jump in and step up to help with understanding how this is impacting organizations versus just like, oh, there’s 10 less people here now. So that must be better. 00:41:42.51 [Aubrey Blanche]: Right. So I would say yes. And I would go a click deeper. There’s something that analysts can bring to this that folks outside of the field might not, which is it’s not just what could go wrong. But what is the leading indicator that would tell us it is? And what does our data infrastructure allow us to measure? So that, to me, is the really exciting thing, is that an analyst, because they’re deep in the systems, they understand the data, they’re actually able to translate this idea of harm as a theoretical thing into a set of monitoring procedures that would actually tell you if something’s going wrong. Because the impression I don’t want to get is like everything is terrible all the time. like I operate from a slightly different frame is everything could go wrong all the time. And like, if that is my baseline belief, then I personally am motivated to do things to reduce that risk. And so it’s not meant to be doom and gloom, it’s meant to actually just be responsible to say like, So that’s what I would say is like, okay, define the bad. Like in the case of Klarna, it could be something as simple as like, okay, well, we need to track customer satisfaction with individual interactions or successful resolution of issues, right? It doesn’t need to be like a social justice coded metric. You know, in my head, I’m like, are different customers getting different quality of experiences because they’re having different types of problems, et cetera. But like, we’ll start at a baseline of like, do we see a decline in customer satisfaction? But also the analysts can say, hey, given that we’re not sure about this impact, maybe we just do 10% of the objective for two or three months to actually measure that before we make a decision about whether this is a broader kind of initiative, whether that’s workforce reduction or redeployment or implementing technologies. customer service chatbot as an example. So I think that’s where analysts have a really special and unique role and quite frankly really powerful to lead their organizations to be more responsible. And there’s also a lot of debate about like making the ethical case versus the business case. The reality is they’re both tools and which one works will depend on the organization you’re in. I’ve worked with organizations that go full in on the ethics case and the leaders get upset when you talk about financial returns. And I’ve worked with organizations on the other side that are like, this is about like the board reports every quarter. And I’m like, cool, if I need to explain this to you in money, it’s fine as long as we get to the outcome that we all agree is good, which is you know, creating customer value, which is smooth operations, which is avoiding screwing people over. Like as long as we end up in the same place, the path we take is kind of real. 00:44:25.31 [Moe Kiss]: I think that’s like literally a one-on-one in how data practitioners work, which is always like figuring out what does the person making the decision care about, and then how do you frame your analysis and your recommendation in a way that speaks to the thing they care about. 00:44:39.02 [Aubrey Blanche]: Totally. But you’ve just proved the point that I made earlier, which is that We already have most of the skills to do this well. 00:44:47.38 [Moe Kiss]: Oh, burn. Look at you full circle. 00:44:50.64 [Aubrey Blanche]: Yeah, not to be like, I was right, but I think that’s actually more everybody else is already capable of being right. 00:44:57.13 [Moe Kiss]: So one thing we haven’t talked like, I feel like we haven’t gotten into the actual nitty-gritty, like I could spend another five hours talking to you. But one of the things that we did chat about as we were like preparing for the show was about agent to AI and giving up agency. And I would really love to hear your thoughts about those trade-offs about how people give up agency and what folks are willing to give up in terms of speed. Is the sacrifice worth it? And I feel like we’ve touched on that, but maybe not as deeply, particularly with the agent agai example. 00:45:34.65 [Aubrey Blanche]: Yeah. This is a bit of a conjecture, but I feel strongly about it. But if anyone wants to yell at me in the comments, I’m happy to be proven wrong. I’m thinking about the study that Anthropoc put out called Disempowerment Patterns, and what it showed is that people very often gave up agency to AI. which I find really concerning because there’s other research that shows that the majority of people don’t actually understand how LLMs work. If you don’t know LLMs, they just predict what the next word most likely is. They’re really good at producing things that look like language, but they don’t actually know anything. There’s no conscious, there’s no intention behind it. It’s literally just like, I think that people give up agency because they don’t actually understand the problem that they’re faced with. I think there’s also research that shows that the people who are most likely to give up agency are the least skilled at the thing that AI is doing. So there’s this concept called like, I think it’s AI acceptance, which is like the rate at which someone accepts the output of the AI versus challenges it or changes it. And there’s a strong correlation between the expertise that someone has in the domain and their likelihood to challenge AI. So someone with more expertise rejects AI because they have the ability to evaluate the quality of the output. Whereas if you’re not an expert, you actually don’t have the underlying knowledge to understand whether the output is valid or not. And so you tend to defer to the model. Contrary to what’s happening, the ideal behavior is that you only use AI in places you have the expertise to evaluate the output. But that’s not what happens. 00:47:19.77 [Moe Kiss]: The total example that’s coming to mind is using AI for data analysis, right? Because I absolutely will go back and forth. But like you said, I probably have a stronger threshold of AI acceptance because I’m an expert in that area. So I know when something’s not right or something’s off or I have that intuition and that experience the 30 years built up, maybe not 30, I’m not that old. But I had that experience built up. Whereas for someone else who’s trying to use an LLM to do data analysis, they’re much more likely to accept it on face value. And therefore, the risk increases because they don’t have the expertise. 00:48:00.62 [Aubrey Blanche]: Oh, I never knew that’s what that was called. Yeah. And you think about the idea that even some basic practices that you would practice in data science or kind of analytics is that you run, let’s say, I don’t know, the last time I programmed and did analysis, it was in R. So I don’t know how out of date that is. I’m still, I’m probably starting like I’m coming from the 1800s. But like, You run your code, you still do some cursory checks to make sure nothing weird or unexpected has gone on. So that goes back to my point that like data folks have the skills to deal with this, which is like, I never trust an AI output. So I’d use it for research and editing and all these things. But I’m always asking AI to produce links. I always go read the original source of anything that’s being analyzed or presented to me with an LLM. but it speeds up my acquisition of stuff on a certain topic so I can spend more time analyzing and less time searching. So that’s like an example where I’m expert enough to know that LLMs bullshit me. And so I always have just a dot of skepticism that what I’m reading is true. And so again, that’s an attitude that you can build, which is trust is earned. And I have not seen evidence that this technology is deserving of our full trust. But again, I really think teaching AI literacy has to become a basic skill that’s taught in primary schools. I think it was in Oakland. This is a bit of a tangent, but I promise I’ll come back. There was a focus around public health in Oakland, and so they did a really interesting community-based thing where they found that teeth brushing was really highly correlated with a bunch of other positive health outcomes. I don’t know the science behind it, but clean mouth, much better body function. And so they actually made it so that they ran these community programs or became normalized for community members to teach each other about three facts about teeth brushing that were shown to promote better brushing behavior. And I think that’s actually what we need. We need to think about AI literacy as a public health problem. and to say like there was a baseline of competence that we want everyone to have. And I think to your earlier point because of how fast the technology is moving, my personal belief is that organizations have a higher ethical responsibility to teach their employees safety behaviors because the reality is the government can and does not move fast enough to achieve those things on a scale that we want. And I don’t think it has to be like, you don’t need to spend $250,000 on responsible AI training. literally run it for free where you’re like, here are the three AI behaviors that we encourage. One, like always check out puts. Two, be careful with sensitive and confidential information. Don’t put it in tools that aren’t locked out. Like you can teach people that in 10 minutes and reinforce it over time. Again, corporate leaders have the skills to pull off setting expectations for their business. This is not something that’s outside their realm of the capability of anyone who’s getting paid to lead an organization. 00:51:15.04 [Val Kroll]: I like that. So much to think about. 00:51:20.76 [Aubrey Blanche]: But I think it goes back to, like, you talked about personal responsibility. I think we all have a responsibility. And what that responsibility is changes depending on the power we have access to, the systems we have access to, the work that we’re doing. But I hope that’s an empowering message, which means that you can do a lot. Because think about it can be really easy to go, oh, this is all big and structural and scary and whatever. But imagine if each of us did one slightly more responsible thing every week. That’s actually fundamental systems change, and it doesn’t actually require enormous sacrifices and changes on behalf of any one person, but that takes us getting a collective lens on what it means to achieve safety and responsibility with this tech. 00:52:06.68 [Moe Kiss]: I have a very weird one that has… I have not fully processed this, and so I’m just going to talk about it out loud because I want to get your thoughts because that’s what I use the podcast for. Okay. One of the things that An amazing one on the team, Jennifer, was talking to me about planning and how important planning is. Basically, the analogy she gave me is like, Moe, we need to know we’re going from Melbourne to Sydney. We don’t need to know that we’re catching a plane, a bus, or driving a car, but we need to know we’re going from Melbourne to Sydney. I was like, that is excellent. That’s a framework I’ve used now for how we think about planning. because we might have a path, I promise, I’ll come back. But we have a probable way we’re going to get there, but it might change as we learn new things. I was trying to think about this in the context of AI product development the other week. What was bubbling up in my mind is, Maybe it’s that we don’t know that we’re going from Melbourne to Sydney but we know we want to go from Melbourne to the beach. We just don’t know which beach we want to go to and so what might be different about that process is we need to also figure out how we’re going to get there and then we need to figure out which beach. But then I think the bit that’s been rolling around in my mind is, am I treating AI product development as different to other product development because I’m giving this, what’s the word, uncertainty to it that maybe doesn’t exist? I’m curious, Aubrey, you’ve said a couple of times about AI just being a stack of other technologies and we’ve seen this all before. It’s nothing actually that transformative and a lot of it is hype. I guess I’m just processing live. Am I thinking about it with the hype layer on and actually we’re just going from Melbourne to Sydney or is it slightly different and we do need to have that different mindset? 00:54:06.41 [Aubrey Blanche]: So I think it is slightly different. Like I think most of the things we think about kind of standard software development apply like basic, you know, don’t put your code on the internet unless you’re intentionally open sourcing, etc. Like those principles. But the reality is like traditional software development for the most part, software does what you tell it to do. The unique risk posed by AI is AI sometimes does stuff you didn’t tell it to do. not always. And so the way to deal with that uncertainty is, and what I see happening is some people go, Oh, well, it’s uncertain. So we can’t do anything through our hands up. And like, I think that’s quite silly. We can say, okay, I know that there is a degree of uncertainty and any risk professional will tell you that uncertainty is a fact of the universe. And there are actually very good ways to manage it. And from my perspective, one of the things I’m often telling companies that are saying, you know, I know that I want to go to the beach, but I’m not sure which beach. I’m going to, is to say, but there are certain things you can do to prepare. So for example, you need a mode of transportation. You need to know the rules about how to operate that mode of transportation safely. Do you want to take a bus and get a ticket? Do you need to know how fast you can legally go in New South Wales without getting a ticket on your license? So there’s like things that are knowable that you can plan for and you should do that. But then you also have to operate with an understanding of something will go wrong. and I may or may not detect it with AI. And so the question is, great, what path will I find out that thing has gone wrong? And how do I plan to respond to something unknown going wrong? So like an example from corporate, like corporates have crisis communications frameworks. They don’t know what’s gonna blow up, but they know who they’re going to call when that thing does blow up. what the roles and responsibilities are to responding to that thing. So again, Moe, I think you’re actually right. I think we’re just, we need to borrow from more fields of expertise to manage these things. But a lot of the skills and frameworks that are needed to manage them are not things that need to be invented. 00:56:15.31 [Moe Kiss]: Oh, I love this. But the thing that, okay, the thing that I took away from what you just said is like, If we apply this intentionality as well, it might also help us figure out a better path to get there that minimizes the risk. So like we might figure out taking a bus will mitigate certain risks that driving a car won’t. And so therefore that’s the, oh, okay. I love an analogy. 00:56:40.53 [Aubrey Blanche]: I love how well you played with it too. That was, I really liked your beach analogy. I was like, you just come up and seem cold, but. I don’t know if I want to go to that one. But yeah, I think that’s it. And again, there’s this like underlying thing that happens with people who have expertise in one field. And I would say I tend to see it manifest among certain demographics more than others. Take that how you want. Look at my social media and you’ll figure it out. But people tend to think that because you have expertise in one domain that you therefore make good expert level judgments in other domains. And I think in tech, As an industry, we have so deified engineers to be like, oh, they’re amazing. And let me be clear, engineering is hard. It’s amazing that people can do that. I stand behind that. But because you are amazing at building technology does not necessarily mean that you are qualified to evaluate and manage ethical risk or operational risk. It’s that different people have different expertise, and we need to recognize that the people building often have not been trained or exposed to the types of problems they would need to be able to make expert level decisions about these things, which goes back to my point about committees, which I know is really exciting. But my point is that I do not believe that safety and goodness can be achieved without getting a lot of the right bits of expertise in the room. And someone’s going to tell me they’re going to like, go on chat GPT and like program a suite of agents to function that way. And I’m not really sure if I believe that that’s sufficient. One for the particular reason that like, you know, we’re talking about novel problems and extrapolating outside of samples is a problem every data person knows well. So that’s what I would say is like we need to be really careful and part of it is the underlying value of expertise that’s non-technical. 00:58:33.98 [Moe Kiss]: That feels like an incredible place to wrap, which is impossible because I swear I could sit here all day and just keep chatting about this. But the last thing we do on the show is we go around the horn and we share what’s called a last call. And something that might be of interest to our listeners, not our users, our listeners. Just something you’ve read, you’ve watched that’s interest you. We will, of course, share all the links to Aubrey’s amazing reading list in the show notes. But Aubrey, do you want to go first since you’re our guest? 00:59:04.06 [Aubrey Blanche]: Oh yeah, I’m like fully alone, so if anyone does anything, just like drop everything and watch Heated Rivalry. When it’s good, like come for the hot guys, stay for all of like the completely renorming of like queer media. but also get in on the discourse online. The ethics and the quality and the values that the people who are engaged in creating this show are exhibiting is, I think, transformational in terms of the way media comes into the world and what it does. One, it’s just really fun, but if you’re into more critical analysis and things like that, it’s this rich well of things to think about. For me, ultimately, the way we could do things differently and better. 00:59:46.54 [Moe Kiss]: I love it. Nice. 00:59:47.30 [Val Kroll]: I love that. What about you, Val? This is a medium article. I subscribe to a lot of engineering and product and design content to get the diversity of perspective, not just all analytical content. I actually clicked on this one. I started reading it thinking I was going to hate it. I thought this was rage bait for the analytics crew, but I ended up really liking it. So it’s called, I don’t care what you build and neither should you by Joel Dickinson. And he’s talking a lot about like he has quotes about like, you know, Ronnie Kohavi saying that, you know, only 10 to 30% of experiments or product features actually add any impact. So like, why should we care? And like, who cares about the target? And I’m like, like clutching my pearls. But then he starts talking about the framework that he thinks works. And he was saying, I found that good leadership in engineering boils down to asking relentlessly, how will we know? So not what you will build or what technology we’ll use, don’t show me the architecture, just how will we know if the problem is solved? And I was like, okay, you got me. I love that whole thinking about the problems frameworks and things like that. Anyways, definitely not from an analyst perspective, but I really enjoyed it. It was a good read. It was a fun one. 01:01:02.13 [Moe Kiss]: I have a bit of a weird one today. Normally, it’s something I’ve read or looked at, but this time, I’m going to crowdsource some help. I’ve been thinking a lot about a measurement of AI products. And obviously there is like a wealth of information, but I think the angle particularly that I’m thinking about is as it relates to like user engagement and users like having a successful experience and how that potentially differs from like an AI product versus like a more traditional product and like what that intersectionality is. So I’m not like talking about like evaluating a model, I’m more about like How do we understand if a user has multiple designs that are generated from an AI output? Does that count as someone doing something creative or is that like, I’m just like really trying to wrestle with some of these concepts and I don’t have a firm view yet. So it’s more of a shout out that if folks are coming across interesting articles or perspectives on this, I would love you to share them with me because it’s definitely top of mind, especially like, when you have like AI products and non-AI products in your stack and really wanting to be able to paint a holistic picture. So anyway, that’s just, I thought I’d share my conundrum at the moment. I like it. Okay, so this has been such a wonderful conversation. Epically huge thank you, Aubrey. Like, just love having you on the show. We’re so appreciative. Oh my gosh. 01:02:35.92 [Aubrey Blanche]: Literally call me anytime. I’ll move my calendar to show up. Right. I’m sorry, my diary. My diary. I’m trying to assembly here. 01:02:43.51 [Moe Kiss]: Nice. So what we would love after the show is for you to please come and leave us a review or a rating on your listening app of choice and feel free also to request a sticker on the analyticshour.io link. You can also reach out to us on LinkedIn and the measure Slack, and also through our email contact at analyticshour.io. So that’s a wrap on AI the teenager. A very big thank you once again. And for all of my co-hosts, who today is just Val, keep analyzing. 01:03:23.06 [Moe Kiss]: Thanks for listening. Let’s keep the conversation going with your comments, suggestions, and questions on Twitter at @analyticshour on the web at analyticshour.io, our LinkedIn group, and the Measure Chat Slack group. Music for the podcast by Josh Crowhurst. 01:03:41.02 [Charles Barkley]: Those smart guys wanted to fit in, so they made up a term called analytics. Analytics don’t work. Do the analytics say go for it, no matter who’s going for it? So if you and I were on the field, the analytics say go for it. It’s the stupidest, laziest, lamest thing I’ve ever heard for reasoning in competition. 01:04:08.49 [Val Kroll]: Rock flag and let’s get intentional. The post #292: AI Without Adult Supervision with Aubrey Blanche appeared first on The Analytics Power Hour: Data and Analytics Podcast.

We know what the work of the data practitioner is, right? It’s everything from managing data ingestion to data governance to report development to experimental design to basic and advanced analytics. It’s writing (or vibe-writing?) SQL or Python or R while also being adept at whatever data stack—no matter how modern—is at hand. Of course, it’s a lot more, too! And that’s the topic of this episode: the unofficial, often unheralded, but often quite important “shadow work” of the analyst—the myriad tasks required to effectively glue together all the data work that occurs out in broad daylight to enable the data to truly be useful at driving the business forward. This episode is brought to you by Prism from Ask-Y—your agentic analytics platform for automating analytics, exploring data, creating repeatable workflows, and delivering accurate insights—all without the need for manual query writing. Links to Resources Mentioned in the Show DataTune Conference in Nashville: March 6-7, 2026 MeasureCamp New York: March 28, 2026 Marketing Analytics Summit in Santa Barbara, CA: April 28-29, 2026 Photo by Darwin Boaventura on Unsplash Episode Transcript00:00:05.78 [Announcer]: Welcome to the Analytics Power Hour. Analytics topics covered conversationally and sometimes with explicit language. 00:00:14.91 [Michael Helbling]: Hey everybody, welcome to the Analytics Power Hour. This is episode 291. Who knows what evil lurks in the heart of men? The Shadow knows. Moest of our listeners probably don’t know that callback to the extremely famous radio drama The Shadow, but what they probably will recognize is the work that data and analytics people do that lurks in the shadows of our day to day. That’s not really the job description. It usually doesn’t get recognized, but you do it anyway. Maybe some days you feel more like a janitor cleaning up ugly data or a therapist listening to stakeholders’ frustrations or some sort of data marketer just trying to sell your wares internally. I think we should talk about it. Let me introduce my co-hosts. Moee Kisss. How you going? 00:01:03.93 [Moe Kiss]: I’m going great. Thanks for checking in. 00:01:06.77 [Michael Helbling]: Have you ever heard of The Shadow? The radio show, The Shadow? 00:01:10.88 [Moe Kiss]: It was like from the early… No, but I’m deeply familiar with the sentiment. 00:01:14.45 [Michael Helbling]: Oh, okay. Yeah, yeah. And Val Kroll, welcome. Thank you. Bye, everyone. Go Bears. Yeah. And… Hey, it was close. Tim Wilson, probably the only person that got. 00:01:36.40 [Michael Helbling]: I was going to ask you if you remember. I think first up, maybe let’s talk about what kinds of shadow work have you found yourself getting into in your career? Like what are some of the categories or the types of things you’ve gotten into? And then as we sort of get into that discussion, maybe figure out if we thought it was necessary or not or whether it was good or not. So who wants to start us off with some of the stuff you’ve run into? 00:02:16.22 [Moe Kiss]: Oh, I mean, the one that starts with a capital A, admin. And I think this is potentially more on the internal side. I’m going to be curious to hear reflections. But I feel like there ends up being a lot of cadences in a business. And I think I’ve gotten to a point now where I kind of see it. And I’m like, if as a data team, you start to pick up, I don’t know if admin’s the right word or project management or heckling people to be like, you need to fill out this spreadsheet. Have you done this bit of this deck? All of that. And some people might think that that’s fair. But in a space where you have admin support and folks who are meant to have that as part of their role, feel like I see data, people end up having to fill that gap a lot just to keep momentum moving forward. And it’s almost like once you assume responsibility for it, it’s almost impossible to ever roll it back. 00:03:22.03 [Tim Wilson]: I’ve thought, I mean, there’s one specific part of that. There’s like the input, I need to do admin to get stuff. And then when you first said admin, I was thinking like, user governance, like, oh, somebody needs access to whatever. I feel like there’s an admin part that I think is good for the analyst when An analysis is delivered or something is delivered that is supposed to lead to a decision and an action that for a long time, I’ve felt that the analyst does kind of need to own that because it’s pretty easy for somebody to say, yeah, that’s awesome, but they don’t really necessarily have an incentive, direct incentive to take the action as was prescribed. as an accountability mechanism for the analysts to say, oh, I’m going to be here because I know how to set recurring reminders. I’m going to set a reminder to come back and say, hey, you said that was great. In the next release, you were going to do X or Y. Did you do it? I don’t think that’s admin though. 00:04:32.28 [Moe Kiss]: That’s not admin. What’s the word? That’s checking back to be like, if we said there was going to be some outcome, did we achieve that outcome? I would see that almost as being accountable for measurement and making sure that we hit the success bar and making sure that other people in the business are accountable. I think it’s more when you’re like, I know, Tim, you’re going to have strong views on this. But when you think of monthly reports and cadences like that, and it ends up being about getting people to fill out their section, not, hey, I’m doing the data bit and I’m going to partner with my stakeholder on the commentary or whatever it is, it’s like heckling and following up people and making sure people have done their bit. because ultimately like a data person might be responsible for making sure the reports are not or whatever. I think there’s a difference between like ownership and making sure you’re accountable and like Following up people to make sure they do their job. Oh, this is going to be like a trigger point, Tim. 00:05:37.58 [Michael Helbling]: Well, it’s interesting because I’ve definitely found in my career mode where we would go to the business and we would have like a recommendation or insight from the data, which was all part of our job. A couple weeks later, we’d be in a meeting with the IT department to explain what we wanted to change on the website as a result of that. We’re riding shotgun with the project now. It’s like, wait a second, when do we stop doing the analysis and start being the project managers for the implementation of this? That was when I was like, wait a second, what job do I actually have here? Because you’re kind of like, I’m not now not doing data analytics. I’m now running sort of like an integration task force, if you will. So I don’t know if that’s more like in the line of what you’re talking about. 00:06:22.30 [Moe Kiss]: It’s such a fine line though, right? 00:06:25.18 [Michael Helbling]: Because if you want to see your insight go live, you know, yeah. 00:06:30.08 [Moe Kiss]: And it’s something that I do worry sometimes like data folks are like, here, I’ve got a recommendation. I’m going to throw it over the fence. It’s your choice if you do it. And like not taking ownership. I think part of being a strategic partner is taking ownership and being like, I’ve made this recommendation. We’ve agreed on it. I like, I want to see it forward. And I’m, I’m part of this. I’m accountable to it too, because I’ve made this recommendation. So it is such a fine line between picking up too much of the behind the scenes stuff and what you actually need to do to like see the project or recommendation move forward from a business perspective. 00:07:05.72 [Tim Wilson]: Some of it gets down to just recognizing that if it’s kind of Michael, to your example, it’s when everybody agrees that should happen. I mean, that’s kind of like business 101. If it’s like, well, everybody agrees, but no one actually assigned, there was no ownership assigned. If you can do that in the moment, then a lot of times it’s like, well, who should be doing this? If I wait and we haven’t got it, then it shouldn’t be the analyst. But if everybody leaves and the analyst is saying, well, nobody’s gonna do it unless I step up and do it, That’s a little bit of a shame on the organization, shame on the analyst, but there is that part of like the full life cycle is, does need to go all the way through. So what is the next milestone? Who’s going to do what by when? And then looking at that person and being like, are they going to do it or is somebody going to need to babysit them? Which isn’t, I mean, that’s kind of a reality of business as much as the analyst role, I guess. 00:07:59.75 [Val Kroll]: As you were talking through the admin stuff, Moe, I think the consultancy equivalent of some of the admin work is, can you send me that thing that you told me you were going to send me? Can you send me that thing? Or can I have access to that? Especially if it’s like I need one of your other partners or other agencies to send me or give me access to something. The number of times, like, top of a call, like, okay, moving around a lot. Did you get approval for that one thing? Are we good to move forward with that thing? Which is, like, a lot less connected to meaningful stuff. 00:08:36.49 [Michael Helbling]: Explaining another agency’s data analytics to the client. That’s some shadow work right there. I’ll even just say, listen, I don’t think you want to pay me to explain this to you, so let’s find a different way to do it. Not that I don’t want to help you, but I’ve had many experiences where they’re like, okay, we got this from this. Maybe it’s a different agency that runs a specific program for them, like media or SEO or something, and they’re pulling their own reports. They’re like, how did they get these numbers? And I’m like, okay, so now you need me to go reverse engineer how they pulled these numbers together. And it’s like, oh boy. 00:09:26.22 [Tim Wilson]: That’s not a bad part of shadow work, getting poorly documented, regardless of where it comes from. Somebody wants me to take it forward. The first thing I have to do is basically replicate what was done so that I know what I’m carrying forward, which is just not… Some of that can be addressed by documentation, but that’s like this. There can be this expectation like, well, here’s the number and it links to this dashboard so surely you know everything you need to know. You’re at the starting line. It’s like, well, no, no, I’m still actually back in the locker room trying to get ready to come out to the starting line. 00:10:04.00 [Michael Helbling]: That’s a good point. And getting coordinated so that everyone’s kind of using the same data and everyone trusts the data that’s being presented, whether it’s internal or external, goes to that sort of like, that work in preparation I think is very much a part of what I consider like a data and analytics role to be doing. But sometimes it falls in your lap in a weird way, maybe. 00:10:29.36 [Moe Kiss]: Okay. And I think the thing that comes to mind is the word alignment. So like not all shadow work is shit. Some shadow work is actually very valuable. It’s just the fact that the business doesn’t understand like how consuming it is or how important it is. And alignment I think is one of those things where it really is often about like this business unit thinks this or this client thinks this and this area thinks this and like making sure that everyone is speaking the same language, whether it’s about the metric definition, whether it’s about the outcome of the work or like that alignment pace, I think is incredibly important. But I don’t think it’s always something that the business understands that it’s such a big part of a data practitioners role. 00:11:14.50 [Tim Wilson]: I second that. I mean, I think even the alignment, what is it we ran this campaign? What was it supposed to do? And then the fact that the analysts are like, well, I need to be in the meeting up front like that. We need to make sure everybody’s on the same page of what we’re trying to accomplish. It’s not run it. And then the analyst gets involved because the data now exists so they can pull it and they can provide the answers. that upfront, which I mean, some would say I co-created a consultancy that is geared a lot more around trying to get multiple parties on the same page, so that the analytics work or the experimentation work can be productive and successful is a huge part. 00:12:01.60 [Val Kroll]: I’ll third that motion on sometimes the shadow work is really important to move forward when we first started talking about this topic, the first thing that came up for me and granted I do have very much of a recency experimentation bend. is the culture of experimentation work, how that’s become more prominent, especially on LinkedIn in the zeitgeist about how to be successful with experimentation. But if you think how many other roles around a business have to make space for everything that they’re supposed to be doing after the job description was approved and you were hired. It really is all about like building consensus and getting people excited and a little dose of education, a little dose of this is why you should care about what I do kind of a stuff. 00:12:49.38 [Tim Wilson]: And I feel like sometimes that’s a little… The culture of finance, the culture of accounting. Famous. 00:12:56.70 [Val Kroll]: Famous for going around to get people on board. Well, maybe during budgeting season, but just to go back to the point that it’s not that it’s not important, but it’s usually not the first thing you think of when you’re like, oh yeah, I lead an experimentation team inside of an organization. It’s not that it’s not important, but it’s usually not the first thing that comes to mind. 00:13:17.46 [Tim Wilson]: It does seem like maybe to bridge from that to explaining the realities of the data, which kind of takes two angles. there’s always going to be a presumption that the data is cleaner, more accessible, less ambiguous, which is like, no, our data is a company. It is always wildly more complicated than any kind of new person to it thinks it is. And then there’s the other part of that that is what the data can and can’t deliver. Like the data is the objective truth. So there’s a data fluency component where it does sometimes feel like in analytics, and maybe this is the grass is always greener on the other side. If you’re talking about finance, somebody’s in a financial analyst, somebody would expect that they’re an expert around finance and they can go to them and defer to their expertise. I feel like in marketing and product and digital analytics, sometimes it’s like There’s not a presumption of knowledge of complexity. The shadow work is building trust, building the relationship, walking them at the appropriate pace through why a diff and diff is not appropriate in this situation. educating of the business partners that does feel like it’s a proportionally heavier lift than many other roles. Does that count as shadow work? 00:14:57.94 [Michael Helbling]: Tim, when someone asks, which channel has the highest ROI adjusted for LTV, how long does that take you? 00:15:06.30 [Tim Wilson]: Pull GA4, then export to Excel, write SQL for BigQuery, find my LTV formula. I don’t know, let’s say about three hours in a couple of existential crises. At least two. 00:15:20.99 [Michael Helbling]: This is why ask-wise full-stack approach works. Ask in plain English, prism orchestrates across your stack and applies your saved calculations. 00:15:30.06 [Tim Wilson]: So I’m not manually stitching together five tools like some kind of data Frankenstein? 00:15:35.83 [Michael Helbling]: Nope, everything’s traceable, not a black box. DataState secure, semantic layer, generated code, runs locally. It’s all set up for you. 00:15:46.98 [Tim Wilson]: So for a product with a name that makes you think of a title or asking why, repeatedly, this is pretty sophisticated. 00:15:54.65 [Michael Helbling]: I’m not sure making fun of our sponsor’s name is the move here, Tim. Wait, I did say pretty sophisticated. That’s a compliment. All right, fair enough. Well, go to ask-y.ai, that’s ask-y.ai, and use code APH for priority beta access. Join the rise of the AI analyst. 00:16:19.41 [Val Kroll]: 100%. Yeah, I think so. Or even some of that same concept, the explanation to like backend developers, like you were talking about the business partner audience, but that was one, I think we were talking about this a little bit too much. I know that you have scars, but the story that comes to mind for me, when I worked at the American Medical Association, we were working off of the free version of GA at the time, and we had just gotten an analytics canvas license. to overcome the sampling. So it would like hit like every 30 minutes or every hour or whatever it was so that we could extract air quotes, all the data. And I remember it like there was some, some backend developers were like, Oh, perfect. Now we can just, you can just give us all of your GA data every night and we’ll just throw it into the membership cube. And I was like, it doesn’t work like that. Also, like, what do you mean everything? Like, do you even, but like so many conversations, conversations that got escalated, my boss had to pull me into it. And it was like, you guys, This is not like, I don’t, maybe this is on me at this point for not being able to explain this, but this is a little bit of a nightmare. But the other thing is that membership cube, the ID, the key was the membership ID. And I was like, do you think that the only people who visit our website are members and that they’re authenticating at least once every 30 days? Like you are off your rocker, but it was like, at least, at least three months of my life spent on that topic, if not longer. 00:17:45.90 [Michael Helbling]: And a lot of times shadow work is just cleaning up or trying to clean up a data warehouse you inherited from a previous team or something like that. You know, you walk into an Oregon, they’re like, oh, we want to do this, this and this amazing thing. And you’re like, well, the snowflake instance we have is not going to do any of that till we really clean up a bunch of it. 00:18:05.27 [Moe Kiss]: And you’re helps. 00:18:07.76 [Michael Helbling]: Yeah. 00:18:08.24 [Moe Kiss]: Is this like one of those times where you read my exact life situation that is going on right now around to a huge rebuild of our entire data warehouse for a very specific, like very similar reason, right? Like the data wasn’t structured in a way that we can answer the business questions of today. And so, and I think the thing that’s so hard about projects like this is they’re often huge and very time intensive and unlock the heap of value, but people don’t see the value until like months. 00:18:36.97 [Michael Helbling]: Yes, it’s a long time and it’s hard to go pitch those because it’s not very sexy or very exciting to be like it’s not doing anything but setting up a potential for a future as opposed to delivering a business result. It’s so much nicer to go in and say, hey, here’s this analysis where we can make $100 million more this year if we do X, Y, and Z versus, hey, we need to spend a bunch of money redoing stuff we already have because it’s not doing this, this, and this. Eventually, you can write the business case to show where the value will come from, but man, it’s an uphill battle. I don’t know if that’s shadow work exactly. 00:19:15.27 [Tim Wilson]: I think there’s often, I mean, I will see that example and raise it one with wait for a year. I lived this scenario many times, but the most horrifying one, I think, traumatic one, working with a large pharma company that was using Adobe Analytics, and they said, we’re going to get everything into a Azure you know, data store of some sort. And so many requests, they’d say, oh, we don’t have that yet, but it’s all going in. And they were just locked into these backend developers said, we’re going to take the Adobe’s horribly weird and never really thought through, gotta take the Viz high and Viz low, like stitching like messy, messy, messy data feed data. And they were saying, we’re just gonna pump it in at that raw level. And then we’ll just kind of write SQL queries that people can use. I’m like, the SQL query just to answer how many users came to this page is kind of a beast. But we couldn’t get an audience with them because they were just convinced, which seems very common with developers. I feel like it’s maybe less of an issue if you’re taking an event-driven product analytics perspective, but anytime you’re going to something where you’ve got this de-duping sessionization, developers think of event. They don’t think of the need for stuff to be deduplicated by something. So this idea that, well, we’ll just pump all the raw data in, and then you’ll be set. You’ll just have to write SQL, which then becomes a case of needing to maintain SQL libraries, I think. I don’t know whether, Moee, you’re like, that really doesn’t happen if you’ve done it right, or whether you’re thinking, yeah, no, that happens. Oh, or yeah. 00:21:09.32 [Michael Helbling]: Well, I mean, there’s tools that help with that, like, you know, um, dbt or data form or stuff like that that helps you kind of maintain your sequel and repositories and use it effectively. 00:21:19.79 [Tim Wilson]: But sometimes that’s to me, you’re like, you’ve gone with this, like, let’s Let’s get the full ocean, and then we’re going to add layers on top of it. 00:21:30.99 [Michael Helbling]: And then the downstream is the next question that comes from the business user requires yet another SQL query to be written to build out the next reportlet or whatever. So you put yourself in a pretty challenging chain of events just to get answers to data, which AI will totally solve. So don’t worry. 00:21:51.87 [Moe Kiss]: Literally, that’s about to be my comment. I think the biggest challenge right now is that everyone thinks that you can overcome a shitty data architecture with AI, which is just so fucked and hard to manage because you’re literally that’d be broken unless we have the right data architecture. The same way that we’d need to write a bespoke SQL query or you don’t even know where to point the question because of the way we’ve structured the data. That’s the problem that we need to solve. And yeah, it’s not sexy. Like getting the buying is incredibly difficult for this stuff. And it probably is the hardest. I would say one of the hardest parts of my role right now. 00:22:34.80 [Tim Wilson]: So that is deep because the business partners who ultimately want to get value from it, it’s not going to maintain their attention or technical depth, but the analyst is supposed to be engaging with them and serving them. So the analyst becomes the proxy for the business and is now dealing with the backend. And so they become subject matter experts in an area that has Nothing to do with running analyses or validating hypotheses. It’s just they’re living in that middle tier and there’s just no one else. The shadow has to serve it because there is no one. That’s all there is. That’s the best you got. 00:23:20.60 [Moe Kiss]: spot on and then you end up with like one or two people in the air and the business who know one area and no one else can do it because it’s so complex and there are all these like gotchas so even if you’re going to write a bespoke fickle query it has to go through this one or two people because they’re the ones that know those tables know how to to use it well and like that, then you’ve created your own bottleneck, right? And it’s not an intentional thing. I think often the systems were created with the intent to have a lot of flexibility, but then by having flexibility, you don’t have enough standardization and like, yeah, it’s a chicken and egg. But I would say that is one of the hardest shadow tasks for sure. 00:23:59.33 [Tim Wilson]: There does seem like there’s like a macro thought, this whole topic of the show that it’s like the I feel like I’ve worked with analysts who take the attitude, well, that’s, that shouldn’t be my job. So it’s not my job. So I’m not going to do it. And then it kind of falls through the cracks and doesn’t happen. 00:24:20.80 [Michael Helbling]: So on that meta thing, like there’s something to the idea that like some people by personality are going to be more suited to generalist types of roles versus specialist ones or more drawn to them. And so like, I’m definitely much more of a generalist. So when I find myself running further afield of doing the actual data work and the analysis, it doesn’t bug me at all. It’s actually kind of fun to see something different and do something different for a little while. I sometimes will think about, is this really truly serving our purpose? Are we getting done? We need to get done. But generally speaking, doing those tasks, not a big deal. I feel great about it. But I absolutely think there are people who Like that is much more disconcerting to step outside of the role to do those things and less of something that plays to their strengths and much more plays to like the things they definitely do not want to do. And so like that’s the other issue is just sort of like the person kind of matters a little bit to this too. 00:25:19.40 [Val Kroll]: Yeah. And I don’t think, I mean, at least from my personal experience, it hasn’t been like a conscious choice of like, whether I’m going to step outside or, you know, get in someone else’s lane, but it always feels like I’m tugging on a thread of something that in the moment feels necessary for me to understand what I’m analyzing or to understand root cause of like why that had been a problem. I mean, and a lot of times I personally just get fascinated by like, you know, authentication handshakes and like, you know, all the different nuances in that space. But it always ends up feeling like it’s adding to this like mosaic of my understanding, which always feels like it pays dividends in the future too. So I’ve never, I’ve never tried to quiet that voice. Also, I’m just really nosy. 00:26:10.98 [Tim Wilson]: This reminds me of me going overboard on it, where there were webinars in a company that we think we know what webinars. You have a registration and attendance. This was in a business model where it was not that at all, and it was like bonkers how salespeople would sometimes go into an office and sit and watch the webinars, and there were two or three systems involved. The more I pulled on that thread, it definitely was interesting, but it was like, oh, wow, I was looking at this one table of data and interpreting that attendees meant the number of people who attended the webinar, and that was completely wrong. I wound up writing up It was probably a 10 or 12 page document very, very clearly written because there were all these parties in different places and I thought, nobody has put all this together. I have done the most glorious, valuable. This is so useful. I’m pretty sure not even the webinar business owner. read it. I got probably 25% of the information from her, but I was like, oh, she was excited to explain to me the nuances of the complexity, but I kept digging further and further and saying, aha, look what I, the external consultant, has done to really help you understand what’s going on here. And there was kind of no interest. So that was one where I’m like, it was useful for me. It should have been useful downstream. In today’s world now, boy, I’d be throwing that into an LLM somewhere and saying, that’s really helpful data potentially. But I’m pretty sure that document, I was like, I became the domain expert on something that People cared about webinars, they did not care to hear how messy it was to interpret any of the data that was captured. 00:27:57.26 [Moe Kiss]: The thing that’s resonating with me a lot right now, one of the values that I, I do have leadership values, it’s a weird corny thing. But one of them is be unwaveringly useful. Does anyone, pop quiz, anyone remember where that comes from? 00:28:15.39 [Tim Wilson]: I don’t think so. 00:28:16.75 [Moe Kiss]: Oh, from being useful would be… A good friend, Cassie. Yep. I got it. Ding, ding, ding. Yeah. Yeah. She put it in one of her blog articles and it’s always resonated with me. And I’m completely contradicting myself now because at the start I was like, don’t pick up the admin work. But I’m the first person to be like, if someone’s not doing something and I can add value or move something forward, I’ll normally just end up doing it. So like I am, yeah, a walking contradiction. But I do think there is part of that. responsibility of data folk like I tend to get really frustrated when a data person is like, well, that’s not my job. And I’m like, your job is to help the business make better decisions. So if there’s something you can do to be useful to help the business make better decisions, that is your job. Yeah, I don’t know. That’s just the thing that’s bubbling around in my mind at the moment as we’re, I mean, not relevant to Tim’s example, but more broadly about this area of like sometimes it is about getting the domain expertise. Sometimes it is about documenting something that no one in the business has written down. It’s like, sometimes those things are less useful, but a lot of the times are really useful. 00:29:20.48 [Michael Helbling]: just to give some people who might be listening a chance to sort of be like, well, maybe, Moe, I can’t do that thing or I’m not good at that thing. Is it necessarily that you have to go personally be the one in charge of that as much as be part of helping solve it in some way, see that it gets done? So it’s more of like the ownership taking versus the taking on the role and doing it yourself, just so that people who are very specialized or don’t Yeah, because I have a ton of empathy for people who are like, Michael, I just can’t get up in front of people and talk. Cause like I analyze data and that’s what I like to do. And I very stressed out every time I have to go present something. And it’s like, okay, well then has someone else can do that part, but like you just need to make sure you’re a facilitating it up to the moment where it, where it happens. So it doesn’t have to be you necessarily taking on that role. I don’t know if I agree. So don’t pick presenting something then, something else, like managing the project or something like that. But the point being, like, not every person fits every single role. Like, you don’t have to be a polyglot, if you will. Or a polymath. 00:30:27.29 [Tim Wilson]: What that, I mean, if you… Poly PM. First break all the rules, like the precursor to the now discover your strengths, strengths finder, which… But first break all the rules. I’ve always, to that same point, identifying what needs to happen. I think, Moe, that’s the brilliant way to frame it. What is your job? It is not to write SQL. It is not to develop reports. It is not to deliver results. It is to move the organization forward by helping them make decisions. If you say, well, That means that somebody every Tuesday morning needs to reach out to this one person and ask them a question. Like it can be frustrating. It can suck. But you know what? There’s somebody who’s actually super sociable, who loves to ping people or whatever. Like building up that list is kind of, these are the discrete tasks. Not that somebody’s going to love and relish doing every one of them, but it does at a team level. help start to shift around, like, oh, somebody needs to document these database tables, or somebody needs to ask why Guru, they need to know how that tool works really well, figuring out who gravitates to it. I do think there’s, and I think I was cringing similarly with Michael grabbing a random example, there is a fine line between what is a complete analyst need to be able to do and do even if they’re outside of their comfort zone. So it’s, it gets a little squishy. Which of this is shadow work that like somebody’s got to do it, this person gravitates to it. Which of this is going to be a really ineffective handoff because someone just doesn’t, doesn’t want to write sequel. I mean, they’ll use that example. Somebody, I don’t want to, I’m just not the kind of analyst who’s going to learn to write code. It’s like, cool, then you’re not the kind of analyst who’s going to progress particularly far in your career. So, cool. We got it. 00:32:32.88 [Michael Helbling]: Hey, I’ve gotten pretty far. So, you know, no, now you can’t do it. You can’t. 00:32:39.32 [Moe Kiss]: I don’t want to get into team dynamics too much, but I do think a big part of figuring out the shadow work as a team is figuring out who had strengths for different parts of it and we’re making sure people lean in. I know in my previous team, we had a really big gap of, we didn’t really have someone who was really good at the I would say leadership team documenting stuff, pushing it forward, hyper-organized, being like, hey, Moe, these are all the things we have coming up in this time frame. We very intentionally hired someone that was really strong in that space to complement our team. I think that we really need to be thoughtful of what are all those things, especially the shadow work, because if you put someone on something and that’s their strength, it’s so much easier for everyone. They feel like they’re adding value, that the balance feels better. And to be fair, there are some things that no one particularly wants to do, and then it just comes about making sure everyone takes a turn. 00:33:43.61 [Tim Wilson]: Can we hit on that stuff a little bit? And maybe this administrative work, maybe more broadly, because I think that is the danger there. And I do think I’ve seen stuff written that women are much more likely to get screwed on this one, is that this thing needs to happen. And they’re like, oh, well, it’s admin work, like the latent misogyny Maybe not intentional is, well, Moee’s really good at that, but it’s absolute shit work, and she’s not going to speak up. I think there is that the shadow work that needs to be done that has value, that is being done as efficiently as possible, and there can be some gravitate to it. Shadow work that is has to be done. There is value. No one wants to do it and making sure that that doesn’t fall to the passive nice person because because that can spin out where wait now half of your job is unseen shadow work and you can’t advance in your career. Even though everybody’s like, well, this all needs to be done. But good old Jane is, you know, always there for it, you know, but it’s in the shadows. It’s not getting 00:34:59.87 [Michael Helbling]: Yeah. That’s not visible. So this is actually kind of an interesting pivot, Tim, because as you turn into a leader in your space or leading teams and those kinds of things, your job becomes taking the work out of the shadows for some of the exact reasons you just said, because it needs to be recognized. What’s being done, the people doing it need to be recognized. And then who should be doing it, should be much more strategically thought out as opposed to quote, fallen into just because, oh, so-and-so is more agreeable, so they just take it on without fighting too much, which is just a terrible solution to the problem. So anyways, I thought that was a really great point, Tim. And I think that’s sort of the thing that maybe take away is like, when you turn from an individual practitioner or individual contributor into a leader, you know, when you’re just an IC sitting at your desk, you’re like, wow, do all the shadow work. When you’re a leader, you’re like, we need to take the shadow work and expose it to the light. 00:35:55.99 [Tim Wilson]: That sounds hard. That’s why I’m not going to, I’m not striving to be a leader. 00:35:59.48 [Moe Kiss]: I do think though it also is about like recognition. And like one of the things that I would say like, and I’m thinking of this particular person, like I know at the moment their rating would be very good or like they’re like an assessment of their performance, right? Because I value that work. And so I think where the challenge is is like, when there’s that tension where someone’s like picking up a lot of shadow work, that then is not valued or not given the value that it’s deserved. Whereas I see it as like being incredibly essential. And if you do that shit well, like you can unlock a lot for your your team or the business. And so like, I want to make sure that that’s rewarded and reflected. So it there’s a lot of new ones, though, like, obviously, it’s very dependent on specifically like what paths we’re talking about. And yeah, and many factors. 00:36:50.66 [Tim Wilson]: I’d just like to say to all of Moee’s team who’s listening to this podcast, she’s talking about you. 00:36:54.83 [Moe Kiss]: She values you. 00:36:56.86 [Tim Wilson]: Oh, wow. She gave us the name off Mike and it was your name. So good job. 00:37:00.83 [Moe Kiss]: Stop it. You were so cruel. 00:37:03.66 [Val Kroll]: The other thing that this is making me think about is that when any in-house role that I’ve had, I’ve never reported to an analyst. It’s always been, you know, ahead of digital or someone else who it was really hard to message up not only for myself when I was the IC, but then when I grew my team about all the things that takes like, I’ll say, do you think we just sit there and like convey your belt? Just like analyze, analyze. Like that’s so not all that the job is, right? So there’s a lot more. education in that scenario, whereas I was thinking about your comment, Michael, like with the elevation of analysts and to those leadership roles that there’s a lot more visibility and line of sight. So I agree with you on the accountability we’re going to put on any listener to bring that work out of the shadows and acknowledge and like what you were talking about both. So that’s a really good point. 00:37:55.11 [Michael Helbling]: I think what we’re finding out is that the work has value. Whether we should be doing it or not as analytics people isn’t necessarily all the story. Sometimes you should go back and say, workflow-wise, the solution should be to take this group and pull them into this piece of work. rearrange it and come up with a strategy. My early example, Tim, you pointed out, we exposed basically an organizational workflow flaw when we came up with an insight and then had to go drive the insight through the org. What we exposed was no one had thought about, hey, what if we have an optimization, we want to make a reality? How does that get done in our company? Well, somebody should have probably thought about that, and so that was the work that had to be done was to figure out and create a machine that would take care of that. But it’s the same thing with all the rest of it. It’s sort of like, okay, well, what are the parts that need to move into the right places to get it done? Not necessarily you, the data analyst should do it, but that it gets done because it is valuable work at the end of the day, especially if it’s actually driving impact or decision making in the organization using data, which is sort of like the thing that makes me smile anytime I get a chance to be part of something like that. 00:39:13.85 [Moe Kiss]: Can we talk about data quality? We have not touched on that at all. 00:39:17.96 [Michael Helbling]: It’s usually pretty good. Yeah. I mean, just kind of automatically. Yeah. What do you mean? What was there to talk about? So I’m pretty sure. 00:39:25.89 [Moe Kiss]: I think it’s going. 00:39:26.37 [Michael Helbling]: I think it’s going. 00:39:27.19 [Moe Kiss]: Oh my God, stop. Everyone stop triggering me. 00:39:30.04 [Michael Helbling]: Sorry. Sorry, well. 00:39:33.50 [Moe Kiss]: Just come on. I think the one that I’m specifically comes to mind is Bend sent from a media agency. And I just get so frustrated or from a finance team. 00:39:56.95 [Michael Helbling]: Talk about the highly formatted Excel files you might be receiving. 00:40:03.09 [Tim Wilson]: In wide format when they should be in a long format. 00:40:08.30 [Moe Kiss]: Of course. I’m glad you could all laugh about it. I am not at the laughing stage. 00:40:14.39 [Charles Barkley]: Sorry, well, this is probably a whole episode we need to do on stuff like this. 00:40:20.95 [Moe Kiss]: But it just, I think what’s so fucking hard is that your stakeholder will be like, especially the one that owns the relationship with the media agency. I didn’t get it. They sent a spreadsheet over on Moenday. Like, you’ve got the data. What’s the problem? Like, why is it going to take you a week? And you’re like, Do you know that every single city that they run media in is in a completely different format and we then need to sense check it with our record? No, that is a huge lift. And fuck. Anyway, and then you’ve got some very senior, brilliant data scientist that is spending their time basically QAing data. It’s really frustrating. 00:41:08.94 [Tim Wilson]: That is one of those cases where that’s another shadow that the analysts can fall into where they’re the bridge between the data creation. That data may be created out of some contractual necessity, but doesn’t have any real incentive or stake outside of what’s in an agreement. It’s like, oh, we’ll send you data. We’ll send you data. Check the box. And this is going after media agencies pretty hard, that a lot of times they don’t really under, they’re like, whatever the platforms, you know, trade desk spits this data out or runs into our data warehouse and we’ll just give you a feed. And the analyst is the one who winds up having to explain their data to them. So it’s like another version of that. That particularly is another version of what you were talking about earlier, Michael, where you have to be like, Yeah, how can this possibly be zeros across here? It’s like, wait a minute, I’m now having to reach out to… Everybody seems to assume that it’s coming in fine, but I have to set up time to go three levels deep with some partner to get them to agree that it’s actually a problem or explain to me why it’s not a problem. 00:42:27.06 [Michael Helbling]: I’m literally in a situation like that right now. I ran into a situation just this past week where a company is like, yeah, we’re pretty sure the quality of the data in this system is great, and so I get my hands on it and immediately see three things I’m pretty sure making their data quality really bad. And so you’re literally starting out with sort of like, okay, well, our first conversation is gonna be, guess what? The data you thought was really good? Not good. And there’s a number of fixes we’re gonna need to do before we even start on the things we wanna get further along. And it’s frustrating but real, right? So it’s sort of like, yeah. And then the other one that gets me sometimes is sort of like alerts and notifications, anomaly detection and those kinds of things. That is a part of data, but it’s not really what an analyst does necessarily. 00:43:15.77 [Tim Wilson]: Well, the analyst gets blamed if the data all of a sudden it’s found that something wasn’t there for weeks. They’re like, what were you doing as an analyst? How did you not notice? 00:43:24.44 [Michael Helbling]: Raise your hand if you’re the only one that’s had your own secret dashboard so you don’t get caught up in one of those things. So you have advanced warning of something that’s happening. 00:43:34.23 [Moe Kiss]: I think anomalies is part of our job, but you will keep saying analyst, and I think of data practitioners, whether it’s a data analyst, analytics engineer, data scientist. For example, if there is something in our B2B pipeline that breaks our leads coming through that is absolutely data quality and normally detection and I would expect an analytics engineer to go in and solve that. Absolutely. When we’re doing at the complete other end of like a metric goes up, a metric goes down, that sort of stuff, again, I would expect a data person to go in and kind of debug that. It might, they might not be responsible for the complete like up level, you know, challenge of why that thing is or isn’t working anymore. But like, I would expect someone to be pretty across that if we saw like a number tank or something like that or a number skyrocket. 00:44:28.59 [Tim Wilson]: But, but that’s the, I mean, the way you just framed it, not to, I mean, you’re just speaking off the cuff that a, There is a perception that, yeah, yeah, yeah, they need to catch if a number of tanks are a number of skyrockets. In practice, every time I’ve had a system where it’s like trying to tune where, like there’s not a threshold, then there will be platforms out there that say, look, you can set this at a 95% threshold, set up 100 alerts. I’m like, cool, I’ll get on average five alerts a day. 00:44:58.94 [Moe Kiss]: I’m not necessarily expecting a data person to catch them all. I think that’s a really hard thing. It’s so difficult, right? Because if you have a stakeholder who comes to you and is like, hey, this number declined and you’re the data person who’s like, what? I had no idea. That’s shit. It’s hard for trust. But at the same token, expecting a data person to be able to be ahead of the game on every anomaly is also not an expectation I have. But I would I would basically be like, okay, something has gone wrong here. I’m going to reach out to my stakeholders. I’m responsible for letting them know. I’m responsible for letting them know what we’re doing to investigate, how we’re going to solve it, keep them updated. That, absolutely, I do think is a data person’s role. 00:45:47.72 [Tim Wilson]: I still have the alert turned on for a certain tax preparation company that you and I worked on years ago and like January 12th, their home page was down from Seattle because I just never turned it off. But that was one where they were having sporadic Issues and it was like somebody should be monitoring this and I can go set something up And I had to set up on like my personal account and I just never turned it off. 00:46:12.89 [Michael Helbling]: So literally Michael knows the brand I know the brand it was down for about 35 minutes Yeah You need to do some account access cleanup that’s some shadow work that a lot of consultants have to do Get yourself off of those old GA accounts or Adobe accounts that you’ve been on for years and years that you no longer work with. 00:46:36.08 [Tim Wilson]: No, this was using Site 24 by 7. I was doing like a ping tracker that I set up, so I had set it up. So a third-party tool. 00:46:47.24 [Michael Helbling]: You’re doing third-party data collection. I was using a third-party tool. And they’re probably like, why is our website getting crawled by this website? 00:46:55.54 [Tim Wilson]: But that was, they were sometimes saying like, the tool is down. And I’m like, no, like why is this anomaly in the data? Cause your fucking site went down. Like, that’s not a, cause I think I set up a ping for the footer as well. Cause based on where they had the tagging track, but I think it started with them saying your digital analytics, your web analytics data is bad. And I was like, yeah, that’s weird. What’s going on? It’s like, well, no, the whole site went down. 00:47:21.07 [Moe Kiss]: No, I didn’t once find, though I was working somewhere there was like an issue that I couldn’t figure out like why this number had gone weird or whatever. And then like a month after I left, I figured out why. And it was like completely tangential. I was just working on something different. And I did reach out to let them know. I was like, Hey, this is probably what this was. You should fix it. Here is how to fix it. You’re welcome. I’m not a shit human. I want everyone to have the best data they can. 00:47:48.07 [Tim Wilson]: I also get the, it’s backup. So every time I’ve seen it, it’s come back up quickly. So there hasn’t been a point. 00:47:54.94 [Val Kroll]: Okay. So before Michael wraps, cause you got that look in your eyes. I would love to hear. love to hear people’s thoughts on shadow work, not shadow work for like data fluency, data literacy. We’ll call it, we’ll call it, cause data literacy programs I think are one of the more common ways people talk about it because it is like a whole category of work. Yeah. I like data fluency. I think it’s less obnoxious than data literacy. 00:48:24.33 [Michael Helbling]: Everybody can read and write. 00:48:25.89 [Val Kroll]: Yes. So is it shadow work, not shadow work? I think it’s shadow work, but I think it’s important shadow work. 00:48:31.99 [Michael Helbling]: Yeah, I think it goes back to that sort of like what do you need to do to help the organization take a step forward with data, make decisions, use the data, be effective with the data. And a lot of times that’s building up data fluency in an org or helping people build up their data fluency. 00:48:48.85 [Tim Wilson]: But that’s one where if you try to bring it out of the shadows and say, oh, why don’t we just solve this once and for all and send everybody through a data fluency program, pretty ineffective. So it’s the thing that needs to be in the shadows that is a I mean, not that there’s not the opportunity for some of that training. I feel like I’ve been learning how much, I mean, it’s not, it’s the reality of a short attention span that the more you can have like in the moment, like, let me come up with, let me show you this now. Let me explain this little thing now. Let’s talk about, oh, you know what? When you all people say correlation is not causation, this is like the perfect example. Let’s talk about that for five minutes because that’s a trap you’re falling into. 00:49:30.55 [Moe Kiss]: Tim, it actually makes me think about gender bias training and all the research on that, where lots of companies do gender bias training. It doesn’t necessarily result in any differences in behaviour or attitudes or anything, but it’s a tick the box thing. When we start talking about like data fluency or training or education or whatever it is in the data space, I think what happens when we sometimes roll out those programs with really good intent, it’s a tick the box thing. But again, like those in the moment. 00:50:01.93 [Tim Wilson]: That sounds like the sort of observational woman would make, by the way. 00:50:05.21 [Moe Kiss]: We’re going to send Tim back to training. Those in the moment discussions are actually what I think is makes it so hard because it is shadow work because it’s not like I built a program, I’ve shipped it, I’ve ticked it, it’s done. It’s like every time I talk to the stakeholder, I’m trying to help them get a little bit further in how they think and understand data. And that is like you’re never done, you’ve never ticked the box. And so it does have like a very heavy cognitive load, but it’s incredibly important and probably leads to the best outcome I say without a data informed opinion on that at all. Just like that. 00:50:51.84 [Val Kroll]: I’m actually surprised that you guys are all on the same page. I don’t think it’s shadow work at all, whether it’s bite-sized or a big part of it, because even some of the criteria we were talking about using earlier, if your role is to help the business make smarter decisions, like making sure that you’re connecting what you’re finding, what you saw, what you observed, what you validated, what your recommendations are with like what the business can actually be doing with that information. It feels like it’s a, I don’t know, to put it another way, there was a leader who I worked for at UBS who like the four D’s of product development, like the defined design, develop, deploy. He always said there was a fifth like shadow. not actually a fifth one of adoption. Until you understand how people are using that or if this is a data product or whatever, then you’re not done. The work isn’t done when you ship it. The work is done when you understand and create the feedback loops. I feel like it’s very much in the same vein of how to make sure that your work continues. Michael, the work almost similar to what you were saying, creating the processes so that the team knew how to take advantage of those recommendations. I don’t know how I just feel like it’s not Like you’re not done just when the analysis is complete, or it’s not. 00:52:09.76 [Tim Wilson]: Yeah, shadow is optional. Like it has to happen, it’s just not identified as something. 00:52:13.93 [Val Kroll]: It feels like it’s squarely in the court of, I would expect it to be in a job description. Like, that’s what makes me feel like it’s not shadow work. 00:52:21.44 [Michael Helbling]: Again, that’s where I think some of this work should rise up out of shadow work. But again, it’s about recognition. The importance of it, I completely agree. But Tim’s point, I think, was, will you see it in a job description? Probably not. Or if you do, it’ll be run a once-in-a-quarter training and call it done. And we all know that’s not going to be effective. But it’s spending that time, like I’m realizing this episode that like 90% of what I do is shadow work sometimes. It’s so hard to pin down. Michael works the shadows. That or I just don’t do anything. I don’t know. But I remember I had a very specific instance where I had a review and my boss at the time was like, you’re not spending your time the way that it should be spent. And I had to actually walk him through. If I spent the time the way that they wanted me to, it would lose the company money. And I walked him through step by step. If I actually did it the way you said, the company would lose money as a result of the effort. So what you’re telling me is that you would like the company to lose this much revenue by changing what I do day to day, are you sure that’s what you want? And so it was a really interesting conversation because I was able to enunciate exactly where the value lied in each of those things that I was doing. I could show the outputs of those things. But it was a very interesting conversation because it was like, oh yeah. Now, in that case, I had actually prepped that person ahead of time by showing them exactly how I was going to spend my time. They just ignored it and came back with the template. 00:54:01.66 [Moe Kiss]: What was the outcome, though? Like, what was the end of the story? Did you get off to change it? Or did they, like, be like, oh, I see the value of what you’re doing. 00:54:09.14 [Michael Helbling]: I kept going. Yeah. No, I was, uh, that was a role in which firing me probably wasn’t an option. Probably they felt like it in the moment. I sent an email to the head of HR ahead of that meeting being like, I’m about to chew up my boss. Um, But it worked, and I still had a good relationship with that person afterwards. But it was a situation where they were like, oh, OK, well, never mind then. And I just kept going with what I was doing, since it was made sense. 00:54:36.06 [Tim Wilson]: I will claim to the job description that I think when we read job descriptions and say, well, this is looking for a unicorn that’s ridiculous. Or when we read a job description and say, wow, that looks really good, I bet, I’m thinking through some that I’ve seen, the ones that actually have the shadow work articulated as part of the responsibility is collaborating with the business partners to how to ask questions in an informed way. That actually may be, it would be fascinating to look through some job descriptions that when people say this is garbage and say, is any of the shadow work captured? Hey, this one looked, because you’ve had that reaction where you look at one and you’re like, oh, they get it. Like they actually, they’re describing a realistic and practical role. And I bet that that is because there are nuggets of what you’ll be expected to do, include some of the shadow work we’ve talked about. 00:55:30.36 [Moe Kiss]: Okay, but just a push on that I do agree. I think the challenge though is like In my role we write we write the job descriptions for data people data people are writing job descriptions for data people So you can still have a mismatch with the stakeholder of what they think a data person should be doing like so I’m just saying it’s not like bulletproof 00:55:52.79 [Tim Wilson]: Yeah, but I think if that’s recognized, it’s like, hey, we’ve got a bunch of really difficult, unrealistic stakeholder. We should have in the job description that part of this is collaborating with, not you don’t say collaborating with assholes, but you’re like, You know, collaborating with, educating, informing, iterating with, so I think he can still be captured. 00:56:18.53 [Val Kroll]: He’s ambiguous in challenging circumstances. That’s right. Yeah, exactly. Oh, yeah, there we go. 00:56:24.88 [Michael Helbling]: Sell starter, able to juggle multiple priorities simultaneously. It’s like, ugh. 00:56:32.90 [Tim Wilson]: Often when the hiring manager isn’t an analyst, then that’s why that job description doesn’t have the shadow work in it. And that does some of the. 00:56:39.96 [Michael Helbling]: And it comes out ringing false. Yeah. Well, some of my shadow work is trying to get the show wrapped up on time. So let’s go to do that. 00:56:49.96 [Tim Wilson]: We got to find somebody who’s good at it. 00:56:51.36 [Michael Helbling]: All right. Let’s hand that off to somebody else. All right. Well, listen, Moe and Val and Tim, thank you so much. This is, I think, a really interesting topic. And I appreciate your insights on the show. A lot of work is really important, but doesn’t necessarily get recognized for what it is. And I think that’s sort of where this discussion took us today. So thank you for that. You know, as you’re listening, I imagine you’re thinking some of the thoughts yourself. We’d love to hear from you and you can reach out to us. You can reach out to us on LinkedIn or the Measure Slack chat group or through email at contact at analyticshour.io. And if you’re listening to this on Apple podcasts or Spotify or whatever platform you listen to it, give us a review or a rating or a comment. We’d love to see it, love to hear it, love to hear from you. And of course, a couple other things. We’re not doing less calls, but a couple of things where you can find us coming up this year. is at a couple of few conferences and actually coming up really quickly. So, I know Tim and Val, you all will be at the Datatune conference in Nashville. Is that right? You want to talk about it? 00:58:09.09 [Tim Wilson]: Yeah. It’s a little, it’s a Friday is workshops and Saturday, it’s a Conference, it’s a pretty low cost, low three-figures conference all day. It looks kind of not measure campy from an unconference perspective, but from a enthusiasm and critical people, a lot of people, critical mass of people showing up pretty interesting topics. 00:58:32.32 [Michael Helbling]: What are the dates? 00:58:33.02 [Tim Wilson]: Oh, that would be important. Yeah. 00:58:37.65 [Michael Helbling]: I’m here for you. 00:58:38.81 [Tim Wilson]: I’m here for you. 00:58:40.11 [Michael Helbling]: Talk about shadow work. 00:58:43.88 [Tim Wilson]: What is it? 00:58:46.78 [Michael Helbling]: That’s awesome. And then, of course, Measure Camp New York will be in March 28th in New York City. It’s officially in New York City, not New Jersey this year. 00:58:56.14 [Val Kroll]: Very exciting. Very exciting stuff. 00:58:58.06 [Michael Helbling]: Yeah, it’s going to be a great… Measure Camp is always a great time. Obviously, Val’s super involved with Measure Camp Chicago. Moe with Measure Camp Sydney. Tim with Measure Camp Columbus. Me with not being involved with Measure Camp in any official capacity, but I love going to them. And I think right now Tim and I are planning to be at that one, and that’s March 28th in New York City. And then finally, April 28th and 29th, the whole Analytics Power Hour, or a lot of the Analytics Power Hour folks will be at the Marketing Analytics Summit in Santa Barbara, California, which sunshine on the West Coast. Hello. Get there. We love to. 00:59:35.50 [Tim Wilson]: We got some exciting plans for that. Stay tuned to future episodes. 00:59:39.92 [Michael Helbling]: What’s the drink that you have in Santa Barbara? What’s like a good cocktail for that? 00:59:45.15 [Tim Wilson]: I’m sure it’s some fruity California liberal. 00:59:48.22 [Michael Helbling]: Wine. Exactly. Wine. White wine or rosé on the beach or in the sunshine. 00:59:54.24 [Moe Kiss]: Love this. 00:59:54.70 [Michael Helbling]: Love this from me. I don’t know. I’m terrible at picking out drinks. All right. That’s the show. We’re excited to have brought it to you. And I think I speak for all my co-hosts when I say, no matter whether the work is in the shadows or way out in the open, keep analyzing. 01:00:15.26 [Announcer]: Thanks for listening. Let’s keep the conversation going with your comments, suggestions, and questions on Twitter at @analyticshour on the web at analyticshour.io, our LinkedIn group, and the Measure Chat Slack group. Music for the podcast by Josh Crowhurst. 01:00:33.22 [Charles Barkley]: Those smart guys wanted to fit in, so they made up a term called analytics. Analytics don’t work. 01:00:39.83 [Charles Barkley]: Do the analytics say go for it, no matter who’s going for it? So if you and I were on the field, the analytics say go for it. It’s the stupidest, laziest, lamest thing I’ve ever heard for reasoning in competition. 01:00:53.06 [Michael Helbling]: Lacy Fusion Productions. Lacy Fusion. 01:00:58.51 [Tim Wilson]: That’s our production studio’s sister organization on Southern Hemisphere covering Lacy Fusion Media. 01:01:05.83 [Michael Helbling]: 4th floor productions, Lacyfusion Media. 01:01:08.74 [Val Kroll]: Known for expanding into Australia. 01:01:12.26 [Michael Helbling]: Ken Riverside. And the Lacyfusion Media. Present a 4th floor production. 01:01:21.68 [Val Kroll]: Okay, well screw your green bars. You sound like you’re in this building with a paper cup and a string. All right, love you. 01:01:43.82 [Michael Helbling]: Your temperature, Matt. 01:01:50.70 [Moe Kiss]: She is so cute. 01:01:52.76 [Michael Helbling]: I know. It’s ridiculous. 01:01:55.49 [Moe Kiss]: So cute. 01:02:02.79 [Tim Wilson]: Rock flag and who knows what insights lurk in the tables of our databases. The shadow analyst knows. 01:02:15.79 [Michael Helbling]: Nice. That’s actually pretty close. 01:02:20.54 [Moe Kiss]: I’m like, Damn, you got the voice. 01:02:23.81 [Tim Wilson]: That’s got something. The post #291: The Data Work that Lives in the Shadows appeared first on The Analytics Power Hour: Data and Analytics Podcast.

From a professional development perspective, you should always be learning: listening to podcasts, reading books, connecting with internal colleagues, following useful people on Medium and LinkedIn, and so on. Did we mention listening to podcasts? Well, THIS episode of THIS podcast is not really about that kind of learning. It’s more about the sort of organizational learning that experimentation and analytics is supposed to deliver. How does a brand stay ahead of their competitors? One surefire way is to get smarter about their customers at a faster rate than their competitors do. But what does that even mean? Is it a learning to discover that the MVP of a hot new feature…doesn’t look to be moving the needle at all? Our guest, Mårten Schultzberg from Spotify, makes a compelling case that it is! And the co-hosts agree. But it’s tricky. This episode is brought to you by Prism from Ask-Y—your agentic analytics platform for automating analytics, exploring data, creating repeatable workflows, and delivering accurate insights—all without the need for manual query writing. Links to Resources Mentioned in the Show (Article) Beyond Winning: Spotify’s Experiments with Learning Framework (Article) Two Questions Every Experiment Should Answer (Platform) Confidence by Spotify (Article) Choosing a Sequential Testing Framework — Comparisons and Discussions (Article) Bringing Sequential Testing to Experiments with Longitudinal Data (Part 1): The Peeking Problem 2.0 (Article) Bringing Sequential Testing to Experiments with Longitudinal Data (Part 2): Sequential Testing (Article) Risk-Aware Product Decisions in A/B Tests with Multiple Metrics (YouTube Channel) 3blue1brown by Grant Sanderson (Article) Escaping the AI sludge…why MVPs should be delightful (Conference) DataTune in Nashville: March 6-7, 2026 (Conference) Marketing Analytics Summit in Santa Barbara: April 28-29 (Article) The next data bottleneck by Katie Bauer Photo by Jason Dent on Unsplash Episode Transcript00:00:05.75 [Announcer]: Welcome to the Analytics Power Hour. Analytics topics covered conversationally and sometimes with explicit language. 00:00:15.90 [Tim Wilson]: Hi, everyone. Welcome to the Analytics Power Hour. This is episode 290. I’m Tim Wilson, and I’m joined for this episode by Val Kroll. How’s it going, Val? 00:00:25.38 [Val Kroll]: Fantastic. Excited for today. 00:00:28.80 [Tim Wilson]: Outstanding. Unfortunately, we were supposed to also be joined by Michael Helbling for this show, but he’s gone all on brand for the winner and gotten the flu. Luckily, as we’re into our 11th year of doing this show now, we’ve learned a thing or two about rolling with the punches. And as it turns out, learning is the topic for today’s show. I mean, it’s implicit in all forms of working with data. We’re looking at analysis or research or experimentation results and hoping, just hoping that we come out of the experience with a deeper knowledge of something. I mean, and hopefully it’s something useful, more knowledge than we had before. It’s a simple idea. Sometimes though, it’s a little harder to execute in practice. That’s why we perked up when we came across an article from some folks at Spotify called Beyond Winning, Spotify’s experiments with learning framework. We’re excited to welcome one of the co-authors of that piece to today’s show. Mårten Schultzberg is a product manager and staff’s data scientist at Spotify. He has a deep background in experimentation and statistics, including actually teaching advanced statistics in a prior role for a number of years. So who better to chat with about learning? Welcome to the show, Mårten. Thank you so much. Excited to be here. Oh, right. It’s a borderline giddy about the topic as we were diving into our excitement before we hit the record button. Yeah. 00:01:59.34 [Val Kroll]: We definitely fought over who got to be on this one. 00:02:05.69 [Tim Wilson]: Mårten, in the article that I referenced in the opening, which we’re definitely going to link to in the show notes, it’s a great read, you and your co-authors make the distinction between a win rate and a learning rate for experimentation. That’s the premise of the article is this win rate. this learning rate as a proposed metrics or a metric that’s actually in use. That seems like a good place to start. Maybe can you explain what you were seeing as a drawback to too much focus on win rate as a metric for experimentation programs? 00:02:43.90 [Mårten Schultzberg]: Yes. I think it needs to take a little step back. I think it started with When we rolled experimentation out at Spotify properly, like at scale 2019-2020, we quite quickly realized that one of the biggest wins that we made over and over again was to detect bad things early and being able to avoid them. So using it as a sort of dodge-bullets type of mechanism. And we have used it like that since. It’s one of the biggest reasons why we run so many experiments. We want to avoid shipping bad things that happens, you know, unintentionally. Side effects and stuff like that. And at the same time, I’ve seen over the years a lot of blog posts and papers published about win rates from other companies. Win rates as in the rate of experiments where you find a variant that is better than the previous variant and you ship it. So a clear winner. And so I just felt that it was sort of under celebrated all of the other types of wins that you can make besides finding something that was better than the current version. And I also think that it doesn’t really reflect how most companies, at least the companies I’m familiar with, are actually using experimentation. They’re using experimentation partly to optimize things. So to find winners, to continuously improve something and optimize it. But that’s only one part of that puzzle. The other part of using it as a mechanism for safety and safety net is something that wasn’t, I think, talked about enough. And so that’s sort of where this sprung from. 00:04:22.92 [Val Kroll]: I love that. And the one thing though that I think is, I would love for you to talk a little bit about more, that I think even if an organization was like, yes, like in spirit, I completely agree with that premise, Mårten. It seems like using a metric like learning rate seems squishy. Like win rate is objective. We can tally that in a column and calculate that percentage. But can you talk a little bit about how you thought about the criteria for determining how you say, yes, we learned something from this experiment or how it’s defined. 00:04:56.01 [Mårten Schultzberg]: Yeah, and so yeah, I want to firstly call out that this was a team effort. It was a lot of people involved. It was driven by the central experimentation team at Spotify, but there was also a lot of other data scientists that are actually doing product work that was involved in this discussion. We had a lot of really good discussions actually about what learning means and when you actually get value from an experiment. And so I just want to call that out. And I think We see it as there are essentially three ways that you can learn from a Navy test. One is that you find an obvious winner. So what other people refer to as win rate. So you find a version that is better than the current version. The other one is that you find that the current version was worse somehow, that you detect something bad, that you detect the regression or Often that can be, you know, not only that users didn’t like it, but more that maybe something went wrong with some integration somewhere, so you get latencies increasing or crashes increasing. And so those are quite obvious wins, so the finding better stuff and avoiding worse stuff. And then there is the middle one, which is more nuanced, which is when you run a well-planned experiment and you find nothing. So a neutral experiment, which is, I guess, vague. But what we count there as a win is an experiment that actually had a sample size calculation before that did a proper power analysis and said, hey, I want to have a certain power of finding an effect if it exists. And then they ran that experiment according to that plan, and they found nothing. We also view that as a learning, because at that point, they can actually, with the certainty that they hoped for, say, no, there was no effect from this change. So the neutrality in that case is informative, because you can say, hey, maybe this is not worth pursuing, because we actually ran a proper experiment. If there was an effect of the size that we were interested in, or that we hypothesized, we would have found it. So there are those three cases. And obviously that middle one, the neutral one, is a little bit more complicated. It’s more complicated to implement or to instrument because you need to know what sample size calculations were run and if the experiment actually met the planned sample size and all of those things. Fortunately for us, in our tool, it’s fairly easy to do. But yeah, take some thinking to get that right. 00:07:27.62 [Val Kroll]: I’m literally writing those because there’s so many things I want to dig into. But before going to the 5,000-foot view, I guess I’m just curious about the culture change internally. with so many people with access to run experiments and this appetite for experiments, what was it like to get them to shift away from the win rate to this other new metric that you rolled out? I’m just very curious what that experience was like if there was resistance, if there was excitement, or some people were really questioning it. 00:08:01.99 [Mårten Schultzberg]: There’s always people questioning everything at Spotify, which is one of the things that I love about Spotify. So that’s a constant. But yeah, I think because of the fact that we so early realized that experiment was such a powerful tool to avoid mistakes and to detect bad things early, I think that the sort of common definition of learning was already incorporating that aspect of experimentation. I think a lot of people has sort of over the years learned to, I should not use the word learned, come to appreciate that, yeah exactly, come to appreciate that avoiding something bad is a great learning and something that is super valuable for product development. So I think that part was not so controversial when we developed this metric. I think the neutral one is trickier and there’s also It’s a much more room there for discussions about what should count, should you be super strict about that it should be exactly powered, should you allow some wiggle room, there’s a lot of things that you can discuss there. We were eager to get a very clear and explicit definition out and we were also eager actually to write about it externally because we were hoping that other companies would, and I guess this podcast is a good example of that too, that we could have this discussion because I think it’s been I’m really curious how other people think about this. I’m not convinced that our definition of learning is like the ultimate one or the final one or anything, but I think it’s a good first step away from the more naive, only wins count definition. 00:09:53.50 [Tim Wilson]: The raging cynic in me would be, well, gee, if people realized that’s a way to game the metric would be to run really inconsequential small tests, which at the same time, the analyst in me thinks that, yeah, that happens with analytics a lot, that you’re kind of digging in and trying to find something. You’re like, well, somebody thought there would be some relationship here and we’re just not seeing it. And that can be equally unsatisfying for the analyst. So like, how do you think about neutral being, we were trying something that did have a legitimate chance of being meaningful. And maybe this kind of bridges to another article that you wrote, which is, you know, like, how do you say neutral, but not have neutral become a crutch for, yeah, we’re essentially doing AA tests and, you know, giving ourselves two thumbs up on the learning rate. 00:11:01.44 [Mårten Schultzberg]: That’s a great question. I think we’ve been thinking a lot about what a healthy distribution should look like. A healthy distribution of different types of winds and also the proportion of neutral experiments. And I think that’s actually a super interesting topic. I think depending on what kind of strategy you have here from a product side, you can want to have different distributions. So for example, if you take the If we wait with the neutral one, because it’s maybe a trickier one, but if we think about how many experiments should you find regressions in that you dodge versus the win rate, how should that distribution look? Well, that will depend on a lot of things. But if you’re a company that has everything to win and little to lose, then maybe you can afford to have a high rate of just trying stuff. Because whenever you find a win, it’s going to be quite big because you’re still in early stages, whereas if you’re If you’re a product that is already very mature, then maybe you have other goals for those things. It’s a super interesting discussion to have, and that’s one of the discussions we’re having now with teams at Spotify and other people that are using our experimentation tooling. What should we do with this information? And what’s good and what’s bad? And I think it’s different for different parts, even of organizations within Spotify. What’s good, depending on how they’re looking at it. But for sure, we wouldn’t look at the learning rate only. So we would say we want the learning rate to be reasonable. But then we, of course, should probably aspire for having a high win rate. That’s nothing bad in itself. But at least if we have a high learning rate, we know that we’re not wasting our experimentation efforts. We know that experiments we’re running, we’re actually learning from. If we’re running a ton of experiments that are not powered and neutral, then we will never be able to say these things didn’t have an effect. We can’t separate between if these things didn’t have an effect or if we just didn’t run a good enough experiment to detect it. So on the one hand, you look at the learning rate and say like, hey, we want to utilize our experimentation bandwidth really well. So we want to have a high learning rate at all times. But then at each quarter, you can look at this metric and the distribution of these outcomes and say like, hey, you know what, we’re dodging a lot of bullets, but we’re almost never finding something good. Should we rethink our strategy or even more, if we’re finding a ton of neutral results and we see more and more neutral results in some part of the organization, maybe we’re hitting diminishing returns and we should try something different. Maybe we found some kind of local optima, maybe, or something like that. So I think it can be a quite strategic instrument if you have all of these, the distribution of all of these outcomes as part of the learning metric. 00:13:52.61 [Michael Helbling]: You know what’s worse than writing SQL? Probably writing that same SQL for the third time because you forgot where you saved it. 00:14:00.54 [Tim Wilson]: or explaining to an LLM for the 10th time that your GA4 medium field is a mess because three different interns had three different naming conventions. 00:14:10.59 [Michael Helbling]: Yeah, like organic, organic underscore social or, I mean, it’s like a crime scene of good intentions. 00:14:18.34 [Tim Wilson]: Which is why Askwise Skills feature really helps. 00:14:22.20 [Michael Helbling]: Record that data cleaning nightmare once as a skill, reuse it across different datasets, portable expertise, and their jam memory system remembers context, like the July data is doubled or use the product table, not staging. Exactly. 00:14:39.18 [Tim Wilson]: It’s context focused, not just code focused. Plus your data never touches the LLM. Semantic layer generates code that runs locally. 00:14:48.33 [Michael Helbling]: where your data presumably won’t judge you for that medium field situation. We can hope. We’re going to ask-y.ai. That’s ask-y.ai. Use code APH to jump the wait list and stop paying the context switching tax. 00:15:07.37 [Val Kroll]: That’s making me think as you were talking about that, that like even within an organization, like you were saying like companies who have everything to gain or you know, I think everything to Nothing to lose. I forget exactly. I never get that right. Well, apparently I can’t either. But even within Spotify, thinking about the different product teams that if it’s a group that’s working on the cancellation flow and thinking about retention, they’re probably having very different distribution of those outcomes as their goals or targets versus playlist creation which is like such an established user pattern is that like how you customize some of those conversations from like the center of excellence experience like perspective to kind of consult with those teams. 00:15:56.79 [Mårten Schultzberg]: Yeah, let’s say so, but I also add that there’s a lot of centers of excellence when it comes to experimentation at Spotify. Fortunately, we have many parts of the organization that have super strong experimentation organizations or champion groups or nerds. I like to think about it. I mean, look who’s talking. But anyway, no, but so I think, so that discussion happens locally in a lot of places and a lot of people are having those discussions. So it’s not like sometimes we get, you know, questions about how to think about things. And also, one interesting aspect of this metric is that sometimes you might find that You know, if you’re actually, we didn’t talk about, there’s one outcome here that we didn’t talk about, which was the, when you get an invalid experiment where something is wrong with the setup of the experiment. That’s the final sort of outcome in this learning framework. So you didn’t learn because something went wrong. For example, something went wrong with integration. Maybe you got imbalanced treatment and control group assignment for some reason, or you don’t get all of the data that you should get or something like that. And that’s of course an outcome that is the least fun one, so to speak. It’s just like, yeah, we couldn’t get this integration to work well enough. So we have used that one and worked really hard on getting that to as close to zero as possible. We want it to be possible for anyone to run a really high quality experiment. With Spotify running experiments on so many different devices and apps and combinations of those, it’s really tricky to always nail those things, but it’s obviously an important signal. So whenever that one is high, that’s something that teams come to us with and say like, hey, we don’t get our integration to work as well as we want to, how can we improve these things? And also when it comes to the neutral aspect, the quality of the sample size calculator starts mattering a lot. So whenever someone sets up an experiment and we try to predict what sample size they need, it’s a prediction, right? We’re looking at historical data saying like, yeah, well, given how historical data has moved, the variation in that data and the means and the treatment effects that you say you’re interested in finding, we think that you need to run your experiment for this long to reach this many users. And that’s a prediction that takes a lot of things into account, but it can always be improved probably. So that’s also a conversation that we sometimes have when people are like, in our use case, the sample size calculator is not good enough. 00:18:33.77 [Tim Wilson]: But that is a case where you, that’s one where you would come back. Like what is the scenario where you run it, they’ve got a MDE, they’ve got the estimates, you’ve got the sample size calculator, it says run this. If it comes back, I’m trying to understand the distinction between, actually, we probably just didn’t run this long enough versus, well, for what we ran and what parameters we put in, it’s a neutral result. Is there a distinction there? 00:19:13.12 [Mårten Schultzberg]: I can speak a little bit to it. In practice, when we do the sample size calculation, I don’t know how technical and nerdy I’m allowed to get here, but given the name of this podcast, I’m going to go deep. 00:19:26.24 [Tim Wilson]: We don’t want to hit if Matt Gershauf would have to think about it for a minute, that’s a little bit too technical. 00:19:34.37 [Mårten Schultzberg]: No chance, no chance. This is bread and butter for him, promise. No, so we never know the variance of the treatment group, right, before we run the experiment. We can always just think like maybe it will be a homogeneous treatment effect, or we could, I suppose, speculate about how the treatment will affect the variance, but it’s always gnarly, it’s difficult to do. So what we do always in practice, I think everyone essentially is saying like, let’s presume that the treatment effect is homogeneous. In practice, of course, when we start running the experiment, maybe the treatment effects only part of the treatment group, which will then disperse the distribution. If we have a beautiful distribution to start with, but some people get the large treatment effect, you will make the variation of that distribution larger. So the variance in that group will be larger. So the required sample size will go up. We do, in confidence in our experimentation tool, we do both. So we have the pre-experimentation sample size calculator, which uses historical data to make this prediction. And then during the experiment, we’re also collecting the data from the experiment and running the subsize calculation continuously. I actually wrote a paper about that. I think there is a blog post about that too. If someone wants to nerd in on that, that it’s actually valid to look at the power during the experiment. It’s a peaking that is non-problematic. You can look at that. Anyway, so you have those and you might have a big discrepancy then. So when you start the experiment, you might think that, hey, I can run this for two weeks. I will reach my whatever 10,000 users that I need. But then when you run it for a week, you realize that like, no chance. I will reach much less or I will need much more, maybe more likely. I thought I needed 10,000, maybe I need 40,000. And that’s just not possible given the traffic that I have on this page. And in that case, it might be done a conversation about like, hey, how can we make this better? And so one way that we do it in practice is that we say like, okay, maybe instead of us trying to predict it, you can point to a similar experiment. If you know you have a similar experiment, we’re changing the same kind of thing. But yeah, it’s a tricky thing. It’s a truly difficult problem to make good sample size estimations. 00:21:43.79 [Val Kroll]: And one thing that I found interesting, because there’s definitely like two different camps here, is that I hopefully I’m not putting, correct me if I’ve interpreted this incorrectly, that you do allow for multiple success metrics in this, which I know makes that a little bit more complicated. And I think it also talked about adequately powered guardrail metrics, deterioration metrics, quality metrics, which not a lot of organizations do or have the capability to do, but that was like, oh, well, definitely enough to talk a little bit about that. But how do you handle the multiple success metrics, especially if you’re looking at things further into the funnel that have a lower incidence? How do you think about that layer? 00:22:27.60 [Mårten Schultzberg]: Yeah. This is a rich topic. We have a framework for this that we have developed over the years. And it’s also a paper that is, I think, about to be published. It’s an archive, at least, where we go through exactly all of the details of how we’re handling the multi-metric that we call decision framework, statistically. But I can give the short version of it. So essentially, what we’re saying is that we have an explicit decision rule for the multi-metric setup. So we have success metrics and guardrail metrics. So success metrics are metrics that you want to improve, and guardrail metrics are metrics that you don’t want to harm. And so, for example, at Spotify, maybe we want to improve the music consumption, but we don’t want to harm the podcast consumption. We don’t want to do it at the expense of podcast, for example. So if you’re making a new music recommendation algorithm, you don’t want to harm any other consumption. And so the decision rule is essentially that at least one of the success metrics should have improved and none of the guardrail metrics should have been harmed. There are a lot of nuance here, because for the garter and metrics we’re using so-called non-inferiority tests, which makes everything much more complicated to talk about, but leaving that aside, it means that when we’re talking about power and false positive rate, we’re talking about the false positive rate and the power for that decision rule. So we’re saying we want that decision that we would make based on this rule. So at least one of the success metrics are significantly better, and none of the garter and metrics are worse. We want that to be the false positive rate of intent, and we want to have the power to detect given the sample size. So we have to make the adjustments for multiple testing corrections accordingly, and then we have to make the power and sample size calculations accordingly. things to fiddle with there. But in principle, since the guardrail metrics all have to be not harmed, they are not giving you additional chances of succeeding, so you don’t have to correct for them in the same sense. But at the same time, you have to power them simultaneously, because all of them has to show simultaneously that they weren’t harmed if you’re using non-inferiority tests. I’m deliberately avoiding going in too much to know if you were to test because it’s like such a tongue twister to talk about. But if you’re interested in… You still said it eight times. 00:24:51.78 [Tim Wilson]: Good. 00:24:52.60 [Mårten Schultzberg]: Yeah, no, but that was… No, but it’s tricky. So, yeah, so that’s how we do those things. So it’s a bit messy, but… 00:25:02.95 [Val Kroll]: So back to the culture side of this, how do you coach product teams to not just pick 50 success metrics? Because they are so excited about this new feature. It came from up high, and we really want this to, we want to find some success. And obviously, there’s a statistical part of it, like the correction, but culturally, how do you guide that conversation away from? No, it shouldn’t be like a pick list of up to 75 metrics to find something that went quote unquote up. 00:25:36.58 [Mårten Schultzberg]: Yeah, yeah, yeah. No, I mean, this is a conversation that we have. I think it’s Spotify. It has settled, but like this is a conversation that we have from time to time. And I think it’s a It’s a sort of healthy discussion to have because it’s not… I think this is more tricky than it might seem. I want to give the answer that, no, but of course you should just have a discussion and decide on the metrics. I’ll come back to that because that’s ultimately what we do a lot at Spotify, but there is more to it. There is also the fact that we’re making a lot of changes and we are truly interested in any kind of effect that it has. It’s a true statement that if actually this change that I made affected a metric that I didn’t think about, like some weird metric, weird from my perspective metric, if that was truly the case, I would want to know. So from one perspective, I can really understand this. I want to look at all of the metrics and just see which one that I affected. But then on the other hand, you get this obviously super hard problem of like, cursor dimensionality type issues here where you’re looking from too much, so you’re either just going to find noise, or you’re going to find noise, and then you have to control that, and then you’re going to have, instead, very low power to find things. But I think there is merit to the type of experiments where you’re just like, I just want to see what happens when I do this. And I don’t really care. Of course, I care what it is that happens, but I am ultimately interested in all things. But in practice, of course, this is hard. So again, at Spotify, it’s not like the central experimentation team, which I’m part of. building the tooling, we are not dictating these things. It’s rather the other way around that we are, I like to think about it as that we are sort of cultivating what the teams that are doing experimentation are thinking about this. So we have a lot of discussions with them. So the way it works as Spotify is that we don’t decide the defaults and how things should work in the platform. It’s rather that we talk to all of the product teams that are experimenting the 300 teams in various forms and then We collect what they’re saying, and we’re refining it, and then we’re putting that into the tool. So when it comes to this, how many metrics you should have, there’s not one answer at Spotify. It’s different in different parts of the organization. But in most of these parts, there have been very explicit conversations where people have talked about, like, hey, how should we trade off here? actually getting super high precision in the things that we know we’re interested in versus getting interesting insights and stuff that we could be interested in. And this is sort of traded off in various parts of the organization and in various projects, depending on how and what stage those projects are. If it’s like a very new product, then you probably see, or you often see experiments with much more metrics because you’re just interested in understanding what happens when we ship something like this, what kind of behavioral changes does this cause? Whereas when we’re optimizing something, then we’re like, okay, we know pretty well what we need to measure here to do this and to optimize this in a healthy way. 00:28:38.19 [Tim Wilson]: to Spotify, massive user base, a lot of the ability to design, to try to cover and still be sufficiently powered seems doable. I’m thinking of a client we had that was in that same boat. It still feels like the risk, the slippery slope, fishing expedition of let me tell myself a story that I just want to see if it impacted anything. And the understanding required that if you go on a fishing expedition, you are, I think, if I understand correctly, your false positive rate can go way up because you detect noise as a signal, which then when you detect it, you get really excited. Nobody can rationalize why this metric changed. It turns out it was noise. Now we’ve wound up doing negative. We’ve learned something incorrect potentially, unless you have the discipline to say, if we’re going to chase that, we need to come up with a theory, and we need to have the rigor to validate that theory before we accept it as fact. That just feels coming from an analytic side, similar sort of thing. If I just point the machine at all the data and it finds anomalies or finds patterns, there’s a very good chance that it’s detecting noise that just happened to hit at a point where it can show some statistical merit. Somehow, some part of me is just terrified. While I love getting comfortable with We looked for X, we did not find X. That is still a learning and let’s work with our business partners to acknowledge that’s a learning and not have them just chasing for everything. That also feels like a challenge, you know? 00:30:40.48 [Mårten Schultzberg]: Yeah, no, no, I mean, I agree with everything you say, but I also feel like, I mean, I have the same uncomfortable feeling in my body when I think about this, like, let’s look at all of the metrics from a statistics perspective. But I also just like, I really want to, I also think it’s a cop out, not projecting on you now, Tim, but for myself to say like, you know, to say like, you know, We can only look at the metrics that we decided before because we decided that we found nothing. Let’s move on because it’s also obviously true to me somehow, even though I can’t come up with this is how you should do it and this is how it won’t lead to these incorrect learnings that you mentioned. But it feels like it’s a hard argument to make when someone says, yeah, but I looked at some other metrics and I learned something. And then you’re like, maybe you did, maybe you didn’t. And I can think about ways that you could do this. You could do sample splitting and stuff. You could take one part of the sample and look for groups. And then you could validate those findings in another part of the sample and stuff like that to make it much more plausible. Again, you would have the issue then of having lower powers actually find things, or lower precision at least. I just don’t want to be too much of a… Curious? Yeah, or like a grumpy statistician kind of person. But I do, I mean, I agree. I have the same feeling and I haven’t seen anyone do it well. So what I’ve seen is that people have used the argument of saying like, yeah, we must be able to be able, you know, it must be possible to learn more and then just throw all of the metrics at it. And then I think they’re just as well. Like that’s just as bad as not doing it, I think. So I don’t have an answer to it, but Maybe someone smart listens and then they can call me. 00:32:22.39 [Val Kroll]: Yeah. Let us know in the comments. 00:32:24.08 [Tim Wilson]: I mean, I, I mean, I, and I don’t know that this is the answer, but I’m, it does feel like, well, if you, if you throw that at it and you find something figuring out how to have the, the step, which is probably a combination of a data scientist or a statistician with the product manager to say, we need to come up with a, plausible theory as to what’s causing that surprising thing. And we need to have somebody with their bullshit meter turned on. Cause I mean, I’ve certainly watched people find things and they come up with a bullshit theory. They’re like, well, this is clearly happening. Cause obviously like left-handed people, when they’re in the Southern hemisphere, it makes sense that they would prefer the color blue, you know, and something that’s, It’s a theory that fits the data, but it’s not a theory that holds up to human scrutiny. 00:33:25.16 [Mårten Schultzberg]: I think one thing that I’m excited about is replication. I think if you have a streamlined enough way to run experiments and you have your velocity throughput for experimentation high, then one true possibility here is to replicate, to just say like, okay, I looked broad and deep here and I found something. I believe in it. I think I’ve made my people in the sudden hemisphere argument, but I believe it. And then for anyone who would say, I believe in it to the extent that I will now launch a new experiment, take 10% other people or a new random sample and run it again with only that metric or only the new metrics that I care about. And if I can repeat it, then I will ship it. Then I would be like, yeah, go for it. 00:34:16.19 [Tim Wilson]: Or potentially, if the theory is, well, it was this kind of incidental thing that happened to be part of it, but it wasn’t the core focus when we run an experiment where I’ve doubled down on that to say this should now I should now really detect a strong signal because it’s backing that up. 00:34:39.73 [Mårten Schultzberg]: That sort of touches a little bit on the other blog post that you mentioned that has to do with what the intent with an experiment is. I haven’t really talked about it yet. 00:34:51.12 [Tim Wilson]: Let’s talk about that one. Boy, I got giddy on that one too. 00:34:56.65 [Mårten Schultzberg]: Should you want me to give the TLDR on that one too? 00:34:59.45 [Tim Wilson]: Yes, please do. 00:35:01.50 [Mårten Schultzberg]: Yeah, so the idea with that one is I often have like it has sort of come from a lot of the conversations that we’ve had with people running experiments, talking about the learning framework. And then people are like, hey, we have a lot of neutral experiments here. We run high quality experiments, but we don’t find things. And so one thing that I’ve sort of identified from working with teams that Spotify, but also externally other companies is that People are often sort of starting to optimize the idea in their head before they’ve tried that the idea is at all something that will affect the user users. And so what I mean by that is that people are, you know, when they identify something that they think is like, this is important for our users, like, let’s, let’s use a stupid example, like a button color or something, you know, like we think it’s important. And then immediately, instead of saying, OK, we should first answer the question, is it important or not? Do users care or not? Instead, they immediately start thinking about which color is the best. And so they jump from, we have no idea if people care about this to having the conversation about which color is the best. So sort of presuming that people care at all which color this has, besides having a high enough contrast so you can see it. And so this blog post was me just trying to formulate that, like the distinction between identifying if an aspect of your user experience is something that you can optimize if it has sort of an effect on users in any way, people care about it on the one hand, and optimizing that once you have identified that it’s something that people care about on the other hand. So sort of identifying something versus optimizing something. And so I think that this thing that we talked about now is a little bit about maybe if you run an experiment where you thought something, you thought that it was important with some aspect, or you tried to optimize it, and then you find something something new, some metric that you didn’t anticipate to move. That might cost the sort of idea in your head to be like, hey, maybe there is a mechanism here that people care about. Maybe people actually care about how many items we show on this screen. I was thinking about the ranking, but as a side effect of that, we showed more things. So we saw that, I don’t know, lower down that the list clicks increased or something like that. And maybe that’s an indication that this is a mechanism that people care about. I think this going in between the states of identifying something to optimize and optimize the thing you have identified and doing that explicitly and deliberately is something that a lot of product teams would benefit from. It’s easy to fall in the trap of trying to do both at once, I think. 00:37:48.86 [Tim Wilson]: Totally. Is a cousin to the optimizing I mean, the framing of say, which is kind of a, I think it might even be in the article, like the case for taking a bigger swing, take the big swing first, make sure that connects, even if it’s a, you know, in a while, it’s like, yes, there’s something here, now we can tune it. And that, I think of it from a, I mean, from a marketing analytics perspective, where companies will say, Let’s just try it out and see what happens. It’s kind of a death knell. It’s going to be an underinvestment in a new channel or a new tactic where logically, it’s going to be really hard to detect a signal because it winds up getting kind of tempered down to a pretty subtle change. The logic is, well, if this thing actually matters, then we can make a nominal investment and we’ll see this outsized lift as opposed to saying, does this matter at all? Double down on it for some period of time. Go hard. See if you actually see something and then say, okay, we definitely need to be in this channel or using this tactic or doing this to the user experience. Now we need to sort of figure out Did we actually spend twice as much as we needed to, we can get the same? Where are the diminishing returns? It does feel like culturally it’s a tough, human nature is risk averse. Saying, try something and know that you’ll find that it is okay to find that it didn’t work. A big swing with a neutral result feels like it has a lot more merit than a little small tap with a neutral result. That’s the fun in that. 00:39:39.89 [Mårten Schultzberg]: That’s precisely it. This actually what provoked me to write it was discussions about the neutral outcome in the learnings framework where people are like, people are like, yeah, but neutral is no fun. I don’t care if it was powered or not. I don’t want neutral. And that got me thinking, well, if you don’t like the neutral result, it means that the question you posed wasn’t interesting enough. Because I would be like, if I’m convinced as a product person that people care about this thing in our app, if I change this, people are going to care. And then I make a drastic change and nobody cares. I’ve run the experiment, I have high precision in my estimates and nobody cares. If that’s not the learning to be excited about, I don’t know what is, to be honest. That really shows that I’m 100% off with my understanding of what people care about, which is truly strong learning. But on the other hand, this change that I made was like, yeah, I really think our users care about this aspect and I made a minuscule change to it and I didn’t find anything. I might think for a long time about if this was the right change that I made, or if it was… You just get stuck in weird things. But one way that I have sort of sold that, because I agree that people are risk-averses to run both. If you run a neighbor test, people tend to want to be like… But I think I know what users like. I want to go for the identify and optimize at the same time version of this thing, where… I try to choose the right value for my customers or my users. But I also say, just also add then, if you haven’t actually identified that this is something that people care about or that matters for your business or where it might be. Add the more sort of provocative version. I call it maximum viable product, I think, because of course, this has to be reasonable. If you make some button larger than the screen, then of course, you’re going to see some change. So it has to be within the limits from what is still a usable function, but that is still extreme. So the maximum change that you think is like, but this is still, this is not 00:41:48.80 [Tim Wilson]: You’re saying doing that within kind of a multivariate, say we’ve got our control, we’ve got what the optimized and identified at the same time version, and then we have an identify only version. And it’s okay if that identify version detects like the biggest effect, you can say, yeah, that was kind of hedging to make sure that that we got something out of it. And if that one that was identified and optimized simultaneously didn’t, then we’re probably still on a good track. It just turns out we’re not so omniscient that we can come up with the perfect variant in one shot. 00:42:30.52 [Mårten Schultzberg]: I think it’s smart also from, I mean, a lot of companies at least, Spotify and other companies that I work with, they’re all struggling with having big enough sample size, right? both because they have limited traffic, but also because they’re interested in small effects, generally speaking. But the nice thing about making a very drastic change is that it should have a large effect. If you’re making this maximum viable change, then that should cause a large effect. So you should be able to say, yeah, but now I pull this lever as hard as it’s possible to pull. So this should cause maybe 5% change, like whether it’s good or bad. And so you can maybe run smaller experiments. If you’re in a situation where it’s hard for you to know what you should, like you have a hard time finding bandwidth essentially for optimizing things, then I think it’s a smart idea to do these more drastic changes to identify what you should then spend larger experiments on optimizing. Because the truth is that when you start optimizing, even if it’s a nice convex surface for this thing, button size or something, the closer you come to the optimum there, the larger samples you’re going to need to be able to identify those steps. 00:43:44.98 [Val Kroll]: It seems like the framing that I really liked in this article is the building the right thing versus building the thing right. And it feels like the stakes couldn’t be higher in everything you guys are just talking about in a product context because it’s not just about changing a button color. In a lot of cases, this isn’t about UX. It’s about adding additional features or different capabilities. and you’re hoping to impact things like customer lifetime value, not just did they get to the next screen, right? So it’s not just like checkout flows, right? I think I was thinking about this. I’ve actually spent more time than the average human should thinking about the changes that have been happening lately inside of my United app. So I’m United Loyal, I fly United and the app has been changing a ton lately. And we went from, there was one place where I could change my seat to every single screen within this app. I can change my, which I do appreciate. I’m definitely someone who loves feeling a lot of control over changing my seat. But I’m like, what were the conversations that happened internally that said, you know what? The user needs to be able to change their seat while they’re checking in their bag, while they’re checking to see what gate their flight is at. Anyway, just to bring this back to an actual question, building the thing right, and maybe the feature is great, the new functionality that you’re adding, but maybe you have gone about it the wrong way, which has impacted the ability for someone to understand What exactly this is capable of? Maybe it was a micro copy issue, or maybe it was in the wrong place in the flow, which feels more like optimization. Even though this framing and big swing versus small change, that sounds really objective. If you put them side by side, that’s clear. I’m especially interested because now you are in a product role to get a little meta about it. How do you think about what is, when would you ever recycle a concept in a different context? Because it does feel like the optimization killed your ability to understand if it was viable. 00:45:59.02 [Mårten Schultzberg]: The truth is here that this is difficult. I think especially starting with that building the thing right versus building the right thing. Some things you have to do quite a lot of building to even check if it’s the right thing. If you’re building a new feature, there might be a lot of things that you have to get in place to even see if it’s something that people cares about. once you’ve seen that they care about it, like maybe they don’t like it. And that’s because you haven’t built it right yet. So like, I mean, it’s this is a very stylized blog post, of course. But the truth is, is much more muddy. So yeah, I mean, in practice, I think that one of the things that have been discussed a lot that Spotify and other places is like, okay, but with experimentation, where is the room for the product intuition and making bets on things and stuff like that? And I’ve always liked to say that these are completely uh you know they’re they’re augmenting each other they’re helping each other like it’s no they you can make you can have this strong intuition still and you can make these bets what experimentation helps you with this actually validating that your bet was good and helping you change your direction if it wasn’t good and so What I’m trying to say is that, of course, sometimes, and maybe not even rarely, when we’re building experimentation tooling, we have to build for quite some time before we can answer either of these questions. And it’s hard to disentangle them even. So I’d say that we build a completely new feature for experimentation, then some new methodology or something. It’s hard to even have What’s the dimension along here I can test if this is a lever worth pulling? That’s maybe a question more for market research or user research, all those kinds of things. Yeah, so that’s the truth. I think it’s just a lot of, I think the teams that I’m writing this blog post for that I’m thinking about are the teams that sort of have a product already and they’ve been owning it for a while and they feel a bit stuck in terms of like they’re not getting the sort of return of investment rate that they would like from their expectation. They see that they have a lot of neutral results and they’re wondering if they should run much longer experiments or what they should do about it. But yeah, I don’t know. Felt like partly cop out from your question there. 00:48:30.79 [Val Kroll]: No, no, it’s good. It’s, I mean, there’s no clear question. 00:48:34.94 [Tim Wilson]: Come on, Ben. I mean, it’s, he basically said that it’s like, it’s like intuition with experimentation combines. It’s kind of like you need to combine like the facts and the feelings. 00:48:45.65 [Val Kroll]: I knew exactly where you were going with that when he said. Come together. 00:48:50.46 [Tim Wilson]: Cheesy. So. 00:48:51.58 [Val Kroll]: Okay, so. Before I lose the thread because I last question, by the way, because we’re, we’re don’t do that to me. No, no, no, no. I’ve got like three more, but I’ll go fast. I’ll go, we’ll go fire around. Okay. So you’re talking about, um, uh, no one really likes the neutral results talking about some intuition with product. I’m going to talk about those outcomes. So obviously if there’s a win positive outcome, it ships. If it hurt the experience, it doesn’t ship. If there was an issue with the test set up, you hit an SRM or whatever, it doesn’t ship neutral. I want to talk about that. Are there scenarios where the product intuition says, even though this was neutral, it makes sense for where the roadmap is going or some decisions we’re making from branding, like maybe we’re This is building towards a bigger bet in the larger ecosystem to make things easier to share, more social. How do you think about the ship or no ship kind of action as it relates to those neutral results? 00:49:51.28 [Mårten Schultzberg]: It’s a great question. My general recommendation there is that as long as you’ve decided before you run the experiment that you’re going to ship if it’s neutral, I’m all good with it. I think that there’s a ton of situations where it makes sense to ship something if it didn’t change anything, especially if you’re building infrastructural type changes or if you’re building towards something. We’re building a lot of Spotify, building out AI features, as everyone else I suppose, but there’s a lot of changes that we’re making to our infrastructure just to be able to support features that we’re planning to build. And when we’re making those changes, the idea is that we’re hoping that nothing will change. Maybe we’re doing stuff to make things faster or something like that, but that’s a bonus if it changes anything at any point. So there’s a lot of changes that we are expecting won’t make any difference. So what we do then is that we essentially run what we call rollouts where we only have guardrail metrics, actually. So we say, as long as As long as we can prove that we didn’t harm these metrics, we’re going to ship it. So then by using the rollout, you’re sort of declaring your attempt from the beginning that like, hey, we’re planning to ship this as long as it’s not bad, which can sort of be a quite nice way to just make it explicit. That’s completely fine. But then again, I think that I just want to add a small caveat here that they also, I’ve heard a lot of product people at Spotify and other places talk about that like, And this, even maybe if a metric doesn’t look great or if it’s neutral and stuff like that, there is this, I think, almost human fallacy to say, like, this is strategically imported, let’s ship it anyway. And so I think it’s, even though that’s true, and I think that’s why it’s sort of an easy fallacy to fall into, or like it’s an easy trap. That can be true, but I think everyone should think about how large proportions of the things we ship should be shipped from the argument this is strategically important. Pretty small proportion is my general sense. 00:51:58.64 [Val Kroll]: Everyone gets three here. Something like that. 00:52:00.88 [Mårten Schultzberg]: I would love if I could give people a budget for those kinds of things. I think it’s all about trying to avoid the pitfalls of changing the objective when you see the results. We do that all the time at Spotify. We’re shipping a ton of things that are neutral. A lot of them are shipped with rollouts where we just explicitly say, we are planning to ship this thing for some reason. It might be business statistically or we have to improve our back end to scale for more traffic or whatever it might be. We’re going to ship it. So we just want to know that we’re not harming things. 00:52:40.72 [Val Kroll]: I like that. Okay, so Tim, I’m sorry. I have to sneak in. So what you’re talking about here is a very nuanced It feels like a nuanced analytical discussion. Should this be a rollout or how should this be exactly validated? How do you think about the education? Because you’re not talking about an audience of 400 people who are deeply steeped in the analytics or the rationale for why you’d make some of those choices. How do you think about the education piece to these different product teams? 00:53:14.68 [Mårten Schultzberg]: Yeah, I mean, it’s super important. So I’ve spent, I wouldn’t say majority, but a very big portion of my time at Spotify building educational material and mechanisms for this. I think that we have, I think, two strategies for this. I think the first one is to keep the the tool as simple as we possibly can, so have as few options as possible. So we’re talking about a lot of nuanced stuff here, but we also have removed a lot of stuff from our platform and simplified a lot of stuff and removed a lot of options, so made it quite opinionated. to minimize the things that people actually have to understand and know. So that’s one side. On the other side is that we have very explicitly and deliberately built educational material and tooling for experimentation for many years. So with confidence, we have this whole boot camp of self-serve courses. We’ve also given a bunch of courses. We have something called Quick Starts, which is a very basic tutorial for like, this is how you run an experiment, this is how you run a rollout. and those kinds of things. I know it’s a super important thing, but I think it has to come from two sides here. You have to try to make the thing that people should learn as simple as possible because people don’t have time. People have a lot of other things that they need to be good at and learn and understand, and then you have to create the material so that they can learn those things that they have to learn. That’s our solution to that. I mean, we have thought a lot about that. There’s a lot of things that everyone that joins Spotify is onboarded to experimentation immediately, and they go through certain what’s called golden paths at Spotify, which is like onboarding to certain things. And so if you’re a mobile developer, then you learn how to work with our feature flags in mobile, and you run an AA test as part of your mobile engineer. onboarding, for example. So there’s like, we have infiltrated the whole organization with experimentation onboarding and materials. And that has helped. 00:55:22.91 [Tim Wilson]: Wow. Wow. And Val, I’m going to have to put some duct tape. I was like… And we’re going to have to move to wrap. But I just have seven more. Val and the role of Moee Kiss on this episode. 00:55:35.21 [Val Kroll]: Yeah, right? 00:55:36.49 [Mårten Schultzberg]: No. I have zero stress at least, so don’t worry about me. 00:55:42.56 [Tim Wilson]: Well, this great discussion, I love sort of the thinking about what are we doing, why are we doing it, and how can tooling and education and culture and framing all sort of come together. So thanks for coming on for this discussion. But before we leave, the last thing we do on the show is go around the horn and we share a last call, something that might be of interest to our users. And Mårten, you’re our guest. Do you have a last call you’d like to share? 00:56:20.13 [Mårten Schultzberg]: Yes. So one thing that I’m completely, like I have been for actually for many years, but now renewed is the YouTube channel Three Blue One Brown. If I’m not the first one, that just makes me happy because it’s the best. The thing that I’m particularly thinking about now is the videos on Transformers and LLMs. This YouTube channel is essentially a channel that visualizes a bunch of math. That sounds maybe not fun, but it is so insanely good. They have a long series on linear algebra that I think if I would have actually seen it when I was taking linear algebra, it would have helped me a lot. But they also have a bunch of super, super nice things on LLMs and Transformers, which I think is like If you are, like most people, like hearing that word many times and you have like, yeah, it’s some kind of neural net. Maybe I haven’t used a neural net once or twice, but like you have no idea really how it works. Those videos are so very, very good. So I recommend them highly. 00:57:33.50 [Tim Wilson]: We have reached out to have, we had an exchange trying to get him to come on the show. I think it might have been around to talk about neural networks. He was in the process of like moving. So he’s on our list to try to get him on. That’s a- Good reminder. That’s a good one, good reminder to go back because they are, they’re like, I’ve sampled some of those and I’m like, this is so clear. And how does a human being have the time to produce something like this? 00:57:59.15 [Mårten Schultzberg]: Yeah, I mean, Grant Sanderson who has that channel. I mean, he seems to be like one of the true geniuses alive. Like, I mean, just a side note here is like he’s doing this super nice like animations of math and you just built that library himself, the library he built. It’s just… 00:58:19.19 [Tim Wilson]: Come on. We’re going to use this call out when this comes out to reach out to him again and say, hey, come chat with us. 00:58:27.48 [Mårten Schultzberg]: I would listen 100%. 00:58:29.52 [Tim Wilson]: Awesome. Val, what about you? Do you have a last call? 00:58:33.83 [Val Kroll]: I do. And it’s actually related to today’s episode. So this is a medium article published on Unix Collective, article written by James Skinner. It’s called Escaping the AI Sludge Why MVPs Should Be Delightful. And there’s a lot in here, one of the cases he makes is that like using AI is just like regurgitating like we’re not going to get to that delight level if we’re just, you know, using AI to help, you know, develop those different. net new versions that are being tested within a product context. But he talks about the MLPs. I’m obsessed with MVP’s, Mårten, I should tell you, just understanding different people’s perspective. But the MLP is the minimum levelable product. And he also referenced one, the minimum viable whatever, because there’s so many acronyms related to this, with people trying to figure out exactly what that level of fidelity should be, what type of investment you should make before you experiment. He does talk about experimentation at the end, which I do love, but there’s a lot of really good examples. And I love reading from that design product perspective. So, but it’s a, it’s a good read, about 10 minute read. So it’s a good one. And Tim, how about you? Do you have a last call for today? 00:59:47.36 [Tim Wilson]: I’ve got a smidge of housekeeping and a last call. So we are now like into month number two of 2026, which means we’re heading into a conference season. Actually, I am, sitting in Budapest, Hungary as you were listening to this, if you’re listening to it when it came out. A couple of analytics power hour conference attendee appearances coming up in Nashville. If you’re in the States, there’s the Datatune conference that Val and I will both be attending on March 6th and 7th. Some critical mass of the Analytics Power Hour crew, we will be recording a show with a live audience at the Marketing Analytics Summit in Santa Barbara, California on April 28th and 29th. Those are PSAs more than last calls. My last call would be friend of the show, past guest, Katie Bauer, the wrong but useful sub-stack wrote a post called The Next Data Bottle Neck, which I thought it was a unique and really thought-provoking take on the whole drive towards conversational analytics and not the will it or won’t it or the technical challenges of it, but when looking at what people are asking for and why they actually seem to be mundane requests that they seem to be kind of just simple data fetching requests, not these super nuanced things. So she has a lot of musings that can be a little unsettling for the analyst, but then she actually kind of wraps by making the case that really it goes back to good analysts really thinking about the business deeply. So it’s a worthwhile read. So I was a threefer, but I’ve labeled two of them as being a housekeeping writer in the last class. 01:01:45.78 [Val Kroll]: Can I ask one more question then? 01:01:48.19 [Mårten Schultzberg]: That’s how you get airtime in this show, right? 01:01:51.33 [Tim Wilson]: I’m drunk on power. Is Michael as drunk on Tamiflu? Tamiflu? Tamiflu? Tamiflu? I don’t know what the flu medications are. Yeah. By the time this comes out, he will be back to good health and he will vow to never get sick again and cede the mic to me. So this was great. Thanks again, Mårten, for coming on. This was a really fun discussion. 01:02:17.28 [Mårten Schultzberg]: My pleasure. It’s really nice. Thank you so much for having me. 01:02:21.69 [Tim Wilson]: Awesome. Everybody get your Spotify subscription up to speed. This is what’s driving Spotify’s next round of growth is the confidence podcast appearance. 01:02:33.83 [Mårten Schultzberg]: Quarterly call coming, so like, please. 01:02:35.93 [Val Kroll]: There you go. 01:02:38.11 [Tim Wilson]: Perfect. If you are listening and you’ve enjoyed this show or other shows, we would always love a rating and review. I’ll do a little call on audible and read out this one from Apple Podcast that just T5272018 left. It was titled Smart and Funny. And it was love the insights and laughs I get from this podcast. You all have a high bar for analysts and the value they can add, which I so appreciate. And you share all of that perspective via hilarious and authentic banter. Keep it up. Wait, let me check. That is our podcast. Yeah, that is this one. So that was kind of nice. We’ll always love to get ratings and reviews. Theoretically, that is how we expand the reach of the show, that and recording video and putting them on YouTube. So we’ll just double down on the ratings and reviews. If you’re a fan of the show and would like to have a sticker for your laptop or water bottle or whatever, you can go to analyticshour.io and request a sticker. We’ll ship one over. If you have something to say, a thought for a topic, criticism, your own little witticism that you’d like to share, you can reach out to any of us or the show as a whole on LinkedIn. You can catch us on the measure slack or you can just send an email to contact at analyticshour.io. So, with that, for Val and for Michael in absentia from his sickbed, I’m Tim Wilson and no matter what your reason, whether you’re identifying or you’re optimizing or you’re being just aggressively neutral in your findings, you should always keep analyzing. 01:04:24.79 [Announcer]: Thanks for listening. Let’s keep the conversation going with your comments, suggestions, and questions on Twitter at @analyticshour on the web at analyticshour.io, our LinkedIn group, and the Measure Chat Slack group. Music for the podcast by Josh Crowhurst. Those smart guys wanted to fit in, so they made up a term called analytics. Analytics don’t work. 01:04:49.43 [Charles Barkley]: Do the analytics say go for it, no matter who’s going for it? So if you and I were on the field, the analytics say go for it. It’s the stupidest, laziest, lamest thing I’ve ever heard for reasoning in competition. 01:05:03.47 [Tim Wilson]: Yeah, we’ve sent, Australia is the one that’s the real Australia. 01:05:08.27 [Val Kroll]: Singapore. 01:05:08.99 [Tim Wilson]: We’ll take weeks. Singapore, one made it all the way to Singapore, came back to Ohio. Never came to me, turned around and went back to Singapore. So it was like eight weeks. 01:05:22.69 [Val Kroll]: The box was like smashed. The gift wasn’t ruined, but the box was in shambles. 01:05:29.07 [Tim Wilson]: There is now more packing material. I did change after seeing that. It’s a process update. 01:05:35.73 [Mårten Schultzberg]: I guess I should save all of my comments about it for the actual recording. 01:05:40.98 [Val Kroll]: Yeah, we’ll get into it for sure. I’m very excited. 01:05:43.92 [Mårten Schultzberg]: It wasn’t that terrible. The distortion wasn’t that terrible. 01:05:47.21 [Val Kroll]: So every time you do that while we actually record, because you’ll definitely be doing that multiple times, I’m just kidding. 01:05:52.61 [Mårten Schultzberg]: Yeah. 01:05:54.07 [Val Kroll]: Yeah, it looks like. Last for me. 01:05:55.93 [Mårten Schultzberg]: Part of your signal yelling at you. 01:05:58.98 [Val Kroll]: Your guests. 01:06:02.77 [Tim Wilson]: All right, let’s try it again. 01:06:12.37 [Val Kroll]: Rock flag and focus on those learnings. The post #290: Always Be Learning appeared first on The Analytics Power Hour: Data and Analytics Podcast.

That darn data. It’s so complicated and fragmented and gap-filled and noisy that no amount of time is ever enough to truly get to the bottom of all of its complexity. As a result, it’s pretty easy to fill all of our time handling as much of that underlying data messiness as possible. At what cost, though? It’s easy for the analyst’s connection to the business to suffer as they get mired (too) deeply in the data and lose sight of the broader business needs. In this episode, the gang had a chat about business acumen—what it is, how to develop it, and why it’s a must-have for any data or analytics role. This episode’s Measurement Bite from show sponsor Recast is a brief explanation of identifiability—what it is and how to check for it using simulation—from Michael Kaminsky! This episode is also brought to you by Prism from Ask-Y—your agentic analytics platform for automating analytics, exploring data, creating repeatable workflows, and delivering accurate insights—all without the need for manual query writing. Links to Resources Mentioned in the Show (Book) Psychology of Intelligence Analysis by Richards J. Heuer, Jr. Harvard Business School Online Credential Of Readiness (CORe) program (Book) The Innovator’s Dilemma: The Revolutionary Book That Will Change the Way You Do Business by Clayton M. Christensen (Book) Humankind: A Hopeful History by Rutger Bregman (Book) Lord of the Flies by William Golding (Podcast Episode) The Real-Life Lord of the Flies on the Everything Everywhere Daily podcast (this is not the episode Tim referenced on the show, but he found it while trying to track that down) (Podcast Episode) Chaos on the Throughline podcast (possibly/probably the episode Tim was remembering…and it also references Humankind: A Hopeful History!) (Video) In the Long Run, Everything is a Fad: Benn Stancil (Small Data SF 2025) (Podcast) Knowledge Distillation Episode #4: Michael Helbling on 20 Years in the Trenches, Why Trust Is the Real Bottleneck, and The Difference Between Data Retrieval and Actual Analysis (Conference) DataTune Nashville – March 6-7, 2026, in Nashville, TN (Conference) Marketing Analytics Summit – April 28-29, 2026, in Santa Barbara, CA Commoncog Photo by Adeolu Eletu on Unsplash Episode Transcript00:00:05.76 [Announcer]: Welcome to the Analytics Power Hour. Analytics topics covered conversationally and sometimes with explicit language. 00:00:14.56 [Michael Helbling]: Hi everyone, welcome. It’s the Analytics Power Hour and this is episode 289. There’s a comfortable trap that a lot of us in this industry could fall into. We spend hours debating the merits of different data tools, what the best visualization is for a data set, or whether our tracking pixels are firing just right. Data is our craft, and we love it. But there is a harsh reality, and it usually hits about three to five years into a career. You can build the most rigorous model or beautiful dashboard, but if you can’t explain how it actually helps the company drive revenue or reduce churn, it kind of doesn’t matter. We like to complain about how our business stakeholders lack data fluency. But maybe we need to flip the mirror and ask ourselves the hard question. Do we lack business literacy? So that’s what we’re going to talk about. Business acumen. Is it the missing link that turns a data analyst into an actual strategic partner? We’ll talk about it. And to do that, let me introduce my co-hosts, Val Crawl. Hi, Michael. Hey. And of course, Tim Wilson. Hello. Hello and welcome. And Moee Kisss. 00:01:27.32 [Moe Kiss]: Howdy, team. 00:01:28.68 [Michael Helbling]: Hey, and I’m Michael Helbling. All right. I think maybe to kick things off, we should start with like a brief intro to maybe a definition about what we mean when we say business acumen in the first place. Anybody want to take a first stab at it? 00:01:45.65 [Tim Wilson]: I think as we were thinking about this, the little light bulb that went on for me is there’s two types of business acumen and one of them being knowledge of business itself. This is how business, this is how finance works, this is balance sheet income statement, cash flows, marketing. four P’s and more whatever aspect of the business, which is kind of, this is something that you build over time and take throughout and are sort of truisms or practices in business. And then there’s another type of acumen, which is like knowledge of your business, the company you’re working for or with and understanding where they are unique, they all are doing, they overlap and that your business is operating in the context of the broader business, but there is what are the specific external and internal challenges that your business is facing. So to me, that’s, I don’t know if that’s a definition, but that’s kind of two flavors of it. 00:02:54.38 [Moe Kiss]: But if you had to pick, if you had to stack rank, Do you think one is more important than the other? 00:02:59.80 [Tim Wilson]: I feel like they leapfrog as they go along, because you can’t be super incredibly deep in one without being deep in the other. If you were trying to say, I’m just going to know everything about the organization I work for, at some point you’re going to run into the finance team who’s going to be talking about revenue recognition. 00:03:26.36 [Michael Helbling]: Yeah, because I think on the first level, the first one is concepts. So how does a P&L work? And then the second one is context. How does it work in our company? And maybe that would frame it like that. And I don’t know if I could pick, Moee, to your question, which of those is more important. I feel like you need both of those wherever you’re going to be if you’re going to really, truly own business acumen in the place where you are, where you work. 00:03:53.84 [Moe Kiss]: It’s funny, though. One of the things that my mind goes to is, is there also like a third category, which is like knowing your type of business, like your industry, right? So like you can have a really deep experience in like e-commerce or FMCG or, I don’t know, insurance or government or something. And that’s like a different type of knowledge and experience that you can apply as well. 00:04:17.17 [Michael Helbling]: Oh yeah. I think that’s true. I still boil that into the context, you know, awareness of how your business or your vertical works. 00:04:25.82 [Tim Wilson]: But I think that how you’re running is a good, the FMCG or CPG is a good example when you, the number of FMCG brands I’ve worked with who’ve said, we want to be like and they insert B2C retailer. And it’s like, well, yeah. And they’re like, yeah, all we need to do is get direct information about our individual buyers. I’m like, you’re selling soap. That’s where on earth would you have that permission? I mean, maybe that goes back to the concepts of saying, well, you’d understand what the limitations are in the context of this vertical and what you have to do instead. But as you’re talking, I just realized this week had a case where two completely different verticals, but both of them had a franchise model was kind of the way they were working, completely different spaces. And I was kind of like pleasantly surprised to realize that there were some similarities into how that sort of corporate franchisee relationship worked and was managed that was shockingly parallel. And I haven’t worked with that many franchises, but I was like, oh, wait a minute, does every brand that operates on a franchise model or do most of them have this sort of setup. And that went from learning it for one and then kind of stumbling across it for another. But applying those patterns, which I think, Michael, you’re having those concepts and saying, okay, how does that concept apply in this specific context is a, I love that phrasing. 00:06:08.26 [Moe Kiss]: It’s funny though, the one thought I do have about knowledge of your business. Well, I have multiple thoughts. One is, I suppose the first being, it can really unlock a lot in terms of, let’s say hypothetically, you’re talking about maturity or data usage and how great it is in the company. You ask folks to give you a score of zero to 10, right? Zero totally should house and 10 being amazing industry leading. most data-driven, I’ve got air quotes for those listening along, company in the world. And it’s like, you might have knowledge that actually your company can only get to an eight, like a 10 is just not possible at your company for various factors. But it’s interesting because at the same time I say that, I also think it can danger the work that you do and how you provide it sometimes by absolutely biasing your approach. Like you think of Richard Harris’ book, The Psychology of Intelligence Analysis, and for that reason they tell analysts to move around because if you know something too well, you can also make mistakes. 00:07:12.87 [Michael Helbling]: I think, Moe, that’s an incredible point because it’s very natural for people to get locked in on whatever function they’re in. As analysts, it’s an amazing experience to see different parts of the business and build context around those and see how they work together and build out knowledge across. Finance and the way they look and analyze data is different than how marketing does versus how merchandising does versus how sales does. All these different functions within the business look at it different ways. You can become a much more fully featured business analyst by taking time in each of those. You can sometimes get a little bit too I don’t want to say stuck, but there’s a bias that can influence how you even approach or think about what’s possible in terms of insight or action or recommendation that then leaves your analysis not as fresh or aggressive enough. I don’t know the right way to say that good, well, but that’s what I think you’re trying to say too. 00:08:22.77 [Val Kroll]: I think that’s a good connection point between some of the concepts we’re talking about and what that means to the analyst connecting it to their work. I think what you were just talking about, Michael, with the recommendations and the actions, when I think of business acumen and analysts and like building the skill, I think one of the first things is just understanding how the business makes decisions. Therefore, you can come up with the best ways to think about framing or recommendations or proposing actions for the business because it’s like, what is your relationship, you know, but everyone, you know, well-tread area between sales and marketing, but like, what is your relationship with other decision-making arms of the business and how is it supporting that, whether you’re in-house or consulting, yeah. 00:09:07.59 [Moe Kiss]: It’s funny. I actually had someone ask that in an interview and I to this day always reflect on it being such a great question. The question was how does the business make decisions? Like who are the key decision makers and like talk me through the process of like how they get signed, you know, folks get signed off for something. And I was like, it actually is such a wonderful question because it tells you so much about the culture and the ways of working. Like it really is an unlock for someone who is trying to gain that knowledge of the business quickly. 00:09:38.21 [Tim Wilson]: I think it’s there cases where I think that’s a love that point and it also is the sort of thing we’re talking about like to be a better analyst, to be thinking kind of not necessarily just who you’re direct partner that you’re supporting and I think they can wind up in a group in a, not a group thing, but kind of caught in a way of this is somebody who may be a mid-level paid media person and they have been poisoned by their media agency as to what metrics matter and they’re not necessarily don’t. thinking through the broader business, how decisions are getting made, what sales is expecting, what is coming back. There can sometimes be a challenge or an opportunity for the analysts to say, I can’t just trust exactly who I’m supporting. I mean, you want to have a positive relationship, assume good intent, but there are definitely plenty of people in business who are operating with blinders on and often the analysts are the ones, if we’re trying to connect the dots, you need to have more than one dot. So that’s kind of a weird, as you’re talking, it’s making me think about sometimes the analysts needing or having value and having a broader perspective in order to do the analysis that may be providing to somebody who has a narrower perspective to help broaden their perspective. 00:11:04.29 [Val Kroll]: No, I like that. And I think even thinking, I completely agree, because even if you think about the group that you’re supporting, it’s always helpful to think about who else might have a completely inverse motivation at that table. even if you think about resources and thinking about budget allocation or time allocation of resources or, you know, marketing wants as many leads as possible, but sales only wants to qualify leads. So there’s like this tension that I think you can tap into between these different groups to understand, you know, there was a group we were working with where we were working the leaders high up enough that we had one group in the retail group that was focused on the sell in. to the marketplace partners, and then there was a group sitting across the table from them that was only focused on sell-through. And so we had to think about how do these two concepts work together and what is the thread between them that kind of aligns to, again, how the business is making decisions. But I think that those, the motivations, even if it’s not a stated goal, helps you understand the friction and who might be not thinking the same way as the client and the group you’re supporting. 00:12:14.81 [Tim Wilson]: Okay, time for a quick break for a word from our sponsor, Ask Why. You know, we’re allergic to AI hype on this show, so when someone says, AI analyst, my first instinct is, prove it. 00:12:27.57 [Val Kroll]: It’s fair, and that’s why Ask Why caught our attention. It’s not about vibes or magic. It’s about actually automating analyst work, things like generating SQL and cleaning data and exploring data sets from plain English questions. 00:12:41.41 [Tim Wilson]: And importantly, because this matters, Asquai does not send your raw data to an LOM. They build a semantic layer on top of your data, the model writes the code, and that code runs where your data already lives. 00:12:55.29 [Val Kroll]: What’s new and genuinely interesting is that Asquai now has skills and memory. Skills let you save and reuse analytic steps you’ve already taken instead of redoing the same work over and over. 00:13:06.94 [Tim Wilson]: And the memory piece means it can retain context, business logic, assumptions, definitions, so you’re not re-explaining the same setup every single time you ask a question. 00:13:17.58 [Val Kroll]: So the promise here isn’t replacing analysts, it’s reducing the repetitive friction that slows analysts down. 00:13:24.64 [Tim Wilson]: AskY is in beta right now. Go to asky.ai, that’s ask-y.ai, and use code APH to jump to the top of their wait list. This is in Vibe Analytics. It’s tooling for the rise of the AI analyst. Now, back to the show. Can you guys think of times where you all of a sudden realized that you hadn’t fully processed how something worked in the business? And then it sort of made you say, oh, I’m seeing things with the entirely new light because somebody explained that to me. Usually mid presentation of some analysis I thought was really sharp. You’re like, you know what? This is awesome. I just got. Yeah. 00:14:11.98 [Moe Kiss]: I honestly think starting at Canva and getting an understanding of subscription revenue. Like I had worked in Ecom before. Someone bought a pair of socks. The second that item gets delivered, like we have money in the bank. It’s very different when you start thinking about annualized recurring revenue and like the fact that people are trying to extrapolate out what does this mean over a 12 month period versus like this is what has actually hit the bank. I think that was one Yeah, it takes some grappling with and then when you’ve got other revenue streams like print or something that then do hit the bank when the order gets fulfilled, so to speak. I think that’s one that even to this day, sometimes I find stakeholders don’t fully understand. I find it’s really important to be very specific about if we mean annualized revenue or annualized recurring revenue, which is a subset. There’s a lot of nuance in that and it does affect things like when you’re doing marketing, you know, you can’t just like assume, you can’t wait for every time the money lands, right? Like you have to project forward. And so like it creates a lot of complications that I probably never fully appreciated until I worked here. 00:15:24.21 [Tim Wilson]: I mean, this was like kind of blew my mind so bad. And it was actually one that the team, the business partners were making kind of a They were behaving in a way that was illustrating how badly they were looking at the data because they weren’t thinking through how the business worked. This was a company that was kind of of for and by the engineers, sold to engineers, built products for engineers, everyone in marketing, everyone was an engineer. They had engineered this solution. The R&D and engineering teams were combined, and the VP of that team had said, we’re going to start doing a monthly ROI report. And I barely knew ROI. I wasn’t even officially in an analytics role, but I was helping produce this monthly engineering ROI report. And this entire, which was a printed report that was pretty thick, that was kind of a nightmare to produce, and they went, product line by product line, and they took the total dollars invested in that month. What was paid to headcount? What were the fully loaded costs for that department? They took the revenue from that department for that month, and they did the math and said, here’s the ROI. They had been working on it, and it turned out that people were saying, The people who own those product lines were saying, well, they always had an explanation saying, well, we’re doing the R&D now. Then the thing has to go to market and then be produced. And then we’re not going to see revenue from what we’re working on now for months or years. So everything either looked fantastic or terrible, or it was a long time legacy product line that looked totally stable. And that was before I was in analytics and it was Actually, this was while I was getting my MBA and we were talking about revenue recognition is partly what triggered it. And I was like, Oh, Yeah, wait a minute. This is like the whole company is kind of whiffing, well-intentioned, but aren’t thinking through kind of the nuances of how it works. Smart enough, great products, they would kind of do the math after the fact and say, well, this product line looks terrible, but that’s because of X, Y, and Z. It’s like, but you’re looking at data that looks terrible. Shouldn’t you be looking at it in a way where it makes sense instead of distributing it? 00:17:50.88 [Moe Kiss]: So how did you solve it? Like, I guess the thing is when I hear that, I’m like, what’s been built isn’t actually trying to answer the question that was intended. So like, you gotta help me out. How’d you solve it? 00:18:06.20 [Tim Wilson]: So I was a little… Tell the end of that story. Who was like pulling the stuff. I honestly don’t… I think it kind of petered out and they’re like, this report actually doesn’t make much sense. And we need to be thinking through I think they shifted to more of a planning, like an annual plan. This is what we’re going to invest in this product line. And this is what our expectations are. And this is kind of when, so they kind of shifted, I think, to more of a forecasting model so they could forecast the costs and they could forecast the revenue and track against those instead of trying to put them together. But I was a kind of pretty low level. And I was like, this is, and I went to my manager, who was more senior, but he was kind of one of these people who’d been kind of marginalized, like pushed to the side. He’d been around the company forever. And I was like, does this make no sense whatsoever? And he was like, well, Yeah, I think it doesn’t, but we kind of know it doesn’t. It wasn’t like, oh, I’d brought some grand insight. It was kind of like, yeah, we’ve slowly been figuring out that this actually isn’t all that helpful. But there were also plenty of people who were flipping through it and saying, our product line is doing great. It’s like, well, not, I mean, incidentally. 00:19:20.93 [Val Kroll]: It’s a good story when it’s a good story until they’re on the other side of that. 00:19:25.70 [Michael Helbling]: Well, and it happens all the time. Like you mentioned a little earlier too, Tim, like some people grab onto metrics they want to optimize just because it fits within their constraints, but don’t really think further into the business of like, okay, if I optimize for this metric, what are the downstream impacts of that thing? So like, what if I go optimize for customer acquisition in a channel that isn’t really a good fit for a product and returns go way up? Well, now we’re costing the business tons of money over here. that maybe me sitting in my customer acquisition spot never sees or thinks about. Those kinds of things happen all the time. You can optimize for the wrong metric or don’t think through the flow of the money through the system to encapsulate what you’re looking for. That’s why it’s important. 00:20:16.31 [Moe Kiss]: So my mind always goes to the like annoying practical shit, but how do you develop this business acumen? Like I know I probably have ways in my mind, but I think sometimes the frustrating bit is people, like in some cases it does take time, right? Like part of knowing a business really well is the experience of working there. But there are other things that have worked for you all to develop this quickly, especially when you’re not sort of in-house and you need to develop that business acumen fast. 00:20:43.07 [Tim Wilson]: You’ve hired many people, presumably many not coming from a subscription services background. How do you ramp them up outside of the mechanics of the data? What do you have them do? 00:20:58.43 [Moe Kiss]: Why don’t you just turn it back on me? 00:21:03.04 [Michael Helbling]: Well, you had a really good example earlier. 00:21:05.43 [Moe Kiss]: What I do tend to find is part of our onboarding is normally getting to know one of the data spaces really well. So it will be something where you basically have to really dig into the data warehouse to, for example, understand how subscriptions are calculated. How is that revenue calculated? And so often, or if you’re going into the people analytics team, it might be something to do with a particular metric that needs to be calculated for that space. And by really getting your hands dirty on the subject matter, we intentionally do that as part of onboarding. But I have a very different working style than I would say to lots of data people. And the thing for me is talking to people. My advice to every new starter is spend your first month having coffees. I don’t care if you have five coffees a day. I mean, you might want to swap to herbal tea at some point, but like, have as many cat jobs with people as you need to and have a list of questions because for me, I mean, that’s how I absorb information and learn, which is really irritating for other people because it involves talking things through. That’s been my approach, but I know that’s not the case for everyone, right? 00:22:13.56 [Tim Wilson]: I mean, I feel like that’s a pretty common onboarding is giving these are the people you should meet. And I think sometimes that that’s not given because they can’t just go set up coffees because they don’t know who they’re supposed to have coffees with. So it’s on the manager or the team or whoever’s onboarding to say, these are the people and here’s why I think it falls to the analyst. I’m trying to think when I’ve started at places, in-house and I’m like, I guess I’m just going to try to meet them and I’ll come up. I didn’t frame it is I need to deeply understand how they think about the business. And it’s a great opportunity to say, I want to know how Joe thinks about the business who’s in this role. And now I want to go talk to Ann and I want to see how Ann thinks about the business. She’s in a different role. Does it match? Does it jive? By the time you get to the third or fourth one, you’re like, okay, everybody knows that so-and-so is the big dog competitor and we’re never going to beat them on price. That’s consistent. I seem to be hearing inconsistent things here, which means in the business, there’s not agreement as to the value of email marketing or whatever it is. But I think having that framing of starting a new position to say, I’m just trying to figure out, what does everyone agree is the case? Because it probably is. I mean, sure, maybe there’s some every misguided assumption that everybody’s bought into. But framing in that way of like, I want to be able to go and understand how each one of them think about the business means I’m going to learn about our business. 00:23:54.23 [Michael Helbling]: It’s like a flow discovery type of thing because you have to figure out how the money flows to the org, but you also have to figure out how people’s decisions or their work flows. through the org as well. So like to the earlier point about like how do decisions get made? Okay, yeah, what part of the PNL do you care about? What part are you motivated the most by? Because if you can understand motivations, I honestly find that like, Digging into this topic actually helps me increase my empathy for business users quite a bit because I start to understand their motivations for, okay, what are they trying to do? And then I feel like I can find ways to help them that they aren’t even able to really enunciate back to me in the first place sometimes because they don’t really know what I do. But then I can go back and say, okay, here’s three ways we can get you data that actually helps the thing I hear that you’re trying to solve for. And, you know, kind of helps like build some really nice bridges because that’s I’ve always think about it like, okay, well, what decisions are you trying to make? Or kind of like, how are you motivated, right? Like, you know, in a craft sense, like, what are you bonused on? You know, if you hit these targets, is that going to make you happy or make the business happy? And then you do find Tim, to your point, all these disconnects when you start doing that process. And you’re like, what is going on in this organization? This person is motivated and bonus this way. This person is bonus this way. Like, they’re at odds with each other. Who built this alignment? Yeah. 00:25:34.23 [Val Kroll]: And to the other part of your question, Moee, and how do you do that, especially if you’re not in-house? And I would say that there’s pros and cons, advantages, disadvantages, but because you usually start a consulting engagement like that with discovery, like Michael was just kind of talking about, you have permission to take audience with all those people and ask those questions directly. And I do find that it’s like this game of guess who, understanding like asking questions like, Oh, and where does this person sit inside the organization? And after they answer that, it’s like, okay, I don’t need to ask those four. I’m going to put those down. So they’re more in an operations function within marketing. Understood. Got it. Okay. So, and then you’re like kind of building this like understanding as you get to be like direct fire versus like trying to keep a casual over a coffee. Sometimes it’s, it’s almost beneficial to be able to say, this is the intent and purpose of this meeting. I’m going to pepper you with. 20 questions and then similar to how you would at the end of a coffee date if you’re starting to say, who else should I speak with? You have permission for that at the end of stakeholder interviews or that kind of thing. Who else should we be chatting with to fully understand what we’re trying to solve here or what the opportunity is to give it its fullest shape? 00:26:50.81 [Michael Helbling]: Yeah. Do you guys read 10Ks and company decks and stuff like that? 00:26:59.74 [Tim Wilson]: Oh, yeah. Yeah. As a consultant, definitely. And when I was in-house, I would at least would hop on the quarterly conference call when it was public and always… I never did that until I was a consultant. 00:27:13.07 [Val Kroll]: I should have, but I never thought to do it. 00:27:15.09 [Moe Kiss]: One thing that’s on my mind, though, is… As data folks, it’s always like, ask more contacts, keep asking questions, be curious. I do feel like there’s another end to the spectrum though. We had this recently with a partner. It almost got to the point where they kept asking for context and kept asking and kept asking. And it gets to a point where you’re like, you have a lot of information about our business. You know us really well. We’ve worked together. You need to stop asking for context and start coming to the table with some ideas. And I felt that was a pretty fair place to be. And we had a really great conversation. very different culture at their companies. So they were quite cautious. But I was like, it does. It gets to a point where you’re like, I understand that you’re trying to collect this information so you can present something good. But by waiting and waiting and waiting, there are ways you could check in on your thinking earlier that might get us to the outcome. Whereas if you’re always in that collection mode, sometimes I think it can also be problematic. I’m curious to hear perspectives. 00:28:30.54 [Tim Wilson]: I mean, if an analyst, consultant, in-house, whatever shows up and hasn’t even made an attempt to sort of figure it out, I think it’s always much more useful to say, with whatever knowledge I have of whatever this space and drop me in somewhere completely foreign to me, I can still say, well, I would assume that it works like X. And I think it’s, I’ve found it more useful to say, let me put forth where, how I assume it works, but let me also illustrate that I don’t know for sure. This is a logical way to think it, and that gives them the, okay, you’re trying to think, you came prepared, and that’s not sitting in a vacuum, like go and try to figure it out. Who are your competitors? I think that is actually where chatGPT can be a huge, Just going to say that. 00:29:20.16 [Val Kroll]: Yeah. 00:29:20.98 [Tim Wilson]: Go and spend some time with that, whatever you’re doing, and then say, okay, is this how this works? And then they’re reacting and saying, oh, that’s like 70% correct. But what you’re missing is this one other piece. But yeah, it would be pretty annoying. And people are like, what’s our list of stakeholder interview questions? And it’s like, well, here’s your stakeholder interview questions. But here’s the research and prep you damn well better do before you show up in that room because You are taking their time. You want to find out what they can tell you. 00:29:53.48 [Michael Helbling]: And to be honest, Moe, I think I probably struggle with the opposite problem of what you’re describing, which is I tend to have ideas of what I think the solution is before I’ve gone in depth enough to really understand the context sometimes. And so I find I have to hold myself back. from being like, oh, I think I see it, here’s the idea, here’s the solution. Instead be like, nope, gather more information, gather more information so that you don’t miss important nuance. But you’re absolutely right. There’s diminishing returns to trying to like measure four times and cut once versus iterate through like kind of what you talked about, Tim, like create an iteration, expose your assumptions and allow there to be a flow of information back and forth on top of something. And in the old saying, it’s always easier to edit than it is to create, right? So if you get something set up, then people can react to it and give you so much more valuable information as opposed to like them sort of wondering like, what the heck is this person doing in here not talking about anything? You know, so yeah. 00:30:57.46 [Val Kroll]: Yeah, the one thing that I’ll say though, if you were trying to think through, what is this striking this right balance? If you’re in an analysis validating some assumptions, some hypotheses, do you feel like the recommendations you can make to the audience you plan on delivering this to feels like something that is a lever they could pull, something that they potentially actually have control over because there’s been so many times where we’ll see examples of work where it’s like making recommendations that are so off from like what the person could actually do that it makes it feel. It almost invalidates the first 10 slides because it’s like, do you even know But what that team does or how long ago they had to make that decision, that is so unhelpful. 00:31:46.37 [Tim Wilson]: We had a business context case where we wouldn’t have known we were talking to the CEO and we said, Would there be an appetite for geolift tests? You’re not doing it. That would address this issue. And she said, absolutely not. And here’s why. And it was a damn good read. I was like, cool. And she wasn’t upset. She was like, yeah, I would love to do that. And here’s why we can’t, based on the nature of our business and we’re like, boom, ding, ding, ding, like better understanding of kind of what parameters we were operating in. So I don’t know, Michael, I feel like even if you bring and say, I’m bringing a potential solution, my assumption is this is probably not workable, but If it gives you something to react to, to tell me why it wouldn’t be workable, that’s going to help both of us. And they very well may be saying, well, that’s 30% workable and I love that. And I never thought of that. This other 70% won’t work. So. But I think Moee you’re hitting on like the balance of bringing stuff and getting stuff back. Like it does need to be a not just going and expecting to be it all being kind of a pull. It needs to be kind of a back and forth. 00:33:03.46 [Moe Kiss]: How do you bring it back though? I’ve had so many times people have walked in and presented stuff and then like, here, this is the insight and that thing you should do next. And I’m like, cool, we knew that. That’s not helpful. Or like the example you just gave Tim, that’s not feasible here for reason X. When you’re in that situation and you realize that you’ve done this, How do you course correct? How do you repair that relationship and I suppose your credibility to some degree? 00:33:34.50 [Michael Helbling]: You’re technically correct, but business wrong. 00:33:39.54 [Val Kroll]: Well, the example that Tim gave, that was in a discovery session that that happened. So that was, to Michael’s point, exploring solutions, but making sure that we weren’t going to get to the end of some piece of work, making a recommendation for a geolift test. And they’re like, get the hell out. Press the buzzer and your seat ejects you. But I’m thinking in a consulting context, if you make a recommendation that is so off base and you hit that button live, I think part of it would be unwinding some of your thinking as quickly as you can to show why you got to that conclusion, because there might be just a slightly different path forward that still relies on some of those same assumptions or some of the dots we were able to connect because, again, you went too far, but I also think that sharing some of that context and then maybe coming back with a couple questions to understand what might be a right fit. But I think it’s turning it into a discussion as quickly as possible is where my head first goes to it, because that’s the only way that you’re going to dig yourself out and be able to make a sharper recommendation next time. I don’t know. 00:34:50.70 [Tim Wilson]: Some of that goes to the stakeholder management. If that’s happening at a super high stakes, there were probably some relationship and there were some planning stuff that you whiffed on. If you’re going to go and present it, who did you vet it with beforehand in a lower stakes to make sure that the assistant or the team member say, hey, I’m going to present this? This is kind of a big deal. We’re excited to say, well, we’re going to present this amazing thing, but it may completely backfire. Better to figure out, okay, who are the people looking around whether it’s in your group or another group that you have the relationship that they seem to have their finger on the pulse of the business? that they would be a good sanity check. I think just knowing who those people are, that same company years ago, when I managed the BI team, had a lady who had been an analyst supporting sales forever. And she knew everything about how they worked. She knew the data inside and out. She knew what they cared about. She knew all the personalities. And she was kind of gold for the team. If anything was going to get presented to sales. It was like, you better run that past Shelly because she’s going to make it better and she’s going to make sure like it needs to be Shelly approved before you go put it to the business. And that was because she was just a super seasoned approachable analyst. in the team. If it went through her, it was going to be good, but it can also be somebody on who you’re presenting to. If I’m going to present to the head of sales, maybe whoever my buddy who’s in sales management, I should run it by them first to make sure that I’m not about to have the eject button hit on me when I take it to the higher stakes. 00:36:41.20 [Michael Helbling]: We’ve lost so much from in-person meetings because one of my big signals is like when the most senior person picks up their phone, you course correct instantly. Like you’re like, okay, lost you. Now I need to get you back. But even sometimes people will be like, you present all your ideas and then you don’t hear anything. And that’s almost like even worse. Like they don’t, they just write you off as, okay, that guy’s an idiot. and you don’t get any feedback, that’s brutal. And then basically you’re just trying to scrape all your way back in to those conversations after that. And it’s really, it’s just a rough thing because trust is so hard to build and being influential in a business is so hard to build. And so that’s why like kind of all your points, Tim, are super important. Like do the prep, actually run through it with somebody who can give you great feedback on it, collaborate with somebody else on like, okay, this is the analysis I’m thinking about. This is the direction I think it’s going. You know, does this ring true? What do you think this will look like in the room? I remember back in the day when I was a business analyst, when we had an important analysis to present, we would do pre-reads with all of the like most of the people who would be in the meeting just to make sure there was no big alignment issue with what we are going to present to the bigger team. Now, you can’t do that with every single analysis. There’s not time to do that. But if it’s like an important one that’s really going to drive a big decision, like, yeah, grab that director, grab that director, grab that director, make sure they don’t come in with, oh, you’re missing in a very important piece of context. that’s going to derail this whole thing before you even sit down and present this to the broader or leadership team or whoever, and that can really help you. And you just have to think through who might have impact or who might have something to say about it. 00:38:30.77 [Tim Wilson]: If having that sort of forum, and this again, I’m going back a ways, but having managing a BI team, which was where the business analysts lived, and we would do in our I think weekly, every other week, staff meeting and we would do the whole like have somebody present the analysis they were working on or had done and it would have the senior people because there was a good way for the for kind of cross-training in the junior analysts. So as you’re describing that, Michael, there’s like the, I’m Michael presenting this analysis. These are the three experts who I want to make sure I run it by. Who are the three up-and-comers or who are other lines of business who should be there who probably, probably aren’t going to weigh in, but they should learn by listening which I know sounds like Pollyanna like where are we going to find the time for people to witness this but if you’re looking through the lens of saying we need to understand the business. It was a manager that same company as the first one who said, you know, as analysts, sometimes we need to know the business better than our business partners. We need to have a broader understanding of where the moving parts are. And we’re sitting in a central function where we can have that. But that means there needs to be time expended to actually learn that broader context. 00:39:54.02 [Moe Kiss]: And it’s funny, I think one of the things we haven’t touched on though is understanding business timing, which I feel like is almost its own whole area because, and this is something that’s incredibly top of mind for me right now is about like speed to decision. And so what I do observe is, you know, the data folks going away, wanting to put their best foot forward, wanting to like work on this really, incredible, complicated pace of analysis. And it’s like, well, the business needed a decision made last week. And so now you’ve presented it and the decisions are even made. And sometimes I find coaching people through that where it’s like needing to understand the level of rigor you need and the speed, the level of confidence for the business decision being made. I find is one of the most I don’t know, underrated is probably the wrong word, but I feel like it’s something that people kind of forget a bit about and it’s one of the ones that trips up data folks very often. 00:40:55.01 [Val Kroll]: That goes back to Michael’s point about empathy, right? That’s building the empathy or building your business acumen helps build. Okay. Building your business acumen helps build your empathy for those stakeholders because you’re really in tune with the decisions that need to be made, the pressure that they’re under, the risks that they’re considering taking or being forced to take. 00:41:20.14 [Michael Helbling]: And should guide basically what kind of analysis you end up doing. Is this a massive project where we’re gonna go six months on this? Or is it sort of like quickest, dirtiest, some information to help you make a decision tomorrow? Because you’re so right, Moee. That’s one of those ones where I’ve seen this a lot where somebody rolls in with an analysis that would have been great three weeks ago and now is completely out of priority. And especially in a growing, like a fast growth company, That happens almost overnight. It’s like, oh, we moved on. We’re already onto something else, not even talking about that anymore. And you just look so out of place at that point. It’s terrible. 00:42:02.53 [Tim Wilson]: You do, but to be fair, and this is a separate challenge, and I feel like we’ve brought this up We have as the industry at large this belief in the truth and precision and magic of the data that the challenge of saying, I’m going to give you directional crude stuff, but it shows that x is greater than y. What will be heard by the business is that is the X is greater than Y in an absolute truth perspective. That’s a separate challenge. We can have marketers railing about speed to decision. What they’re thinking is I want the super precise in depth, all that detail, which probably wasn’t that hard. You just need to give me the right model or put the right AI agent on it. Uh, I need it now. And if you say, well, I can give you something now, but it’s going to be, it’s going to be pretty blunt and it may be wrong. And it’s going to be a little risky. They’ll be like, fine. So you’ll get, what you’ll get me right now will be perfect just cause I asked for it harder. So that may be a whole other episode of, you know, decision, decision science, decision skills, you know, 00:43:17.66 [Moe Kiss]: If we move really fast, are you comfortable we’re going to be wrong 30% of the time, 60% of the time? Really getting folks used to having those conversations and realizing, sure, if you need a decision tomorrow, I will give you something, but there will be assumptions and I can list them out very explicitly, but it is back at the napkin. That’s what we can do with the time we’ve got. It does require a whole different level of I don’t know, maybe maturity is the word or depth of understanding. 00:43:49.02 [Tim Wilson]: But I think it’s the other pieces that getting that understanding of at the core, what are the big boulders that we as a business are trying to push can help say, these are the things I need to always be thinking about and trying to figure out ways to bring those to bear as opposed to every request that comes in. I need to slot that into the context, the decision speed, all these other pieces. There’s another part of the deeper you get that business context, the more you can just make smarter decisions about where you’re spending your time. Which things are total throwaway? I have to spend 15 minutes on this, but it doesn’t matter. It’s never going to come up again because this was a one-time thing. with one pointless question being asked, oh, this other thing, I need to respond just as quickly in 15 minutes, but I’m also gonna keep working on it to give a more thorough answer because this, if we can crack this nut, then we really will have moved the needle. I feel like I’m talking all in abstractions and I have various examples floating through my head that I can’t figure out how to generically articulate them. 00:45:05.61 [Val Kroll]: Oh, it’s good. We’re following Tim. But the thing that this conversation, this part is making me think about is when speaking of empathy is when you’re delivering an inconvenient truth. Like when you’re saying like, oops, sorry, that painted door test like in the experimentation world says it’s not worth building out that feature or it happened a lot in like my market research days when they were doing concept tests and it’s like, ooh, sorry, everyone kind of attributed that to your competitor. It’s like not cutting through. It’s like not giving you the credibility and it’s like, shit, like what do we do? Like how do we go? And so I think like this is your other opportunity and I don’t have perfect answers here, but to be a good partner of Like what this could mean and how you can help tell the story and, you know, delivering, helping your business partners deliver the message to their higher ups. I think that’s one of the areas where you get a lot of points on the partnership realm for again, having empathy for what this means to that team’s roadmap or budget or their planning. So that’s never, never a fun moment to have to be in. 00:46:13.40 [Tim Wilson]: Sometimes it’s also a good opportunity to pull in somebody from another group. I think the number of times that the analyst gets hits with something and you’re like, I understand what you’re trying to do and you deeply want to answer this question and answer it well, and we’re not the most equipped to do that. We should go to the research team. We should go to the experimentation team. They may already have something. We understand what you’re trying to get at. I don’t know that they’re going to have a simple quick fix either, but let me loop them in, provide them the context, see what their thoughts are, see if something can happen on. on that front. I feel like that happens a lot of times with behavioral data where the question that comes in, and if you really understand what they’re thinking, you’re just trying to cram the big behavioral data answer something that is really an attitudinal data question. But you have to understand what they’re really trying to get at. 00:47:14.08 [Michael Helbling]: This is a question for the team. So we’ve been talking a lot about data analysts. Do we think everybody in the data org should be building a business acumen or there’s some roles that it doesn’t matter as much? 00:47:30.28 [Tim Wilson]: Everyone. Everyone, data engineers should be. 00:47:34.91 [Moe Kiss]: I mean, I literally, I had this conversation yesterday where a team of engineers have built out probably one of the most useful data sets that I could imagine and they left out a particular property. Because like they didn’t understand that that’s how that’s like the connecting fabric for everything we need to make that data useful and you’re like And that’s the business acumen. That’s the like here is how it’s gonna get used here. It’s here’s how it’s gonna like work with our systems here is how a Product manager is gonna answer a business question and it’s like Yeah, anyway, I’m on the very strong fence of everyone. 00:48:14.82 [Tim Wilson]: I think that’s the battle to fight against that I watch. This is like the path that the analysts will get pulled down, the data person will get pulled down as they get into the complexity of the data, which is interesting in and of itself. It is this interesting engineering challenge. There is plenty of understanding and exploration to be done and clever solutions to be come up with. And that doesn’t require going too, too far out of your comfort zone. You can just dig in and figure it out. And then they just sort of spiral down into, hey, I’m getting smarter. I’m getting a deeper understanding of the business when really I’m getting a deeper understanding of the way the data is landing in various tables, which is important and needs to be known. But it’s easy to get. I mean, I think that happens. I mean, Adobe Analytics, CJA. Google Tag Manager. There’s an infinite level of spiraling deeper and deeper into the data that can be done, and it’s got to take a conscious effort to say, you know what, for the next hour, I’m just going to go understand what the business gives a shit about instead of figuring out that how to better normalize this one metric. And that takes that takes conscious effort. 00:49:32.16 [Michael Helbling]: Yes. It’s kind of why I asked the question because I’ve run into people who kind of will be like, well, in my role, I don’t have to. And what’s always surprised me is like, I’ve never, yeah, like that or not maybe explicitly, but like no interest. And it’s so weird because I see the connection immediately from any data role you can think of. But I’ve seen data scientists do this. I’ve seen data engineers do this. Even to a certain extent, even analytics engineers or analysts who kind of live within their function. But I think it’s because you get down into the layers of complexity or arcane knowledge of the specific tools that you’re interested in. And then you sort of feel like that’s enough. The challenge would be, I don’t think you’re going to have a fulfilling of a career if you don’t spend some time trying to go up into the business itself and understand it. Going back into that, we’ve talked a little bit about how to do that. Do you think there’s benefit in going and getting an MBA if you’re a data analytics person? or under education. 00:50:45.22 [Tim Wilson]: I guess it has to be an MBA specifically. 00:50:47.35 [Moe Kiss]: I’d love to get one, but that’s more just like interest. I don’t think it’s a requirement, but I’m interested. 00:50:56.56 [Tim Wilson]: Having gotten one before I was officially in analytics and kind of stumbling backwards into it and finding it very interesting, and it was largely just because I wanted to go do something else. And every passing year, I find more like, wow, there was some good, like how did I just take one micro economics class and 23 years later, I’m still pointing back to some of the game theory that happened in that and seeing those patterns in the world. I don’t think it is a, you must, like it’s an investment of time, but there were things that I was doing Then when I was taking it that I had no idea was like getting is embedded in my brain. And I think did give me deeper understandings of different aspects of the business. So I’m a fan, but I’m also not a reliable assessor. Um, you guys, and I’ll take the FedEx commercial and say, so easy an MBA can do it, you know. 00:51:59.08 [Val Kroll]: Well, as someone who doesn’t have their MBA, I’m similar to Moee. It’s something that’s always interest me, but my very small-scale proxy for ways that you can get deeper in some of that outside of some of these conversations and two different roles that I’ve had In the past, similar to how you go through these exercises to get closer to what the experience is in your customer shoes, I’ve had sitting in rotation with some people. Some of my business partners are stakeholders. When I was at UBS within the investment bank, I spent a couple of days with ride-alongs with some of the research analysts themselves from the top of the morning to the end of the day. My mind was blown within the first five minutes because the analyst that I was following said, oh, I can’t take the subway into work. And I was like, what do you mean you can’t take the subway? We live to New York City. I’m like, what do you mean you can’t take the subway? And he’s like, I literally can’t afford to not be reachable during these hours. And so I have to be able to hop on a call because that’s how I service my my customers and I was like, that is crazy. But I just felt like I, speaking back again, the theme of empathy here understood so much deeper, like what was at stake for him or others in that role and how data, I’m thinking like, well, what do you mean you didn’t read the thing that I sent you and he can’t even get on the subway, right? Not really going back to answer your MBA question, Michael, but just talking about like, how do you find like really deep ways to extract some of that? And that’s, that was the thing that, that cropped up in my, my head like rotations. 00:53:39.62 [Tim Wilson]: I feel like I should also throw in that there are all the very executive ed stuff, and some of those are just ways for schools to just print money. It’s really hard to know what’s garbage and what’s not, but my wife went through one of those mini-MBA type thing, Harvard’s Core Program, capital C-O-R, lowercase e. courses in semesters, but it was six months or something. It was totally doable. She’s in a PMO type roles. She’s like, this has been really useful because the people that I’m working with are sometimes talking about useful to get just a surface level accounting background. surface level finance background, surface level marketing. And that was a much, much lower lift. And even as she was doing, she’s like, oh, this is like really useful. And then there was other stuff that she was like, I could give two shits about this. I don’t think I’m ever going to need it. But so I think there are those in that, but also it was the learning style. If somebody is a, if you are, I need the structure of some sort of a program, there are scads of them out there. 00:54:57.84 [Michael Helbling]: Yeah, I think I was more negative towards MBAs earlier and now I’m more ambivalent. I’m personally not gonna probably pursue one at any point in time, but I could see where they’d be useful. So I’m more open to them now. Probably because I know you, Tim, that’s helped me learn to love MBAs. He’s not so bad. Yeah, exactly. All right, we’ve got to start to wrap up. So hopefully this has been a good discussion about business acumen and you feel like you’ve got a couple of directions. I think now that AI is so prevalent, I feel like any business question you could think of, you could get a really decent answer, at least at a base level from an AI. So those kinds of tools are things like, You don’t have to go buy a whole book on the innovator’s dilemma, although you should read that. But you could also just get a rundown on a topic pretty easily with AI these days. All right. Before we jump into last calls, I want to take a quick break and have a chat with our friend, Michael Kaminsky, from ReCast. There’s a media mix modeling and geolift platform helping teams forecast accurately and make better decisions. You’ve heard Michael sharing a lot of bite-sized marketing science lessons over the past few months and they hopefully are helping you measure smarter. Well, let’s get over to you one more time, Michael. 00:56:17.88 [Michael Kaminsky (Recast)]: Before running any analysis, we need to ask, do we have the right data and model to answer the question we care about? Often we don’t. The problem is really common in economic analyses where you want to estimate a demand curve from your data. But if you only have data on price and quantity sold, you can’t actually determine how much is driven by supply changes versus demand changes. The model can’t be identified statistically with just sales data. You would need data on some other external factor that only affects supply or only affects demand in order to be able to identify the effects you care about. Identification issues can stem from data limitations or model structure problems, and in very complex models, these issues can hide in the structure of the model really easily. Even in simpler models, multicollinearity or correlated variables can make models practically unidentifiable due to insufficient data variation to estimate parameters. My favorite way to check for identifiability issues in a model is via simulation and parameter recovery exercise. we can simulate data where we know what the values of the parameters of interest are, since we use them to simulate the data, and then we can check if our model can accurately estimate those parameters from the data. If it can do that consistently, we don’t have identification issues, but if the model fails to recover, then we know we have a problem. So the takeaways are, be thoughtful about your ability to learn the parameters of interest from your model, and use simulation exercises to check for identifiability problems in your analysis. 00:57:38.46 [Michael Helbling]: All right, thanks, Michael. And for those of you who haven’t heard, our friends at ReCast just launched their new incrementality testing platform, GeoLift by ReCast. It’s a simple, powerful way for marketing and data teams to measure the true impact of their advertising spend. And even better, you can use it completely free for six months. Just visit www.getrecast.com slash geolift to start your trial today. All right, let’s do some last calls really briefly. Moe, let’s start with you. What’s your last call? 00:58:13.36 [Moe Kiss]: Okay. So about a month ago, there was a really horrific incident that happened in Bondi in Australia. And it’s been a really shitty time in both our city and our country. But I’ve been really grateful because it just happened that I started reading this book a few days before. which is called humankind, a hopeful history. I can’t say it’s first name, so the author is Breakman. But his main premise is that human nature is fundamentally good. And I won’t give away the whole book, but what I will say is he, one of my favorite examples is he talks about Lord of the Flies. So if you’re not familiar, a bunch of kids, deserted island, they all end up blowing up at each other. And he found a real life Lord of the Fly situation, which I can’t remember if it was five or six kids in the Pacific and how they actually cooperated and came together, never let the fire go out and ultimately ended up getting saved. And I just It just happened to randomly be a book that I read at the right time. So if you’re feeling a little bit pessimistic at the moment, it’s definitely one that I recommend just to remind you of the good in people. 00:59:37.46 [Michael Helbling]: Yeah, that’s a good reminder. Thank you, Moe. All right, Tim, what about you? What’s your last call? 00:59:43.11 [Tim Wilson]: There was a podcast about the real life guys and I can’t find it. 00:59:48.09 [Moe Kiss]: Oh, I’m dying to hear it. 00:59:49.28 [Tim Wilson]: It tells their story and they actually went back and talked to a couple of them because the guy, the kids had like stolen a boat or something and then they got. They stole a boat. Yeah, they stole a boat. But they wouldn’t actually talk to one or the two of the people who were still around. So I’ll track it down and throw it in the show notes as well. So mine would be just super entertaining. I realized that if Ben Stansel is on stage describing a ham sandwich, I will pay money to go see it. He did a talk. at the Small Data San Francisco 2025, 17 minutes long. The title is, in the long run, everything is a fad. It is like, it zips along, it’s grounded in kind of the Jordan Childs Olympics, third bronze or not. How long was she challenged? Gymnastics thing, but it is so good. And his point actually, Moee gets to kind of that decision making. He kind of makes the case that Maybe AI and LLMs being good at sort of picking up the vibes from stuff, that that may be a more effective thing for people to make decision is just decisions quickly getting a sense of the vibes, even if it’s not hard and quantitative data. He has a whole thing about kind of different generations and what they think is kind of the core of making decisions. But it is like 17 minutes that just blows along and it’s He is just one of the most engaging presenters ever, and I can’t recommend it highly enough. 01:01:27.22 [Michael Helbling]: Nice. Thank you. All right, Val, what about you? What’s your last call? 01:01:33.59 [Val Kroll]: It’s a twofer, but I’ll be quick. So part one is, Michael, you took a break from the APH, Mike, to join another podcast recently. We’re not too jealous. But it was a new podcast, Knowledge Distillation from the Ask Why team. And Michael appears on episode four, which is about trust and bottlenecks and the difference between data retrieval and actual analysis. And Michael, we loved it. And so wanted to give you a shout as one of my two last calls here. 01:02:12.29 [Tim Wilson]: Why don’t you bring that kind of quality to this this podcast? 01:02:16.06 [Michael Helbling]: I don’t know, Tim. Maybe there’s someone always talking over everybody. 01:02:20.01 [Tim Wilson]: Damn it, Moe. I’ll talk to her about it. 01:02:26.85 [Val Kroll]: We’ll take that offline. And then the second one is for an upcoming conference. DataTune is in Nashville. Two-day conference March 6th and 7th. Tickets are on sale. But it’s all about data, analytics, AI, some of the things you’d expect to see. And Tim and I will actually both be there. So we’re speakers at that, and we’re looking forward to it. So if you’re looking for something in March, we’ll see in Nashville. 01:02:56.60 [Tim Wilson]: Nashville. 01:02:58.55 [Val Kroll]: How about you, Michael? What’s your last call? 01:03:00.28 [Tim Wilson]: What should we actually say? Is there any other conference that we’re going to be at? I’m trying to think. 01:03:04.91 [Michael Helbling]: Well, I believe we will be at the Marketing Analytics Summit in April. The Analytics Power Hour will be there. And I believe that’s April 28th and 29th, if memory serves. And we’re excited because it is actually the 25th anniversary of that particular conference. It went by a different name when it’s early days, but now it’s called the Marketing Analytics Summit. And so we’re pretty excited to celebrate with the industry. Go get tickets, come. It’s going to be a blast. There’s some amazing speakers already lined up. And this little crew will be regaling you with our chit chat and something. We’ll come up with a topic. All right, my last call, let’s just come back to sort of the business acumen thing because I got all my business acumen the hard way. I like finding resources or places where I can learn things or I consistently find good information about this kind of stuff. And one of those places is common cog.com, which is run by Cedric Chin, who has been a guest on the show before. They evaluate cases, they write a big long form articles. So there’s a lot of explanation. There’s a community. So there’s a lot of discussion. You can talk to other business people. So I really enjoy that website, common cog.com. So if you’re in a role where you’re trying to ramp up on your business acumen, that might be a great resource to potentially leverage as well. All right. Well, this has been fun. I think this is something we come back around on semi-regularly, but I think it still maintains its importance in the life of anyone working in data and analytics. So thank you, everybody, for putting your time and effort and thoughts into this episode. All of you, thank you. 01:04:55.86 [Tim Wilson]: Thanks for having us, Michael. 01:04:57.36 [Val Kroll]: Yes. 01:04:58.42 [Michael Helbling]: Fun as always. You don’t have to say anything. I’ll take it from here. No. Too much dead air. Couldn’t take it. Couldn’t take it. That’s right. Let’s talk a little bit about how you can reach out to us. We’d love to hear from you. And the best way to do that is on our LinkedIn page or on the Measure Slack chat group or You can also email us at contact at analyticshour.io and you know what, we also take a look and see what ratings and reviews people leave for us on the various platforms. So if you leave us a review, we’re excited to see that also. I’d love to hear those also. All right. Go get better at understanding the business. It’s going to help your career. It’s going to make you more influential. It’s going to make you more impactful. And I think I can speak for all of my co-hosts when I say, no matter what your level or years of experience, keep analyzing. 01:05:57.53 [Announcer]: Thanks for listening. Let’s keep the conversation going with your comments, suggestions, and questions on Twitter at @analyticshour on the web at analyticshour.io, our LinkedIn group, and the Measure Chat Slack group. Music for the podcast by Josh Crowhurst. So smart guys wanted to fit in. So they made up a term called analytics. Analytics don’t work. 01:06:22.10 [Charles Barkley]: Do the analytics say go for it, no matter who’s going for it? So if you and I were on the field, the analytics say go for it. It’s the stupidest, laziest, lamest thing I’ve ever heard for reasoning in competition. 01:06:35.36 [Michael Helbling]: The most accurate business acumen. It’s interesting, Tim, I forgot that you had an MBA. 01:06:43.63 [Moe Kiss]: Like, Tim had a whole book on it. 01:06:47.59 [Michael Helbling]: Is that what that book is about? His MBA? at some point this afternoon coffee is gonna kick in and I’ll be ready to start the show. Moe is literally recording this in a rain forest right now. 01:07:06.91 [Moe Kiss]: I mean there are fewer places I’ve recorded like under the kitchen bench in that place in Italy. 01:07:13.78 [Tim Wilson]: There are so many trees and some of them are even organic. There’s the coat tree. 01:07:17.70 [Michael Helbling]: You got the, yeah, decision tree. All right. That would actually be a cool gift. 01:07:24.99 [Moe Kiss]: Random forests, you know. 01:07:26.95 [Tim Wilson]: Random forests, exactly. Giving everyone a decision tree. That’s right. It’s my, fits in the envelope better. Jump. 01:07:37.05 [Val Kroll]: You can use it at least a little bit. I knew you were gonna say that, Michael. 01:07:43.70 [Michael Helbling]: We belong in the same generation. That’s right. It includes what? Jump to conclusions, Matt? 01:07:54.71 [Val Kroll]: Office space? We were joking about office space yesterday, Tim. 01:07:57.75 [Michael Helbling]: Yeah, come on. 01:07:58.81 [Val Kroll]: When Tim was working on postal codes, cleanliness, whatever project he was at nationwide, I just imagined he was Milton in the basement. 01:08:10.98 [Michael Helbling]: And it was one where like in my memory, I was told I could use Excel at a reasonable volume. 01:08:22.96 [Tim Wilson]: So good. Well, it’s interesting because of the lag and you being in the southern hemisphere, we’re actually hearing what you say before you think it. It’s kind of interesting. 01:08:33.91 [Michael Helbling]: No. Which one do you think will… Moee, just try whatever you think will work best. I don’t… We don’t care. 01:08:42.77 [Moe Kiss]: The reason I use this room is because it has no air con, so no whirring sounds. And it has hypothetically some sound things up. If I go to a meeting room downstairs, the Wi-Fi could be better, but my internet could be really shitty. 01:09:00.12 [Val Kroll]: How many trees will be there in that room? 01:09:06.67 [Michael Helbling]: All right, we’ll think of cool tree jokes while you get reset up. Oh, you guys are going to the DataTube conference. Okay, good. I’m glad some of us are going because I completely whiffed on submitting anything to that guy when he reached out to us about it. And then I remembered it like a month later and I was like, oh, crap, I missed the deadline for that completely. 01:09:44.37 [Tim Wilson]: I’m glad the top has some decorum. Wait, it’s a day-long workshop? 01:09:54.36 [Michael Helbling]: You both are doing a whole day workshop? Or maybe it’s a half day. 01:09:58.43 [Val Kroll]: First time hearing about it. 01:10:00.73 [Michael Helbling]: Oh my gosh. 01:10:09.85 [Tim Wilson]: Rock flag and concepts and context. The post #289: The Imperative of Developing Business Acumen appeared first on The Analytics Power Hour: Data and Analytics Podcast.

If there’s one thing that we absolutely knew would be coming along with the increased interest and use of AI, it would be… more acronyms! And, along with the acronyms, we pretty much could predict that we see a lot of online flexing through casual dropping of said acronyms as though they’re deeply understood by everyone who’s anyone. We tackled one such acronym on this episode: MCP! That’s “model context protocol” for those who like their acronyms written out, and Sam Redfern joined us to help us wrap our heads around the topic. You see, MCP is kinda’ like some other more familiar acronyms like API and XML. But, it’s also like… fingers? Sam’s enthusiasm and explanation certainly had us ready to dive in! This episode’s Measurement Bite from show sponsor Recast is an explanation of model robustness from Michael Kaminsky! This episode is also brought to you by Prism from Ask-Y—your agentic analytics platform for automating analytics, exploring data, creating repeatable workflows, and delivering accurate insights—all without the need for manual query writing. Links to Resources Mentioned in the Show Cursor History doesn’t repeat itself, but it often rhymes Zed Agentic Engineering Series MeasureCamp GitHub’s Official MCP Server Zed ACP Opencode.ai (Podcast) Good Hang with Amy Poehler (including the Rachel Dratch episode) Bhavik (Bhav) Patel Manas Datta Superweek Christopher Berry Adversarial Poetry as a Universal Single-Turn Jailbreak Mechanism in Large Language Models Normally, the image we drop on an episode is a photo taken by a human, and we attribute it accordingly. This time, given the topic, we just couldn’t resist, though: we threw the entire transcript at Nano Banana with a minimalist prompt to see what it came back with. Episode Transcript00:00:05.75 [Announcer]: Welcome to the Analytics Power Hour. Analytics topics covered conversationally and sometimes with explicit language. 00:00:15.15 [Michael Helbling]: Hi, everyone. Welcome. It’s the Analytics Power Hour. This is episode 288. We spent the last decade putting walls around our data, securing it, governing it, putting labels on it. And now the AI revolution walks up and is like, hey, can I see all that? Today, we’re going to discuss Moedel Context Protocol, or MCP. I mean, it’s an open standard. It promises to stop all the copy, paste madness and let AI talk directly to your data systems. Is it the end of the data silo or just the beginning of a new governance headache? Well, we’re going to try to establish an MVP for PMF of MCP all in one hectic hour. All right, let me introduce my co-host, Val Kroll. How are you doing? I’m good. This is going to be an interesting one, yeah. All my acronyms, yeah, that was fun. All right, Tim Wilson, always a pleasure. Likewise. All right, I just used the acronyms to make myself sound smart. That’s all. Let’s get that in early. And I’m Michael Helbling. All right, well, we need a guest, someone to help us dive into this topic. And we’ve got a great one. Sam Redfern is a staff data scientist at Canva, currently working on search and recommendations there and previously marketing measurement. Prior to that, he has held data roles at both Meta and IAG, and today he is our guest. Welcome to the show, Sam. 00:01:36.31 [Sam Redfern]: Thank you very much, Michael, Tim, and Val. We’re really excited to be here. First time caller, long time listener. Oh, that’s awesome. 00:01:43.76 [Michael Helbling]: Well, we’ll ask questions and take our answers off the air. No, no. 00:01:48.48 [Michael Helbling]: So. 00:01:51.27 [Michael Helbling]: Sam, I’m excited to talk to you about this, because obviously, all things AI are very of the moment, and everyone sees the term MCP. But I think if we just take a step back, maybe you could just fill us in on what exactly an MCP is, model context protocol, where did it come from, give us some background on the whole concept to establish the conversation today. 00:02:17.34 [Sam Redfern]: No worries. Super pumped to be talking about this. Let’s take a step back and think about what this is solving, in a sense. We’ve had access to these large language models tools for a little while now. In the early days of GPT2 and GPT3, before chat GBT, these things were like word calculators in a sense. I really like the analogy that it’s like you put the numbers into your calculator and you get the equation out. This is the same for almost words. Early large language models acted like that. And the innovation in the open AI space was to basically feed the output back into the input and make this resumable format. And what’s interesting about this whole, so let’s step back from MCP and branding and letting technical teams come up with the term for things because this is how we got NFTs as a term at the same time. But the core problem to be solved with these is that we’ve got something that feels a little bit like a person in the sense that you will give it some words and it will respond with some words back. Could you give that agent or that large language model the ability to do something other than just converse? And so the first application of tool use, in a sense, in large language models, actually came from the open-source LangChain team. And for those who don’t know what LangChain is, it’s a framework for building agentic experiences. And so you can have your anthropic model your open AI model or whatever you want, and then you allows you to piece together bits of technology to add context into the large language model to try and get it output. And so in April 2023, PR was submitted to the Landtrait project, allowing it to open up for the large language model to take a browser URL and to go have the back end application request the contents of that HTML and then bring it back into the context window of the agent itself. And if you think about it as like the core thing that it’s trying to do is it’s trying to get this I think about it as fingers in a sense, is it’s trying to give the large language the ability to touch something, a bit of information, bring it closer to it for it to understand. That’s at its core what these MCP tools are. MCP is like a brand term through Anthropoc. To say it’s a standard is to be very generous. But I’m really bullish about the concept of giving these large language models access to tools for them to be able to solve problems. 00:05:22.34 [Val Kroll]: So I have to admit that I really did think that MCP was just like the API for LLMs. But the more I was looking into this, it was kind of understanding that those fingers like you use in your analogy is really giving it more access than just here’s this endpoint. And it’s just a one-time thing. Can you talk about some of the things that you give it access to with those fingers or to grab to kind of give a little bit more color to what it actually does or what it’s capable of, if that makes sense? 00:05:54.59 [Sam Redfern]: Yeah, absolutely. So this is in the weeds of how they work. I think the context to understand is if you’ve ever done work inside of large enterprises and you’ve tried to create an application access username and password, it’s a total pain in the butt. to go get this thing. And then your infrastructure team is like, well, you’re going to change the password every three weeks. And then you have to have a cryptographic token to do something. And it’s in some sort of space. And the reason why, actually, Anthropoc had a previous attempt at tool use in October 2024, a month before MCPs were announced, where they had a system that would take over your browser and move your cursor around. And the reason why MCPs are running on your local computer is that your user account has access to all these systems. That’s why the early paradigms of these systems are as close as possible to the end user’s system. So the analogy I give on the fingers, in a sense, is inside these MCPs, you can give it access to any number of… The standard allows you to sort of use representatives any number of tools. And you have bits of information, right? So when the agent starts, it’s basically given a list of all the tools that the MCP server has available to the agent. And so has the name of the tool. And so let’s just do a really simple example of like us, like in our agent environment, we have two tools, right? One of them is called a saw and the other one’s called a drill, right? In our SOAR tool, we would describe the name as SOAR. We would say the description is it cuts wood in a single direction across a line. Then the inputs to that is the position of the wood and the depth. That is a finger for lack of a better term. That’s our first tool and our second tool might be drill. That allows you to drill a hole through the piece of wood. 00:08:03.11 [Val Kroll]: Okay, so I have to ask one more. Sorry, I’m hogging the air. But I guess the one other thing that I’m struggling to grasp a little bit is what was the need for standardization of this, like the protocol? Can you talk about what that is solving? Because what you shared in that analogy was great. I’m absolutely going to use that, and I’ll give you credit every time. but like why was there a need to standardize outside of like you know enterprises you know would feel more secure with that or you know the governance would be easier but is there any more more to it than just that piece? 00:08:39.22 [Sam Redfern]: In the adoption curve, we are so far away from the governance piece on this stuff. There’s a bunch of companies right now that are trying to put governance around these systems, and I’m sure at some point we’ll talk to maybe some of the potential downsides of the standard if we want to call it that. But the reason why Anthropic went down this path is In the technical details around how the LLM is trained, they have been doing this work of training the large language model to use this like special escape set of characters. So when the large language model is like Okay, I think what I need to do is use the saw tool and then it has this string of characters saw tool string of characters, and that indicates to the sort of the agent that’s hosting the large language model. Okay, I have to take the text below this. and send it into the tool itself with the input parameters that it needs for it. Anthropic had this huge lead because they’ve done the work of training their large language models to for tool use and using their reinforcement techniques to basically say, this is what you have to do. And this was this huge lead that anthropic had for a couple of years, in a sense. Everything feels like it’s a couple of years. It’s really about eight months, right? Until other people started trying to solve this problem. Opening, I had their own sort of call procedures kind of method, like I think they were called functions. And it was a very similar kind of thing. Like anyone could have come up with a standard. The core problem they were trying to solve is how do you give the large language model a hand for it to basically make decisions about what information is pulled towards it or what information or what actions it takes when it’s pushing out. 00:10:36.40 [Tim Wilson]: This is slowly coming a little more into focus and still pretty damn fuzzy for me. I know recently it seems like there’s been a lot of chatter about Google Analytics having an MCP server. Is that the right terminology? That is something that that the Google team said, we’re going to produce this to basically make the, this is a saw. This is what it does. These are the inputs. And it’s just a much a lot. I mean, your analogy was very, very simple. Is it as simple as that, which has me going back to saying, well, when Val said, It’s like an API for LOMs. It sure sounds like an API for LOMs. And I’m missing where that analogy is breaking down. 00:11:35.02 [Sam Redfern]: I think you can use it. There’s a lot of analogies of talking about these NCP tools as being the early dates of APIs and stuff like that as well. I think there’s an extra bit of the near direction of where these systems are moving, which is more interesting in the API part, but just to come back to that. The way I think about it, APIs is a great way of talking about it, and there’s lots of people doing weird fun things with these tools right now. If you remember, I think some of us on the call are old enough to remember the early days of web 2.0 and people were making APIs for like the weather and it was open and everything was fine and you know we were very far from like the standardized way of we think about this sort of stuff now right like it’s uh The way you design an API is very standardized now. I think the thing that’s different is one, we’re dealing with this huge amount of non-determinism, right? And we’re dealing with all of these different terms and terminologies that exist. So I think everyone on the podcast might have heard of the term agent, right? And so an agent is the idea where you have a resumable output. You have like some text that is the system prompt, and then you have this resumable conversation. There’s another term that’s being formed right now called a harness. A harness is an idea where you have an agent and you have a tool plugged into the side of it. That has a domain of knowledge attached to it at the same time. Cursor is an agent. The claw desktop is an agent. Oh, sorry, sorry. Cursor is a harness, right? It’s got access to all these different tools. I think the, I actually think of NTP and where it’s at right now is more akin to these digital document formats like XML, right? So we started with XML and the number of people who are writing XML these days is almost none. However, the amount of change of this standardized document format then brought us to JSON and now it has unfortunately brought us to YAML and Markdown. We are at the XML stage of this development is that this is going to tool use conceptually attaching to large language models through agents and harnesses is That is going to stay for a long time. Whether it’s the MTP standard or someone comes along with a better standard, then we’ll see how that goes. 00:14:12.44 [Michael Helbling]: You know how developers got the AI engineer role? It’s time for the rest of us. I think we’re witnessing the rise of the AI analyst. 00:14:22.32 [Tim Wilson]: OK, does that just mean asking a chatbot to do math? Because I have Excel for that, Michael. 00:14:28.43 [Michael Helbling]: Well, no, Tim. I’m talking about Ask Why. It’s full stack analytics. I ask a question in plain English, and the product prism orchestrates the whole thing. You can pull in data from Excel or BigQuery. 00:14:42.10 [Tim Wilson]: Hold on. You’re sending BigQuery data to an LLM? Security is going to have a heart of track. 00:14:48.17 [Michael Helbling]: Well, that’s the best part. Ask why doesn’t upload your data. Explain? Well, it creates a semantic layer. It sends the context to the LLM. The LLM writes the code, and that code runs locally on your data. Your actual numbers never touch their servers, so it’s totally traceable. 00:15:07.61 [Tim Wilson]: So, I get the automation, but my data stays safe and secure? 00:15:13.21 [Michael Helbling]: Exactly. Plus, it remembers context, so as you automate routine tasks, it stores those, so you don’t have to explain it all again the next time you do that same task. 00:15:23.83 [Tim Wilson]: OK, I’m listening. 00:15:25.81 [Michael Helbling]: Where do I get it? Well, it’s even beta right now, and you can go to ask-y.ai. That’s ask-y.ai. You can get ahead of the curve and join the ranks of the AI analysts. 00:15:39.33 [Tim Wilson]: And because we like you guys, use code APH when you sign up, and our friends at Ask Why will put you at the top of their wait list. 00:15:48.17 [Michael Helbling]: Yep, stop pasting data into black boxes. Get Ask Why. 00:15:52.99 [Tim Wilson]: that I’ve had the XML question as well as whether it was, because I remember that being coming from an HTML world and then XML came out and it’s like, look, XML doesn’t give a shit about what you’re rendering in a browser, but it is this structured world. So I feel like, and then JSON, I sort of understood because of the XML. And that makes sense because there were, there was talk of saying different, applications uses would say using this XML structure, let’s define kind of specifically how that is going to be used in the context of this financial services thing. So you’re saying that is also a useful if imperfect analogy? 00:16:40.80 [Sam Redfern]: Yeah, look. Uh, history sort of rhymes more than copies, right? Like, um, you know, it’s, it’s going to, um, uh, like, It’s the first time in software that we’ve had this amount of non-determinism to deal with, right? You think about what success has meant in software development or data before, and it’s like some human’s ability to remember some random function as part of a library and be able to write that code as fast as possible and do it in as perfect as close to grammar as possible. The problem with this new world that we’re going into is the skill of dealing with non-determinism is not closely overlapping with that historical set of skills. It’s going to feel different and people who talk about the vibes of a model, there’s some truth in it in a sense. Getting a feel and it’s true with tool design. When you’re building an MCP, so we’ll just go back to the MCP paradigm, And so you’re thinking about the tools that you’re building and the fingers that you’re giving this agent and this harness access to. When you’re starting out, you’re doing kind of the early playful part of programming, in a sense, right? Where you’re just like, oh, does this connect to this? And when I run through it, what problems do I see with it? I do think that, like, It feels like those early days of these standards and people are playing around with them. And then when you want to get serious about the thing that you’re building, you’re then looking through the agent logs, you’re seeing what tools it’s calling, you’re seeing the parameters, you’re seeing how many times it correctly passes in the correct parameters and everything like that. 00:18:30.07 [Tim Wilson]: So let me hit the non-determinism point. And maybe I’m going to go back to using the Google Analytics MCP as an example that the The client, the application that’s hooked, the agent that is hooking in and using the MCP, say it’s a LLM, it’s core. And maybe I’m missing the non. I’m thinking of that as being it’s a it’s a probabilistic thing, but it’s hitting the MCP to get stuff back. Is the MCP necessarily also kind of non deterministic? Or is it no, like the inputs may be kind of floating around a little bit, but and or maybe it just depends on what it’s an MCP for. If in the Google Analytics example, I would say if the input is users in the last month that the MCP would say, well, as long as that, or is it that input’s going to come in with the little squishiness in it, and it’s up to the MCP to say, I got to figure out what I should go and pull and return. 00:19:39.91 [Sam Redfern]: Yeah, and I think this is a good point to talk about this interesting dimensions that’s coming up recently. If I was going on a league team, which I’m definitely not, and I don’t think I’ve used Google Analytics for over a decade, and I missed many of the big transitions. I’m probably not the best person to talk about GA, but So the Google Analytics MCP is going to have this problem where they have these dimensions and measures and breakdowns, and then they’re obviously trying to do it on the cheap on the inside. And so they’re only storing some of the information. So they’re going to have a basic report tool, so it’s going to be called the report tool. And the name of it is going to be traffic analysis. And it’s going to be put the date range in, and then it’s just going to return back a very minimized array of what the traffic has been. 00:20:41.10 [Tim Wilson]: Report tool, that would be like a saw, but there’s going to be separately a drill. When you’re saying a tool, that is a tool that’s available as part of the MCP. Okay. 00:20:53.24 [Sam Redfern]: That’s right. Now, where this gets interesting and anthropic announced for lack of a bit of time on the, I think a couple of days ago, announces code execution with MCPs and allowing you to actually write code. This is the thing that I’ve actually found over the last couple of weeks, which is it’s way better getting the LLM to write code to interact with APIs or SQL or something along those lines than it is to actually give it access to all of these tools and all of these intermediary steps. If and you might have seen this from your own experience of, you know, you’re probably spending a little bit less time hacking away at a piece of sequel, getting it to form exactly what you want it to these days. You know, you’re probably spending a bit more time of like here’s all the table space that I’m working on. here is a thing that I want to query, here’s what the outputs look like, and then you’re sort of having this sort of feedback loop where you’re doing that work. And my guess is, is that if you wanted to build a more sophisticated MCP, and if you were Google, you would actually lean into this concept where you would let the agent go build a little piece of Python code or JavaScript or something along those lines to query a bunch of known API endpoints to form the data back in the way that you want it to. Snowflake has its MCP server and Canva would make something called data MCP, which takes all of this data information we have and allows the LLM access to understanding how to use it. you’re really doing this piece of work around context engineering and you’re trying to think about like what is the LLM going to put into this tool for then it to get this output out. And so Michael to say, and so your sort of question here is that the MCP tool itself is deterministic, right? So it is an application in the traditional software sense. I think a lot of these MTP tools are sending data outside of out to the internet, right? They’re connected into API or a SQL database or something along those lines, but you can have deterministic tools inside of the MTP server as well that’s locally that’s connected. So you could just have like a calculator tool and it just adds numbers together and then returns exactly the right number out. You know, we’ve heard the joke of counting the number r of r’s and strawberry, right? So you could have a little Python function that just counts the number r’s and strawberry and it’s just like, all right. the R counter tool, put the word strawberry in, and this is how many R’s you get back. And so this is the whole idea of, and this is, I understand I’m going to do shout outs later on, but Zed, which is an ID, has an agentic engineering series. And what they’re saying, And I just think that they have this really great framing of the problem that we’re working on today, which is how do you take the advantages of non-deterministic systems and couple it with the advantage of deterministic systems to get something more than the sum of its components? 00:24:00.95 [Val Kroll]: So I would love to take a little bit of a turn because you’ve teased it a little bit, but I’m so eager to hear some of your favorite use cases and examples and that you can talk about. And I know you mentioned before, it’s still early days, but if you could talk about some of the things that you felt like weren’t possible for or too much effort for what it was worth in the past, but now it’s unlocked this or solved this for you. 00:24:27.20 [Sam Redfern]: I’ll talk a little bit about some of the stuff we’ve got working inside of Canberra at the moment. We use a large database vendor, which I will keep their name out of just so they don’t get in trouble. They’re going down this path of building out their own separate MCP tools and stuff like that. But what’s interesting and what I think is a big opportunity for people in this space is that building these tools for your organization is actually the critical skill that we’re going to see in the coming period of time. Every organization or company is really different, in a sense. And they made a database choice five years ago, and they made like a data transformation tool twice like four years ago. And you have all the incremental knowledge and information that’s built on top of that, which is going to kind of be unique to your organization in a sense, right? And I think there’s going to be vendors who are building tools in this space. But if you’re sort of a midsize company, like I think something to really think about is building customized versions of these tools that actually work with the flows of your organization and really having teams that are like building, thinking about these agenting and engineering practices on how to actually automate parts of the work that people don’t want to do. So some of the use cases that I’m sort of playing around at the moment, just so just the last week I’ve been working on using the Altair Python visualization library and actually building a Python based sort of sandbox environment for it to run the Altair code. And so The way this works is you just put the SQL statement of the data that you want to pass into the Altair code, and then you have a Python sandbox part of the field of the tool. It just puts 300 lines of Python into it, and it builds the visualizations out of it, right? So using these really nice Python visualization libraries has always been a pain in the butt unless you tattoo the way every part of the application works on your arm so you can know exactly how it works. But again, we don’t have to worry so much about grammar anymore, because if you feed these systems examples, they can come back and help you visualize this way faster. And I think that’s where, you know, Zed has this post where they, their concept is leverage, not magic, right? And what we’re trying to do is we’re trying to take our staff members and we’re trying to make them move faster and explore more in a shorter period of time to get to a better end outcome. Just on other sort of like interesting fun use cases, I’ve been, At home, I’ve got my own little sort of home lab kind of thing and stuff like that. And part of playing around with that is there’s all these command switches and stuff like that. And so I built a custom NCP server for home that documents all the different applications that I use in the CLI. And it has all of the context of it. I basically put in a free text field of what I’m trying to do. It then uses a search engine to search over the data. It then takes that context, puts it into the LLM, and then the LLM goes and gives me all the command switches and stuff like that. Another one is, I think the reason why Moe recommended me for this is I gave a presentation at MeasureCamp about, at MeasureCamp, you have to come up with a talk title, otherwise people don’t show up. I said that MCP is the real apex predator for your job in 2026. And obviously, you know, I don’t think that’s true. But so I wrote that someone was like, you need to give a talk sound. I’m just like, fine. So I wrote that part at 11 o’clock, I then vibe coded up a maybe the game battleship. And so I made a version of battleship that uses MCP tools as the the fingers that the LLMs can use to play against each other. And then I built an agent harness where I could get different vendors, MCP tools to actually fight the game battleship against each other. And then you could actually watch the turn by turn thing of watching them compete against each other. And they would then have their thinking of like how they thought about the strategy of the other player. And then you could like change the prompt and be like, okay, you are going to be the random player and you’re just going to do the most random things you possibly can. And then the other one is like the most strategic player of like, this is the common moves in battleship. I’m going to do this and stuff like that. And it’s just, it’s like, like, it’s a whole new paradigm of like weird things to play around with. And like MCPs are just like this, this layer for you to like do this joining between deterministic and non-deterministic systems. But once you start playing with these systems and you’re finding different ways of interesting things to do with them, like, you know, it’s, it’s, it’s, It puts the fund back into sort of like the early stages of programming again. 00:29:38.16 [Val Kroll]: So you giving the context of like deterministic, non-deterministic is really that’s helping it crystallize a little bit in my mind what some of this is. But I do want to go back to when you first started talking about some of these examples, you were talking about how every organization is different. Everyone’s working with a different stack. Everything has to be kind of in context of what’s going on inside your organization. Can I just go back and just repeat a little bit one of my earlier questions. What is the value then of the standardization versus it being custom inside of your organization? Is it just about the ability to leverage those other tools that are using MCP or is there internal benefit too? I guess I might be just, it’s not clicking for me. I would love to just hear you share. 00:30:27.67 [Sam Redfern]: No, no, no, no. Okay, so MCP is the thing that allows you at the moment until someone comes up with a better standard. And we should probably talk about some of the downsides of MCP as well. Just all the criticism, sorry, the criticisms as they stand of MCP, but MCP at the moment allows you to bridge that gap between deterministic and non-deterministic systems. You’ve got the vendors, and the vendors are going out, and they are building their own customers. MCP is right, because what they want, because what they’re getting, is they’re getting pressure from leadership teams of like, okay, we need to get some AI in this organization right now. I have called desktop on my computer and I want to be able to query this information directly from Claude and have that come back to me in a sense, right? And MCPs is like a path you can go down there, but the problem is that you’re wrestling with the non-deterministic nature of these systems when you do that type of connection in a sense, right? And what you actually, this is where building up that practice inside of your organization is really important because when you get into the detail of how these systems work, you realize that giving senior leadership teams access to the raw thing that directly queries the database is problematic for lots of different reasons around you’re not able to check it, you can’t give it a system prompt, all that sort of stuff. So MCPs, the standard, if we can call it that, is just the little connecting block, in a sense. the practice of what I’m talking about about the non-standardization inside of organizations is that you can totally go use the vendor solutions, right? But it’s always going to feel like it’s through a fuzzy piece of glass, because unless you’re doing exactly what the vendor has, right? Like, if you’re If you’re 100% Google shop, and you’ve never used anything other than Google, then maybe the Google stack is going to be great because that’s what they’re going to be using internally. And they’re going to be copying from that of how they’re building it. But I don’t know about you, but most organizations I ever interact with is it sort of like a collage of different solutions. And the money is in getting them to connect together, right? Yeah, templatized versus custom kind of. 00:32:38.05 [Tim Wilson]: Altair, take Canva, take Snowflake, take Google Analytics and MCP for those. Is there the option that Canva or Altair or Snowflake or Google, and those are intentionally very wildly different types of platforms, they can sort of create an MCP, one of these connectors that say this is kind of generic, but there’s also an option that I as a user of Canva with access, I could also build my own MCP, my own connector, or is it okay? 00:33:20.36 [Sam Redfern]: So you’re raising on a really good topic, which is called context pollution, right? So there was a criticism of GitHub’s MCP server where it has like 24 some number of tools, right? And the majority of people use like three, right? And so I push. If you get into the details of how NTPs work is that they repaste the entire list of tools and the tool descriptions and all this sort of stuff with every single message that goes through, right? Because they are trying to solve the context engineering problem of, like, The LLN needs to really know what all the tools are at every single turn. And you can go and add 70 tools to an agent harness. And you should do that to watch it not work, because it’s very entertaining. And you can suddenly watch all of your contacts disappear and have all sorts of problems, right? 00:34:24.92 [Michael Helbling]: You got extra tokens to spare. 00:34:27.04 [Sam Redfern]: Go ahead. You know, somebody would go win the token awards or whatever it is. I did the other day, right? You know, some people need the token trophy. That’s great. But the like your job as an engineer or a technology person inside of an organization is to do it in the most sensible, reliable way possible, right? And you’re trying to harness these nondeterministic systems to get the best outcome. And part of the reason why you go and build that custom harness that is fit for purpose for your organization and has different flatters depending on exactly the task that you’re trying to get into is you’re trying to, it’s in the name, you’re trying to take that harness and you’re trying to constrain this non-determinic system to only work in this particular domain. I think a lot of people, when they talk about these MCPs, they’re talking about it from their experience of having clawed desktop on their computer and connecting it up to JIRA or some other thing like that. That’s a totally valid use case. I don’t have to open JIRA anymore. I think the favorite part of my job now is when someone assigns me a ticket, it’s the only thing I have connected to the clawed desktop on my computer. and I interact with all of my JIRA tickets through Claude. I’ve solved the Atlassian interface problem by just never having to open it. That’s for me as a human here, but when we’re talking about making these systems do useful things in your organization that you can’t convince engineers to pick up, or it’s really boring work, or it’s testing work, or something like that. That’s where these systems kind of shine, right? We’re not trying to put someone out of a job or anything like that. What we’re trying to do is we’re trying to get that as tasks that are not particularly enjoyable, like documentation, testing tracking, all this sort of stuff, like building custom harnesses around that to help engineers make the best possible decision when they’re building something, That’s the real advantage of these tools. 00:36:30.89 [Michael Helbling]: Yeah. And I think, Tim, also, the Google Analytics example is tricky because it’s very limited. And it basically is an API layer to Google Analytics. It’s not really giving you more MCP-ish type of interaction. So I think it adds to the confusion a little bit. Because it’s like, you can do all the same things you can do with the Google Analytics MCP server with their API. It’s just call this function. But instead of you writing the query to the API, the LLM does it for you. But it’s not more stuff. 00:37:04.57 [Tim Wilson]: How does the GitHub MCP server? So I have two questions. One, it sounds like a lot of platforms maybe of those, if they’re 24, and I’m assuming it’s not exactly 24. However many tools there are within the GitHub MCP server that a lot of them are just a layer to the API, and maybe there are some that aren’t. But if you were saying, we do want to use a GitHub MCP server, this has got too much, would it be? I’m going to get their MCP server and I’m going to whittle it down and then probably check it back into GitHub just to make things confusing. But do MCP servers get, there could be the official Uber generic one developed by the platform and then somebody says, yeah, I need to make one that’s just a much narrower scope and maybe add some flavor on it or does it not work that way? 00:37:58.20 [Sam Redfern]: Yeah, great question Tim. And this is why the agent, like talking about agentic engineering and harnesses is really important because in a harness, you say, these are the MCP servers only have these tools, right? And so you can take the GitHub MCP server and you can say, here’s your three tools, deal with it. Sorry, I just, you know, let’s not get into the accuracy of AI overviews, but according to in June 2025, apparently exposes 51 plus tools. Okay. No, okay. You know, like, but this is like, if you’re in a world where you Do you narrow that down to a very limited set of the tools? And you can see this in the cloud desktop thing. If you load an NCP server on the cloud desktop, you have little switches where you can turn on and off tools in a sense. The intention of these systems is that you narrow the scope down to exactly the problem that you’re working on and just that. But what’s interesting, Tim, is why bother having the GitHub? If you’re doing the coding yourself and you’re using it inside a cursor or something along those lines, it’s like, why bother adding the GitHub MCP server at all? Why not just get the LLM to execute something in the command line with all the command switches of the GitHub CLI? 00:39:13.50 [Tim Wilson]: Also, is this getting us to the downsides? 00:39:18.24 [Val Kroll]: I was just going to say, he risks you a bit chomping at the bit, Sam. Let’s hear it. 00:39:24.80 [Tim Wilson]: I mean, there’s part of it feels like this is, it’s new enough and wild, wild west to know there’s like, there could be a governance issue that it would be very easy to embed MCPs into an organization and they’re not well thought out. They’re not well built. They, they, I mean, that just, it seems like there’s just a governance thing like anything that gets rolled out. that one Sam creates something and all of a sudden the entire organization is dependent on it. And maybe Sam’s not very good at, you know, or just half-assed it on a weekend or something like, I don’t know. So that’s my, I’m throwing that out that it seems like there’s a governance risk when you’re being, you’re able to do this stuff so quickly and roll it out. Is that one of the downsides? 00:40:13.13 [Sam Redfern]: I mean, building, and this is something Canva does really well, right? Like building AI tools for people to use in the application is just really different to building stuff internally. Like we are still at the early days of this stuff and Canva’s ecosystem team building out really strong solutions to the space of whether canvas empty server or client or something like those lines. So there was like the professional teams who are like building this sort of stuff for external consumption. So in open AI, you can, you know, you can. use the LLM to interact with it. You can use GPT-5 or whatever they call it these days to interact with Canva and modify your designs and stuff like that. That’s all using this style of tool technology in a sense, right? And there’s a lot of governance that’s been there, right? There’s a lot of thinking about permissioning, thinking about what information we’re giving to the LLM, what actions we’re giving to it, what are the actual actions that change something. One of the downsides of giving the LLM access to your terminal command line is that it could just delete all the files in the directory or something on the inside. I think my favorite one is where an engineer is trying to get the LLM to write the code so it passes all the tests and so it solves the problem by deleting the tests and it’s just like problem solved, I’m done. a technically correct, the best type of correct, but actually, no, that’s not what I wanted. And so this is why that agent harness framework is really useful because that’s where we’re like, here is this domain of a problem. Here is this very finance set of tools. Here’s how I want you to sort of exactly work on this particular part of the problem. And I don’t want you to have this like long chain where you’re sort of like jumping between things. I just want to create the agents of instance, have it solve one or two problems, and for the agent instance to end, so then we can move on to the next problem. That is why we’re trying to solve and harness this non-deterministic nature. Some of the criticisms of this MTP standard is like, one, it’s not a standard, right? Like a standard, if we think about it as from the Internet Enduring Task Force, the W3C is like a collection of for-profit companies coming together and sending some of their best engineers to basically have like a very disgruntled call with a bunch of other engineers from other sort of for-profit companies, right? Like I don’t, as far as I understand, it’s kind of just anthropic building this internally, polishing stuff on their blog, they picked up FastMTP, which is like an open source thing. And they just said, all right, this is the standard. We’re going to use this as our standard library and sort of extend it out and stuff like that. you know, coming back to that governance is like, this is not a structure that feels like it’s ready for like, you, it’s buyer beware on the internal corporate governance stuff. And the way you design these systems is really important. And so some of the data tools I build internally, it has no ability to write information to the database, right? Because that is completely like, we’re just not ready for that kind of world, right? And maybe in really select kind of instances where there’s a really strong, hard-to-surrounder and you have like a checking endpoint and all sorts of other stuff like that and things like that as well, things like Langchain are really useful. But we need to, that’s why I think most of the value of this space is still in the internal application use cases, in a sense. That’s where you can do a more experimentation and worry less about the strangeness of the internet and the internet and AI and all the problems attached to that. But when you’re developing these tools internally, you have a team of you know, a handful of engineers and like, you can make their lives like tangibly better because they don’t have to put context into their mind at a particular part of the problem. And they can just have that answer come back. And even if it’s right 90% of the time, it’s probably better than when you got your junior data scientists sort of do it in the first place anyway. So, you know, that’s that’s the like, that’s the challenge. As far as One of the other criticisms of Anthropox MCP is that it doesn’t have authentication baked in. Moest of the MCP, so it’s counterintuitive having this term of MCP server and client in a sense, right? And what happens is you’re literally running a little application. You’re like, you know, Python run this like Damon or something along those lines. And then that is just talking to the it’s like running a local web server in a sense, right? And that is the security model that has been sold for in the early days. And that’s why the Zed has their ACP agent. What is it Zed? ACP. I think in a couple of years, we’re going to be talking about the agent client protocol as maybe a better way of building this sort of stuff. Everyone sort of agrees that the ACP protocol is probably a better representation of where we’re going in this space. And it is an open standard. I don’t think they’re sort of like the W3C or the internet engineering password style of standard. Back to that XML example, I don’t know about you. I don’t read a lot of XML these days. We’re going to be moving to something else, but I’m very bullish on the concept of tool use in these applications and giving large language models these fingers to do things. 00:45:53.98 [Tim Wilson]: Is there any movement? It sounds like Zed has stepped in and done a little bit of this. Is there any movement to say, like the W3C had its different groups and came together that we should get? 00:46:08.82 [Val Kroll]: Didn’t Google and Microsoft and OpenAI, didn’t they all adopt it? Am I totally misunderstanding what that means? 00:46:16.54 [Tim Wilson]: that it’s another thing to say, here’s our, here’s our, we got to solve authentication. We got to have a recommendation and a standard for how authentication is going to be handled means they can’t just say, like it’s not there. That’s something that new that needs to be incorporated in a way that they say, yeah, we think this is generally going to work for most. We can all work with this, right? Cause that’s, I mean, it’s not, it’s not a static. I mean, I guess let me ask that question. It’s MCP. How static is it? Like when HTML came out, it wasn’t like, okay, we’re done. Well, there was a bunch of other stuff that was needed and browsers added functionality. And so it was kind of, it naturally had to evolve and is MCP the same thing that it needs to, yeah. 00:47:04.94 [Sam Redfern]: Yeah. So there’s a really interesting part of this, which is that there is a, I think there is a recommended output format to MCP servers that as part of their standard. But I think what’s interesting about this is that like because of the non-deterministic nature of these systems. And because you can, I don’t know if you’ve ever played around with this, but it’s always good fun is to, you can have the start of your question in XML, then you can do the middle bit in YAML and then the end bit in JSON. And the large language model doesn’t skip a beat. And it’s just like, oh yeah, sure, I understand this, right? Like it doesn’t matter as much is kind of the context of this problem, right? Because We required these standards in the early days of the internet because they were purely deterministic systems with incredibly strong grammars. I just don’t think it matters as much anymore. That’s why I don’t think there’s been the same pressure to standardize because you don’t need to standardize in the same way. The only thing that matters is do you pass the tool call threshold in your large language model. And I think it may be, you know, like rather, rather than a very deliberate standard like TCP IP or IPv6, I think it’s going to be more along the lines of the QWERTY keyboard, which is like, we just kind of picked it because it was there first, not because it’s better. and MCP will probably change to something else in the future, right? But the primitives that make it interact with the large language model, I think, are now baked in enough that it would be surprised to see if we move away from that. And all we’re going to do is we’re going to find new ways of taking those primitives and doing this code execution thing. So I gave my example of this charting sort of like MCP extension that I built. Like all we’re going to do is we’re going to take the same primitives, but then we’re going to like do wildly different things with them that people didn’t think was possible before. 00:49:06.08 [Tim Wilson]: And at some point, that will have shifted to a point that it’s got a new label. And it’s like, oh, remember, it was just MCPs. Now we have something else, which is grounded in all that we learned from MCP. Okay. 00:49:17.86 [Sam Redfern]: Yeah. Now it’s going to be ACP. It’s going to be, who knows, whatever. I look forward to watching this name change over time. I’m sure there will be an XKCD comic at some point. There’s the end pop on standard XKCD comic, and we are not immune from that paradigm, which has been true in software for long enough. 00:49:40.98 [Tim Wilson]: I’ve brought up that particular strip, I think, on two of the last four episodes. One was on semantic layers and one was on 00:49:48.77 [Michael Helbling]: Yeah, we’ll check back in in six months because certainly things will have shifted quite a bit. All right, we do have to start to wrap up. And as we do that, let me jump into a quick break with our friend Michael Kaminsky from Recast, the Media Mix Moedeling and GeoLift platform helping teams forecast accurately and make better decisions. Michael’s been sharing bite-sized marketing lessons over the past few months to help you measure smarter. Over to you, Michael. 00:50:17.39 [Michael Kaminsky (Recast)]: When we perform statistical analysis of data, what we really care about is that we are discovering actual truths about the world, not random artifacts of the particular data set we’re looking at or the analytical methods we’re choosing. We want generalizable analyses, the kind where independent researchers answering the same question would converge on similar results. This is all another way of talking about a hugely important idea in model building or statistical analysis. Robustness. Without robustness, even a small tweak at assumptions or small changes of the data will spit out dramatically different results. Results that aren’t showing true causation or reflecting reality, but just picking up random noise. So how do we put this into practice when doing statistical analyses? We can randomly resample from our dataset or even randomly drop small amounts of data and see if the results are being driven by one particular outlier observation. Similarly, if we’re running a regression analysis with control variables, we can check how sensitive the results are to different control combinations. If the findings change dramatically depending on which controls we include, we should be skeptical of the overall results. The more robust our results are as things change, the more we feel confident that other analysts or researchers will end up drawing the same conclusions and the better chance we have of finding some underlying truth. 00:51:31.49 [Michael Helbling]: Thanks, Michael. And for those who haven’t heard, our friends at ReCast just launched their new incrementality testing platform, GeoLift, by ReCast. It’s a simple, powerful way for marketing and data teams to measure the true impact of their advertising spend. And even better, you can use it completely free for six months. Just visit www.getrecast.com slash geolift to start your trial today. All right. Well, one of the things we’d like to do is go around the horn and share a last call, something that might be of interest to our users. Sam, you’re our guest. Do you have a last call you’d like to share? 00:52:06.23 [Sam Redfern]: Well, I mean, obviously, thinking about this sort of this agentic engineering thing. OK, so I’m going to get in trouble by doing two. One of them is go to z.dev and go read about and always a problem. Go to z.dev. and go check out under resources and they have their agentic engineering series about the future of software development. I think it’s a great grounding of where we’re going in this industry and I think they lay out a really great vision of what this could be. The last one is that I don’t actually like Zed’s agent. I think one of the most important things here is to go get your hands dirty with these systems. They are just so much fun. And if you’re a bit of an old techie, it doesn’t matter as much about the grammar anymore and really just spend some money on tokens and explore it. And in that vein, I actually think the best agent you can get for nothing is OpenCode. And so I think it’s opencode.ai. Yeah. OpenCode, I think, right now is one of the best agent harnesses that you can possibly go and build things. They’ve got really interesting things like the ability to define subagents that you can give different prompts and contacts to. And so if you want a really great base agent to play around with, to go and then build really interesting harnesses, can’t recommend the OpenCode thing enough. Nice. Thank you. 00:53:29.31 [Michael Helbling]: All right, Val, what about you? What’s your last call? 00:53:32.87 [Val Kroll]: So mine’s a total left turn. But I have just been really enjoying lately the Good Hang podcast with Amy Poehler. It’s been around not quite a year yet, I don’t think. But if you are just in the need of a good laugh, I am telling you, you will walk away from those with a stomach ache. The Rachel Dratch episode, I legit did a spit take. It is So funny. So anyways, she just has like lots of different celebrities on to talk about all different topics and it’s quite enjoyable. So does she have an MCP server? No, I’m saying keeping it light. We’re keeping it light. Yeah, that’s great. You need that. 00:54:18.51 [Michael Helbling]: All right, Tim, what about you? What’s your last call? 00:54:23.62 [Tim Wilson]: So I’m going to do kind of a mix of like the human side of things, just because we’re starting off the year. So now hopefully people are looking forward to what human people they’re going to go see in various places like in-person conferences. So I will plug that I am, I’m getting to return to Super Week this year, which I have missed for the last couple of years. And that’s a missed in-person and in Spirit, so superweek.hu. It’s February 2nd through 6th in Budapest. And then I’m going to double that up with just a couple of good follows. I feel like we need more humor on LinkedIn. And there are two guys who are both very reliably putting in just short, random, funny things, and also some good content. So I’m going to plug Bov Patel, Bovick Patel, and Manas, Dada, DA, TTA. He does all sorts of like something like finger guns, but every time he does something, he has a different sort of something guns at the end of it. So they’re just good follows to put a little less bloviating in your LinkedIn feed would be those two guys. And what did they teach you about B2B sales there, Tim? And they definitely make cracks about that along the way. 00:55:47.29 [Michael Helbling]: What about you, Michael? What’s your last call? Well, I’ll be curious, Tim, to hear whether or not MCP servers come up at Super Week, which I’m sure they will. My last call is AI related. I just was hanging out with my good friend Christopher Berry a week or so ago, and he turned me on to a paper that some folks wrote about how to jailbreak large language models, because sometimes you just need it to give you the recipe for gunpowder or something. And apparently, a really great way to do that is just talk to it in poetry. So, if you add a poem, it will just tell it back to you as a poem and give you the information you want. No questions asked. So, not saying you should do that, but that’s something you should be aware of. We’ll link the paper in the show notes. So, can I sneak in with one last thing? 00:56:37.69 [Sam Redfern]: Yeah, of course. It’s related to jailbreaking large-language models. There’s a community inside of Canberra, a channel where people share their tips and tricks for interacting with these large-language models. There was a thread on how do you get better outputs. Yelling at large-language models, surprisingly works, bribing large-language models surprisingly works. My favorite one is to tell the agent that a much smarter and more sophisticated agent is about to come and check its work. and it should hurry up and make sure that there’s no mistakes before it gets checked. 00:57:12.16 [Tim Wilson]: I so wanted to know that there was some Moe related threat in there that you’d be like, look, if you get this wrong, Moe kiss is going to be disappointed. And they’re like, that, oh my God, that is the ultimate hack. 00:57:25.77 [Michael Helbling]: Add some weights to that name inside of all the large language models inside of Canva. That’s probably a good idea. All right, Sam, this has been outstanding. Thank you so much for taking the time to come on the show. Talk about this topic. This has been great. Thank you very much for having me. It’s been a great time. Yeah, no, and I’m sure our listeners of which there are many will have a lot of questions or things like that. We’d love to hear from you. You can reach out to us on our LinkedIn page or through the Measures Slack chat group or via email at contact at analyticshour.io. as you’re listening or listening to this episode, also leave reviews and ratings. We like to get those as well on whatever platform you listen on. If you want, we also still have some stickers, then Tim will send them to you if you request a sticker over on analyticshour.io. Reach out to us that way. Awesome. Really great. I think this is a more technical topic, but I think it’s still very relevant to everybody in the data space because of the sort of the intersection of AI and data. It’s sort of a thing we’re all talking about. So, Sam, thank you for helping demythologize some of this, if you will, and bring some practical knowledge. I think it’s a huge service. And like you said, it’s changing every day. So, you know, apologies in advance for how outdated this podcast will be in about three weeks, but that’s just the way it works. You got to get started somewhere. The AI nodes take Christmas off. That’s right. Stop updating your LLMs for crying out loud. Yeah. The big news today was that Sam Altman issued a code red for open AI because Gemini is doing so well and they’ve got to get back to getting hard work done. So, okay. Yeah. 00:59:14.15 [Tim Wilson]: Stop this 996 nonsense, right? 00:59:16.81 [Michael Helbling]: Yeah, right. We need some work-life balance. Take a break. Before the AIs take our jobs, we need some work-life balance. No, I’m just kidding. All right. I know that as you go out there and you’re working with data and you’re trying to use AI, it’s always complex and challenging and you’re learning a lot. It feels like the early days of analytics all over again. But I know I speak for both of my co-hosts, Tim and Val, when I say, no matter which MCP you’re using, don’t forget to keep analyzing. 00:59:43.59 [Announcer]: Thanks for listening. Let’s keep the conversation going with your comments, suggestions, and questions on Twitter at @analyticshour on the web at analyticshour.io, our LinkedIn group, and the Measure Chat Slack group. Music for the podcast by Josh Crowhurst. Those smart guys wanted to fit in, so they made up a term called analytics. Analytics don’t work. 01:00:08.09 [Charles Barkley]: Do the analytics say go for it, no matter who’s going for it? So if you and I were on the field, the analytics say go for it. It’s the stupidest, laziest, lamest thing I’ve ever heard for reasoning in competition. 01:00:28.43 [Tim Wilson]: Rock flag and non-determinism. The post #288: Our LLM Suggested We Chat about MCP. Kinda’ Meta, No? appeared first on The Analytics Power Hour: Data and Analytics Podcast.

It’s the most…won…derful…tiiiiime…of the year! And by that, we mean it’s the time of the year when we sit back, look at each other, and ask, “Where did all the time go?!” We brought back a very special someone for this episode as we collectively reflected on the year—show highlights (and what about those shows have stuck with us), industry reflections, and a little shameless shilling for Tim’s book (are you still short on a few stocking stuffers? Order now…!). This episode’s Measurement Bite from show sponsor Recast is a brief explanation of Granger causality (and how it’s NOT actually a causal measure!) from Michael Kaminsky! This episode is also brought to you by Prism from Ask-Y—your agentic analytics platform for automating analytics, exploring data, creating repeatable workflows, and delivering accurate insights—all without the need for manual query writing. Links to Resources Mentioned in the Show #263: Analytics the Right Way (Tim’s book) Analytics the Right Way #272: When the Metric is Calculated and Complex with Dan McCarthy Dan McCarthy’s SoundCloud #282: Using (and Creating!) Data to Understand Pop Culture with Chris Dalla Riva (Book) Uncharted Territory: What Numbers Tell Us About the Biggest Hit Songs and Ourselves Jólabókaflóð #286: Metrics Layers. Data Dictionaries. Maybe It’s All Semantic (Layers)? With Cindi Howson #276: BI is Dead! Long Live BI! With Colin Zima #285: Our Prior Is That Many Analysts Are Confounded by Bayesian Statistics #277: ANOVA? I Hardly Know Ya’! with Chelsea Parlett #268: You Get an Insight! And YOU Get an Insight! with Chris Kocek #076: Insights, Please. Actionable Ones! With Rod Jacka #281: Analytics: The View from the Corner Office with Anna Lee Jeff Bezos’s explain his famous one-way door and two-way door decision making Photo by Vladyslav Tobolenko on Unsplash Episode Transcript00:00:05.76 [Announcer]: Welcome to the Analytics Power Hour. Analytics topics covered conversationally and sometimes with explicit language. 00:00:15.19 [Michael Helbling]: Hey everybody, welcome. It’s the Analytics Power Hour and this is episode 287. Ho, ho, ho, holy shit. Another year is basically over. 2025, I mean, it never even had a chance to slow down and decompress, it feels like. I mean, we’re just running a break next beat, finding out about AI, doing our work, trying to do everything. But regardless, we’re going to try to take a look back and maybe a small peek forward. That’s the analytics power hour year in review episode. And so with no more ado, it’s time to introduce my awesome co-hosts, Moee Kisss, Director of Data Science for Marketing at Canva. How you going? 00:01:00.54 [Moe Kiss]: I’m going pretty good. But yeah, 2025, that was a time. 00:01:04.37 [Michael Helbling]: It felt fast. 00:01:06.67 [Moe Kiss]: Big year. 00:01:07.88 [Michael Helbling]: Big year, I agree. Tim Wilson, Head of Solutions and facts & feelings. Do you agree? Hello. Hello. Hello. Hello. Hello. Quite a year. Yeah. Val Coroll, head of Deliverate facts & feelings. How’s your year going? Gone. 00:01:27.76 [Val Kroll]: Lots of feelings. There were lots of feelings. 00:01:29.79 [Michael Helbling]: Yeah. I agree. And of course, we are missing Julie Hoyer as she enjoys some time off with her new baby. And so we look forward to her coming back next year. 00:01:42.29 [Tim Wilson]: So her year is going sleep deprived, right? Yeah, that’s right. 00:01:48.33 [Michael Helbling]: And of course, as a special treat, we’ve got Josh Crowhurst, Growth Marketing Director at Immanuel Life as our special guest this episode. Welcome back, Josh. 00:02:01.21 [Josh Crowhurst]: Hey, yes, great to be here. 00:02:02.81 [Michael Helbling]: You know, I don’t know Josh if our listeners actually, many of them know the story of how you became involved with the podcast in the first place. So if you don’t mind, I’d like to take a second and just tell people how that happened. 00:02:20.66 [Tim Wilson]: I thought it was gonna be like the 2025 and how you stormed away. Like, keep it as the year in review. 00:02:24.95 [Michael Helbling]: Well, I mean, it’s part of the year in review that Josh finally had to step back from his role with the podcast. So we’re actually really glad that you did rejoin for this one last episode for year in review, which is our tradition. And, you know, if you’re up for it, come back next year. We don’t care, but yeah, Josh stopped being involved. Nicely put, Michael. No, I’m just saying, it’ll be fun. It’s not pressure, it’s up to you. You got a lot going on in life. But no, early 2019, Tim and I were working out how to make the show better, and we thought we needed some help. And so we put out a call for a producer. It was a poorly written job description, one that we did not fully understand. And then- Did Tim fully understood it just to be clear? Well, in terms of like what it would take to do and what we were looking for and all those things, it was just very much like a shot in the dark. To our surprise and delight, we got a response from Josh Crowhurst. And after chatting with him a few months later, because I forgot about the email and didn’t look at it for a while, Josh joined the show as our producer and was with us for, I believe, six years, which is incredible. It’s so amazing. And so now that life has taken Josh in a new direction and he’s growing, he’s obviously stepping into bigger and bigger roles. And it’s so cool to see how your life and career has just flourished. And I like to think maybe I mean, I don’t think that, I don’t know. Anyways. You didn’t even do an audio production at all, yeah. 00:04:10.21 [Josh Crowhurst]: I couldn’t even… All thanks to you, Alves. All thanks to you. 00:04:14.12 [Michael Helbling]: No, not me personally. Just the analytics power hour generally. benefited your career in some way. I’d love to think that, but probably it did. Absolutely. Anyway, we appreciate it and we’re happy that you are able to join us for this episode. Okay, what we do on all these episodes in your review, we like to look back at the year that just went past. We did a lot of shows. We did a lot of interesting shows. We like to talk about some of them, highlight some of our favorite episodes, maybe chat about some of the things that happened this year. So who wants to kick us off? What’s an episode that really stands out for you? 00:04:49.04 [Val Kroll]: Well, obviously we started off our year strong. No show would be complete without Tim Wilson kicking off our year. with the announcement of Analytics the Right Way, episode 263. So that was a big, we were all so excited to see that come to life. And it was super fun to be a part of that episode, since I had the pleasure of working with Dr. Joe Sutherland. And that was just a really fun, big moment, like diving into all of the big themes of the book. But that was the first one of the year, was it? It feels like if it was not for a second. Yeah, starting strong. That was a good one. 00:05:32.77 [Moe Kiss]: I still hope I know about missing that one. 00:05:36.19 [Val Kroll]: We did fight to figure always get to be on it. Yeah. 00:05:40.08 [Tim Wilson]: Well, as other people have released books this year, I realized what a kind of a shit job of ongoing, rolling thunder, you know, promotion of the book. But I was in it for the writing of the book, and I figured it was going to be downhill. Once he showed up on the analytics power hours, a guest, why would there need to be any other promotion? The old APH bump, we like to call it. Yep. Clearly. Dozens, dozens of books flew off the shelf. 00:06:09.47 [Moe Kiss]: I have bought six alone, so I am definitely helping the supplies go out the door. 00:06:15.08 [Val Kroll]: Were those some of your stocking stuffers, Moe, for friends and family? 00:06:21.41 [Tim Wilson]: Folks, it’s not too late. If you’re listening now, you can… That’s right. 00:06:28.07 [Michael Helbling]: for that special someone in your life. The e-book version. 00:06:35.09 [Val Kroll]: Use code APHBump for 10% off. 00:06:39.76 [Michael Helbling]: Don’t say that. Oh, man. Well, I’m glad there’s no other episodes to talk about. Yeah, that was really the one. Let’s talk about that one more. 00:06:54.49 [Michael Helbling]: That was the one. All right. Listen, I have an episode. Here’s the thing. Okay. When we do this podcast, this is the thing I do with a lot of things. When I interview people, when I work with people, when I talk to people, I’m always looking for where their passion lies, what sparks. kind of what makes their eyes light up. And one of our episodes that I really enjoyed and it was a person I’d wanted to get on the show for a long time was Dan McCarthy, which we did episode 272 about calculated and complex metrics. It was a really fun conversation and Dan is so smart and so amazing in his role as a professor in studying these companies and the metrics they produce, especially for public reporting, for stock reporting purposes. But what was amazing was the passion he has for these topics through music. And he has a sound cloud with all these songs on it. And it was sort of after the show was over that he kind of started in on it. But that was sort of where I saw the switch kind of flip into this is fun and up a little bit of light in his eyes about that kind of thing. And I’m sure obviously he enjoys his other work too. But it was just really cool to kind of connect with In the coolest way possible, another data nerd about things they loved about their work and about data. Anyway, so that was just a moment that kind of stood out to me. As far as being a really educational and fun episode, it was just so cool to watch somebody’s eyes light up about things they were passionate about. 00:08:31.86 [Moe Kiss]: I learned so much on that episode and I even probably like a week ago sent it to someone to have a listen. The number of times I get questions about LTV2CAC and like why finance and public companies are like so interested in that specific metric and how it’s calculated. I’m just like, here is a show that I prepared earlier. Please peruse at your own leisure. And I just loved how he did such a wonderful job of really getting into the, I guess, the different perspectives and the complexities that we sometimes face as data folks in a metric that its surface might seem really simple and obvious, but actually can really change a business decision or a perspective of a business by how it’s calculated and how it’s interpreted. And also just to say, like his SoundCloud, the number of data show and tells that I’ve opened with one of those songs, and people are always like, Moe, where do you get these data songs? I’m like, blah. I know people. I know people. So yeah, I definitely had that in my top couple of episode list as well. 00:09:46.52 [Tim Wilson]: Well, that was like my finding him. So I now like see more of his stuff. And he made the point on that episode, and then he kind of continues to make it that when companies stop reporting stuff, it’s not usually for… Sometimes that’s informative. Yeah, that in and of itself. And there’s some kind of hand-waving as to why. And he’s like, but another way to look at it would be, here’s this thing I wrote two years ago that indicated this might be problematic. So yeah, he was a fun one. 00:10:21.38 [Josh Crowhurst]: So on the topic of things that people are passionate about, I think one of the episodes that I absolutely loved and maybe is a bit in line with something that I’m really passionate about was number 282, using and creating data to understand pop culture with Chris Della Riva. So for me, this was honestly probably my favorite episode ever. because it’s like so it’s so right up my alley like it’s in my backyard like it’s like he’s talking about looking up writing credits and production credits on songs and tracking that and this is something that I just do just impulsively like I’m always annoying my friends with pointless surprising facts about songs that Like, did you know Bruno Mars co-wrote Forget You or like, I don’t know, Mark Ronson produced and wrote that song from A Star Is Born? Like, just like shit like that. I’m just always, I’m always looking behind and saying, like, who’s involved in that song? And the idea that there are just people behind the scenes that maybe don’t have mainstream name recognition in a lot of cases, but have really shaped what you’re hearing on the radio or on Spotify for, you know, sometimes for decades. And so, yeah, Chris talks about tracking that and having that in a data set. And I wish I could get my hands on that data because I would absolutely just be pouring up for it. Oh, it’s there. It’s on the show facts. 00:11:52.90 [Tim Wilson]: You can. It’s on the it’s on the show notes page. Oh, my God. 00:11:56.80 [Josh Crowhurst]: Yeah, we found out. I’m out. Yeah. Okay. 00:12:03.33 [Michael Helbling]: I’m diving. Josh liner notes. Crowhurst. 00:12:10.54 [Tim Wilson]: And Michael, you really enjoyed recording that show. Is that, is that right, Michael? You know what? Thank you so much, Tim, for bringing up a sore point. 00:12:20.45 [Michael Helbling]: I just find it hilarious after 11 years of you basically being like, I don’t know anything about pop culture. Like you record that episode instead of me, like come on. 00:12:35.15 [Tim Wilson]: I read his newsletter. No, that was a fair, fair. Anyways, it was. 00:12:39.71 [Val Kroll]: We’re gonna have to rename the show, Year in Review and Erring of Grievances. 00:12:44.32 [Michael Helbling]: This is right. It’s the Festivus Erring of Grievances. 00:12:48.97 [Tim Wilson]: Which Chris’s book is now out. It was not out when we recorded the, but it is. So also, if you’re like somebody, love someone so much that you want to get them analyzed the right way, and a second book that Uncharted territory is now available at booksellers near you. Still available by Boxing Day, probably. 00:13:15.89 [Moe Kiss]: Did you guys have Boxing Day? 00:13:17.75 [Michael Helbling]: No, but it’s the day after Christmas, so you have one more day, so maybe it’ll shift the time. 00:13:23.72 [Tim Wilson]: I don’t know. Everybody has some pretentious neighbor who celebrates Boxing Day, so they can explain to you what it is. Boxing Day is awesome. 00:13:30.31 [Moe Kiss]: You have leftover food and none of the pressure of Christmas Day. 00:13:33.83 [Tim Wilson]: Right. Now, imagine that coming out of an American who’s just explaining how sophisticated they are. 00:13:39.56 [Michael Helbling]: Well, I obviously, with these book recommendations, I would think you’d be talking about Holavoka Flaude. So maybe that’s the holiday. What? Not familiar. Sorry. And it’s an Icelandic holiday where you read books right before Christmas. So there you go. 00:13:56.83 [Moe Kiss]: I was about to say, should I, like, pivot us in a totally different direction and talk about the elephant in the room? 00:14:02.22 [Michael Helbling]: Oh, yeah. I mean… What? 00:14:06.41 [Moe Kiss]: How many episodes you reckon AI came up in? Oh, damn it. I should have actually been prepared. Hold on. And I kept transcripts or some shit. That would have been a good idea. 00:14:16.95 [Val Kroll]: Yeah, use your librarian thing, Michael. 00:14:20.26 [Michael Helbling]: Yeah, well, we don’t have every episode uploaded yet. So it’s still a working process. But thank you, Val, for bringing that up, because it’s an AI project that Tim and I are working on. But I’ve got to say, Moe, it probably came up in probably 75% of our episodes. 00:14:37.03 [Moe Kiss]: You reckon 75%? Everyone put in a guess. I would say maybe higher. 00:14:41.97 [Tim Wilson]: No, I think I’d go 70. I mean, I’m counting. 00:14:48.92 [Val Kroll]: between one, whether it was a topic or it just came up. If it just came up or last calls. 00:14:55.63 [Josh Crowhurst]: Do last calls count? They do in my head. 00:14:57.83 [Val Kroll]: That’s why I got to my number. 00:15:00.86 [Michael Helbling]: I mean, there’s at least 10 episodes that have AI in the title. I’m going to say 90%. 00:15:07.19 [Moe Kiss]: Yeah, it was a lot. Let’s leave everyone hanging and not we can report back in a future day. 00:15:12.81 [Michael Helbling]: That’s right. Guess how many jelly beans are in the AI jar? 00:15:17.87 [Tim Wilson]: So I tried to go on record that I did not commit to that it will be reported out at some future date. So I think the likelihood of that happening is. 00:15:25.73 [Val Kroll]: If any of our listeners want to figure it out, sound off in the comments. 00:15:30.94 [Michael Helbling]: If only we had a producer who could go back through. You know, Tim, as we click champagne glasses on another successful year of the podcast, I think our listeners would agree that you and I almost always agree on things. 00:15:50.15 [Tim Wilson]: What? Absolutely not. I spend half or most of my time on this show, I think, just correcting your misguided thinking. 00:15:57.95 [Michael Helbling]: Well, agree to disagree. But there is one thing we both agree on. AI is starting to reshape our industry. And I think we both call bullshit on nonsense like vibe analytics. Absolutely fucking right. But here’s the flip side. Analysts do have to start using AI. Leveraging LLMs to multiplier capabilities isn’t just interesting anymore. It’s going to be table stakes in 2026. 00:16:22.30 [Tim Wilson]: Which is why I’m actually excited about our new sponsor, Ask Why. Yes, it’s an AI tool, but it’s one where analysts can do real work. And critically, Ask Why is smart about data privacy. They do not send your raw data to the LLM. Right. 00:16:38.86 [Michael Helbling]: Ask Why builds a semantic layer on top of your data and then uses that to generate SQL that answers your questions or helps you build reports on your own data set. It’s currently in beta and it’s evolving fast, but you get the upside of AI and the assurance that your data stays secure. You can actually start leveling up into being an AI analyst, starting with Ask Why. 00:17:00.00 [Tim Wilson]: For a limited time, use the code APH when you join the waitlist, and our friends at Ask Why will move you right to the top of that list. The site is ask-y.ai. 00:17:11.86 [Michael Helbling]: That’s ask-y.ai. So go sign up for the waitlist using code APH. 00:17:19.61 [Tim Wilson]: This isn’t Vibe Analytics. This is the rise of the AI analyst. 00:17:23.82 [Michael Helbling]: All right, let’s get back to the show. Yeah, it is interesting because it certainly, I mean, Moe, I think the point you’re making is like AI was everywhere and always here all year long in 2025. And it seemed to grow in speed and pace throughout the year. 00:17:44.46 [Val Kroll]: Yeah, definitely a topic that came up in the listener survey is people wanting to, wanting it covered, wanting some topics covered there. So I think that creeped into our schedule, informed in 00:17:56.23 [Tim Wilson]: And as my other hat as the fielder of the inbound pitches for show topics, I can certainly say that that percentage was definitely north of 75%. But is it fair to say, and maybe this is my normally optimistic self that you guys are so familiar with, that at the start of the year, the ratio of AI hype to AI specifically in the world of data and analytics, that it was like north of 90% of the AI hype excitement to the, wait a minute guys, it’s not going to be everything in that it’s slowly gotten a little bit more in balance just as the conversation in the zeitgeist around what AI can and can’t do as people have gotten their hands on it and realized limitations or is that me? 00:18:56.78 [Moe Kiss]: I think that’s fair. 00:18:58.12 [Michael Helbling]: It has come back a little. I still think we’re a little out over our skis, though somewhat in terms of AI. I mean, just AI in general, like a lot of people think we’re in a bubble. By the time this comes out, hopefully the stock market hasn’t crashed or anything, but that’s always a thing that people are talking about. It’s like, oh, is this all a bubble? And like the.com boom and bust, kind of an idea. 00:19:23.90 [Val Kroll]: I think it’s like with any trendy thing, it’s like cool to think of all the use cases and all the potential. And then the cool thing is to be like, but you can’t do this, can’t do that. So like, I feel like we’re in that phase of like the LinkedIn. Like I just get so tired, you know? 00:19:39.51 [Michael Helbling]: It’s like the 50th time you hear they not like us. And you’re like, no. 00:19:46.52 [Tim Wilson]: I did see a thing where somebody- That was good, Michael. 00:19:49.97 [Val Kroll]: That was good, Michael. 00:19:54.66 [Tim Wilson]: I read a piece that was saying that instead of a bubble, think of it as a forest fire, which it actually has a lot of bubble tendencies. Well, but it talks about, even if you go back to the internet, the original, the 2000 internet bubble, that it was pointing out that it’s like the bubble burst and it’s not like you’re back where you started. There are Players that were sufficiently hardy and had actually a plan that they, they were like the big trees that actually managed to weather it. And they’re like, yeah, Google, Apple, Microsoft, they’re not going anywhere if the bubble bursts. And then it talked about the ones that are basically just the thin veneer of crap that those are just going to disappear, but that it’s also that correction when it comes, there will be a smarter universe out there and there will be little shoots that come out of it that have can kind of, I don’t know how they refer to it as little green shoots that will crop up. Once all that sort of gets cleared out. It seemed like a useful metaphor. Involved metaphor. But I also find it’s crazy like just having conversations with normies and sort of where the person who’s not kind of has some responsibility to figure it out, how much they’re not, there isn’t real depth of thought. They’re just, I had a friend say, I just use ChatGPT instead of Google search now. And I was like, I don’t have the energy to say you could just use Google search and it would be Jim and I like, if you just want plain text results, and that’s kind of the extent of what they’re doing. although I could also go on some rants as well. 00:21:44.55 [Michael Helbling]: You know, it was interesting to me this year when I would go to different events and like conferences or things like that and see the pace. Like, I remember going to measure camp New York in the spring. And of course, everyone was talking about AI this, AI that, and it was all kind of like, wow, look at all this cool stuff. And then literally from then to the fall and measure camp Chicago, I felt like, we’d already gone through a maturity curve almost with the way we’re discussing AI and some of its use cases. It just seemed like we’re just blasting through the cycle really fast, feels like sometimes. Some places, there’s still quite a bit of hype, but I do think some people are getting their feet on the ground and starting to use it for actual things and starting to understand how to leverage or how to think through use cases effectively. 00:22:39.09 [Moe Kiss]: So that was literally the thing that has been on my mind when I was looking at the episodes that were my favorite. It’s probably recency bias, but they were definitely the ones towards the end of the year. Well, I suppose they weren’t all the end of the year, but like the semantic layer episodes, I thought the topics on BI with Colin were really good and then also loved the one on Bayesian stats with Michael Kaminsky. But part of me wondered, I just felt like, There was this return to us discussing. I want to say quote unquote the basics, but it’s not basics. It’s the fundamentals of data stuff. And is the reason we were discussing that is because like everyone’s trying to go so fast on AI. There was this like not reckoning, but like acknowledgement that to do that well. I don’t want to be like the usual shit of data, bad data in, bad data out, blah, blah, blah, that sort of crap. But I felt like I’ve been giving a lot of thought and energy. And I feel like folks in the industry are about the quality and how we do things well and how we measure if the output is good. And that, of its nature, means we have to have more sophisticated conversations about fundamental data concepts. And I felt like there was a return to that. And maybe that’s Similar to what you’re talking about Michael where like there was kind of a bit of a rush and then people are like Having more sophisticated discussions is probably a good summary. 00:24:09.18 [Michael Helbling]: Yeah. No, I like that framing because I think that’s exactly right. It’s sort of like The early thing I saw was like, well, your own expertise drives results in AI all the time. But it’s sort of like, OK, if you go down to some brass tacks about how to conduct analysis, how to think about data lineage, how to think about traceability, all the things that we teachers are taught as analysts to be able to compose an analysis correctly, follow it through correctly, and deliver out the other side, those are all steps we learned as analysts. And so AI is a part of that process now, but we still have to maintain all of those parts along the way, it feels like. Does that, I don’t know. And maybe AI will get so good, it can do all those steps for us at some point, but I just don’t think there’s ever gonna be any time in the near future, like a black box appropriate approach to analysis. which don’t get me started on the topic of vibe analytics, which is the most stupidest thing I’ve ever heard of in my life. 00:25:16.94 [Moe Kiss]: Well, I think we need to do a spin-off episode on that because I disagree. 00:25:20.29 [Michael Helbling]: Well, it’s probably definitional or semantically, we’re probably in agreement, but yeah, we can probably do a whole show on it. Well, what other? 00:25:32.08 [Josh Crowhurst]: This is something that I’ve also noticed, I think is kind of related is that that using AI, it really drives home to me that you really have to, especially as a manager, you need to have your critical thinking skills switched on because things will start to come up produced by, I mean, especially more junior people in their careers that are I guess more A.I. native will be using this and might at some times skip some of the steps in producing an analysis and they’ll come up with something that sounds really logical. But maybe, you know, they had a conclusion in mind that they punched it into chat GPT and worked backwards at arriving on some logic to present an idea that maybe hasn’t been fully thought through. So this is something where I think we have to be super, super aware of it, right? That there’s a lot of, I guess, convincing sounding bullshit, where if you To go one layer deeper like the thinking just isn’t isn’t there so coming back to the idea of having the fundamentals, but also just being aware that you know, this is this is around us all the time and try to Try to really focus on You know is the logic sound I mean, I think that’s That is there are 00:27:06.46 [Tim Wilson]: when it gets used as, this is something that I don’t enjoy doing, AI gets put out there as being, oh, the grunt and tedious work that you do, AI can do that. Now, I think that’s an overinflation, like how many people are literally sitting there saying, I do monotonous, tedious, repetitive work day in and day out, and no one has come out with a way to streamline that. So this monotonous tedious work gets conflated with, this is work that I don’t really enjoy or I have to kind of think about it. I hate summarizing meetings that are all over the place. Oh, look, Zoom will just record and summarize for me. And it’s like, well, you may hate doing that, but you’re missing what sort of value you should be adding along the way. And I think the same thing goes for If you think that the goal is to get a slide deck produced that looks plausible, then you’re missing what analysis is. There is stuff that is supposed to be hard and that you are having to think through it with that structure as you go. 00:28:22.21 [Michael Helbling]: Yeah, I want to step aside for a quick second and take a quick break with our friend Michael Kaminski from ReCast, the Media Mix Marketing and Geolift platform helping teams forecast accurately and make better decisions. Michael’s sharing bite-sized marketing science lessons over the coming months to help you measure smarter. Over to you, Michael. 00:28:45.82 [Michael Kaminsky (Recast)]: Granger causality might be the worst-named concept in analytics. What you need to know is that Granger causality does not demonstrate causality. Just because some variable passes a Granger check does not mean that it causes some other variable. What Granger causality actually shows is predictive ability. Effectively, the check is looking to see if past values of x can predict y. better than past values of why alone. As an example, let’s imagine we have two time series. One is the time that a rooster crows every morning, and the second is the time of the sunrise. By just eyeballing the data, we can see that the rooster crows consistently a bit before sunrise. Yet, a Granger causality test would conclude that rooster crows Granger cause the sun to come up every morning. The problem is really in the name. It confuses analysts and especially business stakeholders who, understandably, assume that a Granger causality test actually checks for causality. Here’s what to remember, Granger Causality only tests whether one variable proceeds and helps predict another. It says nothing about whether one actually causes the other. 00:29:45.81 [Michael Helbling]: Thanks, Michael. And for those who haven’t heard, our friends at ReCast just launched their new incrementality testing platform, GeoLift by ReCast. It’s a simple, powerful way for marketing and data teams to measure the true impact of their advertising spend. And even better, you can use it completely free for six months, just visit getrecast.com slash geolift to start your trial today. Okay, well, let’s talk about shows we liked maybe that didn’t always touch or didn’t touch fully on AI. What are some topics we liked this year that weren’t necessarily in the AI wheelhouse? And kind of Moee, this is coming off of you talking about sort of this fundamentals kind of an idea. 00:30:32.31 [Val Kroll]: One of the ones that I had FOMO for not being on was the ANOVA, A Hardly Know Ya, with Chelsea. Oh, that was so good. That one was so good. I mean, she’s just a joy. But I don’t know if you guys remember, but she’s one of the things that you guys started with on the episode is that she had a poem. pre-CHAT GPT times Twitter feed poem about ANOVA, which I loved. But she was just so thoughtful in the way that she was describing and getting into all the inner workings and the comparisons with ANCOVA and MANOVA. She’s like, at the end of the day, it’s linear aggression all the way down. And I thought, you guys did a really nice job probing with some really good questions that were very thoughtful from real life experiences that I’d thought. made that episode really good. I’ve definitely listened to that one more than once this year, but that was really fun. It was an easy listen, even though it’s a complex topic. 00:31:28.24 [Tim Wilson]: I’ll throw in the episode 268. You get an insight, and you get an insight with Chris Kocek, which was, I would say, very not AI, because it was so much about a human being pulling things from different directions. And that wasn’t the first. We had Rod Jacka on years, Jacka, Jacka. Chaka, on years ago to talk about what is an insight. So I feel like that’s a perpetual question in our industry. And there are certainly a million AI-powered tools that are like, it’ll find insights for you. And to me, that was like that episode, Chris is not an analytics person. He is coming from much more of a creative and messaging and branding background and getting his perspective on what the many, many facets and the inherently human nature of trying to get some deeper understanding about something. I thought it was a pretty nice corrective to the AI hype. I really liked how he defined an insight. 00:32:36.58 [Michael Helbling]: You know, another one of my favorite episodes, and Moe, you mentioned this one as well, was the one with Michael Kaminski about Bayesian statistics. I think throughout my career, I’ve learned things sort of just sort of by arriving at them, not necessarily being officially trained in them or those kinds of things, just because of how I started in analytics and how I kind of grew into the field. And it was sort of this really big light bulb moment to sort of realize like, wow, the way that I actually approached this stuff is literally what we talked about in that episode. And sort of, for the first time, kind of slammed together in my mind, like made the connection finally like, oh, that’s Bayesian statistics. So it’s just so funny. Yeah, I know what that is conceptually, like, oh, it’s your priors, blah, blah, blah. But as a model for actually doing stuff in the real world, I hadn’t really said, like, oh, I’m Bayesian in the way that I think about that. 00:33:32.67 [Moe Kiss]: It’s funny because I think one of my tendencies, and I always say this to my team, is that I oversimplify things. And I think that’s just part of my role, right, is I’m often trying to communicate something really complex to a leadership team. But I think one of the things that I really loved about that episode is in my mind, I think I had maybe perhaps oversimplified what I understood about Bayesian stats, and Michael brought a level of new depth to the topic that really add a lot of value to me personally. 00:34:04.86 [Michael Helbling]: Yeah. I really liked it. It actually was super applicable. I was literally sitting down with a client not long after we recorded that, and I was able to walk them through a process they could follow where they were in a situation where a frequentist approach would not have worked well. in that context, and I was like, well, here’s some other alternatives. We could actually do something like this, and it actually worked really well. But it’s funny because I probably would have still suggested that, but now I could actually call it what it was, as opposed to being like, I’ve got an idea. Try this. It probably has a name. I just don’t know it. Anyway, it was just really cool to connect the dots on that for me this year. 00:34:51.33 [Val Kroll]: All right, one of the other ones that I’ll throw out there, another recent one that we did was 268, the metrics layers, data dictionaries, maybe it’s all semantic layers with Cindy Hausen. So I have to admit full transparency when we were in our planning for that, I’m like, is that really a whole episode? I’m like, I don’t know. Okay, I’m not on it so I feel, but holy shit. Yes, it was a whole episode because it was with Cindy and it was really, really well done. I love that one so much. 00:35:23.20 [Michael Helbling]: Val, Tim and I both will tell you like we’ve gone into certain episodes over the years and been like, I don’t know about this. And it turns out to be amazing. So like a lot of times a little bit of doubt is almost like an indicator that like something good might happen here. 00:35:38.02 [Moe Kiss]: But also, I think the fact is that Cindy herself is such an experienced data practitioner, has such a depth of knowledge about the technologies and the topic we’re talking about. I mean, I could talk about semantic layers for hours, which I have done with Cindy from time to time. But I think that episode was really strong and really Yeah, semantic layers is a hot topic at the moment. Lots of folks are building things. There’s a DBT product, a Snowflake product. There’s a bunch of similar products that are built into BI tools. It’s a very timely episode, as well, given how quickly things are moving in the industry or maybe, I don’t know, maybe not quickly because we’re like trying to catch up. But Cindy, I think Cindy was just such a wonderful guest for that specific episode and probably is one of my favorites as well. 00:36:31.43 [Tim Wilson]: And the fact that she made the point that one, they’re not new, and two, thinking of it as one monolithic thing, I was like, those were like two. That was big. Very like, ah, this has gotten the label of this is the grand new thing, just roll it out. And I was like, it is the fact that she is very, very politely really fucking annoyed with the cycle of the latest shiny thing being treated as like, this is the thing, the answer. So Josh, you were going to say something. 00:37:06.43 [Josh Crowhurst]: Yeah, a recent one that I particularly enjoyed was 281 analytics, the view from the corner office with Annalie. Yeah, great episode. And I think we were talking about trying to get this like finding the right guess for this idea for Years, maybe? It was a long time. Yeah, that was when I think we were trying to put together for a long time. So when I saw that on my Spotify feed, I was like, oh, I have to listen to this right away. And it was it was worth the wait, for sure. And for me, it really it resonated maybe partly due to some perhaps slightly traumatic recent experiences in my previous company where I had exposure to senior leadership. A few of the things that she talked about like were really sharp. I thought talking about like setting a culture of productive curiosity. I love the term because yeah, I did see it first hand, you know, you’d be in a meeting and the CEO would make an offhand comment. And then people would just spend an inordinate time digging into that, like whatever, whatever the ask was, because, you know, the CEO said it, like I have to, you know, I have to, I have to do this. It might not be something that’s worth spending hours or days looking into. We would come back in the next meeting and the CEO wouldn’t necessarily even remember making the comment. So I kind of learned to level set in the meeting before going and saying, hey, we’re going to look into this. This is the amount of time we’re probably going to spend on it and just sort of set that. Just get that out there before leaving the room. But what Anna said was she She talks about having a level of precision that’s necessary and sufficient for the importance of the decision that’s being made. And then having the self-awareness as a leader to specify that. And then save the team some of the bandwidth. So as an analyst, when you’re in there, if it’s not clear, just state it and get it out in the open and get the alignment. But I love Anna’s perspective that taking ownership as a leader Realizing what you say, people might just take it and run with it and spend a ton of time and you didn’t necessarily intend it that way. So I loved that framing. And then I just thought, yeah, a really thoughtful perspective on what a data-driven culture can look like and how it can be established and driven from the executive level. And just one last. 00:39:40.45 [Moe Kiss]: That is the specific bit that really sung to me was how much responsibility she took as the leader for that culture versus assuming that your data scientists are responsible for the data culture alone. That was one that really stood out. 00:39:59.30 [Josh Crowhurst]: Yeah. No, it made me, I was like, I want to work there. She has such a great way of framing it and thinking about it and communicating her vision on how data can be used and should be used and then setting the example. It was really inspiring, honestly. And then one last thing that resonated, again, going back to my PTSD. But yeah, brief your analysts, right? If you want them to be set up to succeed the first time they’re presenting to the CEO, I’ll say that maybe didn’t happen for me. I might have been passed in front of the whole company group executive committee as a result of that. So please, I don’t think that a vandalist, please do that. That’s great advice. Prevent any, uh, any traumatic pantsings of your, of your team when they’re in a room with the big dogs. 00:40:53.50 [Val Kroll]: Poor Josh. Yeah. The thing that also struck me about that conversation was, I don’t think she realizes how novel her perspective is. Like, she was like, oh, she’s like, of course that that’s what leaders do. I’m like, I was like, can you say that in some of your circles? Like, I was like, where’s the, where’s the link to your jobs posting? I think I even said that, Josh. I was like, hopefully your last call is that you’re hiring. This is awesome. But yeah, no, that was a good one. 00:41:23.59 [Michael Helbling]: That was one of my favorites too, Josh, because it was in a way so affirming of a thing I’ve really come to start believing more and more is that leadership drives data culture more than the data team does. And as the only way to really drive a data-rich culture or data-informed culture in a company is if the leadership is doing it. Because even when you take on the role as a data leader in your company, you can’t force people to become data-driven. They either are, they aren’t. But if the CEO is saying it, well, that makes it a different thing altogether. But yeah, that was a great episode. And yeah, it was a long time coming. That was in our like, every year we’d have that in our list of like, yeah, we got to find somebody that could do justice to this topic. And as analytics people were always thinking like, yeah, what do they think about, you know, when they’re sitting as the CEO, what’s their perspective on data? Do they care? Do they look at these charts and graphs? Like that’s a question I think our whole audience thinks about. Anyways, Anna was amazing. That was 00:42:32.29 [Tim Wilson]: Yeah. Years ago, we had someone who agreed and was ready to come on and then ghosted us like completely. So it was like, yeah. Oh, I forgot all about that. Talk to him. Talk to him later. It was, it turned out his company was like in the midst of, it was about to get acquired. He was like, yeah, I really needed to go darling. I’m like, I don’t know that a email response of like, Hey, actually this isn’t a great time would have, you know, been too problematic, but I don’t know. So yeah, I agree. 00:43:02.61 [Michael Helbling]: Well, what trends are shaping the next year, Moe? God, I don’t know. 00:43:12.78 [Moe Kiss]: I think he’s… I’ve just obviously gone through lots of 2026 planning and thinking about the year ahead. It sounds so boring, but if I had to boil it down to a couple of key things that I’m really thinking about, it is about consistency and making sure that We have really solid consistency in metric definitions and how metrics are calculated and all those sorts. It just sounds boring, but I feel like it’s becoming more important than ever. I think the other thing that I’m spending a lot of time thinking about is, I don’t know, we’re all using AI just for internal efficiency gains and it just feels shit. If you’re using it to write a better email or a Slack message, it doesn’t feel like that is how we can be getting the best from some of these tools. And so thinking a lot more about specifically like the data products we make and how we can better automate. I’ll give you a specific example, which is going to sound really lame. It’s going to sound stupid and lame, but this is the exact thing. We used to keep a list of dashboards, like your top company dashboards. When someone on boards, you can be like, you want to know about this topic or this topic or this topic, you go here. It’s a manual list, it’s paid in the ass, it always ends up outdated, not maintained. I was like, That is a problem where we should be solving with technology, right? And I think that’s probably why I’m so hyper-focused on consistency and all the fundamentals. Because if you want to throw technology, how do we maintain this list without needing someone to go manually update some spreadsheet or whatever it is? How do you understand which your dashboards are being used, which are high value, which are going to answer the right question? To do that, the data that you’re using to build a technological solution has to be very good quality. But yeah, those are just the things that are on my mind going into 2026. Oh, Tim looks for you. 00:45:22.80 [Tim Wilson]: Well, I mean, I believe back on the fundamentals that there still is. It is so easy to get caught up and they were going to keep measure, measure, measure, measure, measure and the complexity kind of explodes. And Moe, you were at a very large massive amount of data digital native company. I have even in the last two weeks have had an experience with a massive company that their issue was much more around internal alignment on what different teams were trying to accomplish. and not the data. Every time the data people would come in, it was just kind of like puking out charts of stuff. And you could see that that wasn’t serving the business. I mean, there were some kind of comical ways in which the data people were very knowledgeable. The visualizations were fine. They could answer questions about the minutia, and that wasn’t remotely what the organization needed. So I think that’s not a direct response. I mean, I think my cringe a little bit like, well, let’s look at which dashboards people are looking at and which metrics like that, that to me winds up being coming up often saying, can AI come up with an engineering solution that’s just going to tell me the insight? 00:46:59.48 [Moe Kiss]: Like it’s kind of like, well, let’s just… No, I don’t agree. I don’t agree. I think the thing Fundamentally, you and I are very aligned that it’s about the business question that you’re trying to answer, right? Like I would say that that’s, I don’t know, I’m getting like a semi-nod. One of the concerns you have is like, Are people leveraging AI to answer a question that could be answered very easily with something that’s already built? And then it comes down to like, this is more about cost efficiency, right? Like, I don’t want someone continually asking a question every day that’s costing us money to run that is sitting on a dashboard that can be easily looked at and interrogated if they just know where it is. It’s about discoverability to answer that question. And so, like, there are multiple problems that you’re trying to solve and it might just be another way to answer that business question. 00:47:59.48 [Tim Wilson]: So I would say it’s not a, I wish it was a trend of 2025, but I think the reason I was kind of having that reaction to answering business questions goes back to momentarily mounts soapbox that the definition is if somebody in the business asks this question, it’s a business question and therefore I need to answer it. How can I answer that efficiently and most effectively? And it becomes a volume play with, and so if you instead totally shifted, I think there’s a a crap ton of questions that are kind of fishing that actually point to a much more fundamental challenge. But so if trying to solve, I mean, it goes to the, and this does come to the AI companies that are like, imagine if you could just sit there with chat GPT and just ask it questions and it would provide responses. And then the pushback winds up saying, ah, but the answers don’t have, they have hallucinations or ah, without this engineering, it can’t provide accurate. And to me, I’m like, That is not the goal in-state, is to have people who aren’t thinking rigorously about what they’re trying to do and are prematurely jumping to the data. I deeply in my soul believe that that is heading down a path of just getting more people wandering through more data to have more meaningless arguments to produce more overly lengthy PowerPoint or Canva or Google Slides decks that aren’t actually moving the business forward. So it’s actually putting fuel on the fire of something that is broken in business horribly, horribly, horribly. 00:49:45.74 [Michael Helbling]: Activity without outcome, maybe. So Tim, maybe the AI product you want to see built is the one that forces more rigorous questioning by guiding people through that process. So be like, why are you asking that question? Oh, interesting, refine that. OK, you don’t really want that analysis because you wouldn’t want to mistake this for this. So maybe you want this analysis. Like, something like that would be. 00:50:12.63 [Val Kroll]: And then at the end, does it turn into like an intake for like an intake system that goes… Oh my God! No! 00:50:20.33 [Moe Kiss]: You and me, I was telepathically communicating with you being like, it kind of sounds like a Jira intake ticket. 00:50:27.98 [Michael Helbling]: She’s throwing gaslighting. That actually looked like something I was already going to say, which was at the end of episode 279, the process of analytics, we have thoughts, that episode, we were talking about that. And at the end of that episode, I was like, now, because of AI, all these processes are going to take on even more importance. And Tim jumped down my throat and said, everyone’s been important. And he wasn’t wrong. But the reality is, is like, To get to leverage AI, you have to do those precursors. To Moee’s point, that return to some of the fundamentals is the trend. Tim was wrong to do that to me on that episode. 00:51:12.03 [Val Kroll]: That’s really the point I’m making. Here’s another poll. Do we think that AI was mentioned more this year or Tim’s blood pressure raising happened more this year? 00:51:27.22 [Tim Wilson]: Wait, what was the first option? What was the first option? 00:51:30.33 [Val Kroll]: Mentions of AI versus Tim’s blood pressure, right? 00:51:32.51 [Tim Wilson]: Oh, blood pressure, yeah. Well, they’re deeply correlated, and there is causation. 00:51:37.01 [Michael Helbling]: Well, at least for Tim’s blood pressure, there’s medications for that. Oh, brother. Well, that’s one trend that will probably continue is that Tim and I will tangle up a couple of times. No, it’s fine. 00:51:52.69 [Moe Kiss]: So I’m probably going to say something again fiery. Also, just to clarify, answering a business question does not mean that we should answer every question raised by the business, just to like caveat the former discussion before I move on to the next question. 00:52:06.60 [Tim Wilson]: But if you’re making it so that they can get to whatever the question, yeah. Okay. 00:52:11.99 [Moe Kiss]: The next topic that I also think is coming up a lot, which very much ties back to the episode with Anna, is about decision velocity. I think that is something that is really, really interesting. And again, Tim makes the point, I work in a very unique position at a company that’s probably not representative of what most companies are that data folks are working in. But it’s very much, how do you use the right level of rigor for the decision that you’re trying to make as a business? And sure, maybe there’s some AI sprinkle salt on the top of that as well. 00:52:51.89 [Tim Wilson]: So I think that making that point of giving getting the business more sophisticated that what are the stakes behind the decision and therefore is a little bit of a signal very quickly because is that. better and desirable than getting a complete answer, but way too late. I think there is starting to be some awareness on the business side that if you just are waiting for the inarguable truth, you’ll just be waiting forever. Although I think there is still the tension between the teams. They can’t get answers to me fast enough. Why can’t I just have an AI? This was secondhand. Somebody said their CMO was like, I just want to have the AI just tell me, you know, give me insights while I’m in the shower in the morning. I just want to get up and have it have sifted through the data. And I’m like, okay, we still have a ways, a ways to go because that CMO is, but that’s not the same thing as decision velocity. 00:53:56.99 [Michael Helbling]: Cause I guarantee you that CMO is not, uh, doing decisions effectively and a good speed. Because they’re doing the gathering of information incorrectly to go after decision velocity. One time, somebody told me that a CEO is just a decision engine. which I thought was actually a really cool way to think about that. We think about executive leadership generally, clearing obstacles for your team and all those things. Decision velocity is a huge part, like not getting yourself bogged down. There’s lots of frameworks for that, like the old Bezos, two-way door versus one-way door, decision matrix, that kind of stuff. There’s these things that can help, but I do think you could look at AI as an enabler of helping you frame or think about speed to decision. Because one of the things my old boss, my guest, used to do, he used to force us to write down decision journals. I don’t know if you’ve ever done this before. Very time consuming and very annoying, and I was always super bad at it. It’s probably why I’m not as good at decision makers as I should be. But it helps you then go back and look at previous decisions and what led up to them and go through that. So not everything is all data analysis. Data informs some decisions and so it would look greater or lesser extent. But to the extent that if I had to use this data, I might have made a better decision. If you’re evaluating your decision capabilities, and I think AI is really well suited to helping you remember some of those things as over time as well. So that could be another way to leverage AI in that context maybe. 00:55:44.57 [Tim Wilson]: Go faster, be smarter. So I think this is your opportunity, Michael, to make a decision to bring the show to a close. You know, 00:55:56.01 [Michael Helbling]: It’s about that time, Tim. It’s hard because I don’t want to because of two reasons. Because we’ve got Josh on the show and I don’t want it to end. And so that’s one part. And then the second part is, it’s the end of 2025. This is our last episode of the year. 00:56:15.13 [Moe Kiss]: Let’s get on with 2026. I am ready for it. 00:56:18.37 [Michael Helbling]: Moe is ready. All right. Let’s shut the door. So we’re done. Thank you all. As you’ve been listening, maybe you have a memory of 2025 you want to share. We would love to hear from you. Or what are you looking forward to in 2026? Same thing. Reach out to us. You can comment to us at our LinkedIn page or on the Measure Slack chat group. or via email at contact at analyticshour.io. We’d love to hear from you. And obviously, thank you, Josh. No show would be complete without thanking you for coming back to be one more special guest one more time. This is fun. 00:56:56.80 [Josh Crowhurst]: Thanks for having me, guys. 00:56:58.98 [Michael Helbling]: Yeah, it’s awesome. It’s awesome. We do. We do. I think you’re still in our Slack. I don’t know if you’ve just abandoned that Slack at all, or you’re still kind of peeking from time to time. 00:57:12.17 [Josh Crowhurst]: Oh, it’s still Slack, but I do still get the emails. Oh, okay. I don’t get analytics hours. Oh, gosh. Oh, yeah. I see those new ideas and suggestions coming through. 00:57:22.53 [Michael Helbling]: I’ll remove you from the email list, I guess, so that you don’t keep getting those. Yeah. Well, we didn’t really have a process for that. So under GDPR, you do have a right to be forgotten, but I don’t want to. All right, and if you listen to the show, leave a rating and review. If you’ve been listening throughout 2025, go to your favorite platform, give us a review, rate the show. That helps other people discover it. And we’ve had a lot of audience growth this year, both on our regular channels and on our YouTube channel. So if you’re ever on YouTube, subscribe to us there as well. We put every episode up on our YouTube channel, as well as some awesome shorts that the team puts together for each episode. don’t know what we’re gonna put together out of this one, but we’ll see. And then of course, for all of my co-hosts, I think I speak for everybody when I say, 2026 is gonna be an amazing year, but no matter what it brings, You know that you can always keep analyzing. 00:58:28.60 [Announcer]: Thanks for listening. Let’s keep the conversation going with your comments, suggestions, and questions on Twitter at @analyticshour on the web at analyticshour.io, our LinkedIn group, and the Measure Chat Slack group. Music for the podcast by Josh Grohurst. So smart guys want to fit in. So they made up a term called analytics. Analytics don’t work. 00:58:53.70 [Charles Barkley]: Do the analytics say go for it, no matter who’s going for it? So if you and I were on the field, the analytics say go for it. It’s the stupidest, laziest, lamest thing I’ve ever heard for reasoning in competition. 00:59:06.71 [Michael Helbling]: I nearly, Josh, on the last episode, did a no-show-it-be-complete without a huge thank you to Josh Norris. 00:59:13.74 [Michael Helbling]: And I switched it. And I switched it at the last second. No-show-it-be-complete without cheap analyzing. 00:59:25.73 [Michael Helbling]: Yes, I know how I did it. 00:59:29.63 [Michael Helbling]: I did a huge thank you door. Yeah. And it’s just a hard cut. No joke, you complete with that. He’s analyzing. 00:59:43.12 [Josh Crowhurst]: Anyway. Yeah, I still need to see music, so I feel like I can see the original setup. 00:59:48.84 [Michael Helbling]: Yeah, you’re in there. 00:59:57.93 [Tim Wilson]: rock flag and let’s raise a glass with tim and mo with michael julie vile five hosts who guide us through the noise and make the numbers ten oh for all our power ours friends for all our power hours. We’ll toast the laughs and insight shared in all those power hours. Voice crack should have picked a different key on that one. That is awesome. That has to be it. That has to be it. That has to be it. That’s the best one we’ve ever done. The post #287: 2025 Year in Review appeared first on The Analytics Power Hour: Data and Analytics Podcast.

Semantic layers are having something of a moment, but they’re not actually new as a concept. Ever since the first database table was designed with cryptic field names that no business user could possibly understand, there’s been a need for some form of mapping and translation. Should every company be considering employing a semantic layer? Is the idea of a single, comprehensive semantic layer within an organization a monolithic concept that is doomed to fail? These questions and more get bandied about on this episode, where we were joined by industry legend Cindi Howson, Chief Data & AI Strategy Officer at Thoughtspot. This episode’s Measurement Bite from show sponsor Recast is an explanation of multicollinearity from Michael Kaminsky! Links to Resources Mentioned in the Show Claudia Imhoff Jill Dyché TDWI Snowflake’s Open Semantic Interchange OBI EE (Comic) On standards: this classic xkcd strip (Post) The metrics layer has growing up to do by Amit Prakash Sonny Rivera (Podcast) The Data Chief Podcast (Substack) we have the data Pro Bono Analytics from INFORMS Photo by Vitaly Gariev on Unsplash Episode Transcript00:00:05.76 [Announcer]: Welcome to the Analytics Power Hour. Analytics topics covered conversationally and sometimes with explicit language. 00:00:14.64 [Michael Helbling]: Hey everybody, it’s the Analytics Power Hour. This is episode 286. It’s Tuesday and I know what you’re thinking. I sure hope revenue and active customers still mean the same thing as they did yesterday. A lot of you know firsthand the pain I’m describing. There’s data ping-pongs around the business taking on shapes and definitions that were never really intended. Well, the semantic layer was supposed to take care of all that. And to be fair, there are some nice, tidy businesses out there doing a great job. But most of us are still trying to figure out where it should live, what it should be written with, and who should own it. So I think we should dig into it. But first, let me introduce my co-host, Moee Kisss. How are you going? 00:00:58.27 [Moe Kiss]: I’m going great. I’m really excited about this. 00:01:01.08 [Michael Helbling]: I’m excited to, and excited to do the show with you. And Tim Wilson, howdy. I think it’s just all semantics. It’s all, oh, so good. Well, that’s an interesting potential cop out. Okay, no. And I’m like, well, to really get into this topic, I think we found the perfect guest. Cindi Howson is the Chief Data and AI Strategy Officer at ThoughtSpot. She was previously Vice President at Gartner, along with many other distinguished roles throughout her career. She is the host of the Data Chief podcast and has authored many books on BI and data. And today she is our guest. Welcome to the show, Cindi. 00:01:45.36 [Cindi Howson]: Thank you for having me, everyone. I’m so excited to be here. 00:01:49.17 [Michael Helbling]: I am excited that you’re here too. You’re kind of, to me, sort of an OG of the data space. And so I love people who can provide as much depth and background and historical perspective on all the things we’re struggling with in the world of data today that were struggles 20 years ago and still remain with us today, but with different tools and names and things like that. But today we’re talking about sort of 00:02:16.97 [Tim Wilson]: Oh, go ahead, Tim. Well, I was going to say, I mean, I, having done my little forensic sleuthing that I saw Cindi speak at a TDWI summit back in 2004, which I think is amazing since we’re both like 35 years old. We just look at him. It was like my entry moving from technical writing to Markham, slightly into web analytics, and then it was like my entree into the world of analytics and BI was that conference. Yeah, so I just felt like I had an appetite. He’s a big fat. I’m gonna say that. 00:02:53.94 [Cindi Howson]: He’s a big fat. It’s the summary. Tim, now I feel like I have to send you an original BI scorecard black bear that I used to use as giveaways for class participation. We’ll see if I still have one. 00:03:08.12 [Tim Wilson]: Now, I couldn’t remember the topic, but I feel like I was coming back and I was like, there were some other lady who spoke and it wasn’t Jill Deshaix and the other person was like the only I cannot remember her name because we wound up actually Claudia Imhoff. Yes, like two months later called and had Claudia like come out and just spend two days explaining data warehouses to our team. So like doing the conference circuit and as a consultant, you’re like, oh yeah, nobody’s really just going to call you up. Like we literally, she’s like, what do you want me to do? And we’re like, you’re smart. Please come just sit in a room and answer our questions for two days. And it was, Our dev team was like thrilled. And I’m so glad that you can remember that was who it was. Okay. Yeah. Okay. Michael. Okay. But now we’re going to do the episode too. 00:04:00.24 [Michael Helbling]: All right. Fast forward just a few years. Yeah. So nowadays, no. So let’s talk about Star Schema. No. Well, I mean, we can start there. So semantic layers, Cindi, obviously everyone talks about those, but there’s a history here. And maybe just to get us started and for people who aren’t as familiar with the concept, maybe just a quick primer on sort of, what does that even mean? What are those? And we can kind of use that as a launching point. 00:04:31.55 [Cindi Howson]: Sure. So in the simplest terms, a semantic layer provides a representation of the business model in business terms to the physical structures in your whatever, data warehouse, data lake, cloud data platform, whatever you want to call it. And it is important that it is in business terms. So if I think about, my German language has only served me well when I look at SAP original tables. VBAP was the customer table in SAP R3. So you could never show a field VBAP to a business person. Instead, you would say, this is customer name. 00:05:21.95 [Moe Kiss]: And I keep hearing the word when people talk about semantic layers of context. And that seems like when you say business terms, is that what you mean? Like the context of the data, how it relates to each other, what the definition is. Is that the same thing? Or when you say business terms, are you thinking about something different? 00:05:42.42 [Cindi Howson]: No, I think it’s both because if we get precise, something like revenue, well, revenue in an inventory and supply chain context, I’m going to look at revenue based on when somebody placed an order. But if I take it in terms of finance, the Office of Finance, they’re going to want to know when that invoice was paid, or if I’m doing, you know, if I’m a cash basis or a cruel basis. So the context of that revenue field matters. Did that answer your question, Moe? Yeah, absolutely. Okay, totally. 00:06:26.17 [Tim Wilson]: Not to head, I struggle with And revenue is a great example because it’s… part of the challenge, it’s like we’re trying to find a technology or a tool or a process to solve for something where the business, the person in finance, when they think revenue, they’re always thinking in a revenue recognition world. And when somebody’s an inventory, they’re always thinking of it another way and don’t even necessarily, and they both may complain that I see reports from the other department and they have revenue and it’s wrong. It becomes a, business understanding challenge that data processing technique mechanism is trying to solve. Or am I just being cynical about that? 00:07:15.78 [Cindi Howson]: No, it is. I could get annoying and say, yes, it is semantics, Tim. But this is where, let’s say, and sometimes people conflate data literacy with technical literacy, which is wrong. But we’re really talking about what does the data mean in a business context? And where does the data originate from? And if I’m talking about an order system versus an invoicing system, sometimes that’s different. And so a finance person is always going to assume I am talking about when it was invoiced. A salesperson is going to come from the context of when is my commission going to get paid. And so we come to the data already thinking about the data through our own lens, our own business function. And yet they may have very, very different meanings. Even somebody that I was working with I won’t name him, but it was hysterical. We’re both working off the same dataset you would have thought. I’m like, why are your numbers different than mine? Here’s what I thought the number was, and you’re coming up with a different number. Yet, in his dataset, he only included software licensing and did not include professional services. I was like, why would you exclude that? I’m really just looking for total revenues related to this particular segment. 00:09:01.05 [Moe Kiss]: Can I then ask, do we risk? I’m going to definitely come full circle on this because it’s definitely been a topic that’s on my mind a lot. One of the things that I’ve seen play out is this very precise, I don’t want to say business domain, but this very specific interpretation of a metric by a particular area of the business. And I’m going to give the typical example in my world, which is like, let’s say you have 12 different products. And so then one team is like, well, we’re going to talk about video MAUs. And another team is like, we’re going to talk about search MAUs. And then another area of the business is, I don’t know, template MAUs. I’m making up all stuff that’s relevant to my world, of course. And then we come up with this fundamental problem of if you summed every department’s version of their metric, we would never end up with all MAUs. But we also end up with these very precise definitions that might work within the business context that they’re in. But then like, I think the thing that I struggle from that viewpoint is sometimes I feel like we over orchestrate things for a specific domain. Then we can’t roll up and think about what’s the bigger picture across the whole company. When we say, MAU, what do we mean? Because they might have had interactions with lots of different products, for example. I feel like semantic layers in this are like there is an overlap here. I’m sure we’ll get to. But do you see that problem playing out a lot? And is that part of why semantic layers are becoming the new hot topic of the moment? 00:10:38.61 [Cindi Howson]: Well, let’s go back and say, you just used a term, Moe. If I was a new employee at Canva or at any kind of SaaS startup, what the heck is it? 00:10:48.20 [Moe Kiss]: Is it MAU? 00:10:48.60 [Cindi Howson]: Or what the hell? 00:10:50.04 [Moe Kiss]: Oh, fair. 00:10:50.44 [Cindi Howson]: A male. A male active user. Well, and maybe even really going to split hairs here and say, well, if I only clicked on the video and so it was a one-second interaction, are you going to count me there as a user or should I have actually watched at least two minutes of the content? So we can parse these definitions a lot of different ways. But I want to come back to why did semantic layers start more than 30 years ago and why are they coming back now? 30 years ago, prior to semantic layers and really business objects patented and won in courts, the first semantic layer and Cognos at the time within Promptu had to actually pay a license fee to them. And prior to this, you had to code your own SQL. You would have to say some cryptic name, by VBAP L333 from this table, and that was terrible. The semantic layer gave report writers a way to click on business terminology to generate the SQL. That was the first purpose of the semantic layer. Now, as the industry moved to, let’s say, in memory tools, With the likes of Click and Tableau, there’s a whole generation of let’s say 10 years, maybe 15 years, where people didn’t think about this. They just loaded their data, did maybe one big SQL extract. loaded it into an in-memory file. And so they were only working with their subset of data. And so, of course, the MAUs meant what I wanted it to mean. And there was this loss of knowledge about what semantic layers are. Now, here we are in 2025, and we’re all trying to build agentic AI systems. And what we’re learning is that without this context or clear business definitions, we have hallucinations. We get incorrect results. So the more context you give the LLM, the more accurate your answers will be. And that is why I think, well, I think semantic layers have become more important because of agentic AI. But also, let’s say before that, cloud data platforms and the whole modern data stack have given rise to, hey, I don’t have to subset my data. I don’t have to load just a small data set into an in-memory engine. Let me get to all of it, whether it’s in Snowflake or Databricks or Google BigQuery. Let me get to all of it. And so people don’t want to move the data, but they do want to trust it, no matter if they’re doing agentic or not. 00:14:09.83 [Tim Wilson]: Well, so this whole notion of context and using agentic AI as an example, is it moving down the path? Will a semantic layer help AI demand some explicit context? If I ask for, tell me how many customers we had last month, will a semantic layer start to say, that’s not enough? I know, Tim, what role you’re in, I can guess what your definition of a customer is, but I’m going to require that you give me more business context in order for me to find the right to pull the right information. Is that a feature of the semantic layer or is that something that’s got to be built in the intermediary tool that’s using the semantic layer to interface with a business user? 00:15:08.27 [Cindi Howson]: Yeah, I follow you, Tim. And this is where I think what people want is one semantic layer to rule them all. And I just think that’s a fallacy. Will I ever see that? I don’t ever see that. What the industry, at least right now, is trying to get to, and I will also say this is the second attempt, maybe the third attempt in the industry. With snowflakes open semantic interchange, is it least let there be a common set of standards so that everyone can interoperate? And that already would be a huge sea change. Otherwise, everyone’s building proprietary integrations. Even, I mean, I will say, working for ThoughtSpot. ThoughtSpot integrates with the Looker metrics layer in LookML. ThoughtSpot integrates with the DBT semantic layer, and that has changed different incarnations. There’s a few others. that some have built integrations with KubeJS, some have built integrations with AtScale, there’s others. But let’s just take those. Well, those are all point solutions. We have to keep up with what is DBT’s latest protocol, what is Looker’s latest protocol. And it would be great if we all just say, here are the approaches that we’re going to use. And so it’s all common rather than point solutions. So that is the vision and the hope of snowflakes open semantic interchange. However, so this is a very long-winded answer, but we will have separate incarnations. And that, I have to say, like every customer conversation I’ve had about this in the last month, they’re like, we only have to have one instantiation. And I’m like, no, you don’t, you do need separate instantiations because every downstream tool and even backend database, they have their own limitations. So if I’m going to create something, a metric called top 10 customers, well, there’s some databases that don’t support a ranking function. So even like Denoto virtualization tried to do this for a while. And it’s like, great. In ThoughtSpot, we have an object called top 10. Well, if I hit the Snowflake database, it’s working. If on the back end it’s hitting, I’m going to forget which database didn’t support it, some variation of SQL server or whatever didn’t support it. Well, then Denoto is like not working, not giving an answer. Or in Looker, we have a very cool visualization, my favorite visualization, a KPI chart. it’s too complicated for the looker metrics layer. So you’re always going to have these separate instantiations of a semantic layer because nobody is going to want to dumb down their semantic layer for the least common denominator. Okay. 00:18:39.39 [Moe Kiss]: I’ve got to make sure I’m following this though. Okay. So what we’re saying is that I guess the thing is like what I’m observing is that folks seem to want to be pulling their semantic layer further and further up in the chain, right? So like you want it to sit less in a downstream tool and more like internally and I obviously have a biased view. But wanting to bring things like semantic layer in-house so that you also have your options open about which way you go with whatever AI you choose to leverage. But what you’re saying is it’s unavoidable that we’re going to end up with a semantic sandwich or cake or whatever you want to call it, where you might have to have something at one layer. And then when you go to a BI tool or some other type of tool or integration, you might end up having to have a second layer just because they have different features or attributes that you want to leverage. Am I hearing that correctly? 00:19:36.60 [Cindi Howson]: Yes. And if by downstream and upstream you mean the database, people want it closer to the database because that’s where the data lives. But then as you get closer to the business decisions, you’re going to have derivations and metrics and context that may not exist in the database. And I would also say, we also have to think about how these things get defined. So working with one team, they’re like, okay, we’re going to build everything out in the database. I’m like, great. So your DBA is going to do all this. Or here I have a really strong SME. And if we bring data mesh operating principles and domain ownership, so I have this great marketing person and they know the differences between a video, MAU, or a web click MAU. And I’m gonna want them to add a little more context to it. So I’m going to want an easier interface. And guess what? That interface does not exist in the one that was designed for the DBA. 00:20:53.38 [Moe Kiss]: OK, but I don’t want to take things in a totally different direction, but I’m like dying a little bit. 00:21:00.96 [Cindi Howson]: I feel like a dymo. 00:21:03.42 [Moe Kiss]: I opened another can of worms, I can tell. The thing I’m really struggling with with this whole discussion about semantic layers It doesn’t feel new, and I feel like what you’ve written about it makes that very clear. But part of me is also really grappling with, is it actually the fact that it’s not net new? Is it the fact that the way we want to use agentic AI on top of our data? Or is it the fact that we have gone towards this data mesh approach with less, I don’t know if structured is the right word. Cindi, you can definitely insert better terminology because you are the queen of exceptional terminology. But we used to have such structured datasets. We had store schemas, they had context. Is part of this just our own doing because we wanted to move faster and have less structure in our data? And so this is just the consequence. 00:22:04.24 [Cindi Howson]: So that was a two-part question. So is it new? Is the semantic layer new? It’s not new. It has gotten more robust over time. And not all semantic layers are created equal. So I can show you one semantic layer, and it only supports a single star schema. Or maybe even worse, it only supports one big table. I can show you another semantic layer and it supports multiple fact tables, different design approaches, star schema, snowflake schema. It even includes capabilities for aggregate table navigation or query compilation so that the most efficient query path is taken. So not all semantic layers are created equal. And I do think that has changed over time. And for sure, the openness has changed over time. So if I go back to the original query tools, whether again, business objects, Cognos, whatever, those were largely closed. Some boutique consultancies had open APIs to access them. OBI EE, their model was open and nobody used it. You could expose it as an ODBC connector to other BI tools, but nobody used it. Performance was not good. So what we have now is definitely more openness, but I do believe it is the agentic part of why we’re demanding, why we need them more. It’ll just make AI better. The second part of your question was then, are we decentralizing these things? And yes, I think that’s part of it too. 00:24:03.60 [Tim Wilson]: This makes me feel like there are This is either going to be just so obvious that it’s dumb. Are there people out there who, if some generic person came in and looked at it, they would say, you have built a wonderful semantic layer, and the people who built it would say, I built something that functioned for what I need. I didn’t know that’s what it was called. On the flip side, that has me thinking that semantic layer, it sounds cool. It gets treated as this binary. If you have one, things are good. If you don’t have one, they’re bad. It sounds like what you’re saying is, you could try to boil the ocean with one grand semantic layer, and it would probably be bad. We treat it as though there’s this label, and if you have it, then things are fixed. But there’s always the gradations of whether you do it well, well-architected and appropriately. That probably happens with everything that gets a fancy new label. 00:25:07.99 [Cindi Howson]: Yeah. Yeah. So I don’t, I don’t know, Tim, like, do I want one mega semantic layer? Oh, please not. So because it becomes overwhelming to maintain and, and it becomes now maybe, maybe if I’m just using natural language, um, to ask questions, I don’t care what it’s hitting on the back end, but I, I, I would be skeptical that that would work. There is a belief that in the industry we’re going to go towards verticalization of some of these semantic layers. So there will be And maybe this is, I kind of bristle it. We throw new terms out there, ontologies. Well, can we just talk about domains? That makes more sense to me, and that aligns with the data mesh. But could we have an insurance industry semantic layer? Could we have a marketing web analytics semantic layer? I think we could. I think we could. We would get to common metrics. The physical pointers ultimately back to which table is it hitting might change a little bit, but I think that business representation, we could possibly get to that. 00:26:43.49 [Moe Kiss]: Okay, one thought, just asking for a friend, of course, that has been on my mind is we can approach this from like a business domain perspective, just like the examples you gave, right? So like, you might have one that’s more like marketing and acquisition, more the one that’s like, I don’t know, finance or whatever, and so whatever else business domain. And the thing that I kind of keep wrestling with though is, Are we just doing this again where we’re overlaying our thoughts about like what a domain is versus the business user and how they want to interact with data? And what I mean by that is like, if I’m a business user, what’s my business question? What are the questions that I want to ask? And let’s say the theme might be I want to ask a question about our users or I want to ask a question about I don’t know. Now I’m going to like struggle to think of a comparative example. I might want to ask something about, someone help me with an answer. A marketing channel. A marketing channel. Sure. A marketing channel or, yeah, like I want to understand something about how experiments have done. Like, are we doing a thing where we’re trying to make semantic layer is representative of business domains that make maybe business sense, but don’t reflect the way that users and our business users want to interact with data when they have questions. 00:28:15.26 [Cindi Howson]: Well, so to me, if you build a semantic layer that doesn’t work that way, what is the point? Like, go home. Because you know SQL, you want to code your SQL, you don’t need a semantic layer. You might want it for some reusability, but the semantic layer gives the business user the ability to ask the questions without knowing SQL. And then it gives the LLM more context to generate better SQL. So all these companies that have tried to do text to SQL without a semantic layer, they’re largely failing. And guess what? They’re adding semantic layers so that they work. So semantic layers bring reusability. That was the original purpose. And then it is a business-friendly interface. And now in agentic AI, it’s the context for the LLMs to ensure accuracy. So if you’re going to give me a semantic layer that is just a bunch of cryptic names, technical names, and it’s not giving it to me in a way that the business sees it, it’s a waste of time. It’s a poorly architected semantic layer. 00:29:37.66 [Moe Kiss]: So hypothetically, if you just took all your YAML descriptions, that probably wouldn’t be good enough because it’s been written by a data scientist in their domain for their own specific domain for use by someone who deeply understands their area. 00:29:51.28 [Cindi Howson]: Well, if they deeply understand their area, there might be a lot of useful context in there. But if it’s a lot of code, and techno babble, then I think it’s going to be less useful. 00:30:06.98 [Tim Wilson]: Back on the, I may blend two things together. The referencing snowflakes open semantic, what is it? Open semantic interchange does feel like That brings to mind the XKCD cartoon about people complaining that there are 13 different standards. We need one standard, and then the next panel is, well, now we have 14 competing standards. There does need to be a first mover or a dominant, is there a race to say, obviously, Snowflake wants to be the owner, the driver of that? I guess the same thing when you talk about verticalization, say something like digital analytics, and you’re like, let’s just have one common marketing digital analytics. Well, now you’re going to have the players in that are all going to say, yes, the way that we think about that data is the way that the industry. the effort to try to have some sense of standards not lead to self-interested competition to sort of pull the market towards whoever’s on point for defining the standard. Or maybe my third example would be the W3C. I mean, we go back 30 years trying to define what HTML is supposed to do. And Microsoft doesn’t even conform to the W3C standards because 00:31:40.40 [Cindi Howson]: Tim, you just answered that question. Microsoft would love us to revert to MDX instead of SQL for the most part. But it is true. So look at who is not part of that. effort. Was Databricks invited to the party? Was Google BigQuery invited to the party? Will they invite themselves? Will they become part of it? Standards get adopted based on who leads it, but then also who uses it and who asks for it. So that’s where when I look at how we prioritize our product strategies, we are very much listening to the customers. And sometimes we’ve gone down rabbit holes. And I’m like, why did we build that integration? So I won’t say which integrations to me were a waste of time. But some of them, I’m like, why did we do that? Because we were trying to, we thought something would have legs to it. We listened to the customer and it never really took off. And then some will change strategies. So, you know, we thought DBT’s initial effort would take off and instead then, you know, they’re on version two and now So Snowflake, hugely influential in the industry. We’re very proud to be part of the committee defining these standards, but we have to see how broadly adopted they are. The market will decide. 00:33:29.23 [Michael Helbling]: And certainly right now, AI is kind of a forcing function for the industry where maybe that hasn’t been or there hasn’t been an imperative like that for a lot of companies. Does that seem fair? 00:33:41.61 [Cindi Howson]: I think that seems fair. Yeah. And there’s more willingness to be open and to focus on where you add the value in this data to insight to action change. 00:33:59.08 [Moe Kiss]: That actually triggers an interesting thought. One of the things that I’ve observed is this push for semantic layers. I feel like it’s come out of left field. I don’t know if that’s fair or not. It just seems to have swirled very quickly. And the products maybe aren’t at a state of maturity where they need to be for what people. I almost feel like a lot of companies are building as they’re gathering requirements as customers are trying to build out with them. Do you think that’s a fair representation? Has this happened before with a particular tool that’s had to develop very quickly because of the pressure? And I feel like AI is the pressure of like, Everyone suddenly needs these semantic layers to make AI quote unquote work. Do you feel like that’s happened with the product development before or is this like a net new thing that data companies are trying to deal with where they’re trying to build at pace while customers are wanting to already leverage and use it? 00:35:00.49 [Cindi Howson]: Yeah. So I don’t want to sound like a commercial and you can edit this out afterwards. All semantic layers are not created equal. Now, fortunately, because ThoughtSpot, whether it was purposeful or luck, ThoughtSpot always generated SQL on the backend. So the semantic layer was always super robust. So, did we get lucky or was it intentional? And the cloud data warehouse and agentic AI has just helped that. Others have only just started to embark on natural language processing. and a gentek AI. And they tried to do it without a semantic layer. And that’s why now they’re dabbling in it. And they’re like, oh, it takes a lot to build this. And some of them, they’re starting out simple. You know one big table. That’s all they can handle and code based. So and I think about I think about a blog actually that our co-founder Amit Prakash wrote about four years ago, I think it was, and it was the metrics layer, which is just a subset of the semantic layer. The metrics layer has some growing up to do. And even as a former Gartner research vice president, I have to give credit to Gartner, they still say that the time to maturity for metrics layers is five to 10 years. That’s a long time. 00:36:45.52 [Tim Wilson]: Yeah. But so how unfair is this parallel to point to master data management as something that I remember having a moment that was, oh, things are getting fragmented. We need to just do an MDM initiative. And I guess to my earlier point, it was kind of a binary, if we do MDM, all these problems get solved. The companies that were already built where they had MDM under the hood anyway because they’d architected their setup well, could do MDM, the ones that had built kind of a hot mess and were then trying to just apply a whole bunch of duct tape and bailing wire to do MDM like never really got there. Is that a fair parallel or am I too much of a stretch? 00:37:45.56 [Cindi Howson]: Yeah, well, so is it a fair parallel? I would just say it’s valid. It’s valid. And I remember, so the first eight years of my life in this industry were at Dow Chemical. And we had a master data management system called INCA. I don’t even remember what it stands for. It was homegrown. And then I worked at Deloitte, and I was like, wait, you don’t have clean product codes? You don’t have a single product hierarchy? You don’t have clean customer data? It was a foreign concept to me that not everyone had clean master data. So I would just say that And Moe, you asked this earlier, so I wanted to come back to this point. Semantic layers right now are mainly for the structured data. But I think there’s a time in the not too distant future that it will encompass also the semi-structured data. And I would say this data is a hot mess, frankly, because we’ve never applied all of these data governance and data management disciplines that we have been applying to the structured data. So I think organizations that had the organizations that are best positioned for the agentic AI era got to cloud. had clean data, had good master data, and then of course the culture and the people change management. If they already did that, they’re already, they have such a leg up. Now we’re throwing generative AI, agentic AI, semi-structured data, a lot more data that we couldn’t get to before. And yeah, it’s not that easy. 00:39:48.31 [Michael Helbling]: It’s nice to know we’ll continue to have jobs going into the future though. 00:39:51.87 [Cindi Howson]: Yeah. That’s why I’m like, what’s everyone worried about not having jobs? They, they just will be different jobs, different jobs. Yeah. 00:40:00.90 [Tim Wilson]: I got one more. This is like, this could be a complete non sequitur, but it, but I feel like Cindi could tee off on this and I want more color because it was, it was from the post that you’d written where the quote was, our industry has also now raised the generation of data analysts who never learned proper data modeling. And I kind of wanted you to elaborate on that. 00:40:23.75 [Cindi Howson]: Well, I’m going to say first, tell me if you disagree or not. But tell me if you disagree or not. But I follow the work of people like Joe Rice and Sonny Rivera, a snowflake superhero. And yeah, it’s, and I work with a lot of, let’s say visualization experts who are just used to one-offs, let me load the data and let me visualize it. They never really learned proper data modeling techniques. 00:41:02.35 [Tim Wilson]: I guess my question is that a way of saying that there are analysts who aren’t really actually thinking about the structure of the data and the ramifications for how the data fits together. They’re just trying to get to and output. I don’t know if I agree or disagree. I probably agree because I’m just generally negative and that’s like a negative statement. 00:41:31.71 [Cindi Howson]: Let’s not take it negative. Yeah. Let’s challenge these people. Let, to me, empower them. So you know what? You’re great at visualization and you’re great at building dashboards. But if you want to continue to have a career on this space, in this space, I want you to learn some data modeling fundamentals. And I don’t care which methodology you follow. Learn some data modeling. That’s on the technical side. But also, we talk about data literacy. We also need to bring in business literacy. And so to me, it’s not just about Where is the data coming from? It is also, how is it used? And that there really might be two different definitions. I mean, when I talk to somebody in airlines, I don’t even, I’m like, oh wow, I think of on time performance. Did it leave the gate on time? or did it arrive on time? Which one is really more important to you? And by the way, when you’re crossing international date lines, that it gets a little more complicated still. So I would say I want these analysts to learn both the skills. 00:42:59.57 [Moe Kiss]: I have one last question. Just hypothetically, if you were implementing a semantic layer, what would be the top three things you’d want to avoid? 00:43:09.34 [Cindi Howson]: The top three things? Okay, well, I’m going to start with the first thing I would want to do, so I’d have to flip it, avoid it, or what do I want to do? 00:43:20.30 [Moe Kiss]: Or you can do the top three things to make it successful, either way, whichever your brain works. 00:43:25.43 [Cindi Howson]: You want to avoid bringing in absolutely everything in the physical storage and exposing that to mere mortals because that’ll be overwhelming. So I always start with who is going to use this. And what are the top questions they’re going to want to be able to ask of it? Not because I’m going to hard code that, but that I’m going to get an idea of the context in which they’re operating. 00:43:56.65 [Michael Helbling]: Cindi, wow. So cool to talk to you. Thank you so much. This has been really, really good. I’ve got a ton of notes that I’ve been writing down. So I know that our listeners probably also get gaining a lot from this episode. All right, well, let me switch gears really quickly because I need to talk about a quick break with our friend Michael Kaminsky from ReCast. The media makes marketing and GeoLift platform helping teams forecast accurately and make better decisions. Michael’s been sharing with us bite-sized marketing science lessons over the last couple of months, and they’ll help you measure smarter. Okay, over to you, Michael. 00:44:38.12 [Michael Kaminsky (Recast)]: Multicollinearity strikes fear into the hearts of many analysts and executives, but it’s also one of the most commonly misunderstood concepts in analytics. Some amount of correlation across variables is expected in most real-world analyses, so it’s critical to understand what multicollinearity is, why it causes issues, and whether or not it’s a problem for your particular analysis. multicollinearity means that two of your variables share some of the same signal. This causes problems for a regression model, which will not know how to allocate credit between the two variables. This can cause challenges when it comes to interpreting the results of your regression. Let’s imagine you’re modeling the drivers of home prices in some geography, and you want to include home square footage and the number of bedrooms as predictors. These two variables share some amount of signal, namely about the bigness of the house. If you include both variables in a simple linear regression, you’ll often get strange results, where one of the two variables is highly impactful with a large coefficient, and the other might be very small or even negative. Slightly different data sets might even cause the variables to flip, which one is positive and which one is negative. This happens because the model doesn’t know how to apportion credit for bigness, which is present in both variables. So you get these strange results. So the core problem of multicollinearity is that when there’s shared information across variables, a simple regression won’t know how to apportion credit between them. This means that you either need to accept more uncertainty in results, or try to change the variables you’re using to account for the shared information. 00:45:52.32 [Michael Helbling]: Thanks, Michael. And for those who haven’t heard, our friends at ReCast just launched their new incrementality testing platform, GeoLift, by ReCast. It’s a simple, powerful way for marketing and data deems to measure the true impact of their advertising spend, and even better, you can use it completely free for six months. Just visit getrecast.com slash geolift to start your trial today. Okay, well, we’ve got that done. One thing we’d love to do is go around the horn and share something we call last call, something of interest that might be of interest to our listeners. Cindi, you’re our guest. Do you have a last call you’d like to share? 00:46:29.26 [Cindi Howson]: Well, I want to ask a question if I can on the last call. And when you think about how quickly our industry is moving and innovating, what do you see as your best method media to keep up? Is it listening to podcasts, reading, substack or medium articles, or how do you feel about books? 00:46:53.37 [Michael Helbling]: Are we supposed to answer that? 00:46:55.23 [Cindi Howson]: Well, I’m looking for feedback because you know, even though I’m a podcast host, I’m a writer at heart and yet is the industry moving too quickly for another book? 00:47:08.29 [Moe Kiss]: Yeah. I mean, I can speak for myself. I listen to podcasts and host a podcast. That’s a big part of how I stay up to date. But I also, I love books. I’m a book person. Probably books more than articles. But you listen to a lot of the books, right? Yes, I do, but that’s just because of my life stage of being time poor. I end up listening to books on Audible a lot. Yeah, for sure. 00:47:34.88 [Michael Helbling]: What about you, House? I would say my number one source is articles. So in my day-to-day travels, I’ll run across an article and then bookmark it and read it later. So I’ll do that. I buy a lot of books and then don’t read them. Oh boy. In fact, that’s… Right behind me. Michael, have you finished the book? I have not finished your book, Tim. Oh, well, you haven’t finished that either. Uh, so yeah, but I, so I don’t, cause for me reading is sort of like an enjoyable pastime. And I, unlike Moe, I can’t pay attention if someone’s reading it aloud or audio books. So I have to sit down and read it. And then when I do finally get a chance to read, I end up reading like sci-fi or fantasy novels instead of business books. So it’s, it’s a tough one. And then of course, of course podcasts are very important. I have to believe that, right? So there you go. 00:48:32.18 [Cindi Howson]: This feels like confessions of a podcast host. 00:48:35.88 [Michael Helbling]: Yeah, that’s right. Exactly. What do you think Tim? I listen to a ton of podcasts. Yeah, he does. 00:48:43.27 [Tim Wilson]: I listen to a ton of podcasts and very few of them are business or data analytics related. So I am very much the subscribe to, I mean, a medium substack. daily weekly newsletter fiend, which starts to feel a little overwhelming. But yeah, so with the occasional book. The books feel like a chore, though. Well, I feel like if someone else is doing a good job. So cool. I’m just going to be clear. So I don’t listen to the podcast, even though I make one. And I don’t tend to read. I struggle to read the books, even though I wrote one. So yeah, I’m the worst. 00:49:23.03 [Cindi Howson]: Wow. So I think Tim summed it up. Wait, are you telling me two-third of our time spent is like a waste of time? Why am I writing books and why am I hosting a podcast? I’m just gonna get on with building stuff. Okay. 00:49:36.79 [Michael Helbling]: I don’t like the data that we’ve uncovered here. 00:49:42.04 [Tim Wilson]: I mean, I get a lot of value out of hosting the podcast because we get to have excuses to say, hey, why don’t you come on and explain semantic layers to us? 00:49:50.66 [Michael Helbling]: So yeah, that is actually doing a podcast is one of the ways I learn new things. So that’s something you could add to the mix. Yeah. So when is your next book coming out? 00:50:03.30 [Cindi Howson]: I don’t know. Can I take a break from the podcast or stop something? I don’t know. I don’t know. 00:50:09.77 [Michael Helbling]: This is what I was trying to figure out. 00:50:12.20 [Cindi Howson]: What should I do next? Yeah. Yeah. 00:50:14.24 [Michael Helbling]: Fair point. All right. Tim, what about you? What’s your last call? 00:50:19.94 [Tim Wilson]: Well, I guess follow on. There is a sub-stack that I discovered a couple of months ago from somewhere that is We Have the Data. It’s kind of silly. It’s kind of data visualization candy, but it’s WeHaveTheData.net. I think it’s a couple of times a week, and it’s just kind of a It’s like NOMLAC news, but data visualizations instead. So they’re pretty lengthy. They’re a collection of often kind of trivial data visualizations, but it’s kind of a fun scroll in my inbox. 00:50:54.66 [Michael Helbling]: Outstanding. All right, Moe, what about you? 00:50:59.06 [Moe Kiss]: I want to do a plug for Cindi’s podcast. I was lucky enough to be a guest back in October and it’s called The Data Chief. And as you can tell, I ended up hanging out after the show and picking Cindi’s brain for like another 30, 40 minutes about all of these topics, which is why she’s here today. And she just has such a range of like really incredible guests. It’s a really different format to our show. So really encourage you to go check out The Data Chief podcast. 00:51:29.54 [Michael Helbling]: I’m standing in, and yeah, we’ll put a link to that in our show notes as well, so people can find it. 00:51:36.37 [Tim Wilson]: You’re supposed to hand her at the beginning. 00:51:37.99 [Michael Helbling]: It’s fine. We’ll hand her all over the place. What’s your last call? Well, I’m so glad you asked him. So a good friend of mine, Mary Gates, actually made me aware of this. So Informs, which I’m sure we’re all familiar with, they have an initiative called Pro Bono Analytics. So I’m a big fan of any analytics initiatives that I’ve been able to be part of them over the years that help nonprofits and allow people to give of their skills and data and analytics to nonprofits and mentorship and things like that. Pro Bono Analytics is an initiative run by Informs. And so I just wanted to give that a shout out. I was not familiar with this before, but it looks like a very cool organization. And so if you’re a nonprofit and you’re listening, that might be an amazing place to partner with them to get help with data initiatives. And if you’re a professional in working in data and you want to find a way to give back, that might be an amazing way to do that. So we’ll put a link to that in the show as well. OK. As you’ve been listening about on this topic of semantic layers, I’m sure you have thoughts. I’m sure you have questions. We would love to hear from you. Go ahead and reach out to us. And there’s three main ways you can do that. You can do that through LinkedIn or the measure slack chat group, or you can email us at contact at analyticshour.io. And yeah, we’d love to hear from you. Cindi, once again, this has been a very information-rich and awesome episode, and primarily because your deep knowledge and expertise in this field. So thank you again so much for joining. 00:53:21.43 [Cindi Howson]: Thank you for having me. I feel like we should do this over a cup of coffee or a glass of wine at some point. 00:53:28.71 [Michael Helbling]: Yes, I wholeheartedly agree. That’s how this whole podcast started was because we’re all drinking at an analytics conference and said, we should put this on the radio. That’s a great idea. That’s right. Another drunken, great ideas. All right. Also, if you are somebody who puts and is not directed at you, Cindi, this is back to the audience. If you’re someone who puts stickers on your laptops or whatever, we do have stickers and we’d love to send you one. You can actually request one on our website so you can go and do that. And then Obviously, no show would be complete without saying a huge thank you to all of you listeners who go out and share ratings and reviews with us and tell us how you’re enjoying the show. So please continue to do that. We look forward to that feedback. We appreciate it very much. All right. As we wrap up, I know that no matter if you’re trying to Build one ring to rule them all type of semantic layers, or if you’re spreading it out across verticals. I know both of my co-hosts, Tim and Moe, would agree with me. You should keep analyzing. 00:54:41.77 [Announcer]: Thanks for listening. Let’s keep the conversation going with your comments, suggestions, and questions on Twitter at @analyticshour, on the web at analyticshour.io, our LinkedIn group, and the Measure Chat Slack group. Music for the podcast by Josh Grohurst. Those smart guys wanted to fit in, so they made up a term called analytics. Analytics don’t work. 00:55:06.36 [Charles Barkley]: Do the analytics say go for it, no matter who’s going for it? So if you and I were on the field, the analytics say go for it. It’s the stupidest, laziest, lamest thing I’ve ever heard for reasoning in competition. 00:55:19.07 [Michael Helbling]: We’ll just do our best with it. It’s why we have an audio engineer. Hi, Tony. Hi, Tony. 00:55:37.04 [Tim Wilson]: Rock flag and semantic layers are 30 years old. The post #286: Metrics Layers. Data Dictionaries. Maybe It’s All Semantic (Layers)? With Cindi Howson appeared first on The Analytics Power Hour: Data and Analytics Podcast.