The Agentic Edge Podcast

Schaun Wheeler and Arpit Choudhury talk about Content Management Systems (CMS) and how they are a treasury of information that can be leveraged for hyper-personalization. They explore how generative AI is transforming CMS, the importance of data quality, and the potential of LLMs in enhancing user experiences. The discussion also covers the challenges of implementing AI in recommendation systems, the significance of integrating external data, and the future of personalization in various applications, including food delivery and travel.KEY POINTS* CMSs are often seen as repositories of content to present on webpages but they can be structured to use data for purposes beyond webpage display such as personalization, communication, and discovery. LLMs can significantly improve CMS data management.* There are opportunities in what you bring in and what you take out of the CMS. Information provided by vendors can be better used by turning it into structured data.* Generative AI can be leveraged to dynamically change the way content is displayed to users each time.* Cleaning your CMS can allow you to extract additional value from it. LLMs enable on-the-fly data cleaning and enhancement without the massive manual effort previously required. They can also analyze content tone, create detailed sub-categories, and generate descriptive tags. This data can power features like sophisticated filters and AI-driven personalization through agentic learning.* The alignment problem in AI is when there is a gap between user expectations and AI recommendations. The hardest part of aligning input to output in an LLM is getting the context right. * In a CMS, an input is anything that resides in the CMS, such as item IDs, item names, and item descriptions. An output is information that is presented to the user.* LLMs work when choosing between a limited number of options. When there are hundreds of thousands of items, LLMs aren't able to sort through all of the information consistently and coherently.* LLMs can be used to clean your CMS data and to further personalize the presentation of information. They can extract tone, feel, and other metadata that enhance both recommendations and browsing experiences.* Lots of information can be pulled in from various sources to address the discovery problem, but it must be presented selectively. Companies that help users find relevant information quickly will retain users better than those who don't.* A/B testing tells you what works for the largest minority of users, so giving everyone that experience is a bad idea. We can use AI agents to learn recommender preferences and cater to the changing needs of each user, as they are scalable and can give each user the required attention.CHAPTERS:* 00:00 Introduction * 01:22 Developments in CMS technology * 03:45 Clean your CMS * 07:34 Leveraging tags for recommendation * 09:42 The alignment problem * 14:53 Applications of LLMs * 19:14 Data extraction capabilities of LLMs * 25:29 Value propositions in food delivery * 28:36 The potential of CMS * 31:34 Integrating external data * 33:45 Agentic applications in CMS * 40:13 Inputs and outputs This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit edge.aampe.com

In order to mimic human decision making, agents need to estimate the probability that a particular intervention will move user behavior in the right direction, but also estimate how much confidence they should put in that probability assessment. The first estimate is easy - or, at least, straightforward. The second estimate is a lot harder, but it sits at the core of how agents balance exploration of new possibilities with exploitation of lessons already learned.There is no purely empirical way to estimate confidence. Frequentists estimate it implicitly, whereas Bayesians estimate it explicitly, but in both cases the estimate has to come from the very squishy realm of prior beliefs. When I was designing how our agents would assess confidence, I had to do some thinking about the properties of statistical distributions and how those map to different expectations about confidence we can be about a lesson learned from just a single intervention. No one wants to stake too much on a single interaction - in any context - but if we stake too little then we end up discounting a lot of valuable information so much that we ultimately end up ignoring it.In the attached video, I walk through some of the ways we've addressed the topic of estimate confidence with our agents. This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit edge.aampe.com

When you're on a team, you tend to figure out problems collectively - you chat with your teammates, compare notes, and swap stories about what has or has not worked in the past. What if the team is a team of agents? How do agents compare notes?An embedding space. An embedding space is a high-dimensional mapping of data points (words, users, events, etc.) to arrays of numerical values. It's like a map, but whereas a typical map can only be 2D or maybe 3D, an embedding space can have many dimensions as you want. This allows you to pick any point in the space, and quickly find other points that share similar values.Aampe agents maintain multiple embedding spaces for different look-back windows: the last 1 day, the last 7 days, the last 30 days, etc. So when an agent is working with a user and the user just won't respond to any outreach, the agent can use the embedding spaces to quickly find other users who *have* responded recently, and get ideas from those user's agents.It's impossible to visualize so many dimensions on a 2D surface, but I did a little dimensionality reduction in the attached video to try to give a little intuition about how embedding spaces work. This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit edge.aampe.com

Paul Meinshausen and Schaun Wheeler talk about the key components behind decision-making and what goes into automating it in this discussion hosted by Arpit Choudhury. They emphasize that successful decision automation includes understanding the nuances of decision repeatability, outcome evaluation, user preferences, and business constraints. They discuss the importance of designing systems that can learn effectively from user interactions, the limitations of current approaches, and the ongoing need for human input to provide crucial context.KEY POINTS: 1. Decisions can be broken down into three components:* The decision set (options of the decision)* The outcome set (what happens based on decisions)* The information set (data relevant to making decisions)2. Decisions can be described to see how they can be handed to machines. This can be done based on criteria such as whether the decision is answer set constrained and the repeatability and frequency of the decision.3. Recognizing which problems are constrained and repeatable can help leverage past experience to tackle them systematically and save a lot of time.4. Identifying the information relevant to making a decision helps constrain the decision set.5. Recommender Systems are well suited for large decision sets with thousands of options. They should share relevant information to learn about users while not overwhelming them. The way information is presented affects both user decisions and system learning.6. In decision automation, humans are needed to provide context and business constraints, and design interfaces that capture meaningful signals from users. We can show AI agents how to make decisions like we do.7. Challenges in Decision Automation:* There's often a mismatch between available data and user preferences.* Business problems often involve opinions rather than facts, making outcome evaluation difficult.* Most software are built so that the burden of making something repeatable and learning from that repetition falls on humans. * When relevant information is not shared with users, it can make it difficult for agents to understand the information influencing users’ preferences and limit effectiveness.CHAPTERS: * 00:00 Machines and decisions* 04:30 What is a decision?* 06:32 Constrained, repeatable problems* 08:00 Components of a decision* 09:03 The answer set* 12:00 Constrained resources* 12:58 Repeatability and frequency* 15:21 Delivering recommendations* 16:55 Evaluating outcomes * 19:44 LLMs making decisions * 22:03 Facts and inference * 24:13 Not just for users * 27:14 The information set * 28:33 How recommender systems evaluate answers * 31:12 Relevant information and why we automate decisions This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit edge.aampe.com

Agentic learners aren't tools or systems or programs. They're additional headcount. As with a a human team, one of the most important aspects of managing a team agentic learners is to know how you can give feedback and instruction. I made a video recently (link the comments) about representing user preferences as two parameters in order to do a random draw from a beta distribution. The probability parameter tells the agent how much an intervention is expected to positively impact user behavior, and the signal parameter tells the agent how confident it should be about that probability.Draw from a beta distribution that has high probability but low signal, and the result may very possibly be a low number. This is what keeps agents from getting stuck in local maxima.However, all of that deals with the explore/exploit tradeoff. That's a common tradeoff of agents to make, because an agent needs to know whether to continue to try as-yet unexplored options, or focus on options that have already proven successful (even if still other options might be even more successful). But in any realistic business context, agents also need to navigate a tradeoff between what a user prefers and what a business needs. While it doesn't do a business any good to push options on a user if the user really hates those options, it can often make sense to give a user their second- or third-choice option if doing so can meet a business objective. To do that, remember this simple formula:v ** (log(t) / log(a))Three parameters:v: the actual value drawn from the beta distribution.a: the anchor value of the distribution - I usually use 0.5, because it's central and intuitive.t: the target value to which to move the anchor.So if v = 0.5 and a = 0.5 and v = 0.66, then using that formula would transform a draw of 0.5 to 0.66. The value of the formula is that is transforms any draw from the distribution, whether it's 0.5 or 0.98 or 0.00023. It effectively uses the anchor and target values to shift the entire distribution.So if you're a business and you need your agents to prioritize the selling of a particular product line, you can raise the target value of the distribution for that product line and agents will prioritize interventions about that product, even if the user's probabilities for that product tend to be lower than the probabilities for other products.By the way, this video mentions a previous video on parameterizing beta distributions. You can find that here: This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit edge.aampe.com

How do you make a computer program show curiosity? You randomly draw from a beta distribution. Yes I'm serious. A core problem in agentic learning is the navigation of the explore/exploit tradeoff. Agents have many options to choose from when trying to engage users, and they always need to balance the goal of taking advantage of lessons already learned with the goal of trying options that haven't yet been tried. If an agent only ever explores, it never optimizes. If an agent only ever exploits, it prematurely optimizes and ends up a local maximum - something that's better than some options, but not nearly as good as it could be. Aampe agents store two different weights for every possible action in each of their action spaces - a probability of influence (roughly analogous to an expected success rate), but also a measure of signal strength, which encodes how much evidence is backing the probability. The agent needs both of those so it can hedge it's bets when it has to operate on low evidence, and can double down when it's able to operate on a high evidence. Also, this video references another video on Interrupted Time Series analysis. You can find that here: This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit edge.aampe.com

Agents that work on the basis of behavioral data need to be able to make judgements about how successful (or not) their actions are. Unlike an A/B test or a multi-armed bandit, where you use success rates over many users to determine the relative value of different options, an agent needs to be able to try one specific action with one specific user and make a judgement call about whether that action inclined the user in the right direction. There's not such thing as a success rate when you're dealing with a single intervention for a single user. Instead, Aampe agents use a version of Interrupted Time Series analysis, simplified for use with sparse data. This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit edge.aampe.com

Schaun Wheeler and DJ Rich delve into the intricacies of building recommender systems in this podcast hosted by Arpit Choudhury. The discussion highlights the steps to developing a recommender system, practical advice for startups, and the evolving landscape of recommender technologies. KEY POINTS: * Identify the Problem: Recommender systems address the "discovery problem" by helping users sift through vast amounts of options to find relevant content quickly. Recognizing this problem is crucial before diving into solutions.* System Components: Recommender systems are complex and involve multiple components such as:* Item Inventory: Detailed metadata about items (e.g., descriptions, categories).* User Interaction History: Data on user interactions with items (e.g., views, purchases).* Recommendation Model: The core model that filters and ranks items based on user preferences.* The Learner: An important component, which trains the model and separates it from the model's deployment phase.* Build vs. Buy: Should one build a recommender system from scratch or use existing solutions? For many startups, buying an off-the-shelf system can be more practical due to advances in data infrastructure and the complexity of developing a bespoke system. Buying a system can also free up resources for other critical areas.* Practical Recommendations for Startups: Instead of getting bogged down by complex models initially, startups are encouraged to start with simpler models and leverage existing infrastructure to implement a functional recommender system.* Innovations in Recommender Systems: Schaun is interested in combining traditional methods with reinforcement learning to enhance system performance. DJ is excited about research addressing causal questions and handling sequential recommendations.REFERENCES:* "Are we really making progress?" This paper is a replication study on recommender algorithms and shows that many DL approaches couldn't be reproduced or could be beaten with linear methods. * "Deep Exploration for Recommender Systems" This paper talks about sequential decisioning for RSs (where you consider more than just one item recommendation). * "Two Decades of Recommender Systems at Amazon" This paper is a retrospective on what's work well at Amazon. This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit edge.aampe.com

Most smaller apps wish they had the resources of a mega-app, but should they?Join us this week as we talk with Thomas le Hardy about his past experience working as Director of Customer Acquisition at Audible (Amazon) and how it compares with his current role as VP of Marketing at Tonsser.During the podcast, Thomas answered questions like: 👉 What were the biggest differences between running marketing at Amazon vs. running marketing at Tonsser?👉 Which tools or resources did Amazon have that a smaller app doesn't, and what are the impacts of not having these tools?👉 What advantages does a smaller app have over a bigger one?👉  Which best practices has Thomas carried forward from Amazon into his current role?👉 What are the biggest lessons Thomas has learned building the marketing function at Tonsser from virtually nothing? Throughout the episode, Thomas shared some great tips for how to work cross-functionally and establish a good foundation that would benefit apps of any size!If you enjoyed this conversation, be sure to connect with Thomas on LinkedIn: https://www.linkedin.com/in/thomaslehardy/To learn more about Tonsser, you can visit https://tonsser.com/---Appthropology is powered by Aampe, the leader in Message-Led Personalization.www.aampe.com This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit edge.aampe.com

Growing a B2C app is difficult, but encouraging users to use an employer-sponsored app brings additional challenges:If a significant number of employees don't use the app, the employer will cancel your contract.Join us this week on the Appthropology podcast as we talk with Martyna Malinowska, co-founder and chief product officer at GajiGesa.During the podcast, we'll cover what GajiGesa is and the predatory loan problem they're looking to solve in Indonesia.Specifically, we'll cover the cultural nuances that GajiGesa is looking to overcome and how they're encouraging adoption of new features from a technological and user engagement perspective.We'll also take a peek at how GajiGesa measures success and what's next on the horizon!If you're looking to increase user engagement in your app and especially if you have a  B2C and B2B element, make sure to give this episode a listen! 🎧To learn more about GajiGesa, please visit: https://gajigesa.com/en/To connect with Martinya, you can find her on LinkedIn here: https://www.linkedin.com/in/martyna-malinowska-frm-a4770442/---The Appthropology podcast is powered by Aampe (www.aampe.com). This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit edge.aampe.com

What are the most common mistakes apps make in CRM? 🤔Join us this week as we talk with Annika Dunaway, Sr. Growth Consultant at Phiture,  about what she's learned in her experience serving in marketing and CRM roles in virtually every industry — from training and education to food delivery and travel and even health and fitness — and how she currently uses all this knowledge combined with her passion for psychology and human behavior to provide practical steps to driving growth for apps of all sizes as a consultant at Phiture.Annika has done everything from helping companies establish their first CRM system to running large-scale A/B testing and campaign optimizations for giant multinational companies that employ tens of thousands of employees and are deployed in over 50 countries around the world.This is an episode you're not going to want to miss!Join us as we discuss:🙅‍♂️ The biggest mistake companies make with CRM🚀 The biggest challenge apps face along the user lifecycle (and how to overcome it)👩‍🏫 What Annika looks for when she starts a new consulting engagement🧱 How to break through drop-off at paywalls🌟 How to successfully reactivate users🌞 Why CRM teams could benefit from an outside perspective🔨 Annika's favorite tool stack💬 The biggest mistake apps are making with their communications🆎 The most interesting thing Annika has uncovered during an A/B test 🙈1️⃣ ...and the ONE thing CRM teams could do TODAY that would have the biggest impact on the success of their apps.So, if you have ANYTHING to do with CRM or app marketing, you're going to want to tune in to this episode! 🎧 (It's like a free Phiture masterclass! 😱)If you enjoyed this conversation, be sure to connect with Annika on LinkedIn: https://www.linkedin.com/in/annika-dunaway-90b21642/Here is a link to Annika's excellent article that we mentioned during the episode ("Why your CRM activities might belong with the product team.")https://phiture.com/mobilegrowthstack/why-your-crm-activities-might-belong-with-the-product-team/To learn more about Phiture, check out their website:https://phiture.com/---Appthropology is powered by Aampe, the leader in Message-Led Personalization.www.aampe.com This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit edge.aampe.com

Managing CRM for an app is one thing. Running CRM for a FinTech app is quite another....especially when you're mostly a team of one....and your app is designed for people to STOP using it. 😳Tune in this week as we talk to Joel Fransson, Global CRM Manager at Anyfin, the app designed to help users take control of their finances by getting lower interest rates on their existing loans and making smart money decisions.During the episode, we'll chat about:📱 Why it's important for Anyfin to have a mobile app.🤝 How Anyfin establishes trust with its prospects and customers.🏃‍♂️ Why Anyfin wants its customers to stop using their app.👄 How Anyfin encourages its current customers to spread the word (while not violating any financial regulations).🚶‍♂️ How Joel works with other teams and sets his priorities as a CRM team of one.We also talk about the bad rap that FinTechs often get, and how they can actually help people live better, more financially free, lives.💲 So, if you're in the FinTech space or just want to learn more about it, save up your pennies and tune in to this episode! 🎧To learn more about Anyfin, you can visit their website:https://anyfin.com/And, if you enjoyed this conversation, be sure to connect with Joel on LinkedIn: https://www.linkedin.com/in/joel-fransson-crm/Appthropology is powered by Aampe, the leader in Message-Led Personalization.www.aampe.com This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit edge.aampe.com

What will determine whether apps survive 2023?Lifecycle marketing. ...but not lifecycle marketing as we've been doing it.Apps came off a high in 2020, but the tides have changed. Now, most apps are feeling the squeeze between heavy resource constraints (both fewer physical resources and less capital during the recession) and worsening ad performance (between data privacy and user tracking regulations - Forbes is reporting a 40% decrease in ad effectiveness to iOS users).How can apps possibly grow in such a market?You need to engage your current user base.Just look at Spotify - Only 43% of their current users are paid subscribers. That means if they could figure out how to engage their current "lurking" user base, they could more than double their revenue without giving Meta a dime.So the question remains — How do we engage our current inactive user base (especially since, to date, our efforts haven't worked)?Listen in this week as Jim goes solo and talks about the approach that the biggest apps in the industry use and how you can apply this same strategy for your app (no matter the size).[👉Hint: It *doesn't* involve user journeys.]Lifecycle marketing is the key to survival in 2023....but we have to do it well. This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit edge.aampe.com

What's the role of lifecycle marketing, and how can you make the biggest impact on your brand?Tune in this week as we talk to Sean Collins, Director of Lifecycle Marketing at BILT Rewards, as we answer this and more!💳 Get an overview of BILT and the awesome value they deliver to their users😱 Learn how BILT discovered that their own name was potentially turning some users off (and how they pivoted).🚵‍♀️ Hear how they create and decide which services to offer to help increase their usefulness and customer loyalty📬 Learn how Sean decides on messaging cadence and how he knows "which messages worked"👨‍🏫 ...and hear Sean's view on his role as a lifecycle marketer and how to be most effective in a Lifecycle Marketing role.There's also some fun bonus content in here, like how to properly measure user LTV so you can make better user acquisition decisions and the biggest lesson Sean's learned through his entire working career! (Who can pass that up? 😅)There's fun for everyone in this episode, so be sure to tune in! 🎧 This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit edge.aampe.com

1 in 4 children in America right now grows up without learning how to read — and we can't just fix it with "more school."In fact, only 35% of public school students are rated at or above 'Proficient' in grade 4 reading, and a full 37% of children arrive at kindergarten without the skills necessary for lifetime learning.Tim Dikun and the folks at Teaching.com don't think that's acceptable, so they're doing something about it.Per the National Center for Education Statistics (NCES), regardless of household income or background, educational success starts at home —Children who are read to at least 3x per week are almost 2x as likely to score in the top 25% in reading compared to children who are not, and students with involved parents are more likely to have better behavior, higher grades, better attendance, and better social skills while being less likely to develop behavioral issues and have low self-esteem, so Teaching.com created an app (Reading.com) to help encourage 1-to-1 reading in a family environment.Reading.com isn't just an "Edutainment app" — something parents give their kids to keep them distracted. It's an app that parents and children do together...which is the very key to unlocking children's educational success.As parents ourselves, Paul and I were so excited to interview Tim and understand Teaching.com's mission, strategy, and the lessons he's learned on the journey to launch Reading.com.This episode is a great listen for parents, educators, and folks who are getting started on the app design process!Nationally, only about half of children between birth and five years (47.8%) are read to every day by their parents or other family members,  let's help turn that stat around with Reading.com. Reading.com can currently be found on the Apple App Store:🍎 https://apps.apple.com/us/app/reading-com-learn-to-read/id1534938305The Appthropology Podcast is powered by Aampewww.aampe.com This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit edge.aampe.com

How can you stand out in a crowded marketplace?How do you effectively expand into new markets?...and how can you make a big impact, as quickly as possible, when taking over in a growth role for an up-and-coming app?Tune in this week as we discuss all of this and more with Siim Pukk, the Head of Marketing and Growth at Fudy, Estonia's premier food delivery app!From auditing and documenting baseline opportunities and issues, to exploring how to create brand partnerships and successfully expand into new markets, we'll talk about how Siim took over the reins and strategized his critical first few months at Fudy....We'll even cover some traditional Estonian Christmas dishes that are...well...more than a little unsettling. 😅This is a great episode for anyone looking to grow their app in a highly-competitive market! Be sure to listen in! 🎧FUDY can be found on the Apple App Store and Google Play:🍎 https://apps.apple.com/us/app/fudy/id1612226150▶ https://play.google.com/store/apps/details?id=eu.fudy.client&hl=en_US&gl=US&pli=1Here's Siim's excellent "First Month at Fudy" LinkedIn article:https://www.linkedin.com/pulse/first-month-head-marketing-growth-fudy-siim-pukk/ The Appthropology Podcast is powered by Aampewww.aampe.com This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit edge.aampe.com

What's the role of data science and behavioral science in mobile apps?How do you prioritize and conduct experiments? ...and how do you effectively interface with marketing? *gasp!*Tune in this week as we discuss all of this with Ben Tengelsen, the VP of Data Science at IntelyCare, the flexible healthcare staffing app!We'll dive into some common misconceptions and pitfalls that can occur from looking at data, when and how to apply different disciplines of data and behavioral analysis, and we'll even cover a few examples of real-life experiments Ben and the IntelyCare team have run to help boost their different business objectives.We'll even discuss the top-secret way Ben decides which requests from marketing to pursue!There's really something for everyone in this episode, so be sure to check it out! This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit edge.aampe.com

This week on the Appthropology podcast, we're talking with Gijs Verheijke, the Founder of Ox Street, Southeast Asia's premier app for buying and selling authenticated shoes and apparel.Tune in to learn Gijs's advice on how to successfully start up a 2-sided marketplace, some of the tools and strategies he uses to keep Ox Street going, and the lessons he's learned along the way.You'll also get a glimpse into "Sneakerhead" culture, see what user metrics matter most, and learn about how Crocs are making a comeback! *gasp* This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit edge.aampe.com

Tune in this week as we talk to Baptiste Malaguti, the CEO and Co-Founder of Kuri, the Climate-Friendly Meal Planning App.We'll dive into why Baptiste started Kuri, Kuri's mission, and how Baptiste was able to quickly scale and get featured in the Apple App Store.We'll also explore how Kuri chose their user-behavior metrics, the way Baptiste thinks about content, and the most interesting thing he's learned on his journey! This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit edge.aampe.com