Evgeniya Sukhodolskaya - Data Advocate, Toloka - Data at the core of all the cool ML

Toloka’s support for Academia: grants and educator partnershipshttps://toloka.ai/collaboration-with-educators-formhttps://toloka.ai/research-grants-formThese are pages leading to them:https://toloka.ai/academy/education-partnershipshttps://toloka.ai/grantsTopics:00:00 Intro01:25 Jenny’s path from graduating in ML to a Data Advocate role07:50 What goes into the labeling process with Toloka11:27 How to prepare data for labeling and design tasks16:01 Jenny’s take on why Relevancy needs more data in addition to clicks in Search18:23 Dmitry plays the Devil’s Advocate for a moment22:41 Implicit signals vs user behavior and offline A/B testing26:54 Dmitry goes back to advocating for good search practices27:42 Flower search as a concrete example of labeling for relevancy39:12 NDCG, ERR as ranking quality metrics44:27 Cross-annotator agreement, perfect list for NDCG and Aggregations47:17 On measuring and ensuring the quality of annotators with honeypots54:48 Deep-dive into aggregations59:55 Bias in data, SERP, labeling and A/B tests1:16:10 Is unbiased data attainable?1:23:20 AnnouncementsThis episode on YouTube: https://youtu.be/Xsw9vPFqGf4Podcast design: Saurabh Rai: https://twitter.com/srvbhr