Powered by RND
PodcastsBusinessLenny's Podcast: Product | Career | Growth

Lenny's Podcast: Product | Career | Growth

Lenny Rachitsky
Lenny's Podcast: Product | Career | Growth
Latest episode

Available Episodes

5 of 299
  • Inside Google's AI turnaround: The rise of AI Mode, strategy behind AI Overviews, and their vision for AI-powered search | Robby Stein (VP of Product, Google Search)
    Robby Stein is VP of Product at Google, where he oversees the core products of Google Search—including the new AI Overviews, AI Mode, search ranking, Google Lens, and more. Previously, he led consumer products at Instagram, where he and his teams built Stories, Reels, Close Friends, and other key features now used by billions.What you’ll learn:1. Why Google’s AI products are suddenly taking off after years of perceived stagnation2. How AI is expanding Search rather than replacing it, contrary to what many predicted3. The three core product principles that have helped Robby build multiple billion-user products4. Inside Instagram’s decision to build its own version of Snapchat Stories5. His mantra of “relentless improvement”6. How Google developed AI Mode from concept to launch in just one year7. Why most teams give up too early on potentially transformative products—Brought to you by:• Vanta—Automate compliance. Simplify security: https://vanta.com/lenny• Jira Product Discovery—Confidence to build the right thing: https://atlassian.com/lenny/?utm_source=lennypodcast&utm_medium=paid-audio&utm_campaign=fy24q1-jpd-imc• Orkes—The enterprise platform for reliable applications and agentic workflows: https://www.orkes.io/—Transcript: ⁠https://www.lennysnewsletter.com/p/how-google-built-ai-mode-in-under-a-year⁠—My biggest takeaways (for paid newsletter subscribers): ⁠https://www.lennysnewsletter.com/i/175041217/my-biggest-takeaways-from-this-conversation⁠—Where to find Robby Stein:• X: https://x.com/rmstein• LinkedIn: https://www.linkedin.com/in/robbystein/—Where to find Lenny:• Newsletter: https://www.lennysnewsletter.com• X: https://twitter.com/lennysan• LinkedIn: https://www.linkedin.com/in/lennyrachitsky/—In this episode, we cover:(00:00) Introduction to Robby Stein(04:46) Google’s recent success with AI(06:08) The evolution of Google Search(09:41) AI Mode and its impact(15:30) The rise of AEO(18:50) Building successful AI products(21:31)   Embodying relentless improvement(30:10) Lessons from Instagram Stories(35:20) Driving growth in established products(40:08) Balancing optimization and innovation(43:39) The journey of AI Mode: From launch to expansion(48:05) Organizational changes and urgency(49:51) AI Mode vs. competitors(51:35) Core product principles(57:07) Instagram’s Close Friends feature(01:03:01) The importance of resources in development(01:06:39) AI corner(01:11:19) Curiosity and learning(01:15:01) Lightning round and final thoughts—Referenced:• Google Gemini: https://gemini.google.com/app• Nano Banana: https://aistudio.google.com/models/gemini-2-5-flash-image• Chat GPT: https://chatgpt.com/• Perplexity: https://www.perplexity.ai/• Google Lens: https://lens.google/• AI Google search: https://www.google.com/ai• Why ChatGPT will be the next big growth channel (and how to capitalize on it) | Brian Balfour (Reforge): https://www.lennysnewsletter.com/p/why-chatgpt-will-be-the-next-big-growth-channel-brian-balfour• Alex Rampell on X: https://x.com/arampell• A 4-step framework for building delightful products | Nesrine Changuel (Spotify, Google, Skype): https://www.lennysnewsletter.com/p/a-4-step-framework-for-building-delightful-products• Look broader, look closer, think younger: Tony Fadell speaks at TED2015: https://blog.ted.com/look-broader-look-closer-think-younger-tony-fadell-speaks-at-ted2015/• Jobs to Be Done: https://www.christenseninstitute.org/theory/jobs-to-be-done/• The ultimate guide to JTBD | Bob Moesta (co-creator of the framework): https://www.lennysnewsletter.com/p/the-ultimate-guide-to-jtbd-bob-moesta• Rinstagram or Finstagram? The curious duality of the modern Instagram user: https://www.theguardian.com/technology/2016/sep/26/rinstagram-finstagram-instagram-accounts• V03: https://v03ai.com/• Pirate GPT: https://www.kickstarter.com/projects/silentmeditation/pirate-gpt/• The Bear on Hulu: https://www.hulu.com/series/the-bear-05eb6a8e-90ed-4947-8c0b-e6536cbddd5f• Dune on HBO Max: https://www.hbomax.com/movies/dune/e7dc7b3a-a494-4ef1-8107-f4308aa6bbf7• Top Gun: Maverick: https://www.imdb.com/title/tt1745960/• Purple pillows: https://purple.com/pillows• Avocado pillow: https://www.avocadogreenmattress.com/products/green-pillow• Justin Bieber’s website: https://www.justinbiebermusic.com/• Scooter Braun’s website: https://scooterbraun.com/—Recommended books:• Competing Against Luck: The Story of Innovation and Customer Choice: https://www.amazon.com/Competing-Against-Luck-Innovation-Customer/dp/0062435612• The Design of Everyday Things: https://www.amazon.com/Design-Everyday-Things-Revised-Expanded/dp/0465050654• Aurora: https://www.amazon.com/Aurora-High-Stakes-Survival-Navigate-Darkness/dp/0062916475• Project Hail Mary: https://www.amazon.com/Project-Hail-Mary-Andy-Weir/dp/0593135202—Production and marketing by https://penname.co/. For inquiries about sponsoring the podcast, email [email protected].—Lenny may be an investor in the companies discussed. To hear more, visit www.lennysnewsletter.com
    --------  
    1:21:37
  • First interview with Scale AI’s CEO: $14B Meta deal, what’s working in enterprise AI, and what frontier labs are building next | Jason Droege
    Jason Droege is the CEO of Scale AI, a company that provides foundational training data to every major AI lab. He previously co-founded Scour with Travis Kalanick and built Uber Eats from idea to $20 billion in revenue. In this conversation, Jason shares lessons from getting sued for $250 billion, discovering restaurant economics by weighing sandwich ingredients, and over 25 years of launching transformative technology businesses.What you’ll learn:What actually happened with Meta’s $14 billion investment in Scale AIWhy AI models still need human experts to improve, and how that relationship is evolvingHow AI models learn from experts building websites and debugging codeThe business lessons from building Uber Eats from zero to $20 billionWhy most enterprise data is useless for AI models todayWhy urgent daily problems beat super-valuable occasional problems when building productsHow to think independently when building new products and businesses—Brought to you by:Merge—The fastest way to ship 220+ integrations: http://merge.dev/lennyFigma Make—A prompt-to-code tool for making ideas real: https://www.figma.com/lenny/Mercury—The art of simplified finances: https://mercury.com/—Transcript: https://www.lennysnewsletter.com/p/first-interview-with-scale-ais-ceo-jason-droege—My biggest takeaways (for paid newsletter subscribers): https://www.lennysnewsletter.com/i/174979621/my-biggest-takeaways-from-this-conversation—Where to find Jason Droege:• X: https://x.com/jdroege• LinkedIn: https://www.linkedin.com/in/jasondroege/—Where to find Lenny:• Newsletter: https://www.lennysnewsletter.com• X: https://twitter.com/lennysan• LinkedIn: https://www.linkedin.com/in/lennyrachitsky/—In this episode, we cover:(00:00) Introduction to Jason Droege(06:01) Jason’s early career and lessons learned(10:27) The current state of Scale AI(12:37) The shift to expert data labeling(17:02) Challenges and strategies in finding experts(18:48) Reinforcement learning and AI environments(28:18) The future of AI and human involvement(31:21) The role of evals(35:25) What AI models will look like in the next few years(41:43) Building Uber Eats and understanding customer needs(48:19) The importance of independent thinking(50:45) Setting high standards for new businesses(53:03) Exploring and selecting business ideas(57:07) The McDonald’s story(01:00:13) The role of gross margins in business feasibility(01:04:49) Why Jason says, “Not losing is a precursor to winning”(01:09:12) Hiring and building teams(01:12:11) AI corner(01:14:47) Lightning round and final thoughts—Referenced:• Travis Kalanick on X: https://x.com/travisk• Scour: https://en.wikipedia.org/wiki/Scour_Inc.• Scale: https://scale.com/• Alexandr Wang on X: https://x.com/alexandr_wang• Why experts writing AI evals is creating the fastest-growing companies in history | Brendan Foody (CEO of Mercor): https://www.lennysnewsletter.com/p/experts-writing-ai-evals-brendan-foody• Brendan Foody’s post on X about knowledge work changing: https://x.com/BrendanFoody/status/1970163503702188048• MIT Finds 95% of GenAI Pilots Fail Because Companies Avoid Friction: https://www.forbes.com/sites/jasonsnyder/2025/08/26/mit-finds-95-of-genai-pilots-fail-because-companies-avoid-friction/• Uber Eats: https://www.ubereats.com/• Stephen Chau on X: https://x.com/thestephenchau• a16z Podcast: https://a16z.com/podcasts/a16z-podcast/• F1: The Movie: https://www.imdb.com/title/tt16311594/• V03: https://v03ai.com/• Careers at Scale: https://scale.com/careers—Recommended books:• The Selfish Gene: https://www.amazon.com/Selfish-Gene-Anniversary-Introduction/dp/0199291152• The Road Less Traveled: A New Psychology of Love, Traditional Values, and Spiritual Growth: https://www.amazon.com/Road-Less-Traveled-Timeless-Traditional/dp/0743243153/• Good to Great: Why Some Companies Make the Leap . . . And Others Don’t: https://www.amazon.com/Good-Great-Some-Companies-Others/dp/0066620996• Thinking, Fast and Slow: https://www.amazon.com/Thinking-Fast-Slow-Daniel-Kahneman/dp/0374533555/—Production and marketing by https://penname.co/. For inquiries about sponsoring the podcast, email [email protected].—Lenny may be an investor in the companies discussed. To hear more, visit www.lennysnewsletter.com
    --------  
    1:24:01
  • How to find hidden growth opportunities in your product | Albert Cheng (Duolingo, Grammarly, Chess.com)
    Albert Cheng has led growth at three of the world’s most successful consumer subscription companies: Duolingo, Grammarly, and Chess.com. A former Google product manager (and serious pianist!), Albert developed a unique approach to finding and scaling growth opportunities through rapid experimentation and deep user psychology. His teams run 1,000 experiments a year, discovering counterintuitive insights that have driven tens of millions in revenue.What you’ll learn:1. How to use the explore-exploit framework to find new growth opportunities2. How showing premium features to free users doubled Grammarly’s upgrades to paid plans3. What good retention looks like for a consumer subscription app4. Why resurrected users drive 80% of mature product growth5. Why “reverse trials” work better than time-based trials6. The three pillars of successful gamification: core loop, metagame, and profile —Brought to you by:Vanta—Automate compliance. Simplify security.Jira Product Discovery—Confidence to build the right thingMiro—A collaborative visual platform where your best work comes to life—Where to find Albert Cheng:• X: https://x.com/albertc248• LinkedIn: https://www.linkedin.com/in/albertcheng1/• Chess.com: https://www.chess.com/member/Goniners—Where to find Lenny:• Newsletter: https://www.lennysnewsletter.com• X: https://twitter.com/lennysan• LinkedIn: https://www.linkedin.com/in/lennyrachitsky/—Referenced:• How Duolingo reignited user growth: https://www.lennysnewsletter.com/p/how-duolingo-reignited-user-growth• Inside ChatGPT: The fastest-growing product in history | Nick Turley (Head of ChatGPT at OpenAI): https://www.lennysnewsletter.com/p/inside-chatgpt-nick-turley• Explore vs. Exploit: https://brianbalfour.com/quick-takes/explore-vs-exploit• Grammarly: https://www.grammarly.com/• Reforge: https://www.reforge.com/• Chess.com: https://www.chess.com/• Everyone’s an engineer now: Inside v0’s mission to create a hundred million builders | Guillermo Rauch (founder & CEO of Vercel, creators of v0 and Next.js): https://www.lennysnewsletter.com/p/everyones-an-engineer-now-guillermo-rauch• Building Lovable: $10M ARR in 60 days with 15 people | Anton Osika (CEO and co-founder): https://www.lennysnewsletter.com/p/building-lovable-anton-osika• Figma: https://www.figma.com/• Cursor: https://cursor.com/• The rise of Cursor: The $300M ARR AI tool that engineers can’t stop using | Michael Truell (co-founder and CEO): https://www.lennysnewsletter.com/p/the-rise-of-cursor-michael-truell• Claude Code: https://www.anthropic.com/claude-code• GitHub Copilot: https://github.com/features/copilot• Noam Lovinsky on LinkedIn: https://www.linkedin.com/in/noaml/• The happiness and pain of product management | Noam Lovinsky (Grammarly, Facebook, YouTube, Thumbtack): https://www.lennysnewsletter.com/p/the-happiness-and-pain-of-product• Kyla Siedband on LinkedIn: https://www.linkedin.com/in/kylasiedband/• The Duolingo handbook: https://blog.duolingo.com/handbook/• Lenny’s post on X about the Duolingo handbook: https://x.com/lennysan/status/1889008405584683091• The rituals of great teams | Shishir Mehrotra of Coda, YouTube, Microsoft: https://www.lennysnewsletter.com/p/the-rituals-of-great-teams-shishir• Duolingo on TikTok: https://www.tiktok.com/@duolingo• Kasparov vs. Deep Blue | The Match That Changed History: https://www.chess.com/article/view/deep-blue-kasparov-chess• Magnus Carlsen: https://en.wikipedia.org/wiki/Magnus_Carlsen• Elo rating system: https://www.chess.com/terms/elo-rating-chess• Stockfish: https://en.wikipedia.org/wiki/Stockfish_(chess)• AlphaGo on Prime Video: https://www.primevideo.com/detail/AlphaGo/0KNQHKKDAOE8OCYKQS9WSSDYN0• Statsig: https://www.statsig.com/• The State of Product in 2026: Navigating Change, Challenge, and Opportunity: https://www.atlassian.com/blog/announcements/state-of-product-2026• Erik Allebest on LinkedIn: https://www.linkedin.com/in/erikallebest/• Daniel Rensch on X: https://x.com/danielrensch• Chariot: https://en.wikipedia.org/wiki/Chariot_(company)• San Francisco 49ers: https://www.49ers.com/• Breville Barista Express: https://www.breville.com/en-us/product/bes870—Recommended books:• Snuggle Puppy!: A Little Love Song: https://www.amazon.com/Snuggle-Puppy-Little-Boynton-Board/dp/1665924985• Ogilvy on Advertising: https://www.amazon.com/Ogilvy-Advertising-David/dp/039472903X• Dark Squares: How Chess Saved My Life: https://www.amazon.com/Dark-Squares-Chess-Saved-Life/dp/1541703286—Production and marketing by https://penname.co/. For inquiries about sponsoring the podcast, email [email protected] may be an investor in the companies discussed. To hear more, visit www.lennysnewsletter.com
    --------  
    1:25:25
  • A 4-step framework for building delightful products | Nesrine Changuel (Spotify, Google, Skype)
    Nesrine Changuel helped build Spotify, Google Chrome, and Google Meet. Her work has helped her discover the importance of emotional connection in building successful products. At Google, she served as a dedicated “delight PM,” a role specifically focused on making products more delightful. She recently published Product Delight, a book that provides a practical framework for creating products that serve both functional and emotional needs. Based in Paris, she now coaches founders and CPOs on implementing delight strategies in their organizations.What you’ll learn:1. Why delight is a business strategy, not just “sprinkling confetti” on top of functionality2. How to identify emotional motivators that drive product retention3. The 50-40-10 rule for balancing delight in your roadmap4. The 4-step delight model5. The origin story of Spotify’s Discover Weekly6. Why B2B products need delight just as much as B2C products7. How to get buy-in from skeptical leaders who think delight is a luxury—Brought to you by:DX—The developer intelligence platform designed by leading researchers: https://getdx.com/lennyJira Product Discovery—Confidence to build the right thing: https://atlassian.com/lennyLucidLink—Real-time cloud storage for teams: https://www.lucidlink.com/lenny—Transcript: https://www.lennysnewsletter.com/p/a-4-step-framework-for-building-delightful-products—My biggest takeaways (for paid newsletter subscribers): https://www.lennysnewsletter.com/i/174199489/my-biggest-takeaways-from-this-conversation—Where to find Nesrine Changuel:• LinkedIn: https://www.linkedin.com/in/nesrinechanguel/• Newsletter: https://nesrinechanguel.substack.com/• Website: https://nesrine-changuel.com/—Where to find Lenny:• Newsletter: https://www.lennysnewsletter.com• X: https://twitter.com/lennysan• LinkedIn: https://www.linkedin.com/in/lennyrachitsky/—In this episode, we cover:(00:00) Introduction to Nesrine and product delight(04:56) Why delight matters(09:17) What makes a feature “delightful”(12:29) The three pillars of delight(13:03) Pillar 1: Removing friction (Uber refund example)(15:07) Pillar 2: Anticipating needs (Revolut eSIM example)(17:21) Pillar 3: Exceeding expectations (Edge coupon example)(18:35) The “confetti effect” and when it actually works(22:02) B2B vs. B2C: Why all products need emotional connection(29:52) The Delight Model: A 4-step framework(30:57) Step 1: Identifying user motivators (functional and emotional)(33:55) Step 2: Converting motivators into product opportunities(34:46) Step 3: Identifying solutions with the delight grid(36:46) Step 4: Validating ideas with the delight checklist(40:22) The Delight Model summarized(42:18) The importance of familiarity (Spotify Discover Weekly story)(45:21) Real examples: Chrome’s tab management solution(51:32) Google Meet’s solution for “Zoom fatigue”(55:02) Getting buy-in from skeptical leaders(59:39) Prioritizing delight: The 50-40-10 rule(1:02:41) Creating a culture of delight in your organization(1:06:45) The habituation effect(1:08:15) When delight goes wrong: Apple reactions example(1:10:21) How delight motivates product teams(1:12:24) Lightning round and final thoughts—Referenced:• Spotify: https://open.spotify.com/• Linear: https://linear.app/• How Linear builds product: https://www.lennysnewsletter.com/p/how-linear-builds-product• Jira: https://www.atlassian.com/software/jira• Asana: https://asana.com/• Monday: https://monday.com/• The Product Delight Model: https://nesrinechanguel.substack.com/p/the-product-delight-model• Revolut: https://www.revolut.com/• How Revolut trains world-class product managers: The “local CEO” model, raw intellect over experience, and a cultural obsession with building wow products | Dmitry Zlokazov (Head of Product): https://www.lennysnewsletter.com/p/how-revolut-trains-world-class-product-managers• Microsoft Cashback: https://www.microsoft.com/en-us/edge/features/shopping-cashback• Superhuman’s secret to success: Ignoring most customer feedback, manually onboarding every new user, obsessing over every detail, and positioning around a single attribute: speed | Rahul Vohra (CEO): https://www.lennysnewsletter.com/p/superhumans-secret-to-success-rahul-vohra• Brian Chesky’s secret mentor who died 9 times, started the Burning Man board, and built the world’s first midlife wisdom school | Chip Conley (founder of MEA): https://www.lennysnewsletter.com/p/chip-conley• Workday: https://www.workday.com/• SAP: https://www.sap.com/• ServiceNow: https://www.servicenow.com/• Salesforce: https://www.salesforce.com/• GitHub: https://github.com/• Atlassian: https://www.atlassian.com/• Snowflake: https://www.snowflake.com/• Data Superheroes: https://www.snowflake.com/en/data-superheroes/• Google Meet: https://meet.google.com/• Andy Nesling on LinkedIn: https://www.linkedin.com/in/andynesling/• Matic: https://maticrobots.com/• Diego Sanchez’s (Senior Product Manager at Buffer) post on LinkedIn: https://www.linkedin.com/feed/update/urn:li:activity:7365014292091346945/• Miro: https://miro.com/• Arc browser: https://arc.net/• Competing with giants: An inside look at how The Browser Company builds product | Josh Miller (CEO): https://www.lennysnewsletter.com/p/competing-with-giants-an-inside-look• Migros Supermarket: https://www.migros.ch/• The rise of Cursor: The $300M ARR AI tool that engineers can’t stop using | Michael Truell (co-founder and CEO): https://www.lennysnewsletter.com/p/the-rise-of-cursor-michael-truell• Building Lovable: $10M ARR in 60 days with 15 people | Anton Osika (CEO and co-founder): https://www.lennysnewsletter.com/p/building-lovable-anton-osika• Linear’s secret to building beloved B2B products | Nan Yu (Head of Product): https://www.lennysnewsletter.com/p/linears-secret-to-building-beloved-b2b-products-nan-yu• Suno: https://suno.com• Snapchat: https://www.snapchat.com/• Use Reactions, Presenter Overlay, and other effects when videoconferencing on Mac: https://support.apple.com/en-us/105117• Dr. Lipp: https://drlipp.com/• How to be the best coach to product people | Petra Wille (Strong Product People): https://www.lennysnewsletter.com/p/how-to-be-the-best-coach-to-product• The Great American Baking Show: https://www.imdb.com/title/tt21822674/• Le Meilleur Pâtissier: https://en.wikipedia.org/wiki/Le_Meilleur_P%C3%A2tissier• The Upside on Amazon Prime: https://www.amazon.com/gp/video/detail/amzn1.dv.gti.3cb8500f-31af-9f4f-5dec-701e086d58e8• The Intouchables: https://www.imdb.com/title/tt1675434/• Yoyo stroller: https://www.stokke.com/USA/en-us/category/strollers/yoyo-strollers• UppaBaby strollers: https://uppababy.com/strollers/—Recommended books:• Product Delight: How to Make Your Product Stand Out with Emotional Connection: https://www.amazon.com/Product-Delight-Stand-Emotional-Connection-ebook/dp/B0FGZ93D9Y/• Factfulness: Ten Reasons We’re Wrong About the World—and Why Things Are Better Than You Think: https://www.amazon.com/Factfulness-Reasons-World-Things-Better/dp/1250107814• STRONG Product Communities: The Essential Guide to Product Communities of Practice: https://www.amazon.com/STRONG-Product-Communities-Essential-Practice/dp/3982235189/r—Production and marketing by https://penname.co/. For inquiries about sponsoring the podcast, email [email protected] may be an investor in the companies discussed. To hear more, visit www.lennysnewsletter.com
    --------  
    1:24:22
  • Why AI evals are the hottest new skill for product builders | Hamel Husain & Shreya Shankar (creators of the #1 eval course)
    Hamel Husain and Shreya Shankar teach the world’s most popular course on AI evals and have trained over 2,000 PMs and engineers (including many teams at OpenAI and Anthropic). In this conversation, they demystify the process of developing effective evals, walk through real examples, and share practical techniques that’ll help you improve your AI product.What you’ll learn:1. WTF evals are2. Why they’ve become the most important new skill for AI product builders3. A step-by-step walkthrough of how to create an effective eval4. A deep dive into error analysis, open coding, and axial coding5. Code-based evals vs. LLM-as-judge6. The most common pitfalls and how to avoid them7. Practical tips for implementing evals with minimal time investment (30 minutes per week after initial setup)8. Insight into the debate between “vibes” and systematic evals—Brought to you by:Fin—The #1 AI agent for customer serviceDscout—The UX platform to capture insights at every stage: from ideation to productionMercury—The art of simplified finances—Where to find Shreya Shankar• X: https://x.com/sh_reya• LinkedIn: https://www.linkedin.com/in/shrshnk/• Website: https://www.sh-reya.com/• Maven course: https://bit.ly/4myp27m—Where to find Hamel Husain• X: https://x.com/HamelHusain• LinkedIn: https://www.linkedin.com/in/hamelhusain/• Website: https://hamel.dev/• Maven course: https://bit.ly/4myp27m—In this episode, we cover:(00:00) Introduction to Hamel and Shreya(04:57) What are evals?(09:56) Demo: Examining real traces from a property management AI assistant(16:51) Writing notes on errors(23:54) Why LLMs can’t replace humans in the initial error analysis(25:16) The concept of a “benevolent dictator” in the eval process(28:07) Theoretical saturation: when to stop(31:39) Using axial codes to help categorize and synthesize error notes(44:39) The results(46:06) Building an LLM-as-judge to evaluate specific failure modes(48:31) The difference between code-based evals and LLM-as-judge(52:10) Example: LLM-as-judge(54:45) Testing your LLM judge against human judgment(01:00:51) Why evals are the new PRDs for AI products(01:05:09) How many evals you actually need(01:07:41) What comes after evals(01:09:57) The great evals debate(1:15:15) Why dogfooding isn’t enough for most AI products(01:18:23) OpenAI’s Statsig acquisition(1:23:02) The Claude Code controversy and the importance of context(01:24:13) Common misconceptions around evals(1:22:28) Tips and tricks for implementing evals effectively(1:30:37) The time investment(1:33:38) Overview of their comprehensive evals course(1:37:57) Lightning round and final thoughts—LLM Log Open Codes Analysis Prompt:Please analyze the following CSV file. There is a metadata field which has an nested field called z_note that contains open codes for analysis of LLM logs that we are conducting. Please extract all of the different open codes. From the _note field, propose 5-6 categories that we can create axial codes from.—Referenced:• Building eval systems that improve your AI product: https://www.lennysnewsletter.com/p/building-eval-systems-that-improve• Mercor: https://mercor.com/• Brendan Foody on LinkedIn: https://www.linkedin.com/in/brendan-foody-2995ab10b• Nurture Boss: https://nurtureboss.io/• Braintrust: https://www.braintrust.dev/• Andrew Ng on X: https://x.com/andrewyng• Carrying Out Error Analysis: https://www.youtube.com/watch?v=JoAxZsdw_3w• Julius AI: https://julius.ai/• Brendan Foody on X—“evals are the new PRDs”: https://x.com/BrendanFoody/status/1939764763485171948• Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences: https://dl.acm.org/doi/abs/10.1145/3654777.3676450• Lenny’s post on X about evals: https://x.com/lennysan/status/1909636749103599729• Statsig: https://statsig.com/• Claude Code: https://www.anthropic.com/claude-code• Cursor: https://cursor.com/• Occam’s razor: https://en.wikipedia.org/wiki/Occam%27s_razor• Frozen: https://www.imdb.com/title/tt2294629/• The Wire on HBO: https://en.wikipedia.org/wiki/The_Wire—Recommended books:• Pachinko: https://www.amazon.com/Pachinko-National-Book-Award-Finalist/dp/1455563935• Apple in China: The Capture of the World’s Greatest Company: https://www.amazon.com/Apple-China-Capture-Greatest-Company/dp/1668053373/• Machine Learning: https://www.amazon.com/Machine-Learning-Tom-M-Mitchell/dp/1259096955• Artificial Intelligence: A Modern Approach: https://www.amazon.com/Artificial-Intelligence-Modern-Approach-Global/dp/1292401133/Production and marketing by https://penname.co/. For inquiries about sponsoring the podcast, email [email protected].—Lenny may be an investor in the companies discussed.My biggest takeaways from this conversation: To hear more, visit www.lennysnewsletter.com
    --------  
    1:46:33

More Business podcasts

About Lenny's Podcast: Product | Career | Growth

Interviews with world-class product leaders and growth experts to uncover concrete, actionable, and tactical advice to help you build, launch, and grow your own product.
Podcast website

Listen to Lenny's Podcast: Product | Career | Growth, The Business and many other podcasts from around the world with the radio.net app

Get the free radio.net app

  • Stations and podcasts to bookmark
  • Stream via Wi-Fi or Bluetooth
  • Supports Carplay & Android Auto
  • Many other app features

Lenny's Podcast: Product | Career | Growth: Podcasts in Family

Social
v7.23.9 | © 2007-2025 radio.de GmbH
Generated: 10/13/2025 - 6:09:42 PM