DataFramed

For forty years, the rule held that transactional and analytical databases had to be separate systems, connected by fragile pipelines that move data from one to the other. That assumption is now being questioned. As AI agents start generating the majority of database activity, the old architecture is being redesigned around speed, scale, and a single copy of governed data. For anyone who works with data day to day, this raises practical questions. Do you still need separate systems for live and historical data? What happens to the pipelines you maintain? And how does your stack change when agents, not people, write most of the queries?
Reynold Xin is co-founder and Chief Architect of Databricks. He is one of the original creators of Apache Spark, where he led the design of GraphX, Project Tungsten, and Structured Streaming, co-designed DataFrames, and served as release manager for Spark 2.0. He holds a PhD in Computer Science from UC Berkeley's AMPLab and a degree in Engineering Science from the University of Toronto.
In the episode, Richie and Reynold explore self-service analytics with Genie, the ontology layer that grounds AI in enterprise data, handling hallucinations, governance and permissions for AI agents, merging transactional and analytical databases with Lakebase and LTAP, real-time analytics, controlling cost through autoscaling, the future of Spark and classic machine learning, and much more.
Links Mentioned in the Show:
• Connect with Reynold: https://www.linkedin.com/in/rxin
• Genie (Databricks data agent): https://www.databricks.com/product/genie
• Genie Ontology / Genie One: https://www.databricks.com/blog/introducing-genie-one-genie-ontology-and-genie-agents
• LTAP (Lake Transactional/Analytical Processing): https://www.databricks.com/company/newsroom/press-releases/databricks-launches-ltap-first-lake-transactionalanalytical
• Lakehouse//RT: https://www.databricks.com/blog/introducing-lakehousert-real-time-performance-unified-lakehouse
• Lakebase: https://www.databricks.com/product/lakebase
• Apache Spark: https://spark.apache.org
• AI-Native Course: Intro to AI for Work - https://www.datacamp.com/courses/introduction-to-ai-for-work
• Related Episode: AI's Impact on Databases - https://www.datacamp.com/podcast/ais-impact-on-databases
New to DataCamp? Learn on the go using the DataCamp mobile app - https://www.datacamp.com/mobile
Empower your business with world-class data and AI skills with DataCamp for business - https://www.datacamp.com/business

Across the AI industry, capability has exploded while trustworthiness has lagged badly behind. The same technology that writes fluent prose can invent a refund policy that was never real, and most of those errors are subtle enough that no one notices. As more teams hand high-stakes work to AI — in banking, healthcare, customer service — the cost of confident mistakes adds up fast. So how common are hallucinations, really? Can chaining models together or adding humans to the loop fix it? And is reliability something you can design into a system from the start?
Dan Klein is the CTO and co-founder of Scaled Cognition and a professor of computer science at UC Berkeley, where he leads the Berkeley NLP Group within the Berkeley AI Research (BAIR) Lab. He previously co-founded Semantic Machines, a conversational AI company acquired by Microsoft in 2018. At Scaled Cognition he built APT (Agentic Pretrained Transformer), a frontier model designed from the ground up for reliable, policy-adherent agentic AI.
In the episode, Richie and Dan explore why AI reliability has lagged behind capability, how hallucinations hide in plain sight, the limits of humans-in-the-loop and LLM-as-judge, building reliability into model architecture, agentic systems and verifiable actions, test-driven agent development, the skills that stay valuable, digital literacy, and much more.
Links Mentioned in the Show:
• Connect with Dan: https://www.linkedin.com/in/dan-klein/
• Scaled Cognition: https://www.scaledcognition.com/
• Berkeley NLP Group: https://nlp.cs.berkeley.edu/
• Code smells (Martin Fowler): https://martinfowler.com/bliki/CodeSmell.html
• Refactoring, by Martin Fowler: https://martinfowler.com/books/refactoring.html
• "Now you have two problems" (Jamie Zawinski quote): https://regex.info/blog/2006-09-15/247
• Lean theorem prover: https://lean-lang.org/
• AI-Native Course: Intro to AI for Work - https://www.datacamp.com/courses/introduction-to-ai-for-work
• Related Episode: How to Build AI Your Users Can Trust with David Colwell, VP of AI & ML at Tricentis - https://www.datacamp.com/podcast/how-to-build-ai-your-users-can-trust
New to DataCamp? Learn on the go using the DataCamp mobile app - https://www.datacamp.com/mobile
Empower your business with world-class data and AI skills with DataCamp for business - https://www.datacamp.com/business

AI agents are no longer limited to automating routine tasks like customer support or report generation. Research labs and pharmaceutical companies are beginning to deploy teams of specialist AI agents capable of designing experiments, analyzing data, and proposing new hypotheses — in some cases producing results that outperform human experts. For data scientists and researchers, this raises urgent questions: Where do AI agents excel in scientific workflows today, and where do they fall short? How do you build an agent that can genuinely innovate rather than just replicate what's already been done? And what does it take to scale a single model into a fully functioning virtual research team?
James Zou is an Associate Professor of Biomedical Data Science, and by courtesy of Computer Science and Electrical Engineering, at Stanford University. He leads the Stanford AI for Science Lab and is affiliated with Together AI. His research focuses on building AI agents for scientific discovery and data science, making AI more reliable and statistically rigorous. He has received a Sloan Fellowship, NSF CAREER Award, two Chan-Zuckerberg Investigator Awards, and faculty awards from Google, Amazon, and Adobe.
In the episode, Richie and James explore how AI scientist agents are already outperforming human experts in scientific discovery, the Virtual Lab framework for building teams of specialist AI agents that conduct real research, teaching models to innovate not just imitate through a new training paradigm called "learning to discover," DS Gym for self-improving data science agents, scaling agentic systems from a single model to a Virtual Biotech with tens of thousands of agents, Einstein Arena as the first competition platform built exclusively for AI agents, converting scientific papers into agent-native MCPs through Paper to Agent, and much more.
Links Mentioned in the Show:
• Virtual Lab (Nature paper)
• Einstein Arena
• DS Gym
• Paper2Agent
• Together AI
• AlphaFold 2 / Nobel Prize 2024
• Connect with James
• AI-Native Course: Intro to AI for Work
• Related Episode: #358 How AI Agents Will Work While You Sleep | Ruslan Salakhutdinov
New to DataCamp?
• Learn on the go using the DataCamp mobile app
• Empower your business with world-class data and AI skills with DataCamp for business

Most organizations know AI matters, but few have turned that conviction into a written plan. Ambition and hope are everywhere; a clear roadmap tied to business strategy is rare. For teams on the ground, this gap shows up as scattered initiatives, tools nobody fully uses, and a lot of activity that never adds up to real value. So where do you actually start? How do you move from a long list of use cases to a focused plan you can execute? And who in the organization should own the job of turning AI into business results?
Charlene Li is a New York Times bestselling author and strategic advisor who has spent more than two decades helping leaders navigate disruptive change. She founded Altimeter Group, has advised 49 of the Fortune 100, and is the co-author of Winning with AI: The 90-Day Blueprint for Success (with Dr. Katia Walsh).
In the episode, Richie and Charlene explore how to get your organization AI-ready in 90 days, why you don't need a separate AI strategy, appointing an AI value owner, creating value beyond efficiency, building AI fluency, Goldilocks governance, why you should kill your AI pilots, and much more.
Links Mentioned in the Show:
Winning with AI: The 90-Day Blueprint for Success
Dr. Katia Walsh (co-author)
Moderna
Konecta
IKEA
Andrej Karpathy's LLM Wiki
Connect with Charlene: LinkedIn
AI-Native Course: Intro to AI for Work
Related Episode: Our Data Trends & Predictions for 2026 with Jonathan Cornelissen & Martijn Theuwissen

New to DataCamp? Learn on the go using the DataCamp mobile app - https://www.datacamp.com/mobile
Empower your business with world-class data and AI skills with DataCamp for business - https://www.datacamp.com/business

AI agents are starting to handle parts of the shopping journey that used to require human judgment — discovery, comparison, checkout. But behind every agent recommendation is a massive, invisible layer of data infrastructure. Product catalogs need to be structured, inventory synced in real time, pricing accurate, and quality signals clear. For data engineers and teams building at companies like Shopify, this shift means rethinking how data flows through systems and what "good enough" quality actually means. How do you ensure data is ready for AI? And how is this reshaping what data teams actually do?
Nell Thomas is the VP of Data at Shopify, where she leads a team of approximately 400–500 people across data infrastructure, ML platforms, data engineering, and data science. Her career spans multiple industries including social media (Facebook), e-commerce (Etsy), politics (Hillary for America, Democratic National Committee), and now commerce. She holds an A.B. in Psychology from Harvard University and an M.A. in History & Sociology of Science from the University of Pennsylvania.
In the episode, Richie and Nell explore agentic commerce and how AI agents are transforming shopping, the role of data in enabling AI-driven commerce, Shopify's Catalog and Universal Commerce Protocol, data quality requirements for agentic systems, how the data team function is evolving at Shopify, changing skill requirements for data professionals, and Nell's unconventional career path from politics to tech.
Links Mentioned in the Show:
- Agentic Commerce on Shopify
- Universal Commerce Protocol (UCP) vs Agentic Commerce Protocol (ACP)
- Shopify Catalog Documentation
- Agentic Storefronts — Shopify Sales Channel
- ChatGPT — OpenAI's Conversational AI
- How Shopify Built Data Infrastructure at Scale
Related
Scaling Data Quality in the Age of Generative AI
New to DataCamp? Learn on the go using the DataCamp mobile app - https://www.datacamp.com/mobile
Empower your business with world-class data and AI skills with DataCamp for business - https://www.datacamp.com/business