The MAD Podcast: How AI Gets Built — with Matt Turck

AGI in 2027. That’s the last line of this year’s State of AI Report, and its author Nathan Benaich (Air Street Capital) is back for the third year to defend it. AI is now building AI: Claude leads 26% of Anthropic’s own model R&D, up from under 1% in February, but Nathan thinks “research taste” is the wall. OpenAI’s evaluation agents ended up inside Hugging Face’s systems, only a Chinese open-weight model could help investigate, and Nathan now argues cyber risk matters more than existential risk. Catching up to OpenAI or Anthropic from scratch costs about a trillion dollars, which is why “you either die trying to get to the frontier, or live long enough to serve inference.” Americans would rather live next to a nuclear plant than a data center, $50B of projects are stalled, and all 27 EU countries together have fewer GPUs than one top US cluster. Sovereign AI is “like a national airline: it’s still someone else’s airplane.” Also: the three-lab race and the fate of the neo labs, models vs. harnesses, why he won’t give a personal AI agent his credit card, the Codex adoption gap and why OpenAI should open a store, enterprises owning their own AI instead of renting it, and the morning a solo GP woke up to twelve expert interviews his agent ran overnight.

(00:00) Cold open and intro
(00:56) Why Nathan Benaich predicts AGI in 2027
(02:28) AI building AI: Claude leads 26% of Anthropic’s research
(04:00) Research taste: can AI pick the right problem?
(08:34) Doomerism is back, but cyber risk beats extinction
(10:13) Prediction: a frontier model’s weights get stolen
(13:05) How OpenAI’s agents ended up attacking Hugging Face
(15:29) Why only a Chinese open-weight model could investigate
(17:33) What frontier-lab cyber defense could look like
(19:08) Self-replicating agents and the risks of AI swarms
(21:16) Can frontier labs agree to slow down?
(24:03) Why communities are blocking $50B of data centers
(24:44) Nuclear plant vs. data center: what Americans prefer
(26:45) Europe’s AI bottleneck: 80,000 GPUs and electricity costs
(28:23) Where Europe can still win in AI
(29:31) Catching up to OpenAI costs $1 trillion
(29:43) Die trying for the frontier, or serve inference
(30:31) How inference chips could give countries AI leverage
(33:34) Why NVIDIA GPUs don’t guarantee AI sovereignty
(36:14) The UK’s problem: “but we have DeepMind”
(37:07) Why European defense AI needs domestic suppliers
(39:41) Why OpenAI, Anthropic and Google keep pulling ahead
(41:17) Where independent AI labs can still compete
(43:26) World models: from robotics training to game engines
(45:39) Would cheaper AI undermine the compute boom?
(46:43) Will better models absorb the agent harness?
(49:41) AI due diligence: 12 expert interviews in 2.5 days
(53:35) The trust problem with personal AI agents
(56:22) Why prompt engineering put people off AI
(59:02) Does AI need its own Genius Bar?
(1:00:25) Codex beyond coding: agents for knowledge work
(1:02:49) Why enterprises want to own their AI, not rent it
(1:05:17) Rapid fire: home robots, NVIDIA returns and Arm

Mistral AI is back in the frontier AI race. Mistral Large 4, nicknamed “Le Chonk,” is a 1-trillion-parameter open-weight model, trained in two months on 3,800 GPUs in Mistral’s own European datacenters — and independently rated the best open model outside China alongside Meta’s Muse Spark Open. After a year when open-weight AI meant DeepSeek, Kimi, Qwen and GLM, October brought a wave of Western open models, with Mistral at the front of it. Pierre Stock, Mistral’s VP of Science and first employee, walks through how the model was built: mixture-of-experts explained, what Mistral borrowed from DeepSeek, reinforcement learning at scale on coding, cyber, finance and chip design, why the RL run is still going, the model that pinged its own trainer from inside the sandbox, open weights vs cyber refusals, why 10-trillion-parameter models are coming, and Mistral Large 5, already in pretraining.

(00:00) Cold open
(00:59) The Western open-weight comeback: Beam, Le Chonk, Muse, Nemotron
(04:09) The US shutdown of Anthropic’s models — “AI is like electricity”
(06:07) Why “Le Chonk”
(07:56) “Mistral fell behind”: the comeback from the inside
(10:29) Training on 3,800 GPUs in Mistral’s own datacenters
(13:24) Meta keeps its best model closed, Mistral opens Chonk
(15:40) Chonk vs Beam, Muse Spark Open and the Chinese models
(18:44) Why Chinese labs are still at the frontier
(21:08) Pierre’s story: FAIR, Yann LeCun, the “Mistral mafia”
(25:14) From compressing networks to a trillion parameters
(27:30) Mixture of experts explained: the hospital analogy
(32:25) 10-trillion-parameter models are coming
(35:29) What Mistral borrowed from DeepSeek
(36:46) One model, several reasoning modes
(39:15) Native multimodality: satellites, manuals, computer use
(41:56) Data, synthetic data and model collapse
(46:32) Reinforcement learning as an exam
(49:59) Where the efficiency comes from
(52:18) Le Chonk is still training right now
(53:39) What RL trains on: coding, cyber, finance, chips
(58:13) Partial rewards and looping penalties
(59:58) Did RL teach anything new? pass@k
(1:03:16) The sandbox story: the model that pinged its trainer
(1:07:44) Open weights vs cyber refusals
(1:12:05) Timestral: time series on a CPU
(1:14:18) AI doing AI research
(1:16:22) Coming up: Mistral Large 5

What is Jev? TypeSafe’s Jev is a new kind of AI model, a “decision model,” that doesn’t write text. It makes fast, cheap, calibrated decisions inside software, and it has taken the AI world by storm: within weeks of launch it became the most copied model in AI, with OpenAI, Cloudflare, Amazon and Databricks all shipping their own versions, and TypeSafe just raised an $870 million Series A at a $7.5 billion valuation. In this episode, Jev’s creator Diogo Almeida, co-inventor of RLHF at OpenAI and now CEO of TypeSafe, explains Jev in plain English: what a “System One” or decision model actually is, why ChatGPT-style language models can’t be trusted inside software (“95% Einstein, 5% Mr. Bean”), how RLHF and RLVR created the problem, what RLCD (reinforcement learning for calibrated decisions) does differently, where the training data comes from, which models Jev is built on, why “just a classifier” is a compliment, and why he calls pre-training “a giant dumpster fire of wasting resources.”

We also cover the copycats and why he thinks they missed the hard part, why benchmarks are useless, whether Jev will go open source, the Jevons paradox business model behind the name, why he doesn’t believe AI is on the path to recursive self-improvement, and why he thinks the AI winter he feared has been averted.

(00:00) Cold open & Intro: “95% Einstein, 5% Mr. Bean”
(01:09) Jev’s breakout launch: why intelligence per dollar matters
(03:25) Why TypeSafe built in stealth for two years
(05:50) “We could have had AGI last year”
(06:48) Why the product carries the marketing
(09:33) What is Jev? Machine-native AI, built for software
(10:53) System One vs. System Two: the limits of reasoning
(12:26) From simple decisions to composable AI
(15:21) Automating email: giant prompts vs. smart if-statements
(20:03) Probabilities, thresholds, and developer control
(23:43) Why Jev is faster and cheaper than chat models
(25:39) “Classification is AI-complete”
(27:16) RLHF: how OpenAI trained models to please humans
(31:28) Sycophancy, refusals, and the hidden costs of RLHF
(34:22) RLVR: why benchmark gains don’t mean reliable automation
(37:51) RLCD: optimizing for machine-native intelligence
(39:54) Why TypeSafe builds its training data in-house
(41:49) The Bitterest Lesson: define the task before scaling
(44:39) Building Jev from open-weight models
(45:56) Is Jev just a classifier? Why that’s a compliment
(48:35) “Build prod, not God”
(50:11) Embedding AI in software, beyond coding assistants
(53:05) Jev’s copycats: matching the interface vs. the intelligence
(55:33) Why benchmarks mislead the users they should help
(58:37) Open weights vs. APIs: how TypeSafe weighs the tradeoff
(1:00:04) Jevons paradox: a new market for machine intelligence
(1:02:50) Why Diogo is skeptical of recursive self-improvement
(1:04:47) Can practical automation avert an AI winter?

AI agents are becoming the biggest users of databases: creating them, querying them, and sometimes deleting them. Andy Pavlo, Carnegie Mellon's database professor and now VP of Database Research at ClickHouse, explains what changes when billions of agents hit the data layer, and why most of the AI world's mental model of databases stopped at vector databases and RAG.

Andy's CMU courses taught a generation of engineers (and, it turns out, the AI models); he writes the year-in-review on databases the whole industry reads; and this summer he joined ClickHouse to found ClickHouse Labs. We cover why vector databases were "just an index," what it means for an agent to create a database, why agents could mean 10 to 100x more queries, whether files or databases should hold an agent's memory, how text-to-SQL went from 60% to 99.5% accuracy, why 60% of open-source databases now have commits from coding agents, and what ten years of self-driving database research taught him. Plus ClickHouse, Postgres, the vector/graph/GPU database verdicts, Larry Ellison, and the Wu-Tang Clan.

Disclosure: FirstMark, where Matt is a General Partner, is an investor in ClickHouse.

(00:00) Cold open & Intro
(01:21) From vector databases and RAG to the age of agents
(04:09) Neon's stat: agents create 80% of databases?
(07:25) Why agents keep deleting production databases
(08:19) Guardrails: the toddler-and-stairs rule
(10:51) The four eras of database volume: 10–100x more queries
(15:10) Agent memory: files vs. databases ("everything is a database")
(18:24) Which database do AI models recommend? The new SEO
(19:25) "It cites me back to myself"
(23:14) MCP for databases, and text-to-SQL from 60% to 99.5%
(27:09) Trust an agent the way you'd trust a junior developer
(27:47) Can AI build an entire database? Opus 4 and the CMU projects
(29:57) 60% of open-source databases now have AI commits
(30:43) Ten years of self-driving databases, Peloton to today
(33:52) What LLMs changed: 85% of the tuning in 15 minutes
(35:42) Should anyone still study databases?
(38:27) Free CMU courses, the DJ, and the Wu-Tang final exam
(42:09) Why start a research lab inside ClickHouse?
(45:53) Why ClickHouse looked like vaporware in 2016
(47:46) What makes ClickHouse fast: columns, vectors, Snowflake's lineage
(52:22) Why Databricks, Snowflake and ClickHouse all added Postgres
(58:15) Vector, graph and GPU databases: thumbs up or down?
(1:08:34) Is the database market stagnant? "A cheetah on cocaine in a Ferrari"
(1:10:14) The relational model is arithmetic; SQL as the new assembly
(1:11:59) Larry Ellison, Linux, and why databases still matter

This summer, AI agents broke out of their test sandboxes at every major lab and hacked real companies, and the safety tests didn't see it coming. Eric Ho is the co-founder and CEO of Goodfire, the Anthropic-backed mechanistic interpretability lab whose new paper, "Models Know When They're Reward Hacking," found that top open-source AI models cheat on agent tasks up to 96% of the time, that a clear "cheating" signal exists inside the model, and that cheap activation probes can catch it, including hacks that chain-of-thought monitors miss. In this conversation, Eric explains reward hacking in plain terms (AI agents as "amoral students with a mostly absent teacher"), why reinforcement learning turned a known problem into a crisis, why chain-of-thought monitoring is fading as models start to think in "neuralese," and why nobody, including the people who build them, understands how these models work.

We then go inside the science: what interpretability actually is, what probes and steering do, how Goodfire found the cheating signal and turned it up and down, and what can be done about it, from real-time activation monitoring to "intentional design," where you give gradient descent a choice about what the model learns. Along the way: a model caught planning its hack to evade its own monitor, a new Alzheimer's biomarker discovered inside a diagnostic model, why only a few hundred people in the world work on this, and Eric's bet that we'll decode neural networks by 2028.

(00:00) Cold open & intro
(01:06) Amoral students with an absent teacher
(02:49) What is reward hacking?
(04:13) Models cheat up to 96% of the time
(06:08) Do models know they're cheating?
(08:10) The Hugging Face hack: why tests missed it
(11:35) "The dumbest models we'll ever deal with"
(11:58) Should AI slow down?
(13:56) Mechanistic interpretability 101
(16:06) Models are grown, not built
(19:10) How labs do alignment today
(21:30) Is this an alignment crisis?
(23:26) Why agents changed everything
(25:38) Chain-of-thought monitoring is fading
(27:49) Neuralese: AI that stops thinking in English
(29:28) A model caught evading its own monitor
(30:42) Open vs. closed models: a frontier problem coming for everyone
(32:11) Activation monitoring in production
(35:00) Learning from superhuman AI
(36:15) The most underrated field in AI
(38:24) What is a probe?
(39:52) What is steering? Golden Gate Claude
(40:53) Why build Goodfire outside the labs
(42:46) Do you need frontier model access?
(44:16) Inside the paper: an MRI for the model
(47:28) Dialing sycophancy up and down
(49:16) Probes vs. chain-of-thought monitors
(51:44) Cutting monitoring costs by 90%
(53:28) So what can we do about it?
(55:41) Giving gradient descent a choice
(57:24) RL from feature rewards
(1:00:03) Silico and Goodfire's business
(1:01:25) A new Alzheimer's biomarker, found inside a model
(1:03:38) Decoding neural networks by 2028?
(1:06:00) What engineers can do tomorrow
(1:07:35) Are we at risk? "I want people to believe"