Detection at Scale

This week is a special episode of the Detection at Scale podcast. I’m usually the one asking the questions, but this time I’m in the guest chair, hosted by Julian Giuca, Panther’s Chief Product Officer. Our conversation covers the journey of building Panther’s AI SOC platform: From the evolution of the SOC from human-led to AI-enabled, the shifts of the last few years that took LLMs from “autocomplete-plus-plus” into genuinely useful agents, how security teams are actually adopting this technology in production, and what we’ve learned building these systems as the operating patterns keep evolving.
The podcast traces back to two early bets, the security data lake and detection-as-code, which we made in 2018 to solve the data scale problem before AI emerged as the next wave. Many years later, those choices turned out to be the exact foundation AI agents needed: detection logic they could read and modify, and a query layer to access huge amounts of helpful context. The work since then has been figuring out what changes when agents are the primary readers of detections, where teams actually want to land on the autonomy spectrum, and what it means for a system to close the loop rather than just close the alert.
One frame I keep coming back to from this conversation is that the risk of not adopting AI in the SOC is greater than the risk of an agent making a mistake. That has not been true for any prior generation of automation, and it changes the calculus for how aggressively teams should move. I hope you enjoy the conversation! Please leave a comment with your thoughts.

Why detections written for humans fail agents: Most detections hand an analyst a step-by-step runbook — check this, then this, then make a call. When you give that structure to an agent, you’re wasting the technology. What works is describing the threat model, the evidence you’d want to see, and the judgment criteria a senior analyst would apply, then letting the agent reason from there.
Closing the alert vs. closing the loop: Closing an alert clears the queue. Closing the loop means the system gets smarter every time it runs. An agent without native access to your detection logic can triage, but it cannot improve the underlying detection that fired, which means alert volume stays flat or grows because nothing is actually learning.

The three-part inflection that moved AI from code autocomplete to agentic work: Reasoning models, tool calling, and MCP converging together is what made agents capable of doing real investigative work in the SOC — not any single capability in isolation, but the combination of all three arriving at roughly the same time.

Architectural prerequisites most teams don’t have yet: Python-based detections and a centralized cloud data warehouse weren’t built with agents in mind, but they created exactly the foundation agents need — detection logic they can read and modify, and a data layer they can query and federate out from via MCP into sources like BigQuery, Elastic, or Snowflake.

Agent autonomy tolerance as a function of workflow risk: There’s no single right threshold for how much independence to give an agent. High-stakes workflows need tighter guardrails; routine triage can run with more autonomy. Teams applying one blanket policy across everything are either underusing the technology or taking on more risk than they realize.

From 50% to 110% alert coverage: Going from monitoring half your alerts to exceeding full coverage — with that extra capacity running as proactive threat hunting agents around the clock — isn’t theoretical. It doesn’t mean fewer security people; it means you need people who know how to work with agents, prompt them well, and encode your team’s expertise into something that scales.

What does it actually take to automate security operations when you're processing 7 trillion log lines daily and a single missed threat could compromise billions of users? Michael Sinno, Director of Detection & Response at Google, explains how his team handles this with less than 1% requiring human intervention through strategic AI implementation. He explores Google's methodical approach to AI autonomy, including fine-tuned models trained on golden datasets, validation through overseer agents, and the critical distinction between traditional automation and agentic AI that exercises judgment. 
Michael also discusses groundbreaking work with Sec-Gemini and Timesketch that enables forensic analysis to surface attack patterns humans would never detect manually. Michael shares concrete metrics like reducing executive incident notifications from 30 minutes to 90 seconds, achieving 95% precision in ticket deduplication, and automating vulnerability coordination from hours to minutes. 
Topics discussed:
Processing 7 trillion log lines daily with less than 1% of a million annual tickets requiring human intervention at Google

Strategic evolution from AI-assisted to AI-led to autonomous security operations using fine-tuned models and golden datasets

Building modular detection agents as pluggable components that can be combined like Legos for specific security use cases

Implementing quality assurance through overseer agents that review other agents' work to ensure precision in security decisions

Reducing executive incident notifications from 30 minutes to 90 seconds using AI-powered summarization and context gathering

Achieving 95% precision in ticket deduplication while managing the trade-off between precision and 38% recall rates

Integrating Sec-Gemini with Timesketch to surface attack patterns in forensic investigations that humans would never find manually

Shifting from traditional detection and response to infer-and-interrupt models that contain threats immediately before escalation

Automating vulnerability coordination workflows from hours to minutes through AI-powered data collection and impact analysis

Distinguishing between traditional automation and agentic AI that exercises judgment rather than following if-then logic

Setting a stretch goal of 70% automation in operations work while focusing humans on novel and complex security challenges

Measuring success through time-to-mitigation metrics and evaluating AI performance against human baseline capabilities

Listen to more episodes: 
Apple 
Spotify 
YouTube
Website

What if the real risk isn't adopting AI agents, but refusing to? James Nettesheim, CISO & Head of Enterprise Technology at Block, argues that principled risk-taking beats playing it safe. James shares Block's journey co-designing the Model Context Protocol with Anthropic and building Goose, their open-source general-purpose agent that enables anyone in the company to write security detections using natural language.
James also explores Block's Binary Intelligent Triage system achieving 99.9% accuracy, their data safety levels framework, and practical strategies for balancing autonomous AI capabilities with human oversight. James offers candid insights about implementing AI security principles, the evolution from tool experts to domain experts, and why open source remains fundamental to Block's mission of economic empowerment and technological innovation. 
Topics discussed:
Co-designing of MCP with Anthropic and developing of Goose as an open-source general-purpose AI agent

Implementing prompt injection defenses and adversarial AI concepts to harden Goose against malicious instructions and attacks

Rolling out AI responsibly through data safety levels modeled after CDC bio-contamination protocols for sensitive data handling

Democratizing detection engineering by enabling anyone at Block to write detections using natural language

Achieving 40% of new detections created with AI assistance through recipes, playbooks, and automated tuning capabilities

Building Binary Intelligent Triage system that analyzes historical alerts and investigations to achieve 99.9% automated triage accuracy

Balancing autonomous AI capabilities with human oversight, requiring PR reviews and maintaining accountability for agent-generated code

Transitioning from tool expertise to domain expertise as the future skill set needed for detection and response professionals

Block's commitment to open source development driven by economic empowerment mission and desire to build accessible financial tools 

Listen to more episodes: 
Apple 
Spotify 
YouTube
Website

Ryan Glynn, Staff Security Engineer at Compass, has a practical AI implementation strategy for security operations. His team built machine learning models that removed 95% of on-call burden from phishing triage by combining traditional ML techniques with LLM-powered semantic understanding. 
He also explores where AI agents excel versus where deterministic approaches still win, why tuning detection rules beats prompt-engineering agents, and how to build company-specific models that solve your actual security problems rather than chasing vendor promises about autonomous SOCs.
Topics discussed:
Language models excel at documentation and semantic understanding of log data for security analysis purposes
Using LLMs to create binary feature flags for machine learning models enables more flexible detection engineering
Agentic SOC platforms sometimes claim to analyze data they aren't actually querying accurately in practice
Tuning detection rules directly proves more reliable than trying to prompt-engineer agent analysis behavior
Intent classification in email workflows helps automate triage of forwarded and reported phishing attempts effectively
Custom ML models addressing company-specific burdens can achieve 95% reduction in analyst workload for targeted problems
Alert tagging systems with simple binary classifications enable better feedback loops for AI-assisted detection tuning
Context gathering costs in security make efficiency critical when deploying AI agents across diverse data sources
Query language complexity across SIEM platforms creates challenges for general-purpose LLM code generation capabilities
Explainable machine learning models remain essential for security decisions requiring human oversight and accountability
Listen to more episodes: 
Apple 
Spotify 
YouTube
Website

Mike Vetri, Sr. Director of Security Operations at Veeva Systems, reflects on transforming SOC investigations through AI-powered data aggregation and building threat operations teams with the analytical mindset required for proactive defense. Mike introduces the C3 Matrix framework for prioritizing security efforts across centers of gravity, crown jewels, and capability enablers, and explains the seven Ds of cyber defense from discovery through deception operations. 

Drawing from 10+ years of Air Force cyber intelligence experience, Mike details why threat operations requires fundamentally different system-two thinking than detection engineering, and how this discipline shift moves organizations from reactive firefighting to proactive threat anticipation. He covers practical examples of AI cutting investigation time by aggregating data from multiple tools, the importance of defense in personnel for operational resilience, and strategies for preventing analyst burnout while maintaining effective security operations. 

Topics discussed:

How AI transforms insider threat investigations by aggregating workstation logs, browsing history, and DLP alerts into single queries

The C3 Matrix framework prioritizes security controls across centers of gravity, crown jewels, and capability enablers based on organizational impact and recoverability

Why threat operations requires system-two analytical thinking fundamentally different from the engineering mindset

The seven Ds of cyber defense: discover, detect, deny, disrupt, degrade, destroy, and deception operations for comprehensive threat mitigation

How deception operations provide the most accurate intelligence by studying adversary behavior in controlled environments

The distinction between threat intelligence and threat operations, and why mature SOCs need teams focused on proactive defense strategies

Defense in personnel ensures multiple team members can handle each security capability, preventing single points of failure

Time-sensitive investigation scenarios where AI delivers maximum ROI by eliminating the need to manually query dozens of security tools

The evolution of cyber threats from technical attacks to psychological warfare using AI to challenge human judgment and decision-making

Why security culture must extend beyond traditional boundaries as AI-powered threats increasingly target HR processes, financial operations, and business functions

Listen to more episodes: 

Apple 

Spotify 

YouTube

Website