The Chief AI Officer Show

Alexander Page transitioned from sales engineer to engineering director by prototyping LLM applications after ChatGPT's launch, moving from initial prototype to customer GA in under four months. At Big Panda, he's building Biggie, an AIOps co-pilot where reliability isn't negotiable: a wrong automation execution at a major bank could make headlines.

Big Panda's core platform correlates alerts from 10-50 monitoring tools per customer into unified incidents. Biggie operates at L2/L3 escalation: investigating root causes through live system queries, surfacing remediation options from Ansible playbooks, and managing incident workflows. The architecture challenge is building agents that traverse ServiceNow, Dynatrace, New Relic, and other APIs while maintaining human approval gates for any write operations in production environments.

Page's team invested months building a dedicated multi-agent system (15-20 steps with nested agent teams) solely for knowledge graph operations. The insertion pipeline transforms unstructured data like Slack threads, call transcripts, and technical PDFs with images into graph representations, validating against existing state before committing changes. This architectural discipline makes retrieval straightforward and enables users to correct outdated context directly, updating graph relationships in real-time. Where vector search finds similar past incidents, the knowledge graph traces server dependencies to surface common root causes across connected infrastructure.

Topics discussed:

Moving LLM prototypes to production in months during GPT-3.5 era by focusing on customer design partnerships

Evaluating agentic systems by validating execution paths rather than response outputs in non-deterministic environments

Building tool-specific agents for monitoring platforms lacking native MCP implementations

Architecting multi-agent knowledge graph insertion systems that validate state before write operations

Implementing approval workflows for automation execution in high-consequence infrastructure environments

Designing RAG retrieval using fusion techniques, hypothetical document embeddings, and re-representation at indexing

Scaling design partnerships as extended product development without losing broader market applicability

Separating read-only investigation agents from write-capable automation agents based on failure consequence modeling

Dr. Ami Bhatt's team at the American College of Cardiology found that most FDA-approved cardiovascular AI tools sit unused within three years. The barrier isn't regulatory approval or technical accuracy. It's implementation infrastructure. Without deployment workflows, communication campaigns, and technical integration planning, even validated tools fail at scale.

Bhatt distinguishes "collaborative intelligence" from "augmented intelligence" because collaboration acknowledges that physicians must co-design algorithms, determine deployment contexts, and iterate on outputs that won't be 100% correct. Augmentation falsely suggests AI works flawlessly out of the box, setting unrealistic expectations that kill adoption when tools underperform in production.

Her risk stratification approach prioritizes low-risk patients with high population impact over complex diagnostics. Newly diagnosed hypertension patients (affecting 1 in 2 people, 60% undiagnosed) are clinically low-risk today but drive massive long-term costs if untreated. These populations deliver better ROI than edge cases but require moving from episodic hospital care to continuous monitoring infrastructure that most health systems lack.

Topics discussed:

Risk stratification methodology prioritizing low-risk, high-impact patient populations

Infrastructure gaps between FDA approval and scaled deployment

Real-world evidence approaches for AI validation in lower-risk categories

Synthetic data sets from cardiovascular registries for external company testing

Administrative workflow automation through voice-to-text and prior authorization tools

Apple Watch data integration protocols solving wearable ingestion problems

Three-part startup evaluation: domain expertise, technical iteration capacity, implementation planning

Real-time triage systems reordering diagnostic queues by urgency

UserTesting deployed 700+ custom GPTs across 800 employees, but Michael Domanic's core insight cuts against conventional wisdom: organizations fixated on hallucination risks are solving the wrong problem. That concern reveals they're building assistants for summarization when transformational value lives in using AI as strategic thought partner. This reframe shifts evaluation criteria entirely.

Michael connects today's moment to 2015's Facebook Messenger bot collapse, when Wit.ai integration promised conversational commerce that fell flat. The inversion matters: that cycle failed because NLP couldn't meet expectations shaped by decades of sci-fi. Today foundation models outpace organizational capacity to deploy responsibly, creating an obligation to guide employees through transformation rather than just chase efficiency.

His vendor evaluation cuts through conference floor noise. When teams pitch solutions, first question: can we build this with a custom GPT in 20 minutes? Most pitches are wrappers that don't justify $40K spend. For legitimate orchestration needs, security standards and low-code accessibility matter more than demos.

Topics discussed:

Using AI as thought partner for strategic problem-solving versus summarization and content generation tasks

Deploying custom GPTs at scale through OKR-building tools that demonstrated broad organizational application

Treating conscientious objectors as essential partners in responsible deployment rather than adoption blockers

Filtering vendor pitches by testing whether custom GPT builds deliver equivalent functionality first

Prioritizing previously impossible work over operational efficiency when setting transformation strategy

Building agent chains for customer churn signal monitoring while maintaining human decision authority

Implementing security-first evaluation for enterprise orchestration platforms with low-code requirements

Creating automated AI news digests using agent workflows and Notebook LM audio synthesis

Utah's tax chatbot pilot exposed the non-deterministic problem every enterprise faces: initial LLM accuracy hit 65-70% when judged by expert panels, with another 20-25% partially correct. After months of iteration, three of four vendors delivered strong enough results for Utah to make a vendor selection and begin production deployment. Christian Napier, Director of AI for Utah's Division of Technology Services, explains why the gap between proof of concept and production is where AI budgets and timelines collapse.
His team deployed Gemini across state agencies with over 9,000 active users collectively saving nearly 12,000 hours per week. Meanwhile, agency-specific knowledge chatbots struggle with optional adoption, competing against decades of human expertise.
The bigger constraint isn't technical. Vendor quotes for the same citizen-facing solution dropped from eight figures to five during negotiations as pricing models shifted. When procurement cycles run 18 months and foundation models deprecate quarterly, traditional budgeting breaks.
Topics discussed:
Expert panel evaluation methodology for testing LLM accuracy in regulated tax advice scenarios

Low-code AI platforms reaching capability limits on complex use cases requiring pro-code solutions

Avoiding $5 million in potential annual licensing costs through Google Workspace AI integration timing

Tracking self-reported productivity gains of 12,000 hours weekly across 9,000 active users

AI Factory process requiring privacy impact assessments and security reviews before any pilots

Vendor pricing dropping from eight-figure to five-figure quotes as commercial models evolved

Forcing adoption through infrastructure replacement when legacy HR platform went read-only

Separating automation opportunities from optional tools competing with existing workflows

Digital identity requirements for future agent-to-government transactions and authorization

Regulatory relief exploration for AI applications in licensed professions like mental health

Extreme Networks architected their AI platform around a fundamental tension: deploying non-deterministic generative models to manage deterministic network infrastructure where reliability is non-negotiable. Markus Nispel, CTO EMEA and Head of AI Engineering, details their evolution from 2018 AI ops implementations to production multi-agent systems that analyze event correlations impossible for human operators and automatically generate support tickets. Their ARC framework (Acceleration, Replacement, Creation) separates mandatory automation from competitive differentiation by isolating truly differentiating use cases in the "creation" category, where ROI discussions become simpler and competitive positioning strengthens.

The governance architecture solves the trust problem for autonomous systems in production environments. Agents inherit user permissions with three-layer controls: deployment scope (infrastructure boundaries), action scope (operation restrictions), and autonomy level (human-in-loop requirements). Exposing the full reasoning and planning chain before execution creates audit trails while building operator confidence. Their organizational shift from centralized AI teams to an "AI mesh" structure pushes domain ownership to business units while maintaining unified data architecture, enabling agent systems that can leverage diverse data sources across operational, support, supply chain, and contract domains.

Topics discussed:

ARC framework categorizing use cases by Acceleration, Replacement, and Creation to focus resources on differentiation

Three-dimension agent governance: deployment scope, action scope, and autonomy levels with inherited user permissions

Exposing agent reasoning, planning, and execution chains for production transparency and audit requirements

AI mesh organizational model distributing domain ownership while maintaining centralized data architecture

Pre-production SME validation versus post-deployment behavioral analytics for accuracy measurement

90% reduction in time-to-knowledge through RAG systems accessing tens of thousands of documentation pages

Build versus buy decisions anchored to competitive differentiation and willingness to rebuild every six months

Strategic data architecture enabling cross-domain agent capabilities combining operational, support, and business data

Agent interoperability protocols including MCP and A2A for cross-enterprise collaboration

Production metrics tracking user rephrasing patterns, sentiment analysis, and intent understanding for accuracy