Patronus AI with Anand Kannappan - Weaviate Podcast

AI agents are getting more complex and harder to debug. How do you know what's happening when your agent makes 20+ function calls? What if you have a Multi-Agent System orchestrating several Agents? Anand Kannappan, co-founder of Patronus AI, reveals how their groundbreaking tool Percival transforms agent debugging and evaluation. Percival can instantly analyze complex agent traces, it pinpoints failures across 60 different modes, and it automatically suggests prompt fixes to improve performance. Anand unpacks several of these common failure modes. This includes the critical challenges of "context explosion" where agents process millions of tokens. He also explains domain adaptation for specific use cases, and the complex challenge of multi-agent orchestration. The paradigm of AI Evals is shifting from static evaluation to dynamic oversight! Also learn how Percival's memory architecture leverages both episodic and semantic knowledge with Weaviate!

This conversation explores powerful concepts like process vs. outcome rewards and LLM-as-judge approaches. Anand shares his vision for "agentic supervision" where equally capable AI systems provide oversight for complex agent workflows. Whether you're building AI agents, evaluating LLM systems, or interested in how debugging autonomous systems will evolve, this episode delivers concrete techniques. You'll gain philosophical insights on evaluation and a roadmap for how evaluation must transform to keep pace with increasingly autonomous AI systems.

Links:
Percival Launch: https://www.patronus.ai/percival
Docs: https://docs.patronus.ai/docs/percival/
Paper: https://arxiv.org/abs/2505.08638

Chapters
0:00 Welcome Anand!
1:15 Percival!
17:20 Online and Offline Agent Tracing
20:40 Complex Agent Traces
23:05 Quick Insights and Deep Research
24:47 Automated Agent Tuning
31:19 LLM-as-Judge and Scalable Oversight
42:24 Agent Inbox for Evals
45:49 Causal Inference and AI
51:24 Percival and Weaviate
56:04 Exciting Directions for AI