AI is rewriting how software is built — and it’s completely transforming how we test it. Traditional QA methods were designed for deterministic systems, but today’s AI-enhanced applications behave in probabilistic, emergent, and sometimes unpredictable ways. How do testers ensure safety, reliability, and trust when the software itself is learning, evolving, and making decisions?
In this presentation, Kevin shares a practical, entertaining, and deeply insightful guide to testing AI systems drawn from real work at FamilySearch, where AI is used to label, extract, and publish data from billions of historical documents. You’ll learn why inputs → outputs is the new golden rule of AI testing, how evals and dashboards replace traditional pass/fail thinking, and how to design guardrails for AI agents before they act unexpectedly.
The session explores:
Why AI features require a different mindset than traditional regression testing
How to measure coverage when AI can generate thousands of test cases per feature
The role of LLMs, agentic systems, and AI managers in modern test workflows
How to safely test semi-autonomous and fully autonomous AI agents
Practical patterns for building evals, categorizing risks, and sharing dashboards
Packed with vivid examples — from ML-assisted data extraction to agents accidentally trying to “hack a Swiss bank” — this talk reframes QA for the age of AI and gives testers a clear blueprint to stay indispensable in an AI-partnered future.

• Why AI features require a different mindset than traditional regression testing
• How to measure coverage when AI can generate thousands of test cases per feature
• The role of LLMs, agentic systems, and AI managers in modern test workflows
• How to safely test semi-autonomous and fully autonomous AI agents
• Practical patterns for building evals, categorizing risks, and sharing dashboards