Daniella's Customer Brain

Honeycomb <> OneAI: Weekly Sync

3 min read

Honeycomb <> OneAI: Weekly Sync

Date: 2025-11-04 | Duration: 20.969999313354492 min | Account: honeycomb

External attendees: jan.j@honeycombinsurance.com, jon@honeycombinsurance.com, rafael@honeycombinsurance.com

Summary

The discussion revolved around the development of a supervisory AI layer aimed at enhancing call qualification accuracy, as described by Rafael Mintz, who highlighted the implementation of a 1,000-line code combined with an LLM to automatically flag misqualified deals. This system reviewed 30 days of calls, identifying only three misqualified deals, which were confirmed as properly disqualified upon review. The team acknowledged the challenges in ensuring AI accuracy and the necessity for manual sampling alongside AI to capture nuanced errors. Efforts to connect AI performance metrics to business value were also discussed, with plans for deeper analysis using Tableau reports. Knowledge sharing on AI supervisory functions was emphasized, alongside a focus on reducing call friction points to improve customer interactions. The strategic direction points towards integrating AI oversight as a crucial growth opportunity for the business.

Key Points

🤖 AI Quality Assurance:: New supervisory AI layer improves call qualification accuracy, featuring a 1,000-line code to flag misqualified deals. 📊 Impact Metrics:: System reviewed last 30 days, identifying three misqualified deals; false positives and negatives expected to decrease. 💡 Business Value Analysis:: Ongoing efforts aim to link AI performance metrics to revenue, requiring deeper data access for clearer insights. 🔄 Knowledge Sharing:: Team seeks insights on AI supervisory functions while ensuring results are adapted for unique business models. ✅ Friction Reduction:: Recent updates show reduced high-priority call friction, improving customer interactions and overall satisfaction metrics. 📈 Strategic Direction:: AI oversight integration is a key growth initiative, driving data-driven decision-making in sales qualification processes.

Overview

  • AI Quality Assurance: New supervisory AI layer improves call qualification accuracy, featuring a 1,000-line code to flag misqualified deals.
  • Impact Metrics: System reviewed last 30 days, identifying three misqualified deals; false positives and negatives expected to decrease.
  • Business Value Analysis: Ongoing efforts aim to link AI performance metrics to revenue, requiring deeper data access for clearer insights.
  • Knowledge Sharing: Team seeks insights on AI supervisory functions while ensuring results are adapted for unique business models.
  • Friction Reduction: Recent updates show reduced high-priority call friction, improving customer interactions and overall satisfaction metrics.
  • Strategic Direction: AI oversight integration is a key growth initiative, driving data-driven decision-making in sales qualification processes.

Action Items

Amit Ben Review Slack message content and provide sorted feedback by tomorrow morning (04:05) Collaborate with John and Rafael to review and potentially enhance their supervisory layer approach based on shared learnings (15:07)

Rafael Mintz Run supervisory AI layer across last 30 days of data and share outcomes (05:39) Continue deeper mapping into Tableau reporting for business value analysis (06:19)

Daniella Block Check available reports related to AI management layers and manual sampling for insights (11:24)

Rafael Mintz & Jon Repka Align on review of existing ~1000 lines of supervisory code and results for false positive verification (15:07)

Amit Ben & Team Apply fine-tuning to the AI system based on team input and revert with updates (20:24)

Graph View