REPORT-chat-message-analysis-2026-06-20.md14.3 KBView on GitHub # Cedar Chat Message Analysis — External Customers Only
**Date:** 2026-06-20
**Window analyzed:** 2026-02-01 → 2026-06-20 (~4.5 months, all available history)
**Corpus:** 3,014 **external-customer** user messages (role = `user`). Internal Cedar dogfooding (`@cedarcopilot.com`) — 3,548 messages, 54% of raw volume — has been **excluded**.
**Method:** Full external-only export from Postgres (`chat_messages` ⋈ `chat_threads`, domain ≠ cedarcopilot.com), LLM classification of every message into a 14-category taxonomy (3 parallel passes), cross-referenced with `agent_tool_calls` / `agent_executions` health metrics.
---
## 1. Methodology & data sources
User messages live in **`chat_messages`** (`role` = `user`), joined to **`chat_threads`** whose `page_key` is the channel discriminator. We kept only `role='user'` messages from non-cedarcopilot.com domains.
> **Coverage gap — read before citing channel totals.** SMS, WhatsApp, and iMessage bot conversations are stored **in Redis only** (`msg_thread:{id}`, 7-day TTL, last 20 messages) — `apps/server/src/services/messaging/thread-history.ts`. Only iMessage threads mirrored into `chat_messages` (14 external msgs) appear here; true SMS/WhatsApp volume is **not durably persisted and cannot be measured.** Customers' own Slack traffic ingested for CRM (`crm_slack_messages`) is a different dataset and excluded — this report is messages sent **to** Cedar's assistant. **First infra action item: persist bot-channel messages to Postgres.**
---
## 2. General overview (external only)
### Volume & growth
| Month | External user msgs |
| ----------------- | -----------------: |
| 2026-02 | 94 |
| 2026-03 | 303 |
| 2026-04 | 631 |
| 2026-05 | 822 |
| 2026-06 (to 20th) | **1,164** |
External usage is accelerating — June alone (partial) is already 42% above all of May, and ~12× February. This is the real growth signal, undiluted by internal testing.
### Who's using it (top external users)
| User | Org | Msgs |
| --------------------------------------------------------------------------------- | -------------- | -------------: |
| <email> | SimCare | 585 |
| <email> | Superglue | 441 |
| <email> | Reform | 426 |
| <email> | Mintlify | 243 |
| <email> | Warp | 147 |
| <email> | ListenLabs | 136 |
| <email> | Strac | 106 |
| <email> | Edexia | 80 |
| max / noah / lindsay @teachwithscout.com | TeachWithScout | 190 (combined) |
| <email> · <email> · <email> · <email> | — | 39–59 each |
**Adoption is concentrated:** the top 3 orgs (SimCare, Reform, Superglue) = **48%** of all external messages; the top 6 = **76%**. Cedar's external story is currently a handful of design-partner power users, not broad-based usage.
### By channel
| Channel | Msgs |
| --------------------------------------------------- | ----------: |
| in-app | 2,743 (91%) |
| task_invocation | 88 |
| mail / home / crm / calendar (legacy keys, Feb–Mar) | 117 |
| slack (since Jun 8) | 35 |
| imessage (since Jun 6) | 14 |
| playground-aop-editor | 14 |
In-app is essentially the entire external surface. Slack and iMessage bots are brand-new and barely used.
---
## 3. Request buckets (what external customers ask for)
Every message classified into exactly one primary category. % = share of the 3,014 external messages.
| # | Category | Share | What it is |
| --: | ------------------------------ | --------: | --------------------------------------------------------------------------------- |
| 1 | **Email drafting** | **40.3%** | Follow-ups, recaps, outreach, rewrites, "make it more X" |
| 2 | Acknowledgment / chitchat | 8.7% | "yes", "looks correct", "keep going", greetings |
| 3 | **Config / memory / behavior** | 7.9% | "remember this", edit AOP/playbook, CC rules, voice, agents |
| 4 | **Meeting / call notes** | 7.5% | "what did X say", pull transcript/recording, pain points |
| 5 | CRM update / action | 6.4% | Close/move deals, MEDDPICC, update Salesforce/HubSpot/Attio, tasks |
| 6 | Sales coaching | 5.1% | Positioning, objections, talk tracks, SAYA/MEDDPICC, competitive |
| 7 | Deal summary / status | 5.1% | "how is this deal going", blockers, recaps |
| 8 | CRM query / filter | 4.6% | "which deals…", overdue opps, find uncreated deals |
| 9 | Document creation | 4.4% | Decks, business cases, MAPs, ROI docs, handoff docs |
| 10 | Next steps | 3.7% | "what should I do next", agenda for this call |
| 11 | Other | 3.2% | Meeting-prep agent invocations, security probes, off-topic |
| 12 | Research (person/company) | 2.5% | Enrichment, pre-meeting intel, ICP checks |
| 13 | Calendar / scheduling | 2.4% | Availability, book meetings, invites |
| 14 | Correction / feedback | 2.4% | "that's wrong", "be more specific" _(under-counts true dissatisfaction — see §5)_ |
### What changed when internal traffic was removed
- **Email drafting jumped from 28.8% → 40.3%.** It's even more dominant for real customers — 5× the next category. This is unambiguously _the product_.
- **Document creation (7.1%→4.4%), deal summaries (7.5%→5.1%), and sales coaching** all shrank — those were inflated by internal dogfooding and the canned demo prompts. Customers do these far less than the team does.
- **Calendar (1.4%→2.4%) and next-steps (2.9%→3.7%) rose** — these are genuinely external behaviors.
- **Config/memory stayed high (7.9%)** — and it's almost all from a few power users _programming Cedar's behavior in prose_ (tone, CC rules, follow-up cadence, deal-stage definitions). A meaningful chunk of paid-customer effort goes into fixing defaults rather than getting work done.
### The intent stack (external)
- **Write for me** (drafting + docs + calendar): **~47%**
- **Tell me about my deals** (meeting notes + deal summary + CRM query + next steps + research): **~23%**
- **Do CRM work** (updates/actions): **~6%**
- **Make Cedar smarter / fix it** (config + corrections): **~10%**
### Representative external requests
- _"draft a follow up to the meeting with malte and philip just now, in german please"_ (superglue.ai)
- _"modify the email to be more attack mode vs asking for feedback… highlight the top things Rippling can't do"_ (warp.co)
- _"Please find all conversations in the demo/negotiating stage that have overdue next steps"_ (strac.io)
- _"can you update meddpicc in salesforce?"_ (mintlify.com)
- _"what did tobias say I should follow up with Jermain on"_ (superglue.ai)
- _"edit my follow up instructions so when I send an email, follow up every 3-7 days"_ (r41.tech)
- _"Act as a sales coach. Review the most recent call and provide feedback"_ (mintlify.com)
---
## 4. What's working ✅
1. **Follow-up drafting is the killer use case** — 40% of external traffic, with daily repeat usage from every power user. The clearest product-market-fit signal in the data.
2. **Voice-matching is the moat when it lands.** Nearly all of the (rare) praise is for drafts that nailed tone — _"looks great", "that's perfect", "yey"_. When the draft sounds like the user, they ship it with one word.
3. **Meeting-history recall is differentiated and relied upon** — 7.5% of messages ask Cedar to recall what was said on specific calls ("when did Postman come up in our calls?").
4. **Core CRM read/update plumbing is reliable.** `select-primary-company` 0.0% fail, `lookup-company-fuzzy` 0.0%, `list-tasks` 0.1%, `get-conversation` 3.1%, `update-next-steps-and-tasks` 0.5%. Overall: ~497k completed executions vs ~4.9k hard failures (~1%).
5. **Power users are investing** — 8% of their messages tune AOPs/playbooks, which only happens when customers intend to make Cedar permanent.
---
## 5. What's NOT working ❌
Among external customers, frustration fired on **4.3%** of messages, bug reports on **3.0%**, and explicit praise on only **0.7%** — a **6 : 1 frustration-to-praise ratio.** Note the low `correction_feedback` category count (2.4%) _understates_ dissatisfaction: much of it surfaces as the next drafting request or a frustrated re-send rather than a tidy "that's wrong." Failure modes, in priority order:
### A. Artifact persistence & visibility — the #1 complaint
Drafts/docs silently fail to save, land in the wrong thread, or render blank — the single most repeated external friction:
- _"wheres draft"_ (hamming.ai) · _"i dont see the draft"_ (strac.io) · _"where is the draft I sent to them today?"_ (superglue.ai)
- _"why didn't my drafts save??? where is it"_ (superglue.ai) · _"the file is not here"_ (mintlify.com)
- Corroborated: **`write-document` fails 8.7%** (5,132 failures), **`draft-email` fails 6.6%** (1,319) — the worst of the high-volume tools.
### B. Automations silently not firing
Scheduled/triggered agents (post-meeting follow-ups, daily agenda, pre-meeting Slack pings) don't run; users only notice their absence:
- _"why are post meeting follow ups not populating in my inbox anymore?"_ (simcare.ai)
- _"Is there a reason why we didn't get notified for the two kickoff calls we had at 2:00 PM Eastern?"_ (simcare.ai)
- _"why didnt this prompt a follow up email"_ (strac.io)
- Corroborated: **20,697 executions stuck in `executing`** (never reached a terminal state) over 90 days; `pre-execution-setup` fails **13.2%** (17,184) — likely an upstream cause.
### C. Send / CC mechanics
- _"that didn't work still noone in cc and also why do you keep saying you can't send the email?"_ (superglue.ai)
- _"yeah its still not working. just update the html file and i will download"_ (mintlify.com)
### D. Deal/entity mis-resolution
Cedar attaches emails/actions to the wrong deal, or creates deals for non-deals:
- _"Are you a fucking idiot? The email I'm providing you is someone that isn't linked to that fucking school at all"_ (edexia.ai)
- _"what? they are not testing the use cases, I am"_ (r41.tech)
### E. Grounding / fact fidelity
- _"thats not accurate try again"_ (reformhq.com) · _"this draft doesn't reiterate how superglue can help and is very surface level"_ (superglue.ai)
- Notably, customers frequently **paste call transcripts inline** because Cedar couldn't retrieve the meeting itself — a recurring workaround that signals a retrieval gap.
### F. Config that doesn't stick
Power users re-teach the same rule repeatedly — a retention risk:
- _"why are you messing this up consistently? I need you to fix this in your knowledge base or agents or whatever"_ (simcare.ai)
- _"can you fix your understanding so this shit doesnt happen anymore"_ (simcare.ai)
### G. Latency / mid-run halts
- _"whats taking so long"_ (mintlify.com), plus recurring "are you stuck / why did you stop".
**Frustration is concentrated in the highest-value accounts** — SimCare, Superglue, Edexia, Mintlify (the power users) generate the most profane, churn-risk-flavored complaints, almost all about _reliability of configured behavior_, not raw drafting quality.
---
## 6. Recommendations (priority-ordered)
1. **Fix artifact persistence & visibility (drafts/docs).** Highest-frequency external complaint _and_ worst tool failure rates. Guarantee drafts surface in the right thread with a visible confirmation — make "where did my draft go" impossible.
2. **Instrument & alert on silent automation failures.** 20,697 stuck executions; investigate `pre-execution-setup` (13.2% fail). Surface failed/stalled scheduled runs proactively instead of letting users discover no-shows.
3. **Make remembered config actually stick.** Repeated re-teaching by paying power users is the clearest churn signal; audit why org/playbook preferences aren't applied consistently.
4. **Close the meeting-retrieval gap.** Customers paste transcripts because Cedar can't fetch them — fixing native Gong/Granola/Circleback/Fathom retrieval removes a daily workaround and strengthens the #4 use case.
5. **Add a pre-flight deal/entity match confirmation before CRM writes** (literally requested by a customer) to kill mis-resolution.
6. **Persist bot-channel messages (SMS/WhatsApp/iMessage)** so cross-channel can be measured at all.
7. **Lean into the wedge:** Cedar is a _follow-up-drafting + meeting-recall_ engine for external users. Make those two paths "never wrong, never lost" before broadening surface area.
---
_Reproducibility: external-only export at `/tmp/chatanalysis/ext_messages.jsonl` (3,014 rows); per-batch classifications at `/tmp/chatanalysis/ext*result*_.json`. Counts from `chat_messages`where`role='user' AND deleted_at IS NULL AND domain <> 'cedarcopilot.com'`. Tool/execution metrics scoped to last 60–90 days from `agent_tool_calls`/`agent_executions` (whole-fleet, not external-isolatable).\*