BUGREPORT-optimistic-timeline-desync.md23.3 KBView on GitHub # Bug: the conversation timeline can permanently claim an email was sent that Gmail never received
## Status — shipped 2026-09-11
Three defects, all fixed, each demonstrated by a test that failed before the change.
**None of them is proven to be what caused the reported incident.** PostHog was checked after the
fixes landed and does not support the link: Zach's entire client-side footprint on 2026-09-11 is a
single session beginning at **17:44:01 with a fresh `/login`** — roughly 60 seconds *after* the
send he was describing and 60 seconds *before* he reported it. Whatever performed that send was not
this browser session, so the client-side defects below cannot be shown to have fired for it. See
**Unresolved** at the end.
> ### ⚠️ CORRECTION — the thread below is NOT the reported email
>
> Zach's screenshot (supplied later) shows the email he meant: **1:28 PM ET to Umar**, about Rho /
> Austin Founder House sponsorship. That is thread **`1a09183e8ef98734`**, not the thread analysed
> in this box. See **"The actual reported email"** immediately after. The analysis below is retained
> because it is correct about its own thread, but it is about a different message.
>
> ### The email on thread `1a0918b989b8f1b6` WAS sent. Do not resend it.
>
> Thread `1a0918b989b8f1b6` (the agent draft `r-2090112338578338896`, created 17:37:21) gained a
> real non-draft message at **17:47:47 UTC**, ~2.5 minutes after Zach reported it missing:
>
> | time (UTC) | `sync.gainedNonDraftMessage` | `gmail.total_replies` | `gmail.labels_count` | `gmail.has_latest` |
> |---|---|---|---|---|
> | 17:46:55 → 17:47:42 (5 syncs) | false | 0 | 5 | false |
> | **17:47:47** | **true** | **1** | **4** | **true** |
>
> `messages_found` stayed at 1 throughout while the label count dropped 5 → 4 — the DRAFT label
> being removed from a draft *sent in place*, not a new message appended. `cedar.analytics.draft.send`
> then fired at 17:47:54 with `match_kind: thread_id` against that draft, from the sync-side
> reconciliation at [email-events.ts:654](../../services/crm/email-events.ts#L654) — which exists
> precisely to catch "user sends a Cedar-drafted email via the Gmail UI" and runs only under
> `direction === 'outbound'`.
>
> **Gmail holds one copy. Resending would deliver a duplicate to a customer.**
>
> This, plus the PostHog session gap, means the incident is most likely: Zach never dispatched a
> send from a Cedar browser session at all, went to Gmail, found the draft and sent it there. The
> three defects fixed below are real and independently demonstrated, but they are hardening — not
> the diagnosis for this report.
## The actual reported email — thread `1a09183e8ef98734`
Identified from Zach's screenshot: **1:28 PM ET to Umar**, Rho / Austin Founder House, Option A
$7,500 Dinner Partner vs Option B $10,000 Headline Partner.
**Gmail accepted and sent it, then the thread disappeared from Gmail — and Cedar did not remove it.**
| time (UTC) | evidence |
|---|---|
| **17:28:57.567** | `sync.syncThreadFromProvider` — `gainedNonDraftMessage: true`, `gmail.messageCount: 1`, `s3.messageCount: 0` (Cedar had no prior copy), `changeVia: contentDiff` |
| **17:28:57.583** | `GET /gmail/v1/users/me/threads/1a09183e8ef98734?format=full` → **HTTP 200**, `messages_found: 1`, `total_replies: 1`, `has_latest: true`, `labels_count: 4` (no DRAFT label) |
| 17:29:07.384 | `cedar.analytics.draft.send`, `match_kind: no_cedar_draft` — ingested as an **outbound** email event |
| **17:44:24.674** | `gmail.users.threads.get` → **404 `notFound`** — "Requested entity was not found" |
| **17:45:20.213** | **404 `notFound`** again — 6 seconds after Zach reports it missing |
So Gmail itself returned the sent message at 17:28:57, and 15 minutes later returned 404 for the
same thread id.
**Cedar is not what removed it.** Every Gmail operation on connection `7bbcf4c9-…` between 17:28
and 17:46: `get` 44, `listHistory` 24, `get.minimal` 15, `getUserLabels` 9, `getEmailAliases` 9,
`list` 7, `modifyLabels` 5, `getDraft` 3, `listDraftThreadIds` 3, `deleteDraft` 2, `createDraft` 1,
`getMessageAttachments` 1. There is **no `threads.trash`, no `threads.delete`, no
`messages.delete`**. The two `deleteDraft` calls (17:29:56, fired twice 3ms apart — a duplicate-
dispatch smell worth its own look) target draft `r-7586165798160331551` on thread
`1a0913c56d532c85`, a different thread.
### Open
Why the thread 404s is **not established**. Remaining candidates:
1. Zach deleted or permanently removed it in Gmail himself.
2. Gmail re-threaded/merged the message into another conversation, retiring the original thread id —
in which case the email still exists in Gmail under a different id, and Cedar's timeline row
points at a dead one. Thread `1a091904bbddd319` gained non-draft messages 4 times from 17:42:31
on Zach's connection and is the nearest candidate destination, unverified.
Option 2 would be a Cedar-facing bug in its own right — a timeline entry deep-linking to a thread
id Gmail has retired looks exactly like "I sent this and it's not in Gmail". Deciding between the
two needs Zach's Gmail (Sent folder / Trash), not more log queries.
| # | Defect | Fix |
|---|---|---|
| 1 | Two of the three send-commit paths swallowed failures with `.catch(() => {})` | single `commitPendingSend()` dispatch point; `onSendComplete`/`onSendError` ride on the pending send |
| 2 | Keepalive guard matched `endsWith('/mail.send')` but the client batches | `lib/trpc-batch-url.ts` parses the comma-joined procedure list |
| 3 | **The teardown guard lived in the composer, which unmounts the instant a send is deferred** | guard moved to module scope beside `pendingSend`, and extended to every pending send, not only ones with attachments |
| | |
|---|---|
| Touched | `lib/trpc-batch-url.ts` (new), `providers/query-provider.tsx`, `modules/drafting/hooks/use-undo-send.ts`, `modules/drafting/components/email-composer.tsx` |
| Tests | `tests/lib/trpcBatchUrl.test.ts` (6), `tests/modules/drafting/undoSendErrorPaths.test.ts` (4), `tests/modules/drafting/undoSendUnloadGuard.test.ts` (5) |
| Verified | 71 suites / 633 tests green; `tsc -b --force` clean |
| Outstanding | size the blast radius from the new `email_deferred_send_failed` event once deployed |
**Fix C (server-authoritative undo window) was considered and rejected.** Routing every send
through `scheduleMail` puts SQS latency inside a fixed 5-second promise and places scheduler
failures on the critical path of ordinary sending. Guarding the tab as unsaved work achieves the
same invariant for every case except a full browser crash, which is explicitly out of scope.
### Defect 3 in detail
`proceedWithSend` calls `setNewEmail(false)` and `onSendSuccess?.()`
([email-composer.tsx:1716](components/email-composer.tsx#L1716)) immediately after
`startUndoableSend` returns. On the compose surface `onSendSuccess` is `handleSendSuccess` →
`handleClose()` ([compose-display.tsx:93](compose-display.tsx#L93)), so the composer unmounts and
its `useEffect` cleanup removed both the `beforeunload` and `pagehide` listeners — while the 5s
timer was still running at module scope.
```text
pendingSend (module scope) ──────────── 5s ────────────> fires
guard (component scope) ──X unmounted at ~0.1s
└─ nothing watching for the rest of the window
```
For that window there was no unload prompt and no `pagehide` handler, so closing the tab or
navigating away meant `forceSendPending` was never called: **the request was never made at all.**
No network failure required, no double-send required — and nothing logged, client or server, which
is consistent with the total absence of `mail.send` spans for the affected connection.
## Summary
The composer writes the outbound email into the conversation timeline and the mail thread
**before any network call**, then defers the real `mail.send` by 5 seconds via a module-level
`setTimeout`. Two of the three paths that commit that deferred send fire it as
`void sendFn().catch(() => {})` — no rollback, no toast, no telemetry. When the send is dropped,
the optimistic timeline event survives forever, so Cedar shows a sent email that does not exist in
Gmail. Zach Moskow hit this on 2026-09-11 and nearly double-sent because nothing on screen
distinguished the failure from a success.
## Symptom & impact
- **Observed**: The email appears in the conversation timeline and the mail thread as sent, the
draft is tombstoned, and the originating task is ticked — while Gmail has no such message and no
`mail.send` ever reached the server. No error is shown.
- **Expected**: The timeline is a projection of Gmail. An `outbound_email` event exists if and only
if Gmail actually holds the message. A send that does not land must roll back and surface an error.
- **Who/scope**: Every user with `undoSendEnabled` (the default path — `startUndoableSend` returns
`false` and sends synchronously only when the setting is off). Confirmed for
`<email>` (connection `7bbcf4c9-…`, org `537a54c6-…`) on 2026-09-11 17:45 UTC.
Frequency is unknown because the failure is untelemetered by construction — see Decision points.
- **Severity**: Silent wrong result, with data-integrity consequences. The CRM timeline is the
surface reps and the agent both read; a phantom `outbound_email` mis-states deal history, and
`resolveTasksForSend` closes the task that would have prompted a retry.
## Root cause
`proceedWithSend` commits every optimistic mutation — the thread patch
([email-composer.tsx:1330](modules/drafting/components/email-composer.tsx#L1330)), the timeline
event ([:1397](modules/drafting/components/email-composer.tsx#L1397)), the draft tombstones, and
the task closure ([:1452](modules/drafting/components/email-composer.tsx#L1452)) — and only then
hands the actual send to `startUndoableSend`
([:1572](modules/drafting/components/email-composer.tsx#L1572)), which parks it in a module-level
`pendingSend` ([use-undo-send.ts:57](modules/drafting/hooks/use-undo-send.ts#L57)) behind a 5s
timer. Only the timer path wires `onSendError`
([use-undo-send.ts:236](modules/drafting/hooks/use-undo-send.ts#L236)) to the rollbacks. The other
two commit paths — force-send when a second send starts
([:187](modules/drafting/hooks/use-undo-send.ts#L187)) and `forceSendPending` on page teardown
([:150](modules/drafting/hooks/use-undo-send.ts#L150)) — both discard the promise with
`.catch(() => {})`, so a rejected or never-dispatched send leaves every optimistic write standing.
The teardown path is additionally near-guaranteed to drop the request: its keepalive guard matches
`parsed.pathname.endsWith('/mail.send')`
([query-provider.tsx:104](providers/query-provider.tsx#L104)), but the client is
`httpBatchLink` ([:170](providers/query-provider.tsx#L170)), whose batched pathname is a
comma-joined procedure list, so the moment `mail.send` shares a batch the request gets neither
`keepalive: true` **nor** the unload fast-path — it instead `await`s `whenSessionSettled()`
([:133](providers/query-provider.tsx#L133)), which the code's own comment says "risks losing the
request."
## Code walkthrough
1. `modules/drafting/components/email-composer.tsx:1330` — `optimisticSendDraft` converts the draft
into a sent message in the Zustand store and the `mail.get` cache. Nothing has been sent yet.
2. `modules/drafting/components/email-composer.tsx:1397` — `appendOptimisticTimelineEvent` writes
the timeline row. **This is the row in Zach's screenshot.** Shape written:
```json
{
"id": "optimistic-<draftSessionId>",
"eventType": "outbound_email",
"direction": "outbound",
"isSignificant": true,
"emailEvent": { "messageId": "", "rfcMessageId": null, "isDraft": false }
}
```
Note `messageId: ""` and `rfcMessageId: null` — the event carries no provider identity, so
nothing downstream can later tell it apart from a confirmed send.
3. `modules/drafting/components/email-composer.tsx:1452` — `resolveTasksForSend` closes the task
that produced the draft, optimistically.
4. `modules/drafting/components/email-composer.tsx:1572` — `startUndoableSend({ sendFn: doSend })`.
Returns immediately; `doSend` has not run.
5. `modules/drafting/hooks/use-undo-send.ts:224` — the send is committed to a 5s `setTimeout`. Only
here is `onSendError` wired to `rollback()` / `conversationRollback()` / `taskResolution.rollback()`.
6. `modules/drafting/hooks/use-undo-send.ts:187` and `:150` — **the bug.** Both alternate commit
paths are:
```ts
void prev.sendFn().catch(() => {}); // :187 — a second send starts within the 5s window
void sendFn().catch(() => {}); // :150 — forceSendPending, called from pagehide
```
No `onSendError`, no rollback, no toast, no capture. A rejection here is indistinguishable from
success to every surface the user can see.
7. `providers/query-provider.tsx:104` — the teardown path's keepalive guard:
```ts
return parsed.pathname.endsWith('/mail.send');
```
With `httpBatchLink` (`:170`) a batch of two or more procedures has pathname
`/api/trpc/mail.send,mail.get`, so this is `false`, `keepalive` is never applied (`:143`), and
`:133` awaits `whenSessionSettled()` before the fetch — during page unload. The request dies,
and step 6 swallows it.
## Evidence (logs & traces)
- **Environment**: `cedar-prod`
- **Affected connection**: `7bbcf4c9-f43e-46cb-b1a2-303b9f72d41f` (`<email>`) —
verified to be his **only** connection over the window, so no sends were filtered out:
```kusto
['cedar-prod'] | where _time > ago(2d)
| extend em = tostring(['attributes.custom']['gmail.user_email']),
sem = tostring(['attributes.custom']['sync.userEmail']),
scid = tostring(['attributes.custom']['sync.connectionId'])
| where em == "<email>" or sem == "<email>"
| summarize count() by scid
```
→ one non-empty group: `7bbcf4c9-f43e-46cb-b1a2-303b9f72d41f` (1646 rows).
- **The send never reached the server.** `mail.send` fires for him on Sep 9 and Sep 10 and stops:
```kusto
['cedar-prod'] | where _time > ago(2d)
| where isnotnull(['attributes.custom']['send.shouldSchedule'])
| extend cid = tostring(['attributes.custom']['connectionId'])
| project _time, ['name'], cid, sched = tostring(['attributes.custom']['send.shouldSchedule'])
| sort by _time desc
```
Key rows (redacted):
```
2026-09-10T19:56:19.552Z mail.send cid=7bbcf4c9-… sched=false OK ← his last ever send
2026-09-10T18:28:41.308Z mail.send cid=7bbcf4c9-… sched=false OK
2026-09-11T17:23:15.989Z mail.send cid=ef16f273-… sched=false OK ← other users fine all day
2026-09-11T17:21:52.134Z mail.send cid=ef16f273-… sched=false OK
```
Zero `mail.send` rows for `7bbcf4c9-…` anywhere on 2026-09-11. Instrumentation is healthy:
`mail.send.sendDraft` fired 53× across prod in 7 days, 9 of them his (last 2026-09-10T19:56:19Z).
- **But Gmail-side sync did see activity on his account**, i.e. the attribution pipeline detected
sends it never performed — consistent with him getting the mail out by another route after Cedar
silently dropped it:
```
2026-09-11T17:37:21.374Z cedar.analytics.draft.create draft=r-2090112338578338896 thread=1a0918b989b8f1b6 src=agent
2026-09-11T17:47:54.602Z cedar.analytics.draft.send draft=r-2090112338578338896 thread=1a0918b989b8f1b6 match_kind=thread_id
```
The `draft.send` detection lands at 17:47:54 — **2m40s after** he reported the problem at 17:45:14.
- **The user report**, from Cedar's own Slack ingestion (`crm_slack_messages`, channel
`C0B4ZN3A68Z`):
```
17:45:02.297Z Zach Moskow "Something funky is happening" has_attachments=false
17:45:14.366Z Zach Moskow "I sent this email but it doesn't show in gmail" has_attachments=true
17:45:16.658Z Zach Moskow "any ideas?" has_attachments=false
17:46:06.909Z Jesse Li "Looking righ tnow"
17:46:37.164Z Zach Moskow "I almost sent it again so yeah"
```
- **Not obtained**: the screenshot on the 17:45:14 message. The claude.ai Slack connector has no
file scope (`slack_read_file` → `file_not_found` for every id; file-type search returns zero rows
workspace-wide), and Cedar's own ingestion stores `has_attachments = true` with
`attachment_refs = null` and `attachments = null` — it records that a file existed but persists
neither the reference nor the bytes. *(Secondary finding, filed separately from this fix.)*
- **No instrumentation was added.** Which of the three commit paths fired for Zach is
**not determinable from existing telemetry** — that is itself the defect: paths 6a/6b emit
nothing on failure. See Decision points.
## Decision points for the human
- [ ] **Scope of the fix.** The stated invariant is "the timeline must never show an outbound email
that Gmail does not have." Patching the two `.catch(() => {})` sites restores error handling
but keeps a window where the client solely owns an unsent email — a hard kill (tab crash, OS
sleep, browser eviction) still loses it with the optimistic writes standing. A
server-authoritative undo window closes that window for good but is a larger change.
- [ ] **Whether to keep optimistic timeline writes at all**, or render them as a visibly pending
state that only a server-confirmed `messageId` promotes.
- [x] **Backfill — resolved: there is nothing to back-fill.** Verified that both optimistic writes
are client-cache only. `useOptimisticTimelineEvent`
([use-optimistic-timeline-event.ts:44-73](../conversations/components/timeline/use-optimistic-timeline-event.ts#L44))
only calls `queryClient.setQueryData` on `trpc.crm.getConversation`, and
`resolveTasksForSend` ([resolve-tasks-on-send.ts](../userTasks/lib/resolve-tasks-on-send.ts))
likewise mutates and invalidates caches with no server mutation. The server-side task closure
is `completeTaskByDraftId`, which runs *inside* `mail.send` and therefore never fired. No
phantom `crm_events` or `user_tasks` rows exist. **The desync is confined to the client
session** — which is still the whole bug, because it outlives the moment of the send by long
enough for a rep to act on it.
**It is also stickier than an ordinary stale cache.** `onSendComplete` deliberately does not
invalidate ([email-composer.tsx:1601](components/email-composer.tsx#L1601)), and the draft
tombstones added on optimistic send cause `applyDraftTombstones` to strip the real draft from
any natural refetch (window focus, staleness) until Gmail syncs it away. So the one mechanism
that would otherwise self-correct the view is actively suppressed.
- [ ] **Telemetry first?** Because failures here are silent, we currently cannot size the blast
radius. Shipping the `captureException` + rollback wiring alone would tell us how often this
fires before committing to the larger redesign.
## Proposed fix
Three candidate shapes, smallest first — **choose before I write tests**:
- **A (narrow).** Route `use-undo-send.ts:150` and `:187` through the same
`onSendError` the timer path uses, and fix the keepalive guard at `query-provider.tsx:104` to
match a batched `mail.send` (parse the comma-joined procedure list rather than `endsWith`).
Restores rollback + the "Failed to send email" toast on every path.
- **B (A + provisional timeline).** Additionally mark the optimistic timeline event pending and
promote it only on a server-returned `messageId`, with a deadline that reverts it and raises an
error if no confirmation arrives.
- **C (server-authoritative).** Send to the server immediately with the 5s delay held server-side —
reusing the existing `scheduleMail` / `mail.listScheduled` machinery — and make Undo a real
cancel call. The intent becomes durable the instant the user clicks Send, teardown can no longer
lose it, and the timeline event can be written server-side from the Gmail response. Requires
reworking the `shouldSchedule` branch, which today deliberately skips task completion
([mail.ts:1720](../../../server/src/trpc/routes/mail.ts#L1720)).
## Test strategy
Jest, `apps/mail/tests/modules/drafting/` (this app is Jest, not Vitest — see
`tests/modules/drafting/undoSendRestore.test.ts` for the existing mock shape).
1. `undo-send-error-paths.test.ts` — `startUndoableSend` with a `sendFn` that rejects; trigger the
force-send path by starting a second send inside the window, and the `forceSendPending` path
directly. Assert `onSendError` runs and the rollbacks fire. **Fails today** (both swallow).
2. `keepalive-batched-send.test.ts` — assert the request predicate returns true for
`/api/trpc/mail.send,mail.get?batch=1`, not only `/api/trpc/mail.send`. **Fails today.**
3. `timeline-never-orphans.test.tsx` — the invariant test: drive `proceedWithSend` with a failing
send and assert no `outbound_email` event remains in the conversation store and the task is
reopened. **Fails today** on the two silent paths.
## Unresolved — what PostHog showed, and the lead it opened
Queried after the fixes landed (project "Cedar Mail", 251598). His complete client event timeline
for 2026-09-11:
```
17:43:59.455 $pageview cedarcopilot.com / ← marketing site
17:44:00.658 $exception cedarcopilot.com / (Sentry f67e1326…, message not forwarded)
17:44:00.676 $pageleave cedarcopilot.com /
17:44:01.615 $pageview mail.cedarcopilot.com /login ← FRESH LOGIN
17:44:02.700 $web_vitals mail.cedarcopilot.com /mail/inbox
17:44:07.731 $web_vitals mail.cedarcopilot.com /home
17:44:24 → 17:45:41 $autocapture ×7 /home
(then nothing until 19:11)
```
Nothing before 17:43:59 all day. For comparison he emitted 299 events on Sep 9 and 400 on Sep 10.
No `email_sent`, no `Reply Email Sent`, no `draft_deleted`, no `undo_action` on Sep 11 at all.
**The discriminator does not resolve.** The absence of `email_sent` was supposed to separate "never
dispatched" from "dispatched and failed". It cannot, because he was not in the instrumented client
when the send happened — he arrived a minute later.
The desktop app does not explain the gap: `windowService.ts:100` loads the remote app with
`loadURL(url)`, so posthog-js runs there too and reports identically (`host=mail.cedarcopilot.com`,
`lib=web`).
**Unproven hypothesis, recorded so it can be tested rather than assumed:** the `/login` at 17:44:01
suggests his session had expired. A `mail.send` rejected by auth middleware would produce **no
`mail.send` span** (the span opens inside the procedure, after the middleware), which matches the
telemetry exactly — and pre-fix, the deferred path's `.catch(() => {})` would have swallowed the
401 silently, leaving the optimistic writes standing. This is consistent with every observation but
is not evidence.
To settle it:
- Sentry issue `f67e132674034b70b66895619f521892` (the message is not forwarded into PostHog); the
Sentry MCP was not authenticated at time of writing.
- 401s on `/api/trpc` for his session around 17:35–17:45 UTC.
- Ask Zach directly which client he was in and whether he had been signed out.
The three fixes stand on their own: they are real defects with failing-then-passing tests, and each
one is a way a send can be lost silently. Whether any of them is *this* incident is open.