mail-sync-oom-fix-verification-2026-07-21.md23.4 KBView on GitHub # Mail-sync OOM fix — production verification (2026-07-21)
**Question asked:** is the worker OOM crash-loop truly fixed in prod, and did any of the
`fix/mail-sync-memory` changes introduce new mail bugs?
**Verdict: the OOM crash-loop is fixed.** Proven by five independent signals, with the two
obvious confounders (a raised memory limit, a still-enabled kill switch) explicitly ruled out.
Two open risks are flagged below — one of which (worker JS-heap growth) needs a decision.
**Deploy boundary used throughout:** prod worker deployment completed
**2026-07-20 23:24:51 PDT = 2026-07-21 06:24:51 UTC**
(task-def `…WorkerTaskDefinition51334463:225`, image `worker-service:ea11df3`).
All 24 fix commits `fae2b879c…35dd783d4` confirmed ancestors of `origin/staging`.
---
## 1. The crash-loop stopped
### 1.1 Task deaths: continuous → zero
ECS service events for `aws-prod-worker-service`. Before the deploy, a replacement task starts
every 2–10 minutes (the crash-loop). After it completes at 23:24:51, **not one task start**:
```
2026-07-20T23:16:57 has started 1 tasks: (a8e39320…) <- pre-deploy churn
2026-07-20T23:14:27 has started 1 tasks: (c4f0d3d7…)
2026-07-20T23:06:53 has started 1 tasks: (9429aafe…)
2026-07-20T23:03:56 has started 1 tasks: (8fcc2a52…)
2026-07-20T22:56:28 has started 1 tasks: (85212daf…)
2026-07-20T22:54:03 has started 1 tasks: (b53de6c4…)
2026-07-20T22:46:31 has started 1 tasks: (e982f972…)
2026-07-20T22:44:03 has started 1 tasks: (7a410566…)
--- 2026-07-20T23:24:51 deployment completed ---
(no further task-start events through 2026-07-21T17:30)
```
All three current tasks have run continuously since the deploy:
```
0da88983… startedAt 2026-07-20T23:22:09-07:00 RUNNING (~18h)
e5e26a74… startedAt 2026-07-20T23:21:44-07:00 RUNNING (~18h)
38f60ff0… startedAt 2026-07-20T23:21:13-07:00 RUNNING (~18h)
```
`list-tasks --desired-status STOPPED` returns `[]`.
### 1.2 Memory: 82–91% sawtooth → flat 15–23%
`AWS/ECS MemoryUtilization`, worker service, 30-min buckets:
```
2026-07-21T05:24 max= 88.8% avg= 19.7% <- last pre-deploy bucket
2026-07-21T05:54 max= 88.1% avg= 19.1%
--- deploy 06:24:51 UTC ---
2026-07-21T06:24 max= 16.6% avg= 8.6%
2026-07-21T16:54 max= 22.6% avg= 17.3%
```
Every 30-min bucket for the preceding **4 days** peaked 81.7–90.8%. Every bucket since the
deploy peaks 15.7–35.7%. The 8 GB ceiling is no longer approached.
### 1.3 No OOM in the logs
Zero matches since the deploy across the worker log group for `OutOfMemory`,
`JavaScript heap out of memory`, `FATAL ERROR`, `Allocation failed`.
### 1.4 Confounder 1 ruled out — the memory limit did not change
| task-def | registered | cpu | memory |
|---|---|---|---|
| `:225` (current, has fix) | 2026-07-20 23:20 | 2048 | **8192** |
| `:224` | 2026-07-20 07:27 | 2048 | **8192** |
| `:223` | 2026-07-18 15:56 | 2048 | **8192** |
Same 8192 MB and same `desiredCount: 3` on both sides, so the percentage drop is a real
reduction in memory used, not a bigger denominator.
### 1.5 Confounder 2 ruled out — sync is genuinely running
This is the important one: memory would also go flat if sync had simply stopped.
- **No kill switch in the deployed task-def.** `DISABLE_PERIODIC_EMAIL_SYNC` is absent from the
`:225` environment (nor is any `MAIL_SYNC_*` override present — the code default is in force).
- **Gmail fetches are happening at volume.** Axiom, last 2 h:
`Gmail API: threads.get started` = **4,944**, `threads.get completed` = **4,669**.
- **The completion gap is normal and unchanged.** Started-vs-completed gap is 4.7–8.6% both
before and after the deploy (3-day, 3-hour buckets) — i.e. ordinary in-flight tail, and
crucially *not* the old signature where poison threads never logged a completion.
### 1.6 Throughput held; the backlog is draining
`aws-prod-mail-sync-queue.fifo`:
```
backlog (avg visible) deleted / 3h
2026-07-20T17:25 3,083 1,198
2026-07-20T20:25 3,303 2,850
--- deploy ---
2026-07-21T08:25 616 1,402
2026-07-21T11:25 622 1,585
2026-07-21T14:25 763 1,513
```
Backlog fell 3,303 → ~620 while the delete rate held steady or rose. Sync is doing *more* work
on *less* memory. Current depth 967 visible / 4 in flight.
### 1.7 The DLQ is old damage aging out, not new failures
`aws-prod-mail-sync-dlq.fifo` currently holds 343 messages, but it is shrinking monotonically
at a constant rate straight through the deploy boundary (456 → 346 over 4 days), and
`ApproximateAgeOfOldestMessage` is pinned at **14.0 days** — the retention limit. These are
crash-loop-era messages expiring. No new arrivals post-deploy.
---
## 2. Error surface after the deploy
Total `level == "error"` rows across all of prod in the 11 h since deploy: **334**. No new error
class appeared. Normalised per hour (BEFORE = 28.9 h window, AFTER = 11.1 h):
| message | before /h | after /h | change |
|---|---|---|---|
| `Gmail Driver operation failed: get` | 16.05 | 1.17 | **↓ 14×** |
| `Failed to create execution for task` | 8.58 | 4.14 | ↓ 2× |
| `[UpdateExternalCrmWorkflow] Driver update failed` | 2.87 | 1.17 | ↓ 2.5× |
| `mirror.upsertEmailThread failed` | 0.66 | 0.09 | ↓ 7× |
| `Gmail Driver operation failed: modifyLabels` | 2.28 | 2.34 | ~flat |
| `Gmail Driver operation failed: getDraft` | 2.32 | 4.77 | ↑ 2× |
| `Gmail Driver operation failed: getEmailAliases` | 1.49 | 4.32 | ↑ 2.9× |
| `Gmail Driver operation failed: listHistory` | 0.93 | 2.70 | ↑ 2.9× |
The large decreases are the expected result of tasks no longer being killed mid-work. The
increases are all Gmail 429s — see risk 2.
---
## 3. Open risks
### RISK 1 — worker JS heap grows ~40–50 MB/h and is not plateauing (needs a decision)
The *old* failure was native/off-heap: heap stayed low while the cgroup ballooned to 6.5 GB in
~5 minutes, killing the process by SIGKILL (exit 137). **That mechanism is gone** — `nonHeapRssMb`
is flat. But the JS heap now climbs steadily on both tasks over their 18 h life
(`[MEM_HEARTBEAT]`, `/aws/ecs/aws-prod-api/worker-service`):
```
task 0da88983…
uptime 0.3h heapUsed 265 MB nonHeapRss 278 MB rss 551 MB 6.7%
uptime 4.1h heapUsed 403 MB nonHeapRss 321 MB rss 769 MB 9.4%
uptime 8.1h heapUsed 574 MB nonHeapRss 276 MB rss 866 MB 10.6%
uptime 11.0h heapUsed 737 MB nonHeapRss 335 MB rss 1085 MB 13.2%
task e5e26a74…
uptime 0.3h heapUsed 516 MB nonHeapRss 257 MB rss 817 MB 10.0%
uptime 4.1h heapUsed 701 MB nonHeapRss 489 MB rss 1233 MB 15.0%
uptime 8.1h heapUsed 757 MB nonHeapRss 538 MB rss 1349 MB 16.5%
uptime 11.0h heapUsed 949 MB nonHeapRss 371 MB rss 1342 MB 16.4%
```
Growth is linear-to-slightly-accelerating (~21, 50, 38, 47, 57, 53 MB/h in successive 2 h
windows) — it is **not** decelerating toward a plateau within the observed window. `heapTotal`
tracks `heapUsed` at ~98%, which is what retained (live) data looks like rather than
uncollected garbage.
- **Projection if linear:** from 737–949 MB at ~45 MB/h, the 6144 MB V8 limit is reached in
roughly **4–6 days**, i.e. around **2026-07-25/26**.
- **Failure mode would differ:** a V8 heap abort (`FATAL ERROR: JavaScript heap out of memory`,
exit 134) rather than the kernel SIGKILL, and on a days-long cycle instead of 14 minutes.
- **Honest caveat — I cannot attribute this to the change.** Before the fix, tasks never lived
more than ~14 minutes, so no long-run heap baseline exists. This growth may well be
pre-existing and simply have been invisible because the crash-loop restarted every task before
it could show. Prod `api-service` memory over 7 days oscillates in an 11–21% band without
monotonic climb, but it is a different workload and restarts on deploys, so it is only weak
evidence.
- **Cheapest decisive test:** leave the tasks running and re-read `[MEM_HEARTBEAT]` in 24–48 h.
If `heapUsed` plateaus, this is warm-up. If it is still climbing linearly past ~2 GB, it is a
genuine retention leak and worth a heap snapshot (`S3_HEAPSNAPSHOT_BUCKET` is already
configured on the worker).
### RISK 2 — Gmail 429s up 1.6×, but confined to one account
Every single rate-limit row in the 40 h window (308 of them) belongs to **one user,
`<email>`** — who was already being throttled before the change:
```
<email> BEFORE 190 over 28.9h = 6.6/h
<email> AFTER 118 over 11.1h = 10.6/h (1.6x)
```
Affected ops post-deploy: `getEmailAliases` (50), `listHistory` (30), `getDraft` (12),
`modifyLabels` (9), `createDraft` (7). This is a plausible consequence of commit `35dd783d4`,
which **dropped the page-1 reconcile throttle so reconcile runs on every load** — more Gmail
calls per session. It is *not* a fleet-wide regression: zero other accounts are rate-limited.
Worth watching rather than reverting; if it spreads beyond this account, the throttle removal is
the first thing to revisit.
---
## 4. Code review — bugs the change may have introduced
A full adversarial review of the 44-file diff was run against the post-merge code. Findings below
are ordered by severity, each cross-checked against prod telemetry where that was possible.
**Important framing:** the three HIGH findings all produce *silent staleness or silently wrong
content* — they emit no error logs. Section 2's clean error surface therefore neither confirms
nor refutes them. They are code-confirmed, not telemetry-confirmed.
### HIGH-1 · `sendMail` stamps the floor onto the client-supplied threadId
`services/mail/send/send.ts:164-176` passes `mail.threadId` to
`patchStoredThreadWithNewMessage`, not the provider's `result.threadId`. The sibling `sendDraft`
gets this right at `send.ts:431` (`result.threadId || data.threadId`).
When Gmail re-threads a `messages.send` (which happens on subject mismatch — the driver has
explicit subject-forcing code at `google.ts:4223-4300` precisely because of this), the message
lands in thread `T'` while Cedar appends a **phantom message to `T`** and stamps `T`'s floor to a
mailbox-global `historyId` *ahead* of `T`'s own. Every later read of `T` then trips
`isStaleReplicaRead` (`sync/threads.ts:96`, `:277`) and is skipped. Because the mirror floor uses
`GREATEST` (`crm/email-threads.ts:281-286`), it is equally unrecoverable — `T` is **wedged**,
showing a message it does not contain, until unrelated activity pushes its real historyId past the
stamped value. This is the "floor latched too high" class the design explicitly set out to avoid.
No test covers `sendMail` floor stamping (`send-draft-floor.test.ts` only exercises `sendDraft`).
Fix direction: use `result.threadId ?? mail.threadId`, and skip the patch entirely when
`result.threadId !== mail.threadId`.
### HIGH-2 · historyId equality-latch makes locally-fabricated label names permanent
`sync/threads.ts:302-317` returns the S3 snapshot verbatim when historyIds are equal — correct
only if the snapshot faithfully mirrors Gmail. It no longer always does: `sync/threads.ts:491-497`
writes `labelMap.set(labelId, labelId)` (**name := id**) and then stamps the authoritative floor
at `:534`, while `labels.ts:52-92` removed the background re-sync that used to repair it.
So applying `Cedar/aop/Renewals` → resolved to `Label_8842` → snapshot stores
`{id:'Label_8842', name:'Label_8842'}` → the equality-latch makes that permanent. Label chips
render the raw `Label_8842`. `threadItem/components/mail-labels.tsx:52-74` reads `label.name`
with no id→name fallback (the fallback at `thread.tsx:397-448` covers only AOP/agent-draft chips).
The change's own design note (`types.ts:113-125`) warns about exactly this hazard; the mitigation
shipped covers the *ids* but not the *names*. `modify-thread-labels.test.ts:174` asserts ids only.
**Telemetry cross-check:** inconclusive by construction — the success path of `modifyThreadLabels`
is not logged (only `google.modifyLabels.error`, 26 rows/11h), and labels live in S3 snapshots,
not in `crm_email_threads` (confirmed: the table has no label column). Verifying this needs an S3
snapshot spot-check or a UI look at a recently-labelled thread.
### HIGH-3 · Pages ≥ 2 are no longer reconciled at all
`list/route-list-threads.ts:169-171` runs the inline reconcile only when `cursor` is null. The
deleted `useListThreadsCheckSync` mounted one `mail.checkSync` **per loaded page**, and
`check-sync.ts:74-77` still contains the cursor→Gmail-`before:` translation built for exactly that
purpose — now dead code, only ever called with `cursor: null`.
Consequence: archive/read/star/label a thread that lives on page 3 from another client, and
Cedar's list keeps showing the stale state until pub/sub or the periodic cron happens to touch it.
The ghost sweep (`check-sync.ts:243-259`) never runs for deep pages.
The deleted test `use-list-threads-check-sync.test.tsx:59` ("mounts one checkSync query per loaded
page") asserted precisely this behavior. `route-list-threads.test.ts` now asserts the *negative*
(deep pages skip reconcile) — the regression is encoded as intended behavior.
### MEDIUM findings
| # | Finding | Location |
|---|---|---|
| M-1 | Send path still early-returns on S3 cache miss; the hydrate-on-miss fix from `112f61d01` was applied to labels but not to send. A reply from a deep page can lose the optimistic write entirely. | `send.ts:84-86` |
| M-2 | The synthetic sent body (`buildSyntheticSentMessage`, raw client HTML, never `preprocessEmailHtml`'d) is indistinguishable from a real one to `reuseBody`, so enrich-by-diff reuses it forever — the user's own sent message renders with uncollapsed quoted history and no theme CSS. | `send.ts:36-67`, `google.ts:1403-1406` |
| M-3 | Synthetic message persisted into the Postgres mirror with fabricated `attachmentId`/no `emailHeaderMessageId`; with `synthetic-${Date.now()}` ids the idempotency guard can never match on retry → duplicate phantom on a retried send. | `send.ts:110-125`, `:165` |
| M-4 | `headChanged` edge-trigger latches `true` (a 404'ing thread keeps `divergent` permanently true), after which trim-to-page-1 **never** fires again for the session. Zero tests — `headChanged` appears nowhere in `apps/mail/tests`. | `use-threads.ts:421-428` |
| M-5 | Throttle removal (`35dd783d4`) gives page-1 an unbounded cost: `threads.list` + `drafts.list` + up to 10 inline `threads.get` on **every** focus/visibilitychange/5-min tick/reset. Alt-tabbing now costs a Gmail list pair. | `route-list-threads.ts`, `check-sync.ts` |
| M-6 | Inline-image route's 100 req/min per-IP limit is far too low for the new per-occurrence URL pattern (dedupe by `filename:size:mimeType` was removed). A 40-reply thread with 2-image signatures ≈ 80 requests → 429s → broken images. `ip` also falls back to the literal `'unknown'`, pooling all such users into one bucket. | `inline-image.ts:41-53, 88-92` |
| M-7 | Agent-draft/task-type stripping now over-reaches: with two pending agent drafts on one thread, sending one strips the label from the whole thread; the `TASK_TYPE_LABEL_OVERRIDES` sweep can remove a user's manually-applied label. | `google.ts:2866-2913` |
| M-8 | `::numeric` casts in the mirror upsert would abort the statement on any malformed `history_id`; both call sites swallow the error, so a label change would succeed in Gmail/S3 but never reach the list. | `crm/email-threads.ts:281-286`, `:475-477` |
**M-8 telemetry cross-check — NOT materialized.** Direct query against prod:
```
nonnumeric_history_id | 0
null_history_id | 187793
total_rows | 341254
```
Zero malformed values, so the cast cannot currently fire. (`GREATEST` ignores NULLs, so the 188k
NULL rows are safe — they simply have no floor yet.) This is a latent robustness issue, not a
live bug.
**M-6 telemetry cross-check — not yet exercised.** Zero `inline-image` rows in Axiom over 12 h,
so the reference path is not yet carrying meaningful traffic in prod; the 429 risk is real but
unrealized so far.
**M-5 telemetry cross-check — weak positive.** See risk 2 in §3: Gmail 429s rose 1.6× post-deploy,
confined to a single account. Consistent with the throttle removal, not yet fleet-wide.
### LOW findings
- `historyIdChanged` (`history-id.ts:80-83`) compares with `!==` on strings while every sibling
helper uses `BigInt`; a numeric on either side would report "changed" on every sync (thrash, not
data loss). Worth a `String()` normalization given it is *the* change signal.
- `sync/threads.ts:475-482` returns `removedLabels: []` on the hydrate-failure branch even though
the removals were applied to the mirror one line earlier — optimistic UI driven off it misses them.
- Inline images are blocked when "load remote images" is off (`email-processor.ts:641-650` exempts
only `cid:`) — pre-existing, but reference URLs now make inline attachments indistinguishable
from genuinely remote images, so the exemption can no longer be fixed with a prefix check.
- Quote-boundary detection (`event-context.ts:184` `indexOf('Show quoted text')`) became
whitespace-sensitive when Markdown conversion started preserving newlines. Not broken today, but
a newline inside that span would silently push every email's full quoted history into agent
prompts, with no failing test.
### Verified clean
- **Inline-image authorization** — session required (401), ownership enforced
`record.userId !== user.id` → 403, connectionId is not a capability, cache key namespaced per
connection, `image/svg+xml` excluded, `nosniff` + `default-src 'none'; sandbox` on responses.
No cross-tenant read, no enumeration.
- **enrich-by-diff cannot drop changed content** — `reuseBody` requires cached `processedHtml`
**and** `isDraft === false` **and** no incoming `DRAFT` label, so both directions of the
draft↔sent flip force re-enrichment; everything but `processedHtml` is recomputed unconditionally.
- **Decode dedupe** is byte-for-byte equivalent to the three `fromBinary` calls it replaced.
- **The floor guard itself** — null-safe (never blocks first hydration), BigInt throughout, and
max-not-coalesce in all four writers. `cf7a73658` is applied consistently.
- **Old base64 threads still render** — existing snapshots keep their `data:` URIs; the backfill
is opt-in by design.
- `crm/email-threads.ts` message/participant/attachment derivation is unchanged.
---
## 5. Fixes applied (2026-07-21)
Both fixes followed test-first: the test was written to encode correct behavior, run to confirm
it FAILED against the merged code, then the fix was applied and the test re-run green.
### HIGH-1 — `sendMail` now stamps the provider's thread
Failing output before the fix (`send-draft-floor.test.ts`):
```
× does NOT patch or stamp the requested thread when Gmail re-threaded the send
→ expected "storeThreadInCurrentBucket" to not be called at all, but it was called with:
[ 'conn-1', 't1', { ... messages: [ { id: 'msent', threadId: 't1' ... } ] } ]
Number of calls: 1
```
Fix (`send.ts`): compare `result.threadId ?? mail.threadId` against `mail.threadId`; patch only
when they agree, otherwise skip the optimistic write and record a `mail.send.rethreaded` span
event. Three new tests cover kept-thread, re-threaded, and no-threadId-returned.
### HIGH-2 — label snapshots now store the provider's display name
Failing output before the fix (`modify-thread-labels.test.ts`):
```
× stores the provider label NAME, not the id as the name
AssertionError: expected 'Label_8842' to be 'Cedar/aop/Enterprise Renewals'
Expected: "Cedar/aop/Enterprise Renewals"
Received: "Label_8842"
```
Fix, mirroring how resolved *ids* are already surfaced:
- `types.ts` — optional `addLabelNames` on `ProviderModifyLabelsResult`.
- `google.ts` — new private `resolveLabelNames(requested, resolvedIds)`. When the caller passed a
name it is authoritative (no API call); only an id passed for a non-system label triggers a
`labels.list` lookup, so archive / read-unread / Cedar-labels-by-name stay free.
- `labels.ts` — passes `modifyResult?.addLabelNames` through.
- `sync/threads.ts` — `labelMap.set(labelId, addLabelNames?.[index] || labelId)`, falling back to
the id when the provider surfaced no name.
### Regression check
```
Test Files 37 passed (37)
Tests 424 passed (424) # src/services/mail + src/lib/driver
```
`tsc --noEmit` reports no new errors. The errors it does show in these files are pre-existing and
verified so against `HEAD`: `Label` is not exported from `types.ts`, `MailManager.getLabel(id)`
takes no `traceContext` while `labels.ts` passes one (same for create/update/delete/normalizeIds),
`HTMLRewriter` is an undeclared global, and `emailHeaderMessageId` is absent from
`IOutgoingMessage`. None are touched by these diffs, which are purely additive.
### Not fixed (per decision)
The reconcile throttle (`35dd783d4`) stays removed — 429s are isolated to one account and the
throttle removal fixed a real staleness bug. HIGH-3 (pages ≥2 never reconciled) and the MEDIUM /
LOW findings in §4 remain open.
## 6. Heap-snapshot investigation — BLOCKED
The intended next step (capture a heap snapshot to identify what the worker retains) cannot be
done without an infrastructure change:
| prerequisite | state |
|---|---|
| `S3_HEAPSNAPSHOT_BUCKET` | ✅ set — `cedar-aws-prod-heap-snapshots-619071350399-us-east-1` |
| snapshot trigger | ⚠️ `SIGUSR2` only (`worker-entry.ts:1257-1265`) — no HTTP/admin route exists |
| ECS exec (to send the signal) | ❌ `enableExecuteCommand: false` on `aws-prod-worker-service` |
| task-role SSM permissions | ❌ no `ssmmessages:*` in any inline or attached policy |
| existing snapshots to analyse | ❌ bucket is empty |
So capturing one on demand needs **both** an IAM change (add `ssmmessages` to the worker task
role) **and** a service update to enable exec — which redeploys and **restarts all three tasks,
destroying the ~18 h of accumulated heap growth that is the evidence**.
Options, in the order I'd recommend them:
1. **Enable exec on the next planned deploy, then take two snapshots and diff them.** Tasks restart
anyway on any deploy, so nothing extra is lost. Snapshot at ~t+1 h and ~t+13 h and diff — a
delta identifies retained objects far more precisely than a single snapshot, and at ~45 MB/h the
growth re-accumulates quickly.
2. **Let the automatic near-heap-limit watcher fire.** `worker-entry.ts:1267-1277` watches `/tmp`
and auto-uploads any `.heapsnapshot`, and V8 writes one as it approaches the limit — so if this
*is* a real leak, the diagnostic uploads itself in ~4-6 days. Free, but the task dies at that point.
3. **Re-measure first** (the cheap test): re-read `[MEM_HEARTBEAT]` in 24-48 h. If `heapUsed`
plateaus there is nothing to investigate.
I did **not** enable exec or change IAM — both are production infrastructure changes beyond the
scope of a snapshot capture, and option 1 makes them nearly free if bundled with the next deploy.
## 7. Method / queries used
- ECS: `describe-services` (events, deployments), `describe-tasks` (uptime),
`describe-task-definition` (memory limit, env).
- CloudWatch metrics: `AWS/ECS MemoryUtilization`; `AWS/SQS ApproximateNumberOfMessagesVisible`,
`NumberOfMessagesDeleted`, `NumberOfMessagesSent`, `ApproximateAgeOfOldestMessage`.
- CloudWatch logs: `filter-log-events` on `/aws/ecs/aws-prod-api/worker-service` for
`MEM_HEARTBEAT` and for OOM strings.
- Axiom (`cedar-prod`): `threads.get` started/completed by 3 h bin; `level == "error"` grouped by
message split on the deploy timestamp; `isRateLimit == true` grouped by `operation` and `email`.