mail-sync-oom-corrected-root-cause-2026-07-18.md7.0 KBView on GitHub # Worker OOM — Corrected Root Cause (mail-sync thread materialization)
**Supersedes** `mail-sync-oom-root-cause-report.md` (2026-07-08 "concurrency-20 fan-out") and
`worker-oom-crashloop-verification-2026-07-16.md` (2026-07-16 "single pathological thread of ~36KB
notification emails"). Both identified the OOM correctly; both mis-identified the *mechanism* and the
specific thread. This report is grounded in the actual CloudWatch `threads.get SUMMARY` logs, which
carry per-message `sizeEstimate`.
## Symptom
`aws-prod-worker-service` (Fargate, 2 vCPU / 8192 MiB, desiredCount 3) OOM-crashloops: memory climbs
~580 MB → 6.5 GB+ over ~5 min with both vCPUs pegged and network flat, kernel SIGKILLs (exit 137), the
in-flight SQS message never acks, redelivers, and kills the next task. `history_start_id` stays pinned
(e.g. `5282071` for <email>) across every retry, proving the cursor never advances because the
batch never completes → the same threads reprocess forever.
## Root cause (one sentence)
`threads.get({ format: 'full' })` returns image parts as **references** (`attachmentId`, no bytes), but
Cedar then **re-fetches every inline image and base64-inlines it into the stored HTML of every message
that quotes it**, DOM-parses each inflated copy, holds the whole thread plus the prior S3 snapshot in
memory, and re-serializes it — turning a thread that is a few MB of text-plus-references into 200 MB+,
processed with no per-message cap and no incremental GC, on every sync pass.
## The specific thread (evidence)
Batch for `<email>` (`source: background-sync`, `thread_count: 9`). Eight threads are KB-scale.
One is the killer — **`19df82a3fa83e1bd`, 25 messages, 147 MB on the wire**, per-message `sizeEstimate`:
```
0.00–0.04 MB May 5 – Jun 9 8 normal emails (intro / scheduling)
8.62 MB Jun 11 "We hosted an epic poker night and Knicks watch..."
8.63–8.70 MB Jun 16 – Jul 13 17 more messages, EVERY ONE ~8.6 MB
```
A large inline image entered the thread on Jun 11 and every subsequent reply re-quotes it, so 17
messages each carry a fresh ~8.6 MB payload. The thread never logs `threads.get completed` — it freezes
mid-processing and OOMs first.
Method: `aws logs get-log-events` on the frozen task's stream, parse `🔍 [GOOGLE-GET] threads.get()
SUMMARY` (google.ts:1355) for per-message `sizeEstimate` (emitted *before* the freeze). 12 `getAttachment`
calls observed in the same stream = inline-image fetches (regular attachments are never fetched during sync).
## How 147 MB becomes 6.5 GB (code walkthrough)
1. **`threads.get(format:'full')` — a few MB.** Large parts come back as `{ size, attachmentId }`, no bytes
(`google.ts:1312`). The 147 MB `sizeEstimate` is wire size; it is NOT in memory yet.
2. **Uncapped fan-out over all messages** — `Promise.all(res.data.messages.map(...))` (`google.ts:1378`)
processes all 25 concurrently; CPU-bound, so GC never runs between them.
3. **The original sin — fetch + base64-inline** (`google.ts:1425`, `:1444`): `getAttachment` pulls the
~8.6 MB image, and `processedBody.replace(cid:… , data:…base64…)` embeds ~11.5 MB into each message's
HTML → **~195 MB of HTML strings**, on every sync.
4. **Full DOM parse of each inflated body** — `sanitizeHtml` + `cheerio.load` (`email-processor.ts:274`)
via `preprocessEmailHtml` (`google.ts:1493`): 3–5 transient copies per message, ×17 concurrent ≈ 1 GB.
5. **Old snapshot loaded to diff** — `readThreadFromCurrentBucket` → `JSON.parse(existing.text())`
(`s3.ts`) of the prior ~180 MB blob: string + parsed graph ≈ 360 MB — to extract message ids + labels
(`threadHasChanged`, `threads.ts:159`, needs only ids/labels).
6. **Re-serialize whole thread to S3** — `JSON.stringify(thread)` + S3 client Buffer ≈ 390 MB (`s3.ts:45`).
7. **No GC** — steps 2–4 are synchronous, event loop pegged, transients accumulate → 6.5 GB → SIGKILL.
Postgres is NOT a multiplier: `upsertEmailThread` stores metadata only (`email-threads.ts:246`).
The agent handoff during sync is already stripped (`stripLargeMessageFieldsForRpc`, `processedHtml:''`),
so the balloon is in fetch/enrich/store, not agent context.
## Verdict on prior reports
| Prior claim | Corrected |
|---|---|
| Poison thread `19ed1b4f…`, thousands of ~36 KB notification emails | Wrong thread (that one is 3 msgs, ~0 MB) and wrong content. Real killer `19df82a3…`: 25 msgs, 147 MB, ~8.6 MB inline-image emails. |
| "one giant thread vs. cumulative batch of 6" (open) | Resolved: one thread (147 MB vs. <1 MB for the other 8). |
| concurrency-20 fan-out + attachment buffers is the mechanism | Partly right; dominant amplifier is intra-thread uncapped `Promise.all` + base64 re-inlining, not breadth. |
| "net RX ≈ 0, nothing downloaded" | `getAttachment` does download the images; the inflation is Cedar's inlining, not a big response body. |
## Fix plan
Concurrency is the accelerant, not the cause. Fixes, in order of leverage:
1. **Stop base64-inlining at sync; store image references.** Read path resolves via a proxy that
**lazy-caches to S3** (content-hash key → the 17 quoted copies dedupe to 1; 7-day lifecycle expiry;
re-fetch from Gmail on miss). Kills steps 3, 4, 6. *This is the fix.*
2. **Enrich-by-diff:** reuse cached per-message bodies keyed by message id; enrich only new messages
(+ the last message's draft→sent flip). Pass `cachedById` into `manager.get()`. Turns per-sync work
from O(all messages) to O(new). Also removes step 3's re-fetch for old messages.
3. **Lightweight manifest for change detection** (ids + labels + per-message content hash) so step 5
stops parsing the 180 MB prior snapshot.
4. **Agent content as structure-preserving markdown** (derived from the existing cheerio tree — no new
dep), stored per-message. Optional for the crash.
5. **Decode dedupe** (`fromBinary` called once) — DONE (`google.ts:~1386`).
6. **Drop the concurrency cap as a memory measure**; keep only a generous, configurable bound for
Gmail API quota / defensive reasons. With small bodies, 20-wide is trivially safe on 8 GB.
## Immediate mitigation
`DISABLE_PERIODIC_EMAIL_SYNC=1` on the worker halts all thread sync and stops the crashloop
(`operations.ts:628`). Sync is idempotent (sync-token based) and catches up when re-enabled.
**On existing poison snapshots (correction):** once Fix 1 lands they **stop OOM-ing** — enrich-by-diff
reuses the cached (base64) message bodies and skips the expensive re-inline + sanitize/cheerio, so a
sync of one of these threads peaks ~360 MB (S3 read + re-store of the ~180 MB blob), not 6.5 GB. They
do **not** auto-shrink, though: the old base64 messages are reused as-is, not re-enriched, so the
snapshot stays ~180 MB (storage cost + that ~360 MB/sync) until migrated. To fully migrate a thread to
the reference format, force a full re-enrich (no cache reuse) in reference mode via
`fetchAndStoreThreadFromProvider(connectionId, threadId)` — see
`src/scripts/backfill-inline-image-refs.ts`. Deleting the S3 snapshot also works (next sync rebuilds
it in reference format).