CedarCopilot wants to merge 2 commits into staging from fix/crm-reconcile-lock-and-active-deal
Live on prod, no production signal yet
OTEL fallback found no prod spans matching linkExternalCrmEventToConversation since deploy.
OTEL fallback found no prod spans matching linkExternalCrmEventToConversation since deploy.
OTEL fallback found no prod spans matching processCrmDealReconciliation / [CRM_RECONCILE] since deploy.
OTEL fallback found no prod spans matching pickActiveExternalCrmDeal / pickActiveDeal since deploy.
OTEL fallback found no prod spans matching pickActiveExternalCrmDeal / pickActiveDeal since deploy.
Libra has verdicts on 0 of 5 tracked behaviors on prod; 5 are still being checked. Libra checks hourly for 3 days after each deploy.
Prod logs, worker-service, 30 consecutive days (the full CloudWatch retention window) , one delivery per day at 03:00:12 UTC, every one of them this line, and not one Starting:
[CRM_RECONCILE] Skipping , another reconciliation run already holds the lock for <date>
There was no concurrent run to contend with. The lock check itself was broken:
lockAcquired = (lockResult as { rows?: ... }).rows?.[0]?.acquired ?? false;
This is drizzle-orm/postgres-js, where db.execute() returns rows as a bare array. .rows is undefined, so ?? false turned "I could not read the result" into "someone else holds the lock" and the job returned early every night. The same file reads purgeResult.rowCount off the top level two functions down, and crm-reconcile-deals.ts:230 casts db.execute straight to an array , .rows appears exactly once in the whole cron directory.
It shipped in #2196, whose stated purpose was to stop this job re-running 96x/day. It took it to zero instead, and the failure logged as a sentence that reads like healthy contention.
Reconciliation is the only thing that writes isClosed / updatedAt / ownerId onto a conversation's linked deals, and those are the fields selection runs on. With it dead, links stayed exactly as linkExternalCrmEventToConversation wrote them , bare , and pickActiveExternalCrmDeal degraded:
isClosed !== true, which is true for every bare entry;updatedAt, so all keys were 0 and the stable sort was a no-op;deals[0].Links are appended, so index 0 is the oldest deal.
Real example , Pirros conversation 4e83a8b9, four linked deals, all bare:
| # | deal | stage | |
|---|---|---|---|
| 0 | Pope Design Group - Missed Meeting | Deal DQ (2024) | ← selected |
| 1 | Pope Architects - 2 | Lost - Company | |
| 2 | Pope Design Group - 1 | Lost - Company | |
| 3 | Pope Design Group - 3 | Discovery | ← the live deal |
A meeting held that morning was attributed to a dead 2024 opportunity. The CRM updater wrote to it (set SQL, changed the DQ reason, moved the initial-meeting date), and the recap email was suppressed because the resolved deal sat in a terminal stage. The orchestrator's own run output shows it noticing and deferring anyway: "the sibling deal 'Pope Design Group - 3' is in Meeting Booked stage and this meeting was clearly the first real meeting for that deal… I'll respect the stage-level override and skip the draft."
Blast radius, measured: 13,008 conversations carry linked deals; 9,794 have at least one bare link; 832 are both multi-deal and bare , the population where the wrong deal can win. 64 users.
1. Fail closed on an unreadable result. Read the lock via requireSingleRow, which throws rather than defaulting. Promotes the existing toRows helper out of services/account-deletion/ into db/result-shape.ts, where a driver-shape concern belongs, and adds the strict variant. "I could not read the answer" and "the answer was no" must not collapse into the same value.
2. Stop links being born blank. Write isClosed / updatedAt at link time from the deal payload the caller already holds, so correctness no longer depends on a later job. Converts linkExternalCrmEventToConversation's trailing positional args (8 of them, several adjacent string | undefined) into a named ExternalCrmLinkDetails object.
ownerId is deliberately not forwarded. The stored field holds a resolved Cedar user id , computeReconcileMetadataUpdate writes the resolved value and deletes it when the provider owner is not a Cedar user, and recomputeActiveFlag uses !!ownerId as its "owner is a Cedar user" test. Forwarding a raw provider id would have marked an open deal owned by nobody on Cedar as active at link time, and active is st
Libra has not measured any production surfaces for this change yet.