Libra CodeHub

CedarCopilot/cedar-mail

Branch: staging

fix(crm): restore daily deal reconciliation and stop selection falling back to the oldest deal

merged#2600CedarCopilot

CedarCopilot wants to merge 2 commits into staging from fix/crm-reconcile-lock-and-active-deal

Live on prod, no production signal yetTimeline and evidence
  1. Opened
    Aug 26, 2026, 7:01 PM
  2. Aug 26, 2026, 7:18 PM
  3. Merged
    Aug 26, 2026, 7:22 PM
  4. Live on prod
    Aug 26, 2026, 7:22 PM
  5. Observed 26 hours
    Aug 26, 2026, 7:22 PM
  6. Pipelines steady after this deploy
    Aug 26, 2026, 7:22 PM
  7. Unobserved

    Live on prod, no production signal yet

    Aug 27, 2026, 9:02 PM

Behaviors Libra is checking

CRM deal links created through manual linking, agent/event workflows, and automatic search linking immediately store available deal name, stage, closure status, and provider update time so correct deal selection does not depend on the nightly reconciliation job.Inconclusivelow confidence

OTEL fallback found no prod spans matching linkExternalCrmEventToConversation since deploy.

prod, checked Aug 27, 2026, 7:30 PM
Relinking an existing CRM deal refreshes only newly supplied metadata, preserves previously known fields when callers have less information, retains explicit open status, and avoids unnecessary conversation metadata updates when nothing changed.Inconclusivelow confidence

OTEL fallback found no prod spans matching linkExternalCrmEventToConversation since deploy.

prod, checked Aug 27, 2026, 8:30 PM
The daily CRM deal reconciliation job correctly reads PostgreSQL advisory-lock results under the production driver so it runs when the lock is acquired instead of silently skipping every scheduled run.Inconclusivelow confidence

OTEL fallback found no prod spans matching processCrmDealReconciliation / [CRM_RECONCILE] since deploy.

prod, checked Aug 27, 2026, 8:56 PM
When no active or known-open deal is available, a linked CRM deal with a name or stage is selected ahead of a bare provider-and-ID stub, while a sole stub remains usable.Inconclusivelow confidence

OTEL fallback found no prod spans matching pickActiveExternalCrmDeal / pickActiveDeal since deploy.

prod, checked Aug 27, 2026, 8:56 PM
Conversations with multiple CRM deals select the explicitly active deal first, otherwise the most recently updated known-open deal, rather than incorrectly treating unknown-status links as open and falling back to the oldest linked opportunity.Inconclusivelow confidence

OTEL fallback found no prod spans matching pickActiveExternalCrmDeal / pickActiveDeal since deploy.

prod, checked Aug 27, 2026, 9:02 PM

Libra has verdicts on 0 of 5 tracked behaviors on prod; 5 are still being checked. Libra checks hourly for 3 days after each deploy.

The daily CRM deal reconciliation has not run since 2026-07-16

Prod logs, worker-service, 30 consecutive days (the full CloudWatch retention window) , one delivery per day at 03:00:12 UTC, every one of them this line, and not one Starting:

[CRM_RECONCILE] Skipping , another reconciliation run already holds the lock for <date>

There was no concurrent run to contend with. The lock check itself was broken:

lockAcquired = (lockResult as { rows?: ... }).rows?.[0]?.acquired ?? false;

This is drizzle-orm/postgres-js, where db.execute() returns rows as a bare array. .rows is undefined, so ?? false turned "I could not read the result" into "someone else holds the lock" and the job returned early every night. The same file reads purgeResult.rowCount off the top level two functions down, and crm-reconcile-deals.ts:230 casts db.execute straight to an array , .rows appears exactly once in the whole cron directory.

It shipped in #2196, whose stated purpose was to stop this job re-running 96x/day. It took it to zero instead, and the failure logged as a sentence that reads like healthy contention.

Why that broke deal selection

Reconciliation is the only thing that writes isClosed / updatedAt / ownerId onto a conversation's linked deals, and those are the fields selection runs on. With it dead, links stayed exactly as linkExternalCrmEventToConversation wrote them , bare , and pickActiveExternalCrmDeal degraded:

  • step 2 accepted isClosed !== true, which is true for every bare entry;
  • it sorted on a missing updatedAt, so all keys were 0 and the stable sort was a no-op;
  • so it returned deals[0].

Links are appended, so index 0 is the oldest deal.

Real example , Pirros conversation 4e83a8b9, four linked deals, all bare:

#dealstage
0Pope Design Group - Missed MeetingDeal DQ (2024)← selected
1Pope Architects - 2Lost - Company
2Pope Design Group - 1Lost - Company
3Pope Design Group - 3Discovery← the live deal

A meeting held that morning was attributed to a dead 2024 opportunity. The CRM updater wrote to it (set SQL, changed the DQ reason, moved the initial-meeting date), and the recap email was suppressed because the resolved deal sat in a terminal stage. The orchestrator's own run output shows it noticing and deferring anyway: "the sibling deal 'Pope Design Group - 3' is in Meeting Booked stage and this meeting was clearly the first real meeting for that deal… I'll respect the stage-level override and skip the draft."

Blast radius, measured: 13,008 conversations carry linked deals; 9,794 have at least one bare link; 832 are both multi-deal and bare , the population where the wrong deal can win. 64 users.

The fixes

1. Fail closed on an unreadable result. Read the lock via requireSingleRow, which throws rather than defaulting. Promotes the existing toRows helper out of services/account-deletion/ into db/result-shape.ts, where a driver-shape concern belongs, and adds the strict variant. "I could not read the answer" and "the answer was no" must not collapse into the same value.

2. Stop links being born blank. Write isClosed / updatedAt at link time from the deal payload the caller already holds, so correctness no longer depends on a later job. Converts linkExternalCrmEventToConversation's trailing positional args (8 of them, several adjacent string | undefined) into a named ExternalCrmLinkDetails object.

ownerId is deliberately not forwarded. The stored field holds a resolved Cedar user id , computeReconcileMetadataUpdate writes the resolved value and deletes it when the provider owner is not a Cedar user, and recomputeActiveFlag uses !!ownerId as its "owner is a Cedar user" test. Forwarding a raw provider id would have marked an open deal owned by nobody on Cedar as active at link time, and active is st

Show production surfaces and changed-file mapping

Production surfaces

Libra has not measured any production surfaces for this change yet.

Changed files → surfaces

  • apps/mail/modules/crm/types/index.tsno production surface mapped
  • apps/server/src/cron/process-crm-deal-reconciliation.tsno production surface mapped
  • apps/server/src/db/__tests__/result-shape.test.tsno production surface mapped
  • apps/server/src/db/result-shape.tsno production surface mapped
  • apps/server/src/mastra/routeHandlers/event-execution/handleExecuteExternalCrm.tsno production surface mapped
  • apps/server/src/mastra/tools/conversation/__tests__/conversationTool.test.tsno production surface mapped
  • apps/server/src/mastra/tools/conversation/conversationTool.tsno production surface mapped
  • apps/server/src/mastra/tools/event-execution/__tests__/createAndLinkEvents-external-crm.test.tsno production surface mapped
  • apps/server/src/mastra/tools/event-execution/assignEventToConversationTool.tsno production surface mapped
  • apps/server/src/mastra/tools/event-execution/createAndLinkEventsTool.tsno production surface mapped
  • apps/server/src/services/account-deletion/db-purge.tsno production surface mapped
  • apps/server/src/services/account-deletion/external-purge.tsno production surface mapped
  • apps/server/src/services/account-deletion/index.tsno production surface mapped
  • apps/server/src/services/account-deletion/result-shape.tsno production surface mapped
  • apps/server/src/services/crm/__tests__/active-deal.test.tsno production surface mapped
  • apps/server/src/services/crm/active-deal.tsno production surface mapped
  • apps/server/src/services/crm/crm-deal-auto-link.tsno production surface mapped
  • apps/server/src/services/crm/external-crm-events.tsno production surface mapped
  • apps/server/src/services/crm/move-event.tsno production surface mapped
  • apps/server/src/trpc/routes/crm.tsno production surface mapped