Libra CodeHub

CedarCopilot/cedar-mail

Branch: staging

Three CRM/worker defects the post-merge prod watch turned up

open#2646CedarCopilot

CedarCopilot wants to merge 3 commits into staging from fix/crm-multiselect-and-vendor-retry

Not deployedTimeline and evidence
  1. Opened
    Sep 1, 2026, 10:27 AM
  2. Sep 13, 2026, 1:39 PM
  3. Merged, live on staging, live on prod, observed
    Pending
  4. Not deployed

    Not deployed

    Pending

Libra has no production signal for this change yet because it has not deployed. Libra checks hourly for 3 days after each deploy.

Found by the hourly post-merge prod watch on the 2026-09-01T09:57Z deploy. None of these came from that deploy , the merged changes (#2641, #2642, #2643) are clean. These are pre-existing defects the investigation surfaced, fixed here on their own branch.

1. A multi-select joined with a comma failed its own vocabulary check

Over 120h, 45 of 69 rejected CRM writes were a multi-select the agent joined with a comma ("Voice,Email" ×12, "Voice,Email,Route Optimization" ×8, "Gusto, Rippling", "Cgl,tech Eo,cyber"), and 7 more were ;-joined on a mapping whose isMultiPicklist flag was never set ("Payroll;Customer Support", "Rippling;Sequoia;Deel").

In every one of them each token was already in the customer's own vocabulary. splitMultiPicklist only split on ;, and the flag was the only thing that enabled splitting at all , so the whole string was matched as one option, the write was refused, and the CRM field silently stayed empty. This is lost data, not a loud failure.

Now: on a field flagged isMultiPicklist, a comma counts as a separator too. The repair is arbitrated by the field's own vocabulary , the semicolon split (the delimiter every CRM puts on the wire) wins whenever it resolves, and the comma split is tried only when it does not. That is what keeps the repair strictly additive: a value that tokenized correctly before still does, and only previously-rejected values can change.

That ordering is load-bearing, not decoration. A whole-value match alone was not enough, because it only catches a value that is ENTIRELY one option , and real vocabularies contain comma-bearing options (Product_Type__c has "Subscription, No Trial"; Opt_Out_Close_Survey_pl__c has "No, send survey"). A legitimate combination like "Subscription, No Trial;Cyber" tokenized to three strings, which refused a write that used to succeed and would have put the mangled "Subscription;No Trial;Cyber" on the wire.

Two existing tests asserted the unflagged path stays broken; they are updated to the new intent. The guarantee that an off-domain token is still refused is kept and tested on both paths.

2. Transient vendor failures were surfaced as failed CRM writes

On 2026-09-01, 14:04,14:16, Superglue rejected 18 writes with Organization tool execution queue is full (500 tasks waiting), try again later, and 26 more failed with fetch failed in the same burst. Its own message says to retry; we did not , so a twelve-minute vendor hiccup became 44 CRM fields that never got written, across 3 users and 2 providers.

executeWorkflow now retries twice (1s, 4s), only on conditions meaning the request never ran: a saturated org queue or a connection-level failure. Anything the CRM actually decided on , a locked record, a value outside a picklist, a missing auth , returns on the first attempt as before, so a rejected write is never re-sent in a loop. Every tool routed through this client is idempotent, so a re-send cannot double-apply.

3. A lost claim race was logged as an error

Failed to create execution for task was the loudest line in the prod error channel (28 in one 3h window), and every one carried Task already claimed or no longer eligible , two workers racing for one scheduled task. The winner runs it; the loser logs. Nothing failed, but it arrived at error level in bursts of ~19.

The outcome is now a named exported constant rather than a bare string, and that one case logs at info. Any other executeTask failure still logs at error, unchanged.

4. The HubSpot driver was inferring multi-ness from the delimiter

updateOpportunityFields in hubspot.ts never calls validateAllowedValue , it validates inline , so the "stop inferring multi-ness" guard added to the shared validator never reached it. It was still selecting the multi-select branch on tokens.length > 1, which meant a single-select field that received "Voice,Email" was written as "Voice;Email": exactly the opa

Show production surfaces and changed-file mapping

Production surfaces

Libra has not measured any production surfaces for this change yet.

Changed files → surfaces

  • apps/server/src/services/integrations/crm/__tests__/crm-multiselect-delimiters.test.tsno production surface mapped
  • apps/server/src/services/integrations/crm/__tests__/hubspot-picklist-tokens.test.tsno production surface mapped
  • apps/server/src/services/integrations/crm/attio.tsno production surface mapped
  • apps/server/src/services/integrations/crm/driver.tsno production surface mapped
  • apps/server/src/services/integrations/crm/external-crm-field-mapping.tsno production surface mapped
  • apps/server/src/services/integrations/crm/hubspot.tsno production surface mapped
  • apps/server/src/services/integrations/crm/salesforce.tsno production surface mapped
  • apps/server/src/services/integrations/superglue/__tests__/superglue-retry.test.tsno production surface mapped
  • apps/server/src/services/integrations/superglue/superglue-client.tsno production surface mapped
  • apps/server/src/services/task-scheduling/execution.tsno production surface mapped
  • apps/server/src/services/task-scheduling/index.tsno production surface mapped
  • apps/server/src/worker/queue-handler.tsno production surface mapped