CedarCopilot wants to merge 3 commits into staging from fix/user-settings-deep-merge
Live on prod, no production signal yet
Strict CloudWatch fallback found 54 recent prod failure log lines for [UpdateExternalCrmWorkflow] Driver update failed (Step updateOpportunityField failed: HTTP 400: [{\), but the same failure had 150 log lines in the comparable pre-deploy baseline. Libra is not blaming this PR.
Strict CloudWatch fallback saw 1,077 success-shaped log lines matching crm, hubspot, salesforce, deal, deals, but no tied operation was present, so Libra is not calling this working.
internal_only
internal_only
internal_only
internal_only
internal_only
internal_only
Libra has verdicts on 0 of 1 tracked behaviors on prod; 1 is still being checked. Libra checks hourly for 3 days after each deploy.
updateUserSettings merged each settings column with COALESCE(col, '{}'::jsonb) || patch::jsonb. jsonb || jsonb merges exactly one level, so a patch aimed at a single sub-key replaced the entire group it lives in and dropped every sibling. Nothing threw, nothing warned, the write reported success.
Reproduced on a live account (Edexia / Daniel Gibbon, 2026-08-25) while intending only to change a timezone:
await updateUserSettings({ userId, notificationSettings: {
emailNotifications: { timezone: 'Australia/Brisbane' } } });
before {enabled: true, timezone: 'America/Los_Angeles', digestTime: '09:00', agendaEmail: true}
after {timezone: 'Australia/Brisbane'}
enabled and agendaEmail are exactly what cron/process-agenda-emails.ts gates on, so the user's Cedar Agenda email silently stopped. This is the same failure class as the incident that mailed 38 users their Slack DMs, except it happens through the correct, single writer. And it is not a corner case: user-setting-specs.ts marks four notification groups nested: true, and every one of them has this shape.
mergedJsonb builds the merge expression recursively: every plain-object value in the patch gets its own || against the value stored at that path, spliced back in with jsonb_build_object. A patch of only scalars generates the same SQL as before.
Still composed as SQL rather than a JS read-modify-write. notification_settings has several unrelated writers (this one, settingsRouter.save, the digest crons), and under pgbouncer transaction pooling a read-then-write races them and drops one side , the same reason switchAgendaDeliveryToSlack patches its one leaf with jsonb_set. Patch keys are bound as parameters, never interpolated, since notificationSettingsSchema accepts z.record(z.unknown()).
| Patch value | Behaviour | Why |
|---|---|---|
| plain object | merged, sub-key by sub-key, at any depth | the bug |
null | replaces , clears a sub-key or a whole group | a merge with no escape hatch can only ever add. {slackNotifications: {notificationChannelId: null}} clears one key, {slackNotifications: null} clears the group. emailNotifications is already declared .nullable() in lib/schemas.ts, so a stored null is a value the schema expects |
| array | replaces wholesale | an array is one value, not a bag of keys. Element-wise merging ['a'] onto ['a','b','c'] would leave all three, so a caller could never shorten a list |
| scalar | replaces | unchanged |
undefined | ignored (not written as null) | it carries no intent. This is the convention extractCategorizedSettings documents ("absent keys are OMITTED, not assigned undefined") and is what JSON.stringify already did to such a key here |
No opt-out flag, because no caller relies on replace semantics. All seven call sites were checked: settings.updateMail/Notification/AgentSettings (the playground editor builds each group by spreading over what it loaded, so a merge is what it means), admin.ts's four agent toggles (scalars), phase5-email.ts (scalars), phase9-notifications.ts (already layers defaults under whatever is stored, so it seeds either way), and applyUserSettingsConfigChange. settingsRouter.save writes its own inline upsert and is untouched by this change.
getExistingSlackNotifications , applyUserSettingsConfigChange used to SELECT the stored slackNotifications and spread the patch onto it in JS before writing. That existed only because the shared writer replaced sub-objects. It raced any concurrent writer, and it only ever covered one of the four nested groups (which is how emailNotifications was left exposed). The writer does it atomically now, so the round trip is gone.
Stale comments in phase9-notifications.ts and its test, which described the old one-level merge as the reason they layer defaults under stored va
Libra has not measured any production surfaces for this change yet.