Libra CodeHub

CedarCopilot/cedar-mail

Branch: staging

fix: prevent duplicate/dropped webhook agent runs on retry or overlap

merged#2676CedarCopilot

CedarCopilot wants to merge 2 commits into staging from fix/hot-lead-webhook-duplicate-and-transient-db-retry

Live on prod, no production signal yetTimeline and evidence
  1. Opened
    Sep 3, 2026, 3:53 PM
  2. Sep 3, 2026, 4:33 PM
  3. Merged
    Sep 3, 2026, 4:39 PM
  4. Live on prod
    Sep 3, 2026, 4:39 PM
  5. Observed 2 hours
    Sep 3, 2026, 4:39 PM
  6. Unobserved

    Live on prod, no production signal yet

    Sep 3, 2026, 4:39 PM
  7. Pipelines steady after this deploy
    Sep 3, 2026, 4:39 PM

Behaviors Libra is checking

Creation of immediate and subagent execution rows retries transient PostgreSQL failures, including failover read-only errors, with fresh query attempts and preserves the fixed run ID.Not checked
prod
Execution status and progress updates retry transient PostgreSQL failures so an agent that already performed an external side effect is not left in a state that causes the queue to rerun it.Not checked
prod
CRM email-thread and label mirror writes retry `read_only_sql_transaction` failures in addition to the existing transient contention errors.Not checked
prod
Webhook deliveries from HTTP, tRPC, board, and Slack sources receive a UUID that is carried through the queue and remains unchanged when the same message is redelivered, with one shared UUID for all registrations matching one Slack event.Not checked
prod
Webhook execution context uses the message's original received timestamp as firedAt instead of assigning a new current time on each queue attempt.Not checked
prod
Webhook dispatches missing either webhookId or deliveryId continue using the existing time-window dedupe scope instead of producing an invalid external-key scope.Not checked
prod

Libra has verdicts on 0 of 8 tracked behaviors on prod; 8 are still being checked. Libra checks hourly for 3 days after each deploy.

What was wrong

Traced from Aspire's inbound-lead-research webhook (2026-09-03), two related gaps in the playbook-webhook automation path:

1. No idempotency anchor for webhook triggers. resolveAutomationDedupeScope had keyed dedup for cron (tick + optional conversation) and event_occurred (CRM event id), but webhook fell through to the plain time-window fallback , and unlike every other time-window caller, a webhook trigger is conversationless, so the query was just "any execution for this agent+user in the last 5 minutes?" with nothing to tell two different deliveries apart. This produced two live symptoms:

  • A retried webhook delivery (SQS redelivery after the DB failure below) re-ran the agent from scratch and posted a duplicate Slack reply.
  • Two genuinely different leads landing within 5 minutes of each other , which happens routinely in a busy inbound channel , silently dropped the second one, deduped against the first's still-running execution. Confirmed live twice on 2026-09-03: NextFoods (Hannah Duong) deduped against Santa Cruz Superfoods' execution, and Raptive (Emma Smits) deduped against Del Taco's, both via [runAgent] Skipping duplicate agent execution / `dedupe_key=[redacted] Neither lead ever got a Cedar response.

2. No retry on a transient Postgres write failure. A ~9-second Postgres failover blip hit updateExecution's finalization write with 25006 read_only_sql_transaction. The agent's Slack post had already succeeded, but the status write failing made the queue consumer treat the whole dispatch as failed and retry it , reproducing the Slack post a second time.

What changed

  • services/agent-action-queue/automation-dedupe.ts , new webhook branch, keyed on (agentId, webhookId, firedAt), no minute-bucketing (unlike cron) since firedAt is now an exact per-delivery anchor, not a resampled clock reading.
  • services/playbook/playbook-webhook.ts , dispatchPlaybookWebhookMessage now passes firedAt derived from the message's own receivedAt (fixed once at first enqueue) instead of leaving it unset, which previously defaulted to Date.now() on every attempt.
  • db/transient-retry.ts (new) , the retry helper promoted from services/crm/email-threads.ts, with 25006 added to its transient-error code list. email-threads.ts now imports it instead of keeping its own copy.
  • services/agent-action-queue/helpers.ts / executions.ts , updateExecution, createImmediateExecution, and createSubagentExecution (same file/shape as updateExecution, on the same hot per-event path) now retry on a transient Postgres error.

How to test

  • New/updated tests: db/__tests__/transient-retry.test.ts (new), automation-dedupe.test.ts (4 new webhook cases), playbook-webhook.test.ts (firedAt threading), automations-service.test.ts (DB-mock wiring test proving runAgent extracts webhookId and writes the composed dedupe key).
  • pnpm --filter @zero/server run types and pnpm deps:check both clean.
  • To verify in prod: watch #hot-lead-demo-request for a burst of 2+ leads within 5 minutes , every one should now get its own inbound-lead-research reply, and a redelivered webhook message should no longer double-post.

🤖 Generated with Claude Code

https://claude.ai/code/session_01HBS8XHFE8jypQBgSzJimRy

Greptile Summary

This PR gives webhook-triggered agent runs a stable deduplication key and adds transient PostgreSQL retries around execution writes while extracting the existing CRM retry helper.

  • Threads each webhook message’s fixed receipt time through playbook execution.
  • Adds webhook-specific execution deduplication using agent, webhook, and receipt time.
  • Retries selected execution inserts and updates for transient PostgreSQL failures.
  • Centralizes transient-error classification and adds regression tests.

Confidence Score: 4/5

The retry behavior appears sound, but t

Show production surfaces and changed-file mapping

Production surfaces

Libra has not measured any production surfaces for this change yet.

Changed files → surfaces

  • apps/server/src/db/__tests__/transient-retry.test.tsno production surface mapped
  • apps/server/src/db/transient-retry.tsno production surface mapped
  • apps/server/src/http/app.tsno production surface mapped
  • apps/server/src/services/agent-action-queue/__tests__/automation-dedupe.test.tsno production surface mapped
  • apps/server/src/services/agent-action-queue/automation-dedupe.tsno production surface mapped
  • apps/server/src/services/agent-action-queue/executions.tsno production surface mapped
  • apps/server/src/services/agent-action-queue/helpers.tsno production surface mapped
  • apps/server/src/services/aop/__tests__/automations-service.test.tsno production surface mapped
  • apps/server/src/services/aop/automations.tsno production surface mapped
  • apps/server/src/services/crm/email-threads.tsno production surface mapped
  • apps/server/src/services/documents/triggers/board-webhooks.tsno production surface mapped
  • apps/server/src/services/playbook/__tests__/playbook-webhook-execution.test.tsno production surface mapped
  • apps/server/src/services/playbook/__tests__/playbook-webhook-sources.test.tsno production surface mapped
  • apps/server/src/services/playbook/__tests__/playbook-webhook.test.tsno production surface mapped
  • apps/server/src/services/playbook/playbook-execution-triggers.tsno production surface mapped
  • apps/server/src/services/playbook/playbook-webhook-sources.tsno production surface mapped
  • apps/server/src/services/playbook/playbook-webhook.tsno production surface mapped
  • apps/server/src/trpc/routes/aop.tsno production surface mapped