Libra CodeHub

CedarCopilot/cedar-mail

Branch: staging

Incident ledger

Incidents Libra paged for this repository, who introduced them, and what fix is in flight.

IncidentEnvSeverityFirst seenLast seenIntroduced byFixStatus
[ERROR] [UpdateExternalCrmWorkflow] Driver update failed
0 users
ProductionhighJun 22Sep 20, 2026, 6:18 AM#2509, mediumnoneresolved
OTel exporter failure: 1 occurrence, concentrated in worker log stream <id>StagingP2Sep 19Sep 19, 2026, 6:11 AMnonenoneresolved
Exporter failures: worker-service, 2 task streams, at 2026-09-18 16:50:49StagingP2Sep 18Sep 18, 2026, 1:04 PMnonenoneresolved
OTLP exporter failure concentrated in worker task streamStagingP2Sep 18Sep 18, 2026, 10:57 AMnonenoneresolved
Cron/worker tasks repeatedly time out and are silently discarded by AWS SWF
9 users
ProductionhighJun 30Sep 17, 2026, 2:25 PM#2397, medium#2104, expiredresolved
Scheduled tasks silently fail — orchestrator agent returns no output, task is permanently marked failed with no retry
98 users
StagingmediumJun 28Sep 17, 2026, 2:15 AMnonenoneresolved
Chat agent streaming fails with Anthropic API 'Overloaded' and 'Internal server error' – users get no response to their queriesProductionP2Jun 30Sep 15, 2026, 11:29 AMnonenoneresolved
error @ GET /v1/projects/cedar-mail-472802/subscriptions/<redacted-token>
0 users
ProductionhighJun 18Sep 13, 2026, 9:14 PM#2277, mediumnoneresolved
To process event is failing: Failed query: \n SELECT \ProductionP2Aug 2Sep 13, 2026, 11:30 AMnonenoneresolved
To process row is failing: Failed query: delete from \StagingP2Sep 12Sep 12, 2026, 6:23 PM#2755, lownoneresolved
Get conversation is failing: synthetic spans since deployStagingP2Sep 12Sep 12, 2026, 12:53 PMnonenoneresolved
Failure concentration: worker-service 148 hydrateExecutionEvents meeting errors, all with _ZodError2ProductionP2Sep 12Sep 12, 2026, 9:51 AMnonenoneresolved
Meeting hydration failures concentrated in worker-service stream <id>: 28ProductionP2Sep 12Sep 12, 2026, 8:52 AMnonenoneresolved
Failure timing: 7 missing-content warnings occurred between 2026-09-09 17:48:02Z and 19:46:21ZProductionP2Sep 12Sep 12, 2026, 6:52 AMnonenoneresolved
Missing-content warnings: 7 distinct meeting IDs, all between 2026-09-09 17:48 and 19:46 UTCProductionP2Sep 12Sep 12, 2026, 2:52 AMnonenoneresolved
Meeting hydration failures concentrated in worker-service streams, including counts of 20, 19, 18ProductionP2Sep 11Sep 11, 2026, 11:52 PMnonenoneresolved
7 failures concentrated in the early post-deploy window from 2026-09-09 17:48:02 to 19:46:21 UTCProductionP1Sep 11Sep 11, 2026, 9:51 PMnonenoneresolved
To persist chat messages is failing: Failed query: insert into \ProductionP2Sep 11Sep 11, 2026, 5:02 PMnonenoneresolved
Get conversation is failing: spans since deployStagingP2Sep 9Sep 11, 2026, 6:27 AMnonenoneresolved
Duplicate reads concentrated in 27 trace_id/object-key pairs; the highest observed pair had 4 readsStagingP1Sep 10Sep 10, 2026, 1:37 PMnonenoneresolved
Update external crm workflow is failing with 401ProductionP2Sep 9Sep 10, 2026, 8:20 AMnonenoneresolved
Strategist agent run is failing: synthetic spans since deployProductionP2Sep 10Sep 10, 2026, 7:20 AMnonenoneresolved
Thread not found or access denied @ trpc.chat.setPrimaryConversation
0 users
ProductionwarningJul 6Sep 9, 2026, 8:49 PM#2058, lownoneresolved
UpsertEmailThread failed (Failed query: \n UPDATE crm_thread_labels tl\n SET latest_message_at = tStagingP2Sep 4Sep 9, 2026, 5:24 PMnonenoneresolved
Agent_execution.create_failed (Failed to create immediate execution): 15 failures vs 23 baselineProductionP2Sep 8Sep 9, 2026, 4:07 PMnonenoneresolved
Sandbox scores, last 30 days
5
160
4
22
3
42
2
6
1
0
0
0
n/a
11