CedarCopilot wants to merge 1 commit into staging from fix/monitoring-alert-noise
Live on prod, no production signal yet
Strict CloudWatch fallback found 1 prod failure log lines for [SUBSCRIPTION] Error setting up Gmail watch (Failed to setup Gmail watch: {\n \), but this domain-wide failure family is not tied to this intent.
Libra has verdicts on 0 of 1 tracked behaviors on prod; 1 is still being checked. Libra checks hourly for 3 days after each deploy.
A 24h sample of #cedar-monitoring (2026-08-26 05:31 ā 2026-08-27 05:31 UTC) held 1,205 alerts across 355 signatures. Roughly a third were cases where the system behaved correctly and paged anyway, which buried the genuine failures.
This fixes the two largest noise sources. Both are alerting bugs, not behaviour changes.
pre-execution-setup ends the workflow when the domain-identification ladder finds no external party to attribute an event to (an internal-only email or meeting). The comment above it already says:
// NO DEAL TO RUN AGAINST IS NOT AN ERROR , but it is the end of this workflow
...and then signals that by throw new Error(...). The alerting layer cannot distinguish that from real breakage, so every internal email paged as a red-circle ERROR. 46 users hit it; top sources were <email> (86), <email> (28), <email> (20).
Fix: a dedicated ExpectedTerminationError. The catch marks the logged tool call isExpected, and logToolCall skips Slack for it. The execution is still marked failed and the call is still written to agent_tool_calls; only the page is suppressed.
The suppression condition is extracted as shouldNotifyToolCallError so the three-way logic (looks-like-failure / recoverable / expected) is named and unit tested, rather than inlined in a function that also does DB writes.
app.ts already suppresses benign webhook rejections by matching parseResult.reason against an allowlist (no_users_found_matching_participants, no_external_participants, ...).
| Driver | Failure sites | Sites setting reason | Alerts/24h |
|---|---|---|---|
| Gong | 15 | 15 | 0 |
| Fireflies | 11 | 11 | 0 |
| Fathom | 5 | 0 | 92 |
| AskElephant | 5 | 0 | 6 |
| Circleback | 3 | 0 | 5 |
With no reason, nothing those three returned could ever match the allowlist, so every rejection paged , including the benign ones the allowlist exists to silence , carrying no reason, no user and no meeting id.
Fix: reason codes at all 13 sites using the existing vocabulary, plus no_fathom_connections_for_users on the allowlist. Circleback's no-user-match maps onto the already-allowlisted no_users_found_matching_participants. AskElephant's remaining reasons are genuine misconfiguration (missing X-User-ID, unknown user), so they keep paging, but are now diagnosable.
These rejections return 401, so the meeting payload is dropped. Two users reported missing post-meeting output on the same day this sample covers. Silencing the benign rejections is what makes the real drops visible; it does not by itself fix them.
tsc --noEmit cleanshouldNotifyToolCallErrorpnpm deps:check reports no violations (1,572 modules)Roughly 1,205 ā ~800 alerts/day, with what remains being mostly real.
An expected termination still marks the execution failed, which feeds health reports and the zero-tool-execution audit. Arguably it should not, but that has wider blast radius and deserves its own change.
š¤ Generated with Claude Code
Libra has not measured any production surfaces for this change yet.