Libra CodeHub

CedarCopilot/cedar-mail

Branch: staging

fix(chat): surface a failed-turn bubble with Retry instead of swallowing chat errors

merged#2804CedarCopilot

CedarCopilot wants to merge 2 commits into staging from fix/chat-failed-message-retry

Live on prod, watching, less than a day leftTimeline and evidence
  1. Opened
    Sep 16, 2026, 7:27 AM
  2. Sep 21, 2026, 5:05 AM
  3. Merged
    Sep 21, 2026, 5:19 AM
  4. Live on staging
    Sep 21, 2026, 5:19 AM
  5. Live on prod
    Sep 21, 2026, 5:19 AM
  6. Observed 16 hours
    Sep 21, 2026, 5:19 AM
  7. Pipelines steady after this deploy
    Sep 21, 2026, 5:19 AM
  8. Pipelines steady after this deploy
    Sep 21, 2026, 5:19 AM
  9. Watching

    Live on prod, watching, less than a day left

    Sep 21, 2026, 9:03 PM

Behaviors Libra is checking

A stream error event now surfaces a retryable failed-turn bubble instead of only logging the error to the console.Degradedlow confidence

Strict CloudWatch fallback found 2 recent prod failure log lines for [processBufferedSlackWebhookEvents] Parked non-recoverable row (No valid connection tokens found for user <id>), but the same failure had 1 log lines in the comparable pre-deploy baseline. Libra is not blaming.

staging, checked Sep 23, 2026, 6:34 AM
A chat run that the user has already stopped is not aborted again by the acknowledgement timeout and does not receive a contradictory failed-turn bubble.Inconclusivelow confidence

Unable to verify the intended behavior from the queried production telemetry: no matching acknowledgement-timeout, cancellation, abort, or chat-failed signals were found. Further telemetry mapping is required before issuing a verdict.

prod, checked Sep 21, 2026, 5:02 PM
The chat message renderer registry recognizes `chat-failed` messages and displays their failure text with an accessible icon-only Retry button and retrying state.Inconclusivelow confidence

Unable to complete the required telemetry verification within the available tool loop.

prod, checked Sep 21, 2026, 6:03 PM
Multiple failure notifications from the same chat turn produce only one failed-turn bubble.Inconclusivelow confidence

Strict CloudWatch fallback saw 41,154 success-shaped log lines matching sendmessage, failedmessagerenderer, agentconnectionslice, failedmessage, initializemessagerendererregistry, but no tied operation was present, so Libra is not calling this working.

prod, checked Sep 21, 2026, 6:05 PM
Clicking Retry on a failed-turn bubble resends the original message through sendMessage with the original text, thread, and captured request parameters rather than using the current composer contents.Inconclusivelow confidence

Unable to verify the behavior: the post-deploy production query returned no matching `mastra.chatStream`/`chatStream` telemetry rows, so there is no positive runtime evidence of Retry-triggered sends or their success/failure health.

prod, checked Sep 21, 2026, 7:03 PM
When sendMessage throws before or around stream creation, the user sees the same retryable failed-turn bubble instead of the previous non-retryable generic error message.Inconclusivelow confidence

Strict CloudWatch fallback saw 15,915 success-shaped log lines matching sendmessage, failedmessagerenderer, agentconnectionslice, failedmessage, initializemessagerendererregistry, but no tied operation was present, so Libra is not calling this working.

prod, checked Sep 21, 2026, 7:05 PM

Failures attributed to this change

No prod customers are affected while this is only in staging. If promoted, this likely touches a customer-facing path; Libra should verify the failed user action before escalating. 0 hits · no retained affected-user count · no retained trace sample.Introducedmedium confidence

single_user

staging, first seen Sep 21, 2026, 9:32 AM
No prod customers are affected while this is only in staging. If promoted, prod impact is unknown because Libra still needs a concrete exception, route, and failed user action before assigning severity. 1 hit · no retained affected-user count · no retained trace sample.Introducedmedium confidence

internal_only

staging, first seen Sep 21, 2026, 9:32 AM

Libra has verdicts on 0 of 8 tracked behaviors on prod; 8 are still being checked. Libra checks hourly for 3 days after each deploy.

Bug

Cedar's in-app AI chat (apps/mail) could fail a turn with zero visible feedback:

  1. Swallowed stream errors. agentConnectionSlice.ts's stream handler had a case 'error': that only did console.error(...) and break , no rethrow. The existing generic error-bubble fallback in the outer catch block never got a chance to fire, because the provider's streamLLM.completion promise resolves normally after emitting an error event (it only calls the handler, then returns without throwing , see providers/mastra.ts's internal try/catch). So a stream error left the user's message sitting there with no reply and no error state.
  2. Requests that vanish before reaching the backend at all. If the POST is dropped by a proxy or the server never opens a stream, there's no HTTP error and no stream error event either , nothing ever settles the turn as failed.

Fix

  • chat-failed message type + FailedMessageRenderer , a new bubble (ShimmerText in its error state, matching the existing "Agent Interrupted" bubble's styling) with a Retry button that resends the original message (same text, same thread) through the normal sendMessage() pathway , using the same overridePrompt replay contract messageQueueSlice already uses for queued sends.
  • surfaceFailedChatMessage , a shared helper (agentConnectionSlice.ts) called from every place a turn can die silently: the stream error case, a genuine exception in sendMessage (replacing the old static "An error occurred..." text bubble that had no retry), and a resumed stream's own error case , instead of duplicating (or, for the stream-error path, omitting) the failure UI.
  • Send-ack timeout (45s) , detects "the request never reached the backend at all": if sendMessage hasn't heard any stream event within SEND_ACK_TIMEOUT_MS of sending, it aborts the stream and surfaces the same failed bubble. This is a one-shot "did anything come back at all" timer, not a per-chunk staleness check , it clears on the very first event (including an early tool-call notification), so a long multi-tool-call turn that's merely slow is unaffected; only total silence from the start counts as vanished. Skipped if the user already stopped the run by hand (no double "Interrupted" + "failed" bubble).

How I decided to detect "never reached the backend"

There's no existing beacon/ack pattern in this codebase for that case (checked: providers/mastra.ts, agentUtils.ts, no AbortSignal.timeout or health-check usage anywhere in agentConnection/). The simplest reliable mechanism given the existing StreamHandler/StreamResponse shape is a client-side "first event" timer: schedule it right after streamLLM() returns, clear it in the wrapped handler on the first event of any type, and treat a fully-silent timeout exactly like a stream error (abort + same failed bubble). 45s is a judgment call , generous enough for a cold start, but open to tuning if telemetry says otherwise.

Tests

  • agentConnectionSlice.failedMessage.test.tsx , stream error → one failed bubble (and exactly one even if the stream errors twice); an exception thrown before streaming starts → failed bubble; ack-timeout fires + aborts when nothing ever arrives; ack-timeout stays quiet (no abort, no bubble) when the user already cancelled the run.
  • FailedMessageRenderer.test.tsx , clicking Retry resends the original threadId/overridePrompt through sendMessage().

Verification

  • pnpm --filter @zero/mail run types , same pre-existing error set as staging (paraglide/react-router codegen +types/@/paraglide/* modules not generated in this checkout); no new errors in any touched file.
  • pnpm deps:check , no dependency violations (1833 modules, 9699 dependencies).
  • pnpm --filter @zero/mail exec jest modules/cedar-os , 31 suites / 443 tests pass, including the 6 new ones.
  • Ran the `thermo-review
Show production surfaces and changed-file mapping

Production surfaces

Libra has not measured any production surfaces for this change yet.

Changed files → surfaces

  • apps/mail/modules/cedar-os/__tests__/chatComponents/FailedMessageRenderer.test.tsxno production surface mapped
  • apps/mail/modules/cedar-os/__tests__/store/agentConnection/agentConnectionSlice.failedMessage.test.tsxno production surface mapped
  • apps/mail/modules/cedar-os/src/store/agentConnection/agentConnectionSlice.tsno production surface mapped
  • apps/mail/modules/cedar-os/src/store/messages/renderers/FailedMessageRenderer.tsxno production surface mapped
  • apps/mail/modules/cedar-os/src/store/messages/renderers/initializeMessageRendererRegistry.tsxno production surface mapped