Why

A user's post-meeting follow-up (Kendall @ Warp) silently never appeared. Root cause: the on-event run hung partway through (a subagent stalled on an in-flight call ~1 min in), sat executing for 30 min, and the stale-execution reaper tombstoned it with no retry , so the recap was silently lost. This is systemic: every reaper sweep tombstones 5,18 hung runs across users, and llm_request_timeout has fired 0 times in 3 days (107 llm_request_failed in the same window prove the log pipeline works), i.e. the stalls are in an unbounded await the per-request LLM timeout can't see , most likely a tool call (external-CRM push / enrichment / MCP).

Full write-up: docs/design/execution-hang-guard-and-self-recovery.md.

What changed

Layer A , bound the hang (activity watchdog). execution-watchdog.ts: a per-execution idle deadline, not a total wall-clock cap , so a legitimately long, progressing run (20+ min, many subagents) is never killed. Every completed step kicks it (wired to onStepFinish); only a step that never finishes (a hung call) lets it fire. watchdog.race() then rejects the awaited generate() the instant it does , regardless of which await is stuck and without depending on the framework threading the signal into tool execution. The signal also rides the existing LlmCallContext ALS, and prompt-cache-fetch composes it into every LLM request so an in-flight socket aborts promptly (subagents inherit the ALS). EXECUTION_IDLE_TIMEOUT_MS (default 10 min of no progress, 0 disables); the loop is separately bounded by maxSteps.

Layer B , recover (self-healing tail). On a stall the orchestrator step marks the run failed and immediately runs attemptEventRunRecovery (services/agent-action-queue/reaper-recovery.ts): re-drive the event once via dispatchReDerivation(scope:'full') , the same proven path a manual refresh_conversation uses , and if the re-driven run also fails, Slack-DM the user ("Processing your recent meeting failed … if this is happening often or it's mildly annoying you, please report it to the Cedar team"). Retry count is derived from data (failed root execs for the same eventId in a 6h window) , no schema change, no marker threaded through the re-exec stack. The reaper stays the backstop for process-restart orphans (where the in-process watchdog dies with the process) and calls the same recovery.

How to verify

• Repro a stall in staging (stub a tool/LLM fetch to never resolve) → execution_stalled logged, run marked failed, re-drive runs, recap appears; a second stall → user Slack-DM; retry_count derivation caps at one retry.
• Confirm a long progressing run is untouched (watchdog keeps getting kicked).
• 61 unit tests green: execution-watchdog (incl. the 20-min-progressing-run case + the stop-progressing case), reaper-recovery, existing reaper, existing prompt-cache-fetch (34, regression). Server types clean on touched files; deps:check clean.

Follow-ups (not in this PR)

• Phase 3 idempotency: verify/strengthen recap-draft + opportunity-creation dedupe so a re-drive can't double-apply (belt for the rare stall-after-partial-output case).
• Observability (Phase 0): boundary heartbeat + tool_request_timeout log to pinpoint which await hangs.
• A3: shared tool-HTTP timeout wrapper (lets the orphaned generate() unwind instead of lingering; now an optimization, not load-bearing).
• Decompose on-event-agent-execution-workflow.ts (now 1041 lines , extract orchestratorAgentStep).
• Regression monitor for reaped/stalled post-meeting runs not followed by a successful re-drive.

🤖 Generated with Claude Code

Greptile Summary

The PR adds an activity-based execution watchdog and a shared recovery tail that re-drives stalled event runs once before notifying the user.

• Propagates the watchdog abort signal through ag