Libra CodeHub

CedarCopilot/cedar-mail

Branch: staging

Diagnose reaped stale executions from their tool calls

closed#2661CedarCopilot

CedarCopilot wants to merge 0 commits into staging from fix/stale-reaper-diagnosis

Not deployedTimeline and evidence
  1. Opened
    Sep 2, 2026, 9:57 AM
  2. Not deployed

    Not deployed

    Pending

Libra has no production signal for this change yet because it has not deployed. Libra checks hourly for 3 days after each deploy.

Every row processStaleExecutionReaper reaped got the same fixed sentence in its output:

still 'executing' 77 min after it began, with no completion recorded , either orphaned by an api-service restart mid-execution, or it exceeded the 30-minute runtime cap.

514 reaps in the last 30 days, ~17/day, all unattributed. That hedge has already cost something: a build item was scoped off its second branch. A pipeline-review agent whose runs take ~3 minutes was reaped once at 77 minutes and read as "exceeded the runtime cap", when its agent_tool_calls rows showed three run-sql calls that all RETURNED inside 14 seconds, then silence. It did fourteen seconds of work. The data to know that was already in the database.

What this does

Classifies each reaped execution at reap time from agent_tool_calls and writes a specific diagnosis instead of the hedge. The classification is also emitted on the reap's existing structured warn log , reap_reason, tool_call_count, last_tool_name, last_tool_returned_at, silent_minutes, worked_seconds , because 500 rows a month are only useful sliced, not read one at a time.

Two diagnoses, not three , the premise had to be corrected

The plan called for a hung tool bucket keyed on "a row with started_at set and result NULL". That predicate cannot fire in this codebase, and shipping it would have written a false diagnosis onto real rows.

logToolCall (tool-calls.ts:144) takes the tool's result as a parameter, so it INSERTs the row only once the tool has already returned. It is the only writer (the sole other INSERT INTO agent_tool_calls in the repo is a test harness), and there is no pre-insert. So:

  • result IS NULL means "the tool returned null/undefined", not "never returned". Measured against production: 0 of 306,760 tool calls in 7 days have a null result, and 0 of the 514 reaped runs do.
  • A tool call that hangs leaves no row at all, which is why the pipeline-review run looked like plain silence.

So the buckets the data actually supports are:

reason30d sharewhat it says
no_tool_activity61 / 514no rows: died before or during its first model call
silent_after_tools453 / 514names the last returned tool, how long the run demonstrably worked, and how long the silence ran

A hang inside a tool, a killed process, and a stuck model call are all indistinguishable from this table alone, so a hedge remains , but only in silent_after_tools, and it now names all three of those causes and leads with what IS known:

Reaped by processStaleExecutionReaper: still 'executing' 77 min after it began. It logged 3 returned tool calls, the last (run-sql) returning 14 s in, then nothing for the remaining 77 min. A tool call is only recorded once it returns, so that silence is one of: the process killed mid-run, the model call after run-sql never returning, or a later tool call that hung without ever writing a row.

Fourteen seconds of work inside a 77-minute run is the fact that kills the runtime-cap reading, and it is now in the row.

Separating those last three causes needs a liveness signal (heartbeat / owning-instance id). Deliberately not built here , no schema change, no new column.

What is preserved

  • The atomic guard is untouched. The UPDATE still re-asserts status = 'executing' AND the staleness predicate; the diagnosis only decides the WORDING of the failure and the log fields. A run that completed or progressed mid-sweep is still skipped, and still logs nothing (covered by a test).
  • No N+1. One batched inArray fetch of the whole sweep's evidence, over three narrow columns , never the arguments/result jsonb. A test asserts exactly two SELECTs for a three-row sweep so this cannot regress into a per-row fetch.
  • STALE_EXECUTING_THRESHOLD_MS and where it is imported from are unchanged.

The one ordering caveat is d

Show production surfaces and changed-file mapping

Production surfaces

Libra has not measured any production surfaces for this change yet.

Changed files → surfaces

  • apps/server/src/cron/__tests__/process-stale-execution-reaper.test.tsno production surface mapped
  • apps/server/src/cron/__tests__/stale-execution-diagnosis.test.tsno production surface mapped
  • apps/server/src/cron/process-stale-execution-reaper.tsno production surface mapped
  • apps/server/src/cron/stale-execution-diagnosis.tsno production surface mapped