Libra CodeHub

CedarCopilot/cedar-mail

Branch: staging

fix(observability): log compute-metric's entry/exit for the HeyTelo chat-hang

merged#2894CedarCopilot

CedarCopilot wants to merge 1 commit into staging from fix/chris-compute-metric-observability

What Libra verified

  • AOP conversion reports scope and history sizes before downstream processing

    Libra ran this behavior against the change and confirmed the check detects when it breaks.

    Technical evidence
    • computeMetric emits the organization, scoped conversation count, and history count after a non-empty aop_conversion scope and history query complete

    Changed code: deal-metrics.ts.

  • A completed compute-metric call records its result size and duration

    Libra ran this behavior against the change and confirmed the check detects when it breaks.

    Technical evidence
    • a successful compute-metric call records its run, user, metric, returned-row count, and elapsed duration before returning the unchanged rows

    Changed code: conversationReasoningActions.ts.

  • Each compute-metric call records its scope and requested metric at entry

    Libra ran this behavior against the change and confirmed the check detects when it breaks.

    Technical evidence
    • the compute-metric action records run, user, thread, organization, metric, grouping, and an empty-scope flag for an org-wide call
    • the compute-metric action records a false empty-scope flag after resolving a requested conversation within the organization

    Changed code: conversationReasoningActions.ts.

Live on prod, watching, 2 days leftTimeline and evidence
  1. Opened
    Sep 22, 2026, 7:43 PM
  2. Sep 22, 2026, 7:53 PM
  3. Merged
    Sep 22, 2026, 8:16 PM
  4. Live on prod
    Sep 22, 2026, 8:16 PM
  5. Observed 0 hours
    Sep 22, 2026, 8:16 PM
  6. Watching

    Live on prod, watching, 2 days left

    Sep 22, 2026, 8:16 PM
  7. Pipelines steady after this deploy
    Sep 22, 2026, 8:16 PM

Behaviors Libra is checking

AOP conversion metric computation records the size of its conversation scope and fetched change-history set after those data are loaded.Not checked
prod
Compute-metric errors are logged with their duration and error message before the existing failure response is returned.Not checked
prod
Each compute-metric invocation logs its entry with run, user, thread, organization, metric, grouping, and scope information before metric computation begins.Not checked
prod
Successful compute-metric executions log their completion duration and number of returned rows.Not checked
prod
Every computeMetric invocation logs its entry into metric dispatch with the organization, metric, grouping, and conversation-filter cardinality.Not checked
prod

Libra has verdicts on 0 of 5 tracked behaviors on prod; 5 are still being checked. Libra checks hourly for 3 days after each deploy.

Bug this instruments (not yet fixed , this PR is diagnostics only)

Christopher Grittner (Admyral/HeyTelo) hung a chat turn on 2026-09-22 (thread thread-1790101387604-j1lemp6) asking for a pipeline KPI report. The agent fired several conversation-analytics compute-metric calls, then the whole generation went silent forever: no error frame, no done, the 20-minute abort timer never fired, and nothing was ever persisted , not even the user's own original message.

I initially suspected a missing statement_timeout on the raw SQL (the same class of bug fixed once before for getRecentExecutionsTool.ts). Checked postgres_logs and pg_stat_statements for the exact window: no evidence of a slow or hung query , that SQL shape has run exactly once in prod history, at 3.2s. So it isn't the DB, or at least isn't a DB query duration problem.

The real problem: computeMetric() and the compute-metric action in conversationReasoningActions.ts have zero logging anywhere , no entry log, no completion log, on either success or failure. Whatever actually stalled (a CPU-bound in-memory reduction, a stuck downstream round-trip, something else), there was never going to be a trace of it. That's why this incident is unresolved rather than merely unresolved-so-far.

What this PR adds

  • conversationReasoningActions.ts , entry log right before calling computeMetric (run_id, user_id, thread_id, org_id, metric, group_by, org_wide), and an exit log on both the success path (row_count, duration_ms) and the error path (error, duration_ms). This is the tool boundary; org_wide directly answers "was this an unscoped org-wide scan."
  • deal-metrics.ts , an entry log at computeMetric's own boundary (metric, group_by, conversation_ids_count), plus a scope-sizing log specifically inside the aop_conversion branch , the one metric that chains a scope query, a change-history query, an AOP-resolution query, and an O(scope × history-per-conversation) in-memory reduction. It's the likeliest of the 9 metrics to stall on an org with heavy AOP churn, so it gets its own instrumentation rather than sharing the generic entry log's granularity.

No behavior change, logging only. The next occurrence of this symptom should leave an actual trail: either the entry log fires with nothing after it (pins the hang inside this boundary, and which metric), or it doesn't fire at all (points further upstream, at the SDK↔MCP dispatch layer instead).

Out of scope

The statement_timeout/abortSignal wiring I originally proposed is NOT in this PR , it's unvalidated against this specific incident (see above) and shouldn't ship as if it were the fix. This PR is the prerequisite: get visibility first, then fix what the trace actually shows.

Test plan

  • pnpm --filter @zero/server run types , clean
  • pnpm --filter @zero/server exec vitest run src/services/crm/__tests__/deal-metrics.test.ts src/mastra/tools/conversation/__tests__/conversationReasoningActions.test.ts , 31/31 pass
  • Watch for the next [compute-metric] / [computeMetric] log lines in Axiom on a recurrence

🤖 Generated with Claude Code

RetriggerConfidence Score: 4/5

The PR appears safe to merge, with two non-blocking observability refinements recommended so future incident traces are accurately scoped and consistently correlated.

Findings

  1. <img alt="P2" src="https://greptile-
Show production surfaces and changed-file mapping

Production surfaces

Libra has not measured any production surfaces for this change yet.

Changed files → surfaces

  • apps/server/src/mastra/tools/conversation/conversationReasoningActions.tsno production surface mapped
  • apps/server/src/services/crm/deal-metrics.tsno production surface mapped