Libra CodeHub

CedarCopilot/cedar-mail

Branch: staging

fix(cron): pick a connection-healthy representative for org-level cron triggers

merged#2516CedarCopilot

CedarCopilot wants to merge 0 commits into staging from fix/sherlock-org-cron-representative-connection-20260817-202652

Live on prod, no production signal yetTimeline and evidence
  1. Opened
    Aug 17, 2026, 8:33 PM
  2. Merged
    Aug 18, 2026, 4:48 PM
  3. Live on staging
    Aug 18, 2026, 4:48 PM
  4. Live on prod
    Aug 18, 2026, 4:48 PM
  5. Observed 2 days
    Aug 18, 2026, 4:48 PM
  6. Pipelines steady after this deploy
    Aug 18, 2026, 4:48 PM
  7. Pipelines steady after this deploy
    Aug 18, 2026, 4:48 PM
  8. Unobserved

    Live on prod, no production signal yet

    Aug 21, 2026, 3:11 PM

Behaviors Libra is checking

Organization-level playbook cron reports run under a linked representative with a usable Google direct-OAuth connection, preventing one disconnected user from aborting the organization's scheduled report when another linked user is healthy.Inconclusivelow confidence

OTEL fallback found no prod spans matching getPlaybookCronConfigs / org-level playbook cron trigger since deploy.

prod, checked Aug 21, 2026, 11:49 AM
When multiple linked representatives have usable Google connections, org-level cron selects the one with the latest connection expiry, with a stable AOP-ID tie-breaker so representative selection does not churn between ticks.Inconclusivelow confidence

OTEL fallback found no prod spans matching pickOrgCronRepresentative / getPlaybookCronConfigs since deploy.

prod, checked Aug 21, 2026, 3:10 PM
Each selected organization-level cron representative is logged with its organization AOP, user, and representative AOP identifiers so operators can audit which identity ran the schedule.Inconclusivelow confidence

OTEL fallback found no prod spans matching [getPlaybookCronConfigs] org cron representative selected since deploy.

prod, checked Aug 21, 2026, 3:11 PM
Org-level playbook cron triggers select a linked representative with a usable Google direct-OAuth access and refresh token, preventing one revoked or tokenless user's connection from stopping the organization's scheduled report when another linked user is healthy.Inconclusivelow confidence

OTEL fallback found no prod spans matching getPlaybookCronConfigs / org-level playbook cron since deploy.

staging, checked Aug 21, 2026, 3:51 AM
When multiple linked users have usable Google connections, org-level cron triggers consistently choose the one with the latest connection expiry, with a stable AOP-ID tie-breaker to avoid representative churn between ticks.Inconclusivelow confidence

OTEL fallback found no prod spans matching pickOrgCronRepresentative / org-level playbook cron since deploy.

staging, checked Aug 21, 2026, 3:52 AM
Representative selection for each org-level cron is logged with the organization AOP, representative user ID, and representative AOP ID so operators can audit which credentials a scheduled run uses.Inconclusivelow confidence

OTEL fallback found no prod spans matching [getPlaybookCronConfigs] org cron representative selected since deploy.

staging, checked Aug 21, 2026, 4:52 AM

Libra has verdicts on 0 of 3 tracked behaviors on prod; 3 are still being checked. Libra checks hourly for 3 days after each deploy.

What was wrong

Org-level cron automations (a cron trigger in an org PLAYBOOK.md , e.g. Pirros' weekly "barriers to buy" report) fire once per tick under a single "representative" linked user. getPlaybookCronConfigs picked that representative with SELECT … WHERE org_aop_id = … LIMIT 1 , no ordering, no connection-health check.

If the arbitrarily-picked representative has an expired/revoked Google connection, building the run's automation context (runPlaybookSectionExecution → buildAutomationContext → generatePreamble → getUserContext) throws "No valid connection tokens found for user …", and the org's entire scheduled report dies before any execution row is created. The failure is near-invisible (it escapes the per-task catch; nothing user-facing is logged).

Evidence (prod, Mon 2026-08-17 07:00 PT tick)

  • Pirros' weekly report agent (0a6c401c) has never produced an execution row.
  • CloudWatch worker log at 14:08:33 UTC: [getUserContext] Could not get active connection for user ZOrHUjz… (<email>): No valid connection tokens , stack: getUserContext → generatePreamble → buildAutomationContext → runPlaybookSectionExecution → processAopAutomations.
  • account row for Peter: Google access token expired 2026-07-30, never refreshed since Jul 22 (he owns the account but doesn't actively use Cedar).
  • 2 of Pirros' 7 reps have dead connections (Peter 7/30, Ross 8/7); the representative is picked arbitrarily.
  • Same failure hit two other orgs at the same tick (representatives RspInxFG8…, tx5YJRft…) , this is systemic, not Pirros-specific.

What changed

apps/server/src/services/playbook/manifest-utils.ts:

  • New pickOrgCronRepresentative(db, orgAopId, now) , joins linked AOPs to their Google account, collapses to one candidate per user, and picks the healthiest representative.
  • New pure, unit-tested rankOrgCronCandidates() , orders by: (1) currently-valid access token, (2) freshest token expiry, (3) most-recently-refreshed account, (4) stable aopId tiebreak (so the choice doesn't churn across ticks).
  • The org branch of getPlaybookCronConfigs now uses it and logs the chosen representative ([getPlaybookCronConfigs] org cron representative selected) so the selection is auditable.

Against Pirros' real data the ranking yields zane > kas > keenan > brett > xander > ross > peter , the two dead reps sort last, and a healthy rep is always chosen when one exists.

How to test

  • pnpm --filter @zero/server exec vitest run src/services/playbook/__tests__/org-cron-representative.test.ts (5 cases, incl. the real Pirros expired-vs-valid scenario).
  • After deploy: on the next org-cron tick, look for [getPlaybookCronConfigs] org cron representative selected naming a rep with a live token, and an execution row for the org report agent.

Residual / follow-up

This removes the common failure (a known-dead rep is never chosen when a healthy one exists). It does not add a runtime fallback if the chosen rep's token is revoked between selection and execution , a belt-and-suspenders retry across candidates could be added later if needed.

🤖 Generated with Claude Code

Greptile Summary

The PR replaces arbitrary org-cron representative selection with deterministic ranking based on Google account metadata and adds focused ranking tests and selection logging. However, the ranking reads a different persistence model from the runtime connection-validity check, so disconnected users can still be selected.

  • Adds a pure candidate-ranking helper and database-backed representative selector.
  • Prefers future token expirations, then expiry and account recency, with a stable AOP tie-break.
  • Adds tests covering valid, expired, absent, tied, and immutable candidate inputs.
  • Logs the selected representative for org-level cron runs.

Confidence Score: 4/5

The representative selection sh

Show production surfaces and changed-file mapping

Production surfaces

Libra has not measured any production surfaces for this change yet.

Changed files → surfaces

  • apps/server/src/services/playbook/__tests__/org-cron-representative.test.tsno production surface mapped
  • apps/server/src/services/playbook/manifest-utils.tsno production surface mapped