The incident

On 2026-08-26 every cron on cron-task-queue stopped for eight hours.

NumberOfMessagesReceived    ~68/hr through 07:00 UTC, then exactly 0 for 08:00-16:35
ApproximateAgeOfOldestMessage   844s at 04:00  ->  43,693s (12.1 hrs) at 16:30
Queue depth                 15 -> 667

ECS recorded no restarts and the containers reported normal memory the whole time, so nothing alerted and nothing recycled them. Webhook-driven work on other queues was completely unaffected, which is why it looked fine from outside.

58 of 71 meeting preps that day fired late, worst case 422 minutes. 40+ users across ~25 orgs got prep after their meeting had already started. Two customers reported it before we noticed.

Two independent causes. One fix each.

1. A handler could hold its polling slot forever

startTaskTokenQueuePoller awaited config.handler(body) inline with nothing around it. A handler that never settles wedges that loop permanently , it never calls ReceiveMessage again. Three wedged loops is the entire outage.

The timeout on the producing SqsSendMessage did not help and never could: it bounds only how long Step Functions waits for the callback token. It expired the token (every drained task logged TaskTimedOut: Provided task does not exist anymore) and never touched the work running on the worker.

handlerTimeoutMs is now required on the poller config, for the same reason SurfaceWorkingContext.taskId is required: a ceiling you can forget is a ceiling that turns out to be missing on the one queue that needed it. The cron lanes read it from the message, which the orchestrator stamps from the CDK task list, so that list stays the single source of truth for both the Step Function timeout and the worker ceiling.

Abandoning frees the slot but cannot cancel the work , JS cannot interrupt an awaited promise. So abandonments are counted, and past a threshold the container hands itself back to ECS rather than leaking orphans that still hold their pg connections.

2. Slow crons and urgent crons shared three polling slots

processCrmDealReconciliation is allowed 60 minutes and processExternalCrmSync 20. They shared a queue and a concurrency budget with processScheduledExecutions, whose whole job is to dispatch meeting prep on time.

Reconciliation being slow is normal. Prep being late is a bug. One queue cannot express that difference, so there are now two: cron-task-queue (fast, concurrency 6) and cron-slow-task-queue (slow, concurrency 3).

The lane is declared per task, not inferred from timeoutMinutes. processAopAutomations declares 20 minutes but has to fire within ±5 of a user's chosen cron minute, so it is fast-lane despite a slow-lane-sized ceiling. Inferring would have quietly broken it.

The fast queue's visibility timeout drops from 70 to 30 minutes, above its longest ceiling (20) so the handler ceiling always fires first and a wedged task is abandoned rather than redelivered and re-run.

Verification

• pnpm --filter @zero/server run types clean
• aws/ CDK tsc --noEmit clean
• pnpm deps:check clean (1566 modules, no violations)
• eslint clean on changed files
• New unit tests for the ceiling, 5 passing, covering the outage shape (a handler that never settles) and the invariant that an abandoned handler's late rejection cannot surface as an unhandledRejection

Deploy notes

• Adds an SQS queue, so this needs a CDK deploy, not just a service update.
• The ~667 messages currently queued predate the timeoutMinutes stamp and fall back to the lane default. They are expired-token cron ticks; the work still runs, only the callback is dead.
• This does not retroactively fix the preps that were missed. Overdue rows for meetings that already ended should be cancelled deliberately rather than left to fire late.
• Still outstanding and not in this PR: a CloudWatch alarm on NumberOfMessagesReceived == 0 for that queue. Either that or `ApproximateAgeOfOldestMes