Libra CodeHub

CedarCopilot/cedar-mail

Branch: staging

fix(playbook-webhook): a Slack Connect channel still fires its registered agent

merged#2595CedarCopilot

CedarCopilot wants to merge 4 commits into staging from fix/slack-connect-workspace-scoping

Live on prod, no production signal yetTimeline and evidence
  1. Opened
    Aug 26, 2026, 2:46 PM
  2. Merged
    Aug 26, 2026, 5:30 PM
  3. Live on prod
    Aug 26, 2026, 5:30 PM
  4. Observed 2 days
    Aug 26, 2026, 5:30 PM
  5. Pipelines steady after this deploy
    Aug 26, 2026, 5:30 PM
  6. Aug 26, 2026, 5:56 PM
  7. Unobserved

    Live on prod, no production signal yet

    Aug 29, 2026, 5:21 AM

Behaviors Libra is checking

AOP context behavior from db/aop-schema.ts keeps working in prod.Inconclusivelow confidence

Strict CloudWatch fallback saw 913 success-shaped log lines matching pipeline, analytics, quota, aop, but no tied operation was present, so Libra is not calling this working.

prod, checked Aug 29, 2026, 4:21 AM
Data model behavior from migrations/playbook_webhook_source_workspace.sql keeps working in prod.Inconclusivelow confidence

Strict CloudWatch fallback found 10 prod failure log lines for [backfillSlackChannelHistory] Backfill failed (Superglue workflow failed: fetch failed), but this domain-wide failure family is not tied to this intent.

prod, checked Aug 29, 2026, 4:21 AM
Slack integration behavior from slack/slack-api.ts keeps working in prod.Inconclusivelow confidence

Strict CloudWatch fallback found 10 prod failure log lines for [backfillSlackChannelHistory] Backfill failed (Superglue workflow failed: fetch failed), but this domain-wide failure family is not tied to this intent.

prod, checked Aug 29, 2026, 5:21 AM

Libra has verdicts on 0 of 3 tracked behaviors on prod; 3 are still being checked. Libra checks hourly for 3 days after each deploy.

The outage

Aspire's inbound-lead agent stopped firing at 12:22 UTC on 2026-08-26, the moment an external member joined #hot-lead-demo-request and made it a Slack Connect channel. Every delivery after that logged cross_workspace_drop and nothing ran:

17:51:22  cross_workspace_drop  channel=C03GC5VHSKS  workspace=T08AHSLJBTL
          webhook=c80e3672…  owner=<email>

Root cause

The guard was approximating. It asked "does the row's owner hold a Slack connection to the workspace this event arrived under" as a stand-in for "is this the channel the row was registered against".

Inside one workspace those coincide. On a shared channel they don't: Slack delivers under one of the connected workspaces, which need not be the one the registration was made from. Verified in production , a message from an Aspire user arrived tagged with Cedar's workspace and was dropped.

Fix

Ask the real question. source_workspace_id records where a row was registered; resolveChannelTeams reads the channel's own host + connected team set, which is the one identity reported identically from every side. A row fires when both it and the incoming delivery appear in that set.

SituationResult
Aspire registered, event arrives under Cedarfires
Colliding channel id, unrelated workspacenothing fires
Channel lookup failsfalls back to the owner check

The tenancy boundary survives because the incoming workspace must also be in the set. Without that, the new proof degrades to "this row is on some shared channel" , which a colliding id also satisfies.

What this does NOT change

Nothing new fires. A registration only exists if someone created one: findSlackChannelSources returns only rows with a matching source_ref, and dispatchPlaybookWebhookMessage then bails at playbook-webhook.ts:182 unless the playbook carries a <trigger type="webhook" id="…"> for that exact id. Cedar's org has zero slack_channel registrations and no such trigger block, so on this channel today the change makes exactly one thing fire: Aspire's. Two orgs both firing requires two deliberate acts by the second org.

Cost and failure modes

  • The Slack call runs only when the cheap owner check has already failed for a row that recorded its workspace, and is cached per channel for 60s. An ordinary single-workspace channel never reaches it (asserted by a test).
  • That TTL is a security bound, not a performance knob. The cached set is what admits a cross-workspace row, so a stale entry is a stale authorization: when a Connect channel is unshared, the removed workspace lingers until the entry expires and its registration still fires. A live lookup corrects immediately, so the TTL is the entire exposure window. Residual: up to 60s after an unshare.
  • A failed lookup is cached for 60s too, so a channel with a permanently stale token doesn't re-ask Slack once per owner per message.
  • A failed lookup returns an empty set → scoping collapses to the pre-existing owner check. A Slack outage can never widen who fires, only narrow it.
  • fetchSlackChannelTeams calls conversations.info directly rather than through the slack-get-channel-info Superglue workflow, whose projection omits the sharing fields and is defined outside this repo.

Also: per-message idempotency

The route dedupes by event_id, which covers Slack retrying one envelope. It does not cover one message reaching Cedar through two installations, which would carry two distinct ids and fire every matching row twice. claimSourceEvent keys on the message ts, stable however many envelopes carry it. Fails open, matching the rate limiter.

Best-effort, not a guarantee. The read and write are separate KV operations, so concurrent envelopes can both claim. KVStore offers no compare-and-set, so an exact claim needs a different store; checkRateLimit accepts the same race for the same reason. What it

Show production surfaces and changed-file mapping

Production surfaces

Libra has not measured any production surfaces for this change yet.

Changed files → surfaces

  • apps/server/src/db/aop-schema.tsno production surface mapped
  • apps/server/src/db/migrations/playbook_webhook_source_workspace.sqlno production surface mapped
  • apps/server/src/services/integrations/slack/slack-api.tsno production surface mapped
  • apps/server/src/services/playbook/__tests__/playbook-webhook-sources.test.tsno production surface mapped
  • apps/server/src/services/playbook/playbook-webhook-registration.tsno production surface mapped
  • apps/server/src/services/playbook/playbook-webhook-sources.tsno production surface mapped