AWS_DOCUMENTATION.md113.1 KBView on GitHub # Cedar AWS Documentation
This document describes the current AWS architecture for Cedar and the exact surfaces that
matter during the migration.
## Current State
- AWS staging app: `https://d2p1ksd83o7plh.cloudfront.net`
- AWS prod app: `https://d28vdqmgxberim.cloudfront.net`
- Staging custom app domain: `https://mail-staging.cedarcopilot.com`
- Staging custom api domain: `https://api.mail-staging.cedarcopilot.com`
- Prod custom app domain: `https://mail.cedarcopilot.com`
- Prod custom api domain: `https://api.mail.cedarcopilot.com`
AWS is now the runtime shape for both Cedar staging and prod. Cedar must stay AWS-only in both
environments: legacy bridge, object, KV, queue, and durable-object fallbacks are not allowed to
silently handle misses.
As of `2026-04-06`, the AWS migration is complete for both staging and prod:
- staging custom domains resolve to the AWS staging CloudFront distribution
- prod custom domains now resolve to the AWS prod CloudFront distribution
- both `mail.cedarcopilot.com` and `api.mail.cedarcopilot.com` serve from AWS (CloudFront + S3 + ECS)
- Cloudflare Workers builds still trigger on PRs but are vestigial and should be removed
## Current URL Mapping
As of `2026-04-03` / `2026-04-04`, the public Cedar URL map is:
- staging app:
- `https://mail-staging.cedarcopilot.com`
- `https://d2p1ksd83o7plh.cloudfront.net`
- both now serve the AWS staging frontend bundle
- staging api:
- `https://api.mail-staging.cedarcopilot.com`
- same AWS staging CloudFront distribution, with `/api/*` forwarded to the staging ALB / ECS
- prod app:
- raw AWS: `https://d28vdqmgxberim.cloudfront.net`
- canonical host `https://mail.cedarcopilot.com` now resolves to AWS CloudFront
- prod api:
- raw AWS: `https://d28vdqmgxberim.cloudfront.net/api/*`
- canonical host `https://api.mail.cedarcopilot.com` now resolves to AWS CloudFront
Both staging and prod custom-domain validation is now valid AWS validation.
## Live Verification
As of `2026-04-04`, the staging custom domain is verified on AWS from both the public edge path
and the AWS control plane:
- public response headers:
- `curl -I https://mail-staging.cedarcopilot.com/health`
- `via: ...cloudfront`
- `x-cache: Miss from cloudfront`
- `curl -I https://mail-staging.cedarcopilot.com/mail/inbox`
- `server: AmazonS3`
- `via: ...cloudfront`
- AWS runtime verification:
- ECS cluster:
- `aws-staging-api-cluster`
- ECS services:
- `aws-staging-api-service`
- `aws-staging-chat-service`
- `aws-staging-worker-service`
- service state after the April 4, 2026 AWS-primary runtime rollouts:
- desired `2`
- running `2`
- API task definition:
`CedarAwsStagingEnvironmentStackCedarAwsStagingAppStackApiTaskDefinition165EFDED:90`
- Chat task definition:
`CedarAwsStagingEnvironmentStackCedarAwsStagingAppStackChatTaskDefinitionDB94FCCB:47`
- Worker task definition:
`CedarAwsStagingEnvironmentStackCedarAwsStagingAppStackWorkerTaskDefinitionA3E4A222:57`
- API image:
`597088032164.dkr.ecr.us-east-1.amazonaws.com/aws-staging/api-service:32d169b65`
- Worker image:
`597088032164.dkr.ecr.us-east-1.amazonaws.com/aws-staging/worker-service:32d169b65`
- AWS log verification:
- CloudWatch log group:
`/aws/ecs/aws-staging-api/api-service`
- recent events show successful Cedar traffic on AWS staging, including:
- `TRPC call: mastra.chatWarm (...) Success`
- `TRPC call: mail.listThreads (...) Success`
- multiple `TRPC call: mail.get (...) Success`
- `TRPC call: crm.searchConversationsMinimal (...) Success`
- `TRPC call: crm.getConversation (...) Success`
- after the `:90` / `:57` rollout, recent CloudWatch scans do not show fresh:
- `agent.getUserLabels is not a function`
- `Could not find API key process.env.ANTHROPIC_API_KEY`
- AWS secret verification:
- runtime Secrets Manager payloads for staging and prod were checked for legacy bridge keys
- `BRIDGE_SECRET`, `WORKER_URL`, `CLOUDFLARE_ACCOUNT_ID`, and `CLOUDFLARE_API_TOKEN` were
removed from the AWS primary runtime secrets on `2026-04-04`
- this alone was not sufficient:
- the existing ECS task definition revisions still referenced `BRIDGE_SECRET`
- that left staging hard-down with `runningCount=0` and CloudFront `503` responses
- staging was restored only after registering new ECS task definition revisions that removed
the stale `BRIDGE_SECRET` secret injection and rolling the services onto those new revisions
- on `2026-04-03`, staging chat also required a follow-up API task-definition rollout: - `/trpc/mastra.chatStream` was executing in-process on the API service for the live staging
path - the API ECS task definition had to inject the same model-provider secrets as the chat
service (`ANTHROPIC_API_KEY`, `OPENAI_API_KEY`, and the other AI runtime keys) - after rolling API task definition `:88`, the live `Could not find API key
process.env.ANTHROPIC_API_KEY` error disappeared from post-deploy CloudWatch logs
- on `2026-04-04`, AWS request/workflow execution was tightened again:
- the AWS mail runtime now exposes the remaining execution methods used by staging
request paths
- core staging routes now go through `getMailRuntime(...).stub` instead of directly
calling `ZERO_DRIVER` in:
- `agent-executions`
- `mastra.executeScheduledExecution`
- CRM thread hydration / recreate-sync paths
- scheduled execution pipeline fan-out
- CRM integrity-heal refresh
- conversation repair refresh
- this was rolled as API / worker service-only ECS updates:
- API task definition `:90`
- Worker task definition `:57`
- this does not mean every Cloudflare-era type or migration tool is gone from the repo yet
- it means the active staging hot path is more AWS-native and no longer depends on
those direct `ZERO_DRIVER` callsites for correctness
## Production Deployment Strategy
The production rollout strategy is designed so a normal deploy does not require Cedar to go dark
for active users.
### 1. Frontend deploys are in-place, but asset-safe
The frontend stays behind one CloudFront distribution. A deploy does not swap customers onto a new
hostname or ask them to refresh onto a different CDN surface.
The publish order matters:
1. upload the new hashed asset files to S3
2. upload the HTML shell after the new assets already exist
3. invalidate CloudFront so new requests see the new HTML quickly
Why this avoids downtime:
- old `index.html` can keep serving while the new assets are uploading
- users who already loaded the old HTML continue requesting the old hashed JS/CSS files, which
still exist until the sync is complete
- new HTML is not exposed until the new asset set is already present
- hashed assets are immutable, while `index.html` is revalidated aggressively on deploy boundaries
Practical effect:
- existing sessions do not get a broken mid-deploy window where HTML references files that do not
exist yet
- new sessions converge quickly onto the new build after the CloudFront invalidation completes
### 2. Backend deploys are rolling ECS updates, not stop-and-replace
The API, chat, and worker runtimes run as ECS Fargate services behind the ALB. In production the
services are configured to keep the current healthy tasks up while replacement tasks start:
- `minHealthyPercent = 100`
- `maxHealthyPercent = 200`
That means ECS is allowed to bring up new tasks before taking old ones down.
The serving path is also health-gated:
- the ALB only registers healthy tasks in the target group
- the API and chat target groups use `/health` checks
- CloudFront continues routing `/api/*` to the ALB, and the ALB only forwards to healthy ECS
tasks
Why this avoids downtime:
- production traffic keeps hitting the old healthy tasks while the new revision boots
- only after the replacement tasks pass health checks does ECS drain and terminate the old tasks
- users should not see a total service outage from a normal container rollout
### 3. Prod keeps a warm floor instead of scaling from zero
Production does not rely on a single cold task appearing just in time for live traffic. The
current baseline is:
- `api-service`: desired `2`, range `2..8`
- `chat-service`: desired `2`, range `2..6`
- `worker-service`: desired `2`, range `2..6`
Why this matters:
- the rollout still has healthy capacity even while one task set is being replaced
- autoscaling can absorb normal load changes without turning deploys into capacity cliffs
### 4. Cutover is separate from routine deploys
Cedar is explicitly not using a shared-write percentage rollout between Cloudflare and AWS.
Instead, Cedar now treats AWS as the only runtime target in both deploy environments:
- `staging` is the AWS staging environment
- `prod` is the AWS production environment
This matters for reliability because it separates:
- the one-time domain/platform cutover risk
- the day-to-day application deploy risk
Once production is on AWS primary, routine deploys stay inside the AWS serving stack and do not
require another customer-visible platform flip.
### 5. What can still cause a bad deploy
The deployment strategy prevents the common "everything goes down during rollout" failure mode,
but it does not make incompatible releases magically safe.
The main remaining risks are:
- a new container revision that becomes healthy but is logically broken
- an incompatible frontend/backend contract during the overlap window
- a schema or data migration that is not backward compatible with the still-serving old revision
- auth/cookie/domain mistakes during custom-domain cutover
Operational rule:
- production deploys should preserve backward compatibility across one rolling window
- domain cutovers should be treated as a separate change with explicit validation and rollback
posture
Bottom line:
- the AWS production strategy avoids planned downtime by combining immutable frontend asset
publishing, CloudFront invalidation, ALB health checks, ECS rolling replacement, and a warm
multi-task baseline
- if a production deploy fails, the expected failure mode is a bad revision that needs rollback,
not the entire Cedar service going intentionally offline during deploy
### 6. Use the fastest safe deploy path
Not every Cedar AWS change should go through the full CloudFormation stack path.
There are three practical deploy classes:
1. frontend-only publish
2. service-only ECS rollout
3. stack / infrastructure deploy
Use them like this:
- frontend-only publish:
- use when only `apps/mail` assets changed
- publishes the new hashed asset set to S3 and invalidates CloudFront
- this is the fastest customer-facing deploy path for UI-only fixes
- service-only ECS rollout:
- use when backend code changed but the AWS infrastructure shape did not
- publish a new image and roll only the affected ECS service (`api-service`, `chat-service`, or
`worker-service`)
- this is the correct path for code-only server fixes because it avoids another CloudFormation
update while still replacing the live runtime cleanly
- do not assume AI provider keys belong only to `chat-service`:
- the API service can still execute `/trpc/mastra.chatStream` in-process on AWS primary
- if that path exists, the API runtime secret contract must also include the model-provider
keys needed by the chat workflow
- if the runtime secret contract changes, a service-only deploy must still register a fresh ECS
task definition revision; updating Secrets Manager without updating the task definition leaves
ECS pinned to the old secret keys
- stack / infrastructure deploy:
- use only when infra or env wiring changed:
- ALB / CloudFront behaviors
- ECS task definitions or service config
- certificates / aliases / DNS-facing config
- IAM / DynamoDB / S3 / SQS / Step Functions resources
- this is the slow path because CloudFormation has to reconcile the full stack and then wait for
downstream resource stabilization
Why CloudFormation feels slow:
- it is not just "push code"
- it may need to:
- synth and publish CDK assets
- update task definitions and services
- wait for ALB target health
- wait for ECS deployment stabilization
- update CloudFront behaviors / aliases
- wait for invalidations and propagation
Operational rule:
- do not use CloudFormation for a code-only API hotfix
- use the service-only fast deploy path instead
- reserve stack deploys for real infrastructure changes so rollout time stays short and the blast
radius stays narrow
## Postgres Pooling
On `2026-04-04`, AWS staging hit a real Postgres saturation incident. The visible symptom was:
- Google auth callback failures returning `Max client connections reached`
CloudWatch showed this was not isolated to auth. The same database exhaustion was also hitting:
- `mail.listThreads`
- `mail.get`
- `mail.markAsRead`
- `labels.list`
- `crm.searchConversationsMinimal`
- `crm.getConversation`
- `chat.getMessages`
Operational conclusion:
- the old Cedar server DB helper was too loose for AWS concurrency
- too many request paths could create Postgres clients independently
- AWS-primary needs one shared `postgres-js` pool per process, not ad hoc client creation at each
`createDb()` call site
Current AWS guidance:
- Cedar now keeps one shared `postgres-js` pool warm per process instead of tearing it down
between sequential requests
- Cedar now resolves pool settings once per `createDb()` call and reuses that resolved config for
both the actual pool options and the pool cache key
- AWS service env now publishes per-service pool caps and timeout knobs via:
- `CEDAR_DB_POOL_MAX`
- `CEDAR_DB_POOL_IDLE_TIMEOUT_SEC`
- `CEDAR_DB_POOL_CONNECT_TIMEOUT_SEC`
- `CEDAR_DB_POOL_MAX_LIFETIME_SEC`
### Pool sizing (updated 2026-04-07)
Cedar connects through Supabase Supavisor (port 6543, transaction pooling mode). Supavisor
multiplexes app-side connections to a smaller set of real Postgres connections. This means the
app-side pool can safely exceed Postgres `max_connections` (90, 87 usable) — Supavisor handles
the mapping.
Production Postgres has `max_connections: 90` (87 available after superuser reservation).
Supabase Pro plan supports ~200+ pooled client connections through Supavisor.
Pool sizes are differentiated by service workload:
**Production** (targets ~90% of Supavisor Pro ~200 connection limit at baseline):
| Service | Pool per task | Rationale |
|---------|--------------|-----------|
| API | 35 | Highest concurrency: handles tRPC requests, webhooks, agent operations |
| Chat | 25 | Agent chat sessions hold connections during tool calls and context fetching |
| Worker | 30 | 8 concurrent SQS pollers, agent workflows, batch sync operations |
| Scale | Tasks | Total Supavisor connections |
|-------|-------|---------------------------|
| Baseline | 2+2+2 = 6 | 2×35 + 2×25 + 2×30 = **180** (~90% of limit) |
| Max autoscale | 8+6+6 = 20 | 8×35 + 6×25 + 6×30 = **610** |
At max scale, 610 exceeds the ~200 Supavisor limit, but max scale is rare and transient.
Supavisor queues excess connections gracefully. If sustained max-scale is expected, upgrade the
Supabase compute tier or increase the pooler connection limit.
**Staging:**
| Service | Pool per task |
|---------|--------------|
| API | 30 |
| Chat | 20 |
| Worker | 25 |
Baseline (6 tasks): 150 total connections.
**Why these sizes, not 5:**
Agent-heavy workloads hold connections far longer than simple CRUD queries. A single agent
action can hold a DB connection for 1-30 seconds (not 10ms) while orchestrating tool calls,
reading context, and writing results. With only 5 connections per task, a burst of 6 concurrent
agent operations would exhaust the pool and block all other DB access for up to 15 seconds
(the connect timeout).
**Why these sizes, not 80:**
The app-side pool controls how many connections one process can open to Supavisor simultaneously.
80 per task × 20 tasks = 1,600 Supavisor connections, which overwhelms the pooler. The current
sizing (35/25/30) keeps baseline usage well under the Supavisor limit while providing enough
headroom for concurrent agent operations.
The `getSharedDb()` helper (in `apps/server/src/db/index.ts`) was added for long-lived class
fields (like `ZeroDB`) that outlive request scope. Unlike `createDb()`, it returns a drizzle
instance without incrementing `activeHandles`, preventing a connection handle leak where
handles were created but never released.
### When to adjust pool size
Increase if:
- CloudWatch shows connection wait times exceeding 200ms consistently
- Agent operations frequently timeout waiting for DB connections
- Confirm headroom first: `SELECT count(*) FROM pg_stat_activity`
Decrease if:
- `Max client connections reached` errors appear
- Check Supavisor connection limit on Supabase dashboard
- Run: `SELECT count(*) as total, count(*) FILTER (WHERE state = 'idle in transaction') as idle_in_tx FROM pg_stat_activity` to check for connection leaks
Code-level fallback (`DEFAULT_AWS_MAX_CONNECTIONS` in `apps/server/src/db/index.ts`) is set
to 25. This is used when `CEDAR_DB_POOL_MAX` env var is not set — a safety net if CDK fails
to inject the env var.
**Deploy connection math:** During a rolling deploy with `maxHealthyPercent: 200`, each
service doubles its tasks (2 → 4). Total tasks go from 6 to 12. Pool sizes MUST be chosen
so that `12 × avg_pool_per_task < Supavisor max client connections`. With CDK defaults
(35/25/30), deploy total is 360. Supavisor Small limit is 400, Medium is 600.
### Zero-downtime deployment configuration (updated 2026-04-08)
Production ECS services are configured for zero-downtime rolling deploys:
| Setting | Value | Why |
|---------|-------|-----|
| `minHealthyPercent` | 100 | All existing tasks stay running until new tasks are healthy |
| `maxHealthyPercent` | 200 | New tasks start alongside old tasks (2 → 4 per service) |
| `healthCheckGracePeriod` | 30s | Containers start in 10-15s, 30s gives 2-3x margin |
| `deregistrationDelay` | 30s | Old targets drain for 30s after deregistration (was 300s default) |
| `circuitBreaker` | enabled + rollback | Auto-rollback on failed deploy |
| ALB health check | 15s interval, 2 healthy / 3 unhealthy threshold | ~30s to mark new target healthy |
**Graceful shutdown:** All three services (api, chat, worker) handle SIGTERM:
1. Set `shuttingDown = true` — health checks return 503, SQS pollers stop accepting new messages
2. Close the HTTP server (`server.close()`) — stops accepting new connections, drains in-flight requests
3. Flush OpenTelemetry tracing (5s timeout race to prevent hangs)
4. `process.exit(0)` — clean exit
5. Force exit after 25s as safety net (before ECS SIGKILL at 30s)
Worker-specific: SQS polling loops check `shuttingDown` flag and break after current long-poll
completes. In-flight messages already received will complete processing before exit.
**Frontend-backend ordering:** `prod-deploy.yml` sequences frontend deploy AFTER backend.
The `deploy-frontend` job has `needs: [plan, deploy-backend]` to prevent serving new frontend
code against old backend APIs.
**Previous issues (fixed 2026-04-08):**
- No graceful shutdown: services only shut down tracing on SIGTERM, didn't close HTTP server
- 120s `healthCheckGracePeriod` was 10x too long (containers start in 10-15s)
- Default 300s `deregistrationDelay` made deploys take 4-5+ minutes
- Frontend and backend deployed in parallel, causing ~4 min version skew window
- Worker service was missing `healthCheckGracePeriod` entirely
## Customer Performance Model
The CDN story is useful, but it helps different parts of Cedar differently.
What CloudFront improves when a customer is far from `us-east-1`:
- DNS, TLS, and the initial HTTP connection terminate at the nearest CloudFront edge POP
- static frontend assets can be served from the edge cache:
- `index.html`
- JS chunks
- CSS
- fonts
- images
- repeat visits benefit from both browser cache and CloudFront edge reuse, which reduces first-load
waiting on the static shell
What CloudFront does not solve by itself:
- same-origin `/api/*` requests still forward from the edge to the Cedar origin in `us-east-1`
- inbox load, thread load, CRM hydration, auth/session reads, chat warm-up, and provider fetches
are still dominated by:
- ECS/Fargate task warmness
- ALB to ECS request path
- Postgres / Supabase latency
- provider API latency
- frontend route-level data fanout
Practical implication:
- CDN improvements matter most for first paint, shell boot, and large static assets
- customer-perceived mailbox speed still depends heavily on origin-side latency and route-level
loading behavior
The right performance order of operations for Cedar is:
1. Keep every customer-visible app alias on the same CloudFront distribution so the browser always
boots one frontend bundle.
2. Keep deploy-time caching safe:
- `index.html` should be effectively revalidated on deploy boundaries
- hashed assets should stay immutable
3. Reduce route boot fanout on the hot paths:
- `mail.listThreads`
- `labels.list`
- `crm.getOrgMembers`
- `crm.searchConversationsMinimal`
- billing warm-up
- execution stream warm-up
4. Keep a warm ECS floor on the hottest services so user traffic does not pay avoidable cold-start
penalties.
5. Only evaluate heavier edge/routing options such as Global Accelerator or multi-region origin
work if origin latency is still the bottleneck after the app-path cleanup above.
Latest app-path improvement deployed on `2026-04-02`:
- Cedar now warms likely-next thread bodies in the browser cache before open:
- hovering a thread row prefetches `mail.get`
- opening a thread also warms the selected thread immediately
- keyboard focus prefetches the active row plus adjacent rows
- draft hover/click uses the same warm-up path
- the inbox also warms the first few visible threads shortly after list paint so the first open is
less likely to hit a cold `mail.get`
- BIMI avatar lookups now resolve by validated domain in the client instead of firing the stricter
email-validated route for malformed display addresses
- speculative prefetches are marked as silent so a background cache miss does not pollute the
global query error surface
- known benign conversation-miss errors are suppressed in the global query logger when the UI has
already handled the state
Why this matters:
- this directly targets user-perceived thread-open latency
- it complements the AWS stack instead of replacing it:
- CloudFront still helps shell delivery
- ECS warm capacity still matters for the origin path
- route/data fanout still determines how fast the mailbox feels after boot
### Performance overhaul deployed on `2026-04-07`
Major performance pass across all four layers of the stack. The goal: make Cedar feel
like a native app. Every optimization targets a real user-perceived bottleneck.
**Tier 1 — Immediate wins (initial load)**
| Change | Impact | Mechanism |
|--------|--------|-----------|
| tRPC request batching | 5-10 HTTP requests → 1 per tick | Removed `maxItems: 1` on `httpBatchLink` |
| Lazy-load email composer | ~200-400KB removed from initial bundle | `React.lazy()` + `Suspense` for TipTap/ProseMirror |
| Lazy-load Three.js Orb | ~600KB removed from initial bundle | Lazy wrapper for `@react-three/fiber` |
| Font preloading | Eliminates FOIT (flash of invisible text) | `<link rel="preload">` for Geist woff2 in `<head>` |
| Server gzip compression | ~60-70% smaller JSON payloads | Hono `compress()` middleware on all responses |
**Tier 2 — Perceived speed (make it feel instant)**
| Change | Impact | Mechanism |
|--------|--------|-----------|
| Thread prefetch on hover | Data ready before click (~200-400ms head start) | `usePrefetchThread()` on `mouseenter`, guarded by `shouldPrefetch()` for slow connections |
| Instant thread header | Thread shell renders in <16ms on click | Preview data (sender, subject, snippet) from list metadata, `MailBodySkeleton` for body |
| Service worker caching | Tab-switch and revisit feel instant | Stale-while-revalidate for API reads with POST body hashing for tRPC cache keys |
| Offline email compose | Send works without connectivity | IndexedDB outbox + Background Sync API, toast indicator for queued sends |
**Tier 3 — Architecture (server-side)**
| Change | Impact | Mechanism |
|--------|--------|-----------|
| Event-driven SSE push | New mail notification latency: 1.5s → <100ms | In-process `EventEmitter` replaces 1.5s database polling, 30s fallback poll for consistency |
| Cache-Control + ETag | Repeat reads served from browser/CDN cache | Per-route `Cache-Control: private` headers + SHA-256 ETag on `mail.listThreads` |
| Server HTML preprocessing | Zero client-side email processing per render | `preprocessEmailHtml()` adds lazy loading, responsive images, strips tracking pixels during ingest |
| Image proxy | Email images served from CDN, privacy-preserving | `/api/image-proxy` with SSRF protection, 24h CDN cache, `loading="lazy"` on all images |
| Progressive thread loading | First message renders in <200ms | `mail.getLatestMessage` endpoint returns 1 message instantly, full thread streams in behind |
**Tier 4 — Advanced optimizations**
| Change | Impact | Mechanism |
|--------|--------|-----------|
| Predictive prefetch | Inbox pre-warmed after archive/delete/send | `requestIdleCallback` prefetches inbox during action animations |
| Route prefetch on hover | Folder navigation feels instant | `prefetch="intent"` on `<Link>` + `usePrefetchFolderThreads()` with 100ms debounce |
| App shell skeleton | UI frame visible in <100ms on cold load | `HydrateFallback` renders full sidebar + thread list skeleton instead of a spinner |
| Public endpoint caching | CloudFront can coalesce identical requests | `s-maxage` headers on health, providers, OAuth discovery, default categories |
| Delta sync | ~90% fewer full thread list refetches | `mail.getThreadListHash` polled every 30s, full refetch only when hash changes |
How these layers interact:
1. **Cold first load**: app shell skeleton (Tier 4) → font preload + lazy composer/Three.js (Tier 1) →
service worker registers (Tier 2)
2. **Inbox browsing**: delta sync avoids redundant fetches (Tier 4) → hover prefetches thread data
(Tier 2) → click renders preview header instantly (Tier 2) → progressive loading streams
messages (Tier 3)
3. **Return visits**: service worker serves cached API data in <5ms (Tier 2) → gzip reduces
revalidation payload (Tier 1) → ETag skips unchanged responses (Tier 3)
4. **Actions (archive, send)**: predictive prefetch pre-warms inbox during animation (Tier 4) →
offline compose queues sends if disconnected (Tier 2) → event bus pushes notification
to other tabs instantly (Tier 3)
Not implemented (requires CDK/infra changes, not application code):
- Edge-side rendering of app shell at CloudFront (would save ~50-100ms on cold loads but
the `HydrateFallback` skeleton achieves a similar effect client-side)
- Switching CloudFront API cache behavior from `CACHING_DISABLED` to origin-respecting
(the `Cache-Control` and `s-maxage` headers are already in place and will activate
automatically when the behavior is updated)
## Staging Custom-Domain Cutover Lessons
Observed during the staging cutover on `2026-04-03` and follow-up validation on `2026-04-04`:
- the staging ACM certificate for:
- `mail-staging.cedarcopilot.com`
- `api.mail-staging.cedarcopilot.com`
is now `ISSUED`
- the legacy Cloudflare Workers custom domains were detached from:
- `cedar-app-staging`
- `cedar-api-staging`
- Cloudflare DNS was changed to `DNS only` CNAMEs pointing both staging hosts at:
- `d2p1ksd83o7plh.cloudfront.net`
- direct validation now passes for:
- `https://mail-staging.cedarcopilot.com/`
- `https://mail-staging.cedarcopilot.com/login`
- `https://mail-staging.cedarcopilot.com/api/auth/get-session`
- `https://api.mail-staging.cedarcopilot.com/api/auth/get-session`
- direct unauthenticated batched tRPC validation now returns JSON with the expected CORS behavior:
- `POST /api/trpc/user.getUserOrganization?batch=1`
- response: `401 UNAUTHORIZED`
- `access-control-allow-origin: https://mail-staging.cedarcopilot.com`
The important lesson was not just "flip DNS and check login":
- some browser failures initially looked like CORS because the browser saw no
`Access-Control-Allow-Origin`
- the actual failing class was upstream `502` / `503` responses on specific routes, which are then
surfaced by the browser as CORS errors because the gateway error page has no Cedar CORS headers
- the clearest example is the execution stream path:
- browser symptom: repeated SSE reconnects plus CORS-looking console noise
- direct probe symptom: `502 Bad Gateway` from the ALB / CloudFront path on
`POST /api/trpc/agentExecutions.stream`
Migration requirement for prod:
- do not treat a browser CORS message as proof that domain mapping is wrong
- first reproduce the route with direct `curl` against the custom domain and inspect the real HTTP
status
- validate both ordinary request paths and long-lived/streaming paths before calling the cutover
complete
Validation rule:
- every cutover validation run must record the exact hostname used in the browser
- every cutover validation run must include:
- app shell load on the custom app host
- auth/session probe on both the app host and the api host
- a normal tRPC probe with an `Origin` header
- an execution stream / SSE probe
## Retro: Staging Domain Flip
The staging domain flip exposed two different classes of problems, and treating them as one issue
would have sent the rollout in the wrong direction.
### What actually broke
- first failure class:
- browser-side surface leakage
- stale public URL handling could pull the browser back onto the wrong hostname / bundle
- second failure class:
- AWS-primary route parity gaps after the alias flip
- some routes looked like CORS or capacity problems, but the underlying bug was inside the
AWS-primary runtime path itself
The most important staging example was `mail.listThreads`:
- the inbox list path was still willing to fall back from AWS S3 thread snapshots into the legacy
bridge object-read path on snapshot miss
- on staging, that fallback produced bridge `r2.get` timeouts
- the user-visible symptom was:
- `500` on `mail.listThreads`
- browser noise that looked like CORS / gateway instability
What the fix changed:
- staging object reads now stay AWS-primary on snapshot miss instead of silently falling
back to the old bridge object path
- the AWS-primary bridge layer now throws immediately if code tries to touch a legacy bridge /
object / KV / queue / DO path
- the inbox list hot path now builds thread previews from Postgres / CRM data first
- thread snapshots are no longer required for the common list render path
- the staging API hotfix was rolled as an API-only ECS deploy, not another stack mutation
What this incident was not:
- it was not primarily a `max client connections` problem
- it was not solved by just raising DynamoDB / RDS / cache-style capacity knobs
- it was not evidence that the domain alias mapping itself was still wrong after the flip
What still remains after the fix:
- the execution stream path (`agentExecutions.stream`) still needs explicit parity validation:
- if a user does not have access to that feature, the correct behavior should be a clean
feature-gated no-op or an expected auth/access response, not noisy reconnect churn
- Cedar still needs a one-time AWS snapshot/object backfill so older objects are present in the AWS
buckets and no longer depend on migration-era source data
Practical lesson:
- after the domain flip, Cedar staging must be treated as AWS-only
- if an AWS-primary route still depends on a legacy Cloudflare object or bridge path for correctness,
that is a parity bug to remove, not a fallback to rely on
- in primary mode the correct behavior is fail-fast, not silent fallback
## Architecture
```text
Browser
-> AWS app URL
-> CloudFront
-> S3 static assets
-> same-origin /api/*
-> ALB
-> ECS api-service (target tracking autoscaling)
-> Postgres / Supabase
-> DynamoDB-backed KV adapters
-> S3 object storage
-> SQS queues
-> ECS worker-service (target tracking autoscaling)
-> ECS chat-service (target tracking autoscaling)
-> external auth / provider APIs
Dedicated migration dashboard
-> CloudFront + S3 host
-> same-origin /api/trpc/cedarAdmin.getAwsMigrationDashboard
-> AWS api-service dashboard payload builder
```
## Topology View
This is the AWS topology in "what concrete primitive is this?" form.
```text
┌──────────────────────────────────────────────────────────────────────────────────────────────┐
│ 1. Browser │
│ User clicks Cedar app URL, loads HTML/JS/CSS, then issues same-origin API requests │
└──────────────────────────────────────────────────────────────────────────────────────────────┘
│
▼
┌──────────────────────────────────────────────────────────────────────────────────────────────┐
│ 2. CloudFront distribution │
│ Type: CDN + edge router │
│ Role: │
│ - default behavior -> S3 frontend bucket │
│ - /api/* -> ALB (readTimeout: 60s, keepaliveTimeout: 60s) │
└──────────────────────────────────────────────────────────────────────────────────────────────┘
│ │
│ static assets │ dynamic requests
▼ ▼
┌──────────────────────────────────────────────┐ ┌───────────────────────────────────────┐
│ 3A. S3 frontend bucket │ │ 3B. Application Load Balancer │
│ Type: S3 bucket │ │ Type: ALB │
│ Role: index.html + JS/CSS assets │ │ Role: route HTTP traffic to ECS │
└──────────────────────────────────────────────┘ └───────────────────────────────────────┘
│
▼
┌──────────────────────────────────────────────────────────────────────────────────────────────┐
│ 4. ECS cluster │
│ Type: ECS cluster running Fargate tasks │
│ Role: hosts Cedar containers │
└──────────────────────────────────────────────────────────────────────────────────────────────┘
│ │ │
▼ ▼ ▼
┌────────────────────────────────────┐ ┌────────────────────────────────────┐ ┌────────────────────────────────────┐
│ 5A. api-service │ │ 5B. chat-service │ │ 5C. worker-service │
│ Type: ECS Fargate service │ │ Type: ECS Fargate service │ │ Type: ECS Fargate service │
│ HTTP/tRPC/auth/webhooks │ │ chat streams / /chat │ │ SQS pollers / scheduled work │
│ apps/server/src/runtime/ │ │ apps/server/src/container/ │ │ apps/server/src/runtime/ │
│ api-entry.ts │ │ index.ts │ │ worker-entry.ts │
└────────────────────────────────────┘ └────────────────────────────────────┘ └────────────────────────────────────┘
│ │ │
│ │ │
├──────────────┬──────────────────────┴───────────────────┬─────────────────┤
│ │ │ │
▼ ▼ ▼ ▼
┌──────────────────┐ ┌────────────────────────────┐ ┌──────────────────┐ ┌────────────────────────────┐
│ 6A. Postgres │ │ 6B. DynamoDB tables │ │ 6C. S3 data │ │ 6D. External provider APIs │
│ Type: DB │ │ Type: DynamoDB │ │ Type: S3 │ │ Type: external HTTPS │
│ Supabase │ │ Cedar KV/state adapter │ │ blobs/files │ │ Gmail/Calendar/CRM/etc │
└──────────────────┘ └────────────────────────────┘ └──────────────────┘ └────────────────────────────┘
│
▼
┌────────────────────────────┐
│ 6E. SQS queues │
│ Type: SQS │
│ async continuation │
└────────────────────────────┘
│
▼
┌────────────────────────────┐
│ worker-service consumes │
│ queue batches and runs │
│ Cedar background handlers │
└────────────────────────────┘
```
### What each box is, in plain AWS terms
```text
Browser
= end user runtime
CloudFront
= CDN + edge cache + path router
S3 frontend bucket
= static site artifact bucket
ALB
= dynamic HTTP load balancer in front of ECS
= access logs to S3 bucket with 30-day retention
ECS cluster
= scheduling/control plane for Cedar services
api-service / chat-service / worker-service
= ECS Fargate services
= each service runs Cedar containers/tasks
= not Lambda
Container / task
= one running Cedar Node process inside Fargate
SQS
= queue, not a workflow engine
DynamoDB
= operational state store, not Cedar's canonical relational DB
Supabase / Postgres
= canonical relational system of record
Step Functions
= AWS-native orchestration in AWS primary mode
= 8 state machines total:
Legacy dispatch (still used as fallback when per-workflow ARN is not set):
1. workflow-runner: dispatches workflow execution to worker via SQS + waitForTaskToken
worker polls workflow-execution-queue, runs workflow in-process, calls SendTaskSuccess/Failure
2. cron-orchestrator: replaces the monolithic hourly cron handler
runs 14 cron tasks in parallel via SQS waitForTaskToken pattern
triggered by EventBridge hourly rule (:00)
3. aop-automations: single-task state machine for processAopAutomations
triggered by EventBridge every 15 minutes (:00/:15/:30/:45)
separate from cron-orchestrator so user schedules at any quarter-hour fire correctly
Per-workflow multi-step state machines (each step.do() is a separate SF state):
3. calendar-sync: setup -> get-sync-token -> fetch-events -> process-events -> save-sync-token
4. cron-sync: setup -> checkAndSync
5. sync-threads: setup -> process-page
6. sync-threads-coordinator: setup -> process-page -> pagination loop -> update-history-id
7. conversation-refresh: conditional AOP branch (root + batch Map) -> conditional search branch
8. conversation-sync: setup -> sent -> inbox -> label Map -> search Map -> meeting Map ->
slack -> crm Map -> cleanup -> conditional finalize Map (two-phase)
Routing: env.ts reads AWS_SFN_WF_* ARNs; if set, dispatch goes to the per-workflow SM
via StartExecution. If unset, falls back to in-process execution via aws-primary-runtime.ts.
```
## Full End-to-End ASCII Diagram
This is the full end-to-end picture starting from an actual user click in the browser and
showing where the request lands in AWS.
### Legend: what the boxes actually are
```text
CloudFront
= AWS CDN + edge router for the public app URL
= serves static frontend assets from S3
= forwards dynamic paths like /api/* to the ALB
= ALB origin timeout: readTimeout 60s, keepaliveTimeout 60s (bumped from 30s default)
S3
= AWS object storage
= used for:
- frontend static assets
- thread blobs
- meeting artifacts / transcripts
- CRM/slack content blobs
ALB
= AWS Application Load Balancer
= the public HTTP entrypoint for dynamic Cedar server traffic
= chooses which ECS service gets the request
ECS api-service / chat-service / worker-service
= ECS Fargate services
= these are long-running Cedar server containers
= each service runs one Cedar runtime role:
- api-service: normal HTTP/tRPC/auth/webhook routes
- chat-service: chat / long-lived stream execution
- worker-service: queue consumers + scheduled work
Container
= one running Cedar server process inside an ECS task
= not a Lambda
= not a Step Functions state machine
Supabase / Postgres
= canonical relational database
DynamoDB-backed KV adapters
= AWS replacement for the Cloudflare KV-style operational state Cedar used before
= current runtime uses adapter code that makes Cedar read/write DynamoDB-backed state
SQS
= AWS queue primitive for async continuation and worker fanout
Cedar workflow / WorkflowRunner / env.<WORKFLOW>
= workflow entrypoint boundary inside Cedar
= in AWS primary mode, the sync workflows below route through Step Functions
```
### Step Functions-backed workflow orchestration
For the current AWS shape described in this file:
- the browser hot path is `CloudFront -> ALB -> ECS/Fargate service`
- static UI still loads from `CloudFront -> S3`
- async continuation is primarily `SQS -> worker-service`
- all 6 Cedar workflow classes (ConversationSync, ConversationRefresh, CalendarSync,
SyncThreadsCoordinator, SyncThreads, CronSync) now have **dedicated per-workflow
Step Functions state machines** where each `step.do()` is a separate SF state
#### Per-workflow multi-step state machines (current model)
Each workflow has its own SF defined in `aws/lib/stacks/workflow-state-machines.ts`.
Each step sends an SQS message to `workflow-step-queue` with a `taskToken` and waits
for the worker to call `SendTaskSuccess`/`SendTaskFailure`.
```text
api-service calls env.WORKFLOW.create({ params })
-> env.ts reads AWS_SFN_WF_<WORKFLOW>_ARN
-> StartExecution(per-workflow SM, { workflowName, workflowRunId, params })
-> SF state for step 1 sends SQS to workflow-step-queue with taskToken
-> worker-service polls queue, executes step.do() logic
-> SendTaskSuccess -> SF advances to next state
-> SF state for step 2 sends SQS to workflow-step-queue ...
-> ... until final step completes the execution
```
The 6 per-workflow state machines and their step chains:
| Workflow | SF Name | Steps |
|----------|---------|-------|
| CalendarSync | `{prefix}-wf-calendar-sync` | setup -> get-sync-token -> fetch-events -> process-events -> save-sync-token |
| CronSync | `{prefix}-wf-cron-sync` | setup -> checkAndSync |
| SyncThreads | `{prefix}-wf-sync-threads` | setup -> process-page |
| SyncThreadsCoordinator | `{prefix}-wf-sync-threads-coordinator` | setup -> process-page -> (pagination loop) -> update-history-id |
| ConversationRefresh | `{prefix}-wf-conversation-refresh` | conditional AOP branch (root + batch Map@5) -> conditional search branch (root + batch Map@5) |
| ConversationSync | `{prefix}-wf-conversation-sync` | setup -> sent? -> inbox? -> label Map@3 -> search Map@3 -> meeting Map@3 -> slack? -> crm Map@3 -> cleanup -> conditional finalize Map@10 |
Key properties:
- Map states use bounded `maxConcurrency` (3-10 depending on the step)
- SyncThreadsCoordinator uses a Choice + Pass loop for pagination
- ConversationSync uses conditional Choice states for each sync phase
- Each step has an individual timeout (2-30 min) set in the SF definition
- All step results are stored at `$.steps.<resultKey>` for downstream states
#### Routing logic (`apps/server/src/env.ts`)
The `buildRuntimeEnv()` function reads per-workflow ARN env vars:
```text
workflowArnMap:
SYNC_THREADS_WORKFLOW -> AWS_SFN_WF_SYNC_THREADS_ARN
SYNC_THREADS_COORDINATOR_WORKFLOW -> AWS_SFN_WF_SYNC_THREADS_COORDINATOR_ARN
CRON_SYNC_WORKFLOW -> AWS_SFN_WF_CRON_SYNC_ARN
CONVERSATION_SYNC_WORKFLOW -> AWS_SFN_WF_CONVERSATION_SYNC_ARN
CONVERSATION_REFRESH_WORKFLOW -> AWS_SFN_WF_CONVERSATION_REFRESH_ARN
CALENDAR_SYNC_WORKFLOW -> AWS_SFN_WF_CALENDAR_SYNC_ARN
```
- If the ARN is set: `workflow.create()` calls `StartExecution` on the per-workflow SM
- If the ARN is unset: falls back to in-process execution via `startAwsPrimaryWorkflow()`
- To revert a single workflow to in-process: remove its `AWS_SFN_WF_*_ARN` env var from CDK
- All 6 ARNs are injected into api, chat, and worker containers by CDK (`app-stack.ts`)
#### Legacy workflow-runner state machine (still exists)
The original workflow-runner SM still exists for backward compatibility:
```text
api-service calls StartExecution(workflow-runner SM, { workflowName, params })
-> SF sends SQS message to workflow-execution-queue with taskToken
-> worker-service polls queue, runs workflow in-process
-> calls SendTaskSuccess/SendTaskFailure to complete the SF execution
```
- Used when `AWS_SFN_WORKFLOW_RUNNER_STATE_MACHINE_ARN` is set but per-workflow ARNs are not
- In this model, all workflow steps run as plain function calls within the worker process
- The per-workflow model supersedes this for observability (each step is a visible SF state)
### Step Functions-backed cron orchestration
The hourly cron handler has been decomposed from a monolithic sequential function into a
Step Functions state machine with 13 parallel branches (processAopAutomations runs
separately every 15 minutes — see below):
```text
EventBridge hourly rule (schedule: cron(0 * * * ? *))
-> Step Function: {prefix}-cron-orchestrator
-> Parallel branches (one per cron task):
├── runEndOfDayHealthCheck (conditional: skips unless midnight UTC)
├── processPeriodicEmailSync
├── processScheduledEmails
├── processReminders
├── processScheduledExecutions
├── processScheduledTasks
├── processExpiredSubscriptions
├── processExternalCrmSync
├── rollForwardStaleCalendarEvents
├── processBufferedSlackWebhookEvents
├── processDailyFollowupDigests
└── processWeeklyCrmActivityDigests
EventBridge every-15-min rule (schedule: cron(0/15 * * * ? *))
-> Step Function: {prefix}-aop-automations
-> processAopAutomations
Fires at :00, :15, :30, :45 so user cron schedules at any quarter-hour match.
getCronAgentsDueAt uses ±5 min tolerance — aligns cleanly with 15-min cadence.
-> Each branch:
1. SF sends SQS message to cron-task-queue with taskToken (waitForTaskToken)
2. worker-service polls cron-task-queue
3. worker executes the named task via cronTaskRegistry
4. worker calls SendTaskSuccess or SendTaskFailure
5. SF branch completes or catches the error
-> All branches run in parallel with individual timeouts (5-20 min)
-> Each branch has a Catch handler so one failure doesn't abort others
```
This replaces the old approach of:
- internal `setInterval` timer in worker-entry.ts
- DynamoDB conditional-put lock for single-instance election
- sequential execution of 15 tasks (failure in task N blocked tasks N+1..15)
Benefits:
- **Isolated failures**: task #3 failing doesn't block tasks #4-15
- **Per-task retry and timeout**: each task has its own timeout in the SF definition (5-20 min)
- **Observability**: execution history visible in the Step Functions console, per-task tracing spans (`cron.task.*`) in Datadog/Axiom
- **No distributed lock**: EventBridge + SF handles single-execution semantics
- **Parallelism**: all 14 tasks run concurrently instead of sequentially
Hardening details:
- `SendTaskSuccess` and `SendTaskFailure` both use `.catch()` to prevent SF branch hangs on transient errors
- `taskToken` presence is validated before dispatch to avoid unhandled exceptions on malformed messages
- `pollersStarted` guard prevents duplicate cron-task-queue consumers if startup fires twice
- Cron-task-queue poller is skipped in development mode (`NODE_ENV === 'development'`)
- `POST /scheduled` legacy endpoint requires `{ "force": true }` to prevent accidental double-execution
- SFNClient is cached as a singleton for TCP connection reuse
- IAM `states:SendTask*` uses `resources: ['*']` (required by AWS, task tokens are opaque)
### Deeper runtime map: what Cedar is actually using in AWS
```text
Public web entry
Browser
-> CloudFront
-> S3 frontend bucket for static files
-> ALB for /api/* and chat paths
Request-serving compute
ALB
-> ECS Fargate api-service
-> ECS Fargate chat-service for /chat* and /trpc/mastra.chatStream*
Async compute
SQS queues
-> ECS Fargate worker-service
Step Functions
-> workflow-runner SM for sync workflows (SF -> SQS -> worker in-process)
-> cron-orchestrator for hourly scheduled tasks (EventBridge -> SF -> SQS -> worker)
Canonical data
api-service / chat-service / worker-service
-> Supabase / Postgres
Operational state adapters
api-service / worker-service
-> DynamoDB tables
- scheduledEmails
- scheduledAgentActions
- pendingEmails
- initialSyncState
- calendarSyncTokens
- genericKv
Blob / artifact storage
api-service / worker-service / chat-service
-> S3 buckets
- threads
- meetings
- crm-content
- slack
Async queues
api-service / worker-service
-> SQS queues
- thread-queue.fifo
- channel-sync-queue.fifo (Slack/LinkedIn/WhatsApp buffer drain wake-up)
- subscribe-queue
- send-email-queue
- send-remind-queue
- scheduled-agent-action-queue
- cron-task-queue (Step Function task token callbacks for hourly cron)
- workflow-execution-queue (Step Function task token callbacks for legacy workflow-runner)
- workflow-step-queue (Step Function task token callbacks for per-workflow multi-step SMs)
External side effects
api-service / chat-service / worker-service
-> provider APIs
- Google
- Microsoft
- Slack
- CRM vendors
- model providers
```
### A. User click on the AWS app
```text
┌──────────────────────────────────────────────────────────────────────────────────────────────┐
│ User in browser │
│ 1. Clicks https://d28vdqmgxberim.cloudfront.net/mail/inbox │
└──────────────────────────────────────────────────────────────────────────────────────────────┘
│
▼
┌──────────────────────────────────────────────────────────────────────────────────────────────┐
│ CloudFront distribution │
│ 2. Serves static app assets from S3 │
│ - index.html │
│ - JS chunks │
│ - CSS │
└──────────────────────────────────────────────────────────────────────────────────────────────┘
│
▼
┌──────────────────────────────────────────────────────────────────────────────────────────────┐
│ Browser boots Cedar frontend │
│ 3. React Router mounts │
│ 4. Frontend decides it needs data │
│ 5. Frontend issues same-origin API calls such as: │
│ - /api/trpc/mail.listThreads │
│ - /api/trpc/mail.get │
│ - /api/trpc/crm.getConversation │
│ - /api/trpc/calendar.listCalendarEvents │
└──────────────────────────────────────────────────────────────────────────────────────────────┘
│
▼
┌──────────────────────────────────────────────────────────────────────────────────────────────┐
│ CloudFront dynamic routing │
│ 6. Path matches /api/* │
│ 7. Forwards to ALB │
└──────────────────────────────────────────────────────────────────────────────────────────────┘
│
▼
┌──────────────────────────────────────────────────────────────────────────────────────────────┐
│ Application Load Balancer │
│ 8. Default route -> api-service target group │
│ 9. Special chat rule -> chat-service target group for: │
│ - /chat* │
│ - /trpc/mastra.chatStream* │
└──────────────────────────────────────────────────────────────────────────────────────────────┘
│
┌─────────────────┴─────────────────┐
│ │
▼ ▼
┌─────────────────────────────────────┐ ┌─────────────────────────────────────────────────┐
│ ECS api-service │ │ ECS chat-service │
│ Handles normal API + tRPC traffic │ │ Handles heavy chat / stream execution │
└─────────────────────────────────────┘ └─────────────────────────────────────────────────┘
│ │
▼ ▼
┌──────────────────────────────────────────────────────────────────────────────────────────────┐
│ Cedar server runtime │
│ 10. apps/server/src/runtime/api-entry.ts or shared chat handler │
│ 11. hands request into apps/server/src/main.ts │
│ 12. Hono route or tRPC router resolves the operation │
└──────────────────────────────────────────────────────────────────────────────────────────────┘
│
▼
┌──────────────────────────────────────────────────────────────────────────────────────────────┐
│ Service/runtime layer │
│ 13. Route handler calls: │
│ - getUserStore() │
│ - getMailRuntime() │
│ - AWS-primary adapters │
│ - workflow / queue publish / object-store helpers │
└──────────────────────────────────────────────────────────────────────────────────────────────┘
│
┌──────────────────────┼──────────────────────┬───────────────────────────┐
│ │ │ │
▼ ▼ ▼ ▼
┌──────────────────────────────┐ ┌──────────────────────┐ ┌──────────────────────┐ ┌──────────────────────┐
│ Supabase / Postgres │ │ DynamoDB-backed KV │ │ S3 object storage │ │ SQS queues │
│ Canonical relational state │ │ operational state │ │ blobs / artifacts │ │ async continuation │
└──────────────────────────────┘ └──────────────────────┘ └──────────────────────┘ └──────────────────────┘
│
▼
┌────────────────────────────────────────┐
│ ECS worker-service │
│ polls SQS and runs handlers.queue(...) │
└────────────────────────────────────────┘
```
### B. Same click, but expanded with Cedar request decisions
```text
User click
-> AWS CloudFront app URL
-> S3-served frontend shell
-> frontend route code runs
-> same-origin fetch /api/trpc/<procedure>
-> CloudFront /api/* behavior
-> ALB
-> api-service
-> api-entry.ts
-> main.ts
-> Hono middleware
-> auth/session resolution
-> tRPC context creation
-> route execution
-> service layer
-> direct read/write to Postgres when relational data is needed
-> direct S3 access when blobs/transcripts/threads are needed
-> direct DynamoDB-backed KV access when operational state is needed
-> SQS publish when work should continue asynchronously
-> provider API call when external mail/calendar/CRM data is needed
```
What each line in Part B means in plain English:
- `AWS CloudFront app URL`
This is the public customer-facing AWS hostname. The browser talks to CloudFront first, not
directly to ECS.
- `S3-served frontend shell`
This is the static React app. At this point Cedar has only served HTML/JS/CSS; no business logic
request has run yet.
- `same-origin fetch /api/trpc/<procedure>`
After the frontend boots, it starts making Cedar API calls back to the same host. That is what
actually loads mailbox, CRM, calendar, settings, and so on.
- `CloudFront /api/* behavior`
CloudFront stops acting like a static file host and instead proxies the request to the dynamic
origin configured for Cedar.
- `ALB`
This is the AWS load balancer sitting in front of the ECS services. It is the first dynamic AWS
compute-facing hop.
- `api-service`
This is an ECS/Fargate service running Cedar's normal server process. Think "the main backend
containers for request/response work."
- `api-entry.ts`
This is the ECS-friendly Node HTTP wrapper. It normalizes the request, handles `/health`, and
passes the request into the shared Cedar server code.
- `main.ts`
This is the real Cedar server entry. Hono routing, auth, tRPC mounting, webhook ingress, queue
handlers, and scheduled handlers all converge here.
- `Hono middleware -> auth/session resolution -> tRPC context creation`
This is the request framing layer. Cedar determines who the user is, builds request context, and
then resolves the specific route or tRPC procedure.
- `service layer`
This is the domain logic layer. This is where Cedar actually decides what data to read or write.
- `direct read/write to Postgres`
Normal relational application data path.
- `direct S3 access`
Blob/object path. Large artifacts do not live in Postgres rows.
- `direct DynamoDB-backed KV access`
Operational-state path. This is not user-facing relational data; it is Cedar's AWS-native
replacement for the Cloudflare KV-style buckets the app previously used.
- `SQS publish`
The current request decided not to finish everything synchronously. It emitted a queue message so
`worker-service` can continue in the background.
- `provider API call`
Cedar is reaching out to Gmail, Google Calendar, a CRM provider, Slack, or another external
system from inside the request.
### C. Chat click path
```text
User opens chat UI
-> frontend starts chat request
-> ALB chat rule matches /trpc/mastra.chatStream
-> chat-service
-> shared chat handler
-> executeChatAgentStreaming()
-> chat agent
-> may read Postgres
-> may query Turbopuffer
-> may call other Cedar tools
```
### D. Backend-triggered path, no user click required
```text
External system
-> webhook or provider callback
-> Cedar AWS HTTP surface
-> CloudFront or direct API hostname
-> ALB
-> api-service
-> main.ts webhook route
-> verification / parsing
-> service logic
-> Postgres
-> S3
-> SQS
-> direct execution helper
```
### E. Queue / scheduled background path
```text
Queue-driven async work:
api-service or scheduled promoter
-> SQS
-> worker-service long-poll loop
-> handlers.queue(...)
-> Cedar background logic
-> Postgres
-> DynamoDB-backed KV
-> S3
-> provider APIs
Hourly cron orchestration (Step Function-backed):
EventBridge (hourly rule)
-> Step Function: cron-orchestrator
-> 14 parallel branches
-> SQS cron-task-queue (with task token)
-> worker-service cron task queue poller
-> cronTaskRegistry[taskName](env, scheduledTime)
-> SendTaskSuccess / SendTaskFailure
-> Step Function branch completes
Long-horizon scheduled state:
-> DynamoDB-backed KV
-> cron task (processScheduledEmails, processScheduledExecutions, etc.)
-> promote ready jobs into SQS
-> worker-service consumes them
```
### F. Cloudflare migration-only shadow path
```text
User on legacy Cloudflare surface
-> Cloudflare request executes normally
-> allowlisted query gets cloned
-> shadow-forwarding middleware
-> forwards cloned query to AWS ALB
-> api-service
-> api-entry.ts logs shadow metadata
-> Cedar request runs on AWS
```
## Runtime Components
### App hosting
- Static frontend assets are published to S3 and served by CloudFront.
- The AWS app surfaces use same-origin API requests so the frontend does not depend on the
Cloudflare API hostname when loaded from the AWS URLs.
### API runtime
- `api-service` runs on ECS/Fargate behind an ALB.
- Health endpoint:
- `/health`
- Auth/session paths, mail reads, CRM routes, and dashboard payload generation all execute
through this runtime.
API request path detail:
```text
tRPC / route handler
-> apps/server/src/lib/server-utils.ts
-> getUserStore() / getMailRuntime()
-> AWS-primary adapters
-> direct Postgres / Supabase access
-> direct mail / calendar / CRM driver calls
-> SQS / S3 / DynamoDB-backed infrastructure where needed
```
Why this design was chosen:
- the Cloudflare architecture used Durable Objects such as `ZeroDB` and `ZeroAgent` as request
routers and actor boundaries
- on AWS, Postgres remained the canonical relational store, so keeping the DO hop would only add
another runtime boundary and another failure mode
- the AWS-primary adapters keep the service-layer contract stable while removing the Cloudflare
bridge from the hot path
- shared AWS-facing code now uses neutral runtime names so the app/API boundary is not forced to
speak in Cloudflare-era terminology
### Shared runtime interfaces
```text
AWS app/API boundary
-> getUserStore()
-> getMailRuntime()
-> getMailRuntimeFromShard() only where shard-aware routing still exists
Legacy workflow/shard internals
-> getZeroDB()
-> getZeroAgent()
-> getZeroAgentFromShard()
-> AWS runtime entrypoints for the remaining legacy method names
```
Why this design was chosen:
- the AWS request-serving surface should read like AWS-native Cedar business logic before the
domain flip
- the orchestration model is now explicitly AWS-first, so the workflow boundary maps cleanly to
Step Functions instead of an implicit in-process shim
- older method names are still accepted at the boundary where they are needed, but the operational
model is now the AWS implementation, not the migration vocabulary
### Background runtimes
- `worker-service` owns queue-driven and asynchronous work.
- `chat-service` owns chat/container-specific server execution paths.
### Deployment automation
```text
pull request
-> GitHub Actions runs AWS staging/prod readiness checks
merge to staging
-> GitHub Actions deploys AWS staging
-> change-aware stack / service / frontend publish
-> staging validation and smoke checks
merge to main
-> GitHub Actions deploys raw AWS prod
-> final separate prod domain flip
```
Why this design was chosen:
- Cedar already uses branch-based promotion, so the AWS automation should match that mental model
- pull requests should show AWS-native build/readiness checks instead of legacy Cloudflare build
statuses
- `staging` is the automatic AWS staging deploy point
- `main` is the automatic raw AWS prod deploy point
- the domain flip remains separate so production traffic can be moved only after the raw AWS stack
is already healthy
### Staging CI/CD
The staging deploy workflow now lives in:
- `.github/workflows/staging-deploy.yml`
It is push-driven from the `staging` branch and uses GitHub OIDC to assume the AWS staging deploy
role. The workflow intentionally does not treat every change as the same class of deploy.
### PR readiness checks
Pull requests now surface AWS-native readiness checks in:
- `.github/workflows/aws-builds.yml`
The workflow publishes four PR checks:
- `AWS Builds / cedar-app-staging`
- `AWS Builds / cedar-api-staging`
- `AWS Builds / cedar-app`
- `AWS Builds / cedar-api`
Those checks are build/readiness only:
- mail frontend bundles are built against staging/prod public URLs
- server runtime entrypoints are built
- the AWS CDK app is typechecked and synthesized for staging/prod
- no AWS staging or prod deployment happens from the PR workflow
Current staging deploy behavior:
- `apps/mail/**` changes:
- build the mail frontend
- upload the bundle to the AWS staging frontend S3 bucket
- invalidate the staging CloudFront distribution
- `apps/server/**` or shared package changes:
- publish new `api-service`, `chat-service`, and `worker-service` images
- deploy `CedarAwsStagingEnvironmentStack` with the new image URIs
- `aws/lib/**`, `aws/bin/**`, or AWS CDK config changes (`cdk.json`, `package.json`, `tsconfig.json`):
- run the staging stack deploy even if no new runtime image is needed
- `aws/scripts/**` changes do NOT trigger a stack deploy (scripts are deployment tooling
picked up at checkout, not infrastructure definitions)
- docs-only changes:
- no AWS staging deploy runs
- ECS service deployments now target fully rolling replacement on both staging and prod:
- `minHealthyPercent = 100`
- `maxHealthyPercent = 200`
- with the current `desiredCount = 2` floor, ECS keeps two healthy tasks up while bringing
replacements online, instead of being allowed to drain staging to zero during deploys
- ECS deployment circuit breaker is enabled with auto-rollback on all services:
- if a new task definition repeatedly fails to start, ECS stops retrying and rolls back
to the last working revision automatically
- this prevents CloudFormation from hanging indefinitely on bad deploys
- Deploy pipeline runs backend and frontend as parallel GitHub Actions jobs:
- `deploy-backend`: publishes images (parallel) + CDK or ECS rollout
- `deploy-frontend`: builds frontend + S3 sync + CloudFront invalidation
- `smoke-test`: runs after both complete
- Docker image publishes run in parallel (api, chat, worker simultaneously)
- ECS rollouts run in parallel with a 600s timeout and circuit breaker detection
- CDK preflight validation (typecheck + synth) runs before the actual deploy
- `node_modules` and AWS CDK dependencies are cached across deploys (keyed on lockfile hash)
- Frontend S3 upload uses two targeted `s3 sync` calls instead of per-file uploads
The reusable deploy entrypoints are now:
- `aws/scripts/deploy-surface.sh` — orchestrates image publish + CDK or ECS rollout
- `aws/scripts/rollout-service-images.sh` — direct ECS task def update + parallel rollout
- `aws/scripts/publish-frontend-assets.sh` — frontend build + S3 sync + CloudFront invalidation
- `aws/scripts/publish-service-image.sh` — single service Docker build + ECR push
- `aws/scripts/plan-surface-scope.sh` — determines what changed and what needs deploying
- `aws/scripts/last-deployed-sha.sh` — the commit each surface was last deployed from, which is what the scope planner diffs against
- `aws/scripts/smoke-test.sh` — post-deploy health verification
Why this split exists:
- frontend-only changes should not wait on unnecessary ECS or CloudFormation work
- backend code changes use the fast ECS rollout path when no infra changed
- infra changes go through CDK so the deployed AWS shape stays declarative
- the scope detection script automatically picks the right deploy class
Required GitHub `staging` environment variables:
- `AWS_STAGING_DEPLOY_ROLE_ARN`
- `CEDAR_AWS_STAGING_APP_DOMAIN`
- `CEDAR_AWS_STAGING_API_DOMAIN`
- `CEDAR_AWS_STAGING_CERTIFICATE_ARN`
- optional: `CEDAR_AWS_VPC_ID`
### Operational promotion rule
The current operator flow matches that branch model:
1. merge into `staging`
2. let `staging-deploy.yml` deploy AWS staging automatically
3. require staging smoke checks to pass on the AWS staging aliases
4. merge the same validated state to `main` so `prod-deploy.yml` deploys raw AWS prod automatically
5. keep the canonical domain flip as a separate operational change
This matters because prod is not supposed to be the first place a staging-branch merge is proven on
AWS. The `2026-04-03` rollout followed this exact pattern.
### Capacity and scaling
```text
staging
api-service 0.5 vCPU / 1 GB desired 2 autoscale 2..6
chat-service 1 vCPU / 2 GB desired 2 autoscale 2..4
worker-service 1 vCPU / 4 GB desired 2 autoscale 2..6
prod
api-service 0.5 vCPU / 1 GB desired 2 autoscale 2..8
chat-service 1 vCPU / 2 GB desired 2 autoscale 2..6
worker-service 1 vCPU / 4 GB desired 2 autoscale 2..6
```
Note: worker-service was bumped from 0.5 vCPU / 1 GB to 1 vCPU / 4 GB because
the esbuild bundle is ~28 MB and pulls in the full application tree via `main.ts`.
This is a known issue — the proper fix is splitting `main.ts` so the worker only
imports queue/scheduled handlers instead of the entire route tree.
- All three ECS services now use target-tracking autoscaling on both CPU and memory.
- Staging autoscaling targets:
- CPU target: `50%`
- Memory target: `60%`
- Prod autoscaling targets:
- CPU target: `60%`
- Memory target: `70%`
- Cooldowns (both environments):
- scale out `60s`
- scale in `180s`
Why this design was chosen:
- Cedar was running as a fixed-size single-task stack before this change, which meant no real
elasticity and poor failure tolerance once load increased.
- API and chat now keep a two-task floor in prod so a single task restart is not the whole
service.
- Worker also keeps a two-task floor in prod so queue throughput and queue recovery are not
pinned to one process.
- CPU and memory target tracking are simple, native ECS policies that work immediately without
introducing more AWS primitives or a separate scaling controller.
- Customer-facing latency is still dominated by ECS/Fargate task warmness and app-level data
loading, especially for inbox open, thread open, CRM tab, and chat stream routes.
- The measured timings below are warm-path browser timings, not isolated container boot timings.
- Cold starts show up when a task is first needed after scale-out or restart, so the prod task
floor exists to keep the hottest routes already warm.
### Storage and operational state
- Canonical relational state remains in Postgres / Supabase.
- Object storage routes to S3.
- Queue fanout routes to SQS.
- KV-like operational state routes through AWS-backed adapters.
Namespace and state ownership map:
```text
Relational system of record
-> Supabase / Postgres
reason: existing canonical application data already lives here
Object blobs and large artifacts
-> S3
reason: native AWS object storage matches the ECS runtime and removes R2 dependence
Queue-driven async work
-> SQS
reason: native queue primitive for worker-service fanout and retry handling
KV-like operational state
-> DynamoDB-backed adapters
reason: replace Cloudflare KV-style access with an AWS-native low-latency key/value store
Runtime logs
-> CloudWatch
reason: authoritative service log sink colocated with ECS in us-east-1
Dashboard telemetry
-> Datadog query path plus CloudWatch fallback in practice
reason: Datadog gives aggregate operator views, but CloudWatch is currently the more reliable
live verifier
```
### Long-delay scheduling pattern
One Cedar-specific pattern does not fit cleanly into a single `KV` or `Queue` row:
```text
Cloudflare scheduling pattern
-> if work is due within the queue delay window:
enqueue directly with delay
-> else:
store lightweight schedule metadata in KV
-> cron later promotes it into the queue once it enters the delay window
AWS current equivalent
-> store lightweight schedule metadata in DynamoDB-backed KV adapters
-> worker scheduled path promotes ready items into the worker/queue execution path
```
Examples in Cedar:
- `scheduled_agent_actions`
- `scheduled_emails`
- `remind_emails`
- task scheduling entries under `scheduled_agent_actions`
Relevant files:
- `apps/server/src/services/agent-action-queue/executions.ts`
- `apps/server/src/services/task-scheduling/scheduling.ts`
- `apps/server/src/services/task-scheduling/promotion.ts`
- `apps/server/src/main.ts`
Important nuance:
- On Cloudflare, the shared scheduling code uses a `12 hour` queue-delay threshold because
Cloudflare Queues allow delayed delivery up to that window.
- On AWS, this should be understood as a `scheduled state + promotion` pattern, not as a claim
that raw SQS delayed delivery is a drop-in replacement for the same 12-hour window.
- So on the primitive map, this belongs conceptually between the `KV namespaces -> DynamoDB`
row and the `Queues -> SQS` row, with the scheduled trigger/promotion path as the bridge
between them.
### External dependencies intentionally still shared
- Supabase / Postgres
- Better Auth
- OAuth providers
- Upstash
## Request Paths
### 1. Frontend -> AWS app URL -> static app shell
```text
GET https://d28vdqmgxberim.cloudfront.net/
-> CloudFront
-> S3 asset bucket
-> index.html / JS / CSS
-> browser boots React Router app
```
This is only the static shell path. No Cedar server code runs until the browser makes a same-origin
API request.
### 2. Frontend -> AWS app URL -> normal API or tRPC request
This is the common path for inbox, thread load, CRM, calendar, settings, and most authenticated
frontend reads/writes.
```text
Browser on AWS app URL
-> GET/POST https://d28vdqmgxberim.cloudfront.net/api/...
-> CloudFront behavior: /api/*
-> ALB: aws-*-api-alb
-> default ALB target group
-> ECS api-service
-> apps/server/src/runtime/api-entry.ts
-> handlers.fetch(request, env, ctx)
-> apps/server/src/main.ts
-> Hono route or tRPC endpoint /api/trpc
-> route/service layer
-> getUserStore() / getMailRuntime() / AWS-primary adapters
-> Supabase / Postgres
-> DynamoDB-backed KV adapters
-> S3 object storage
-> SQS queue publish where async continuation is needed
-> external provider APIs
```
Where in AWS it lands:
- CloudFront is the public frontend edge.
- The ALB is the HTTP entrypoint for dynamic Cedar server traffic.
- `api-service` is the ECS service that actually executes the request.
- The request then stays in-process inside the Cedar server code unless that code explicitly
publishes to SQS, reads/writes S3, or calls an external provider.
### 3. Frontend -> AWS app URL -> chat / streaming execution
```text
Browser chat UI
-> POST same-origin Cedar chat route
-> CloudFront dynamic origin
-> ALB
-> chat listener rule
path ownership: /chat* and /trpc/mastra.chatStream*
-> ECS chat-service
-> shared chat handler
-> executeChatAgentStreaming()
-> chat workflow
-> retrieval / tools / analyzer path
-> Postgres
-> Turbopuffer
-> provider/tool integrations as allowed
```
Why this is separate:
- chat is isolated onto `chat-service` so long-lived streaming and heavier agent execution do not
compete with normal request/response API traffic on `api-service`
- the ALB listener explicitly reserves chat paths for this service in the AWS stack
### 4. Frontend -> Cloudflare surface -> AWS shadow target
This is the migration-only path for allowlisted query shadowing. The user still gets the
Cloudflare response, but AWS also receives a cloned read request for parity validation.
```text
Browser on Cloudflare app/API surface
-> Cloudflare Worker receives /api/trpc/<query>
-> apps/server/src/main.ts captures a clone of the incoming request
-> request executes normally on Cloudflare
-> after successful query completion
-> apps/server/src/trpc/trpc.ts shadow-forwarding middleware
-> apps/server/src/lib/shadow-forwarding.ts
-> allowlist check
-> resolve target by environment:
staging -> CEDAR_AWS_STAGING_API_URL
prod -> CEDAR_AWS_PROD_API_URL
-> forward cloned request with shadow headers
-> AWS ALB
-> ECS api-service
-> apps/server/src/runtime/api-entry.ts
-> logs shadow metadata headers
-> runs the same Cedar request path on AWS
```
Important constraints:
- only allowlisted query procedures are shadow-forwarded
- mutations, auth callbacks, webhooks, queue consumers, and scheduled paths are not part of this
shadow-forwarding slice
- this path exists for migration validation; it is not the steady-state AWS-primary request path
### 5. Frontend calendar request -> AWS API -> provider call
Calendar is worth spelling out because it previously had an AWS parity gap.
```text
Browser calendar page on AWS URL
-> /api/trpc/calendar.*
-> CloudFront
-> ALB
-> ECS api-service
-> calendar tRPC route
-> getMailRuntime()
-> AWS-primary mail runtime adapter
-> createCalendarDriver(activeConnection)
-> Google Calendar APIs
```
Why this matters:
- the route layer still expects the same calendar surface Cedar had before the AWS move
- the AWS-primary runtime now supplies that surface directly, so calendar page loads stay on the
AWS-native request path instead of depending on a legacy Cloudflare-only implementation
### 6. Backend-triggered ingress -> external webhook -> AWS
This is the main backend-triggered HTTP path. The trigger comes from a provider, not from the
browser.
```text
External provider webhook
examples:
meeting notes provider
Slack webhook
Turbopuffer webhook
-> AWS Cedar HTTP surface
-> CloudFront dynamic route or direct API hostname
-> ALB
-> ECS api-service
-> apps/server/src/main.ts webhook route
-> request verification / parsing
-> provider-specific handler
-> may write Postgres
-> may store artifacts in S3
-> may enqueue SQS work for async continuation
-> may invoke direct AWS-primary execution helpers
```
Typical meeting-webhook continuation path:
```text
Meeting provider webhook
-> api-service
-> verifyAndParseMeetingWebhookRequest(...)
-> processMeetingWebhookDirect(...)
-> store transcript/artifact in S3
-> trigger Cedar execution path
-> route/service logic
-> Postgres + downstream integrations
```
### 7. Backend-triggered async flow -> SQS -> worker-service
This is the main non-HTTP AWS backend path once Cedar has decided work should continue
asynchronously.
```text
api-service or scheduled promotion path
-> publish message to SQS
queues include:
thread-queue
channel-sync-queue (channel buffer drain wake-up; optional, cron covers it when unbound)
subscribe-queue
send-email-queue
send-remind-queue
scheduled-agent-action-queue
cron-task-queue (SF cron orchestrator callbacks)
workflow-execution-queue (legacy workflow-runner callbacks)
workflow-step-queue (per-workflow multi-step SF callbacks)
-> ECS worker-service
-> apps/server/src/runtime/worker-entry.ts
-> long-poll SQS
-> handlers.queue(...)
-> queue-specific Cedar business logic
-> Postgres
-> DynamoDB-backed KV adapters
-> S3
-> provider APIs
```
Where it lands in AWS:
- the message does not come back through the ALB
- it is consumed directly by `worker-service` running on ECS/Fargate
- `worker-service` is the AWS home for retries, queue fanout, delayed work promotion, and other
background execution
### 8. Backend-triggered scheduled flow -> Step Functions -> cron-task-queue -> worker-service
```text
EventBridge hourly rule
-> Step Function: cron-orchestrator
-> 14 parallel SQS messages to cron-task-queue (waitForTaskToken)
-> worker-service cron task queue poller
-> cronTaskRegistry[taskName](env, scheduledTime)
-> e.g. processScheduledEmails:
-> scan DynamoDB-backed KV for due items
-> promote ready work into SQS (send-email-queue)
-> e.g. processPeriodicEmailSync:
-> batch sync all active connections
-> SendTaskSuccess / SendTaskFailure back to Step Function
```
This replaces the old monolithic `handlers.scheduled()` approach. Each of the 13 cron tasks
now runs as an independent parallel branch in the Step Function (processAopAutomations runs
in its own dedicated 15-minute state machine):
- long-horizon intent sits in DynamoDB-backed KV state
- near-term executable work is promoted into SQS by the relevant cron task
- `worker-service` then consumes the actual executable message
- failures in one task don't block others
- the legacy `POST /scheduled` endpoint is still available for manual debugging
### 9. Backend-internal fanout inside AWS primary mode
Some Cedar routes still call workflow or runtime entrypoints from inside the API/service layer.
In AWS primary mode, those calls now route through per-workflow Step Functions state machines
where each `step.do()` is a separate SF state.
```text
api-service route / service / workflow
-> env.WORKFLOW.create({ params })
-> env.ts checks AWS_SFN_WF_<WORKFLOW>_ARN
if set (staging/prod):
-> StartExecution(per-workflow SM, { workflowName, workflowRunId, params })
-> SF state for step 1: SQS to workflow-step-queue with taskToken
-> worker-service polls queue, executes step logic
-> SendTaskSuccess -> SF advances to next state
-> ... each step.do() is a separate SF state with its own timeout
if unset (local dev):
-> startAwsPrimaryWorkflow() runs workflow in-process
```
Why this matters:
- the public request may still originate in a legacy Cedar route or service boundary
- in AWS primary mode the orchestration itself is now AWS-native and observable per-step
- each step has its own SF state, timeout, and result path visible in the SF console
- Map states (label sync, search sync, CRM batch, etc.) run with bounded concurrency
- pagination loops (SyncThreadsCoordinator) use Choice states for automatic iteration
### 10. Backend-triggered workflow orchestration -> per-workflow Step Functions
This is the main AWS backend flow for orchestration-heavy sync work.
```text
Backend trigger
examples:
sync thread coordinator
conversation sync
internal scheduler / route service call
-> Cedar route or service layer
-> env.WORKFLOW.create() -> StartExecution(per-workflow SM)
-> SF executes step chain:
-> step 1: SQS to workflow-step-queue with taskToken
-> worker-service polls queue, runs step.do() logic
-> SendTaskSuccess(taskToken, stepResult)
-> SF stores result at $.steps.<resultKey>
-> step 2: conditional / Map / loop as defined in the SM
-> worker processes step, calls back
-> ... until final step completes
-> admin views poll DescribeExecution for per-step status
```
Why this matters:
- the orchestration path is now explicit and inspectable at the individual step level
- failures belong to a named step within a named Step Functions execution
- Map states (batch processing) and Choice states (conditional phases) are visible in the SF graph
- the workflow result store gives the dashboard a durable source of truth for run status and output
### Chat and Retrieval Execution
```text
Chat UI on AWS app URL
-> same-origin chat stream request
-> chat-service / shared chat handler
-> executeChatAgentStreaming()
-> read-only threadId -> conversationId lookup
scope: current user only
reason: keep open-thread questions on the single-conversation path
guardrail: no CRM mutation during chat reads
-> generatePreamble()
-> getUserContext(userId, includeConnectionId=false)
-> in-memory TTL cache
reason: prompt context needs user identity + timezone, not provider connection state
-> cached skill tool registry
reason: avoid rebuilding the same static tool map on every turn
-> chat agent execution
-> direct answer when existing tools/context are enough
-> spawn-subagent(type="analyzer") when content retrieval is needed
-> optional timeline preload using a short-lived DB session
-> search-turbopuffer
-> optional attribute fallback for old namespaces
-> only `aopId` is retry-droppable
-> reason: preserve backward compatibility without masking real schema problems
-> citation enrichment using short-lived DB sessions
-> meeting timestamp enrichment using short-lived DB sessions
```
Why this design was chosen:
- prompt parity with Cloudflare is preserved; the speed work stays in runtime code, not agent
instructions
- open-thread chat requests should resolve to the current user's conversation deterministically,
but a read path must never repair or mutate CRM data as a side effect
- user-context generation does not need `connectionId`, so avoiding that lookup removes latency
from every chat turn without changing behavior
- the analyzer path previously held a DB connection open for the full streamed LLM run; moving to
short-lived DB sessions reduces connection pressure under concurrent AWS chat load
- the Turbopuffer compatibility layer is intentionally narrow: it only retries missing fields that
are explicitly marked optional (`aopId`) so schema drift does not get silently ignored
Relevant files:
- `apps/server/src/mastra/workflows/chat/chat-workflow.ts`
- `apps/server/src/services/crm/email-events.ts`
- `apps/server/src/mastra/prompts/preamble.ts`
- `apps/server/src/mastra/utils/context-formatting/user-context.ts`
- `apps/server/src/mastra/skills/index.ts`
- `apps/server/src/services/turbopuffer/search.ts`
- `apps/server/src/mastra/utils/execution/execute-analyzer.ts`
- `apps/server/src/services/turbopuffer/meeting-timestamps.ts`
## Latest Validation
Most recent full rollout on `2026-04-03`:
- the current `origin/staging` state was synced into `aws-migration`, repaired for AWS deploys,
validated on AWS staging first, and then promoted to the raw AWS prod surfaces
- merge regressions fixed before rollout:
- duplicate helper/symbol leftovers in merged server routes and services
- Step Functions workflow-runner env wiring that created a CDK cycle in
`aws/lib/stacks/app-stack.ts`
- a missing `adminDocumentsRouter` import that crashed the staging API after rollout
- AWS staging rollout record:
- stack: `CedarAwsStagingEnvironmentStack`
- image tag: `staging-merge-sync-20260403-2`
- CloudFront distribution: `E2CQO6F97QJ6M2`
- CloudFront invalidation: `I3Z2QNAPAIDSEXEA6AG7BWIK0T`
- smoke checks passed on:
- `http://aws-staging-api-alb-1395590711.us-east-1.elb.amazonaws.com/health`
- `https://d2p1ksd83o7plh.cloudfront.net/health`
- `https://d2p1ksd83o7plh.cloudfront.net/`
- raw AWS prod rollout record:
- stack: `CedarAwsProdEnvironmentStack`
- image tag: `prod-merge-sync-20260403-1`
- CloudFront distribution: `EKWAET474CHUG`
- CloudFront invalidation: `IZHAU82E4AJXQ8YE4VZD289NC`
- smoke checks passed on:
- `http://aws-prod-api-alb-1448513427.us-east-1.elb.amazonaws.com/health`
- `https://d28vdqmgxberim.cloudfront.net/health`
- `https://d28vdqmgxberim.cloudfront.net/`
Real browser validation was run against AWS prod in Google Chrome on `2026-04-01` at:
- `https://d28vdqmgxberim.cloudfront.net/mail/inbox`
Validated successfully after the calendar parity fix:
- inbox load
- thread open
- CRM tab
- Timeline tab
- Sent folder
- Calendar route
Measured end-user timings from the same pass:
```text
inbox load ~4.9s
thread open ~3.1s
CRM tab ~3.0s
Timeline tab ~2.5s
Sent folder ~3.1s
Calendar route ~3.1s
```
These are warm-path browser timings. They include app-level data loading, route hydration, and
service-side fetch work, so they are a better measure of customer-facing latency than raw process
startup.
Observed backend follow-ups from the same validation:
- calendar route failures were fixed by adding the missing AWS-primary calendar methods
- chat retrieval for open-thread questions was hardened so it uses a read-only, user-scoped
thread-to-conversation lookup rather than the older mutating CRM repair helper
- analyzer chat execution now uses short-lived DB sessions instead of pinning a DB connection for
the full stream
- CloudWatch still shows recurring `handleExecutionStream` poll-loop errors with
`Database connection not initialized`
- several first-load APIs remain slower than ideal and should be tuned after functional
validation
- customer-facing latency is still most sensitive to ECS/Fargate task warmness and per-route data
loading, so the prod floor exists to keep the hottest routes from paying a cold-start penalty on
every restart or scale-out
## Review Notes
The AWS-primary code is materially cleaner than the original migration shim was, but there is
still some migration-shaped code left.
```text
cleaned up now
- AWS chat read path is user-scoped and read-only
- chat preamble avoids unnecessary provider lookups
- analyzer DB lifetime is bounded to discrete operations
- Turbopuffer fallback is explicit about which schema drift is tolerated
still worth improving later
- aws-primary-mail-runtime.ts is still a large runtime adapter
- Datadog-backed dashboard queries still throttle more often than CloudWatch
- the final custom-domain flip should keep the API host under the app domain hierarchy unless
cookie-domain derivation is generalized further
```
## Observability
- CloudWatch is the reliable source of truth for AWS service logs today.
- The migration dashboard has live Datadog query support in code, but the live payload is
currently degraded rather than healthy.
- The dedicated dashboard payload is built in:
- `apps/server/src/services/aws-migration/dashboard.ts`
## Critical Files
| File | Role |
| ---------------------------------------------------------- | ------------------------------------------------------------------------ |
| `aws/lib/cedar-environment-stack.ts` | Top-level AWS environment stack assembly |
| `aws/lib/stacks/app-stack.ts` | App, ALB, ECS, CloudFront, and S3 wiring |
| `aws/lib/stacks/workflow-state-machines.ts` | Per-workflow multi-step SF definitions (6 SMs with step chains) |
| `aws/lib/stacks/data-stack.ts` | AWS data primitives used during migration |
| `aws/lib/runtime-contract.ts` | Runtime env/secret contract for AWS services |
| `apps/server/src/lib/shadow-forwarding.ts` | Cloudflare -> AWS query shadow-forward decision and forwarding |
| `apps/server/src/trpc/trpc.ts` | Post-query shadow-forward dispatch middleware |
| `apps/server/src/runtime/api-entry.ts` | API container entrypoint |
| `apps/server/src/runtime/worker-entry.ts` | Worker container entrypoint (SQS pollers + cron task queue handler) |
| `apps/server/src/container/index.ts` | Chat service/container HTTP entrypoint |
| `apps/server/src/main.ts` | Shared Hono server, webhook routes, chat route, queue/scheduled handlers, cronTaskRegistry |
| `apps/server/src/env.ts` | Per-workflow SF ARN routing (AWS_SFN_WF_* -> StartExecution or in-process) |
| `apps/server/src/container/aws-primary-runtime.ts` | AWS runtime entrypoint for workflow and runtime calls |
| `apps/server/src/services/aws-migration/dashboard.ts` | Migration dashboard payload builder |
| `apps/mail/app/(routes)/cedarAdmin/aws-migration/page.tsx` | Dashboard UI |
## Validation Rules
- Prefer the AWS staging and prod URLs when validating migration behavior.
- Use CloudWatch first for AWS incidents.
- Treat Cloudflare incidents separately from AWS incidents unless the same shared code path
clearly reproduces on both surfaces.
- Track post-cutover cleanup and deferred fixes in [AWS_MIGRATION.md](./AWS_MIGRATION.md).
## Local Dev Against Staging Data
To run local dev against real staging S3/SQS/DynamoDB, set these in your root `.env`:
```bash
# S3 Buckets
AWS_S3_THREADS_BUCKET=cedar-aws-staging-threads-619071350399-us-east-1
AWS_S3_MEETINGS_BUCKET=cedar-aws-staging-meetings-619071350399-us-east-1
AWS_S3_CRM_CONTENT_BUCKET=cedar-aws-staging-crm-content-619071350399-us-east-1
AWS_S3_SLACK_BUCKET=cedar-aws-staging-slack-619071350399-us-east-1
# SQS Queues (get URLs from CloudFormation outputs or AWS console)
AWS_SQS_THREAD_QUEUE_URL=https://sqs.us-east-1.amazonaws.com/619071350399/aws-staging-thread-queue.fifo
AWS_SQS_SUBSCRIBE_QUEUE_URL=https://sqs.us-east-1.amazonaws.com/619071350399/aws-staging-subscribe-queue
AWS_SQS_SEND_EMAIL_QUEUE_URL=https://sqs.us-east-1.amazonaws.com/619071350399/aws-staging-send-email-queue
AWS_SQS_SEND_REMIND_QUEUE_URL=https://sqs.us-east-1.amazonaws.com/619071350399/aws-staging-send-remind-queue
AWS_SQS_SCHEDULED_AGENT_ACTION_QUEUE_URL=https://sqs.us-east-1.amazonaws.com/619071350399/aws-staging-scheduled-agent-action-queue
AWS_SQS_CRON_TASK_QUEUE_URL=https://sqs.us-east-1.amazonaws.com/619071350399/aws-staging-cron-task-queue
AWS_SQS_WORKFLOW_EXECUTION_QUEUE_URL=https://sqs.us-east-1.amazonaws.com/619071350399/aws-staging-workflow-execution-queue
AWS_SQS_WORKFLOW_STEP_QUEUE_URL=https://sqs.us-east-1.amazonaws.com/619071350399/aws-staging-workflow-step-queue
# Optional. Leaving it unset leaves env.channel_sync_queue undefined, which producers
# treat as "no wake-up available" and the channel reconcile cron covers.
AWS_SQS_CHANNEL_SYNC_QUEUE_URL=https://sqs.us-east-1.amazonaws.com/619071350399/aws-staging-channel-sync-queue.fifo
# DynamoDB Tables
AWS_DYNAMODB_SCHEDULED_EMAILS_TABLE=aws-staging-scheduledEmails
AWS_DYNAMODB_SCHEDULED_AGENT_ACTIONS_TABLE=aws-staging-scheduledAgentActions
AWS_DYNAMODB_PENDING_EMAILS_TABLE=aws-staging-pendingEmails
AWS_DYNAMODB_INITIAL_SYNC_STATE_TABLE=aws-staging-initialSyncState
AWS_DYNAMODB_CALENDAR_SYNC_TOKENS_TABLE=aws-staging-calendarSyncTokens
AWS_DYNAMODB_GENERIC_KV_TABLE=aws-staging-genericKv
```
**Prerequisites:** AWS credentials configured with staging resource access (IAM role or access keys for account `619071350399`).
**Leave all AWS vars unset for fully local in-memory operation.** The app falls back to in-memory storage, queues, and KV when these vars are not set.
**Step Functions ARN vars** control whether workflows route through Step Functions or run in-process.
Leave all `AWS_SFN_*` vars unset for local dev — workflows will execute in-process in the worker.
Per-workflow ARNs (`AWS_SFN_WF_*`):
- `AWS_SFN_WF_SYNC_THREADS_ARN`
- `AWS_SFN_WF_SYNC_THREADS_COORDINATOR_ARN`
- `AWS_SFN_WF_CRON_SYNC_ARN`
- `AWS_SFN_WF_CONVERSATION_SYNC_ARN`
- `AWS_SFN_WF_CONVERSATION_REFRESH_ARN`
- `AWS_SFN_WF_CALENDAR_SYNC_ARN`
Legacy ARNs (still used for the workflow-runner and cron orchestrator):
- `AWS_SFN_WORKFLOW_RUNNER_STATE_MACHINE_ARN`
- `AWS_SFN_CRON_ORCHESTRATOR_STATE_MACHINE_ARN`
## Secrets Management
### How secrets work
Secrets are stored in **AWS Secrets Manager** and injected into ECS containers at startup as
environment variables. The deployment pipeline does **not** automatically sync secrets from the
root `.env` — this is a separate manual step.
**Critical:** If secrets are missing or stale, all tRPC routes will return 500 errors because
the rate limiter middleware (Upstash Redis) fails before any handler runs.
### Secret layout
Each environment has a single consolidated secret in Secrets Manager:
| Secret Name Pattern | Key Contents |
|-------------------|--------------|
| `/cedar/aws-{env}/runtime` | DATABASE_URL, REDIS_URL, auth secrets, OAuth credentials, AI provider keys, observability keys, webhooks, integrations (44 fields total) |
The canonical field list is `ALL_SECRET_FIELDS` in `aws/lib/runtime-contract.ts`.
### Syncing secrets
The sync script reads from the root `.env` and pushes values to AWS Secrets Manager.
```bash
# From the aws/ directory:
# Install dependencies (required first time)
pnpm --ignore-workspace install --no-frozen-lockfile
# Dry-run — shows what would be synced and any missing keys
npx tsx scripts/sync-secrets.ts --env staging --mode plan
npx tsx scripts/sync-secrets.ts --env prod --mode plan
# Sync secrets to AWS Secrets Manager
npx tsx scripts/sync-secrets.ts --env staging --mode sync
npx tsx scripts/sync-secrets.ts --env prod --mode sync
```
**Prerequisites:**
- AWS credentials configured for the Cedar account (`619071350399`)
- Root `.env` file present one directory above `aws/`
- AWS SDK dependencies installed via `pnpm --ignore-workspace install`
### After syncing secrets
ECS injects secrets at container startup. Updated secrets require a service restart:
```bash
# Restart staging services
aws ecs update-service --cluster aws-staging-api-cluster --service aws-staging-api-service --force-new-deployment --region us-east-1
aws ecs update-service --cluster aws-staging-api-cluster --service aws-staging-chat-service --force-new-deployment --region us-east-1
aws ecs update-service --cluster aws-staging-api-cluster --service aws-staging-worker-service --force-new-deployment --region us-east-1
# Restart prod services
aws ecs update-service --cluster aws-prod-api-cluster --service aws-prod-api-service --force-new-deployment --region us-east-1
aws ecs update-service --cluster aws-prod-api-cluster --service aws-prod-chat-service --force-new-deployment --region us-east-1
aws ecs update-service --cluster aws-prod-api-cluster --service aws-prod-worker-service --force-new-deployment --region us-east-1
```
### When to sync secrets
- After adding a new env var to `aws/lib/runtime-contract.ts`
- After rotating any API key or credential in the root `.env`
- After initial CDK deploy (creates empty secret placeholders)
- When services return unexplained 500 errors on all routes
## Troubleshooting
### All tRPC routes return 500
**Symptom:** Browser console shows 500 errors on `settings.get`, `connections.list`,
`labels.list`, and every other tRPC route.
**Root cause:** Missing or invalid `REDIS_URL` / `REDIS_TOKEN` in Secrets Manager. Every tRPC
route hits the rate limiter middleware first (`apps/server/src/trpc/trpc.ts:194`), which creates
an Upstash Redis client via `apps/server/src/lib/services.ts:27`. If credentials are empty or
invalid, the middleware throws before any route handler runs.
**Fix:**
1. Verify: `cd aws && npx tsx scripts/sync-secrets.ts --env staging --mode plan`
2. Sync: `npx tsx scripts/sync-secrets.ts --env staging --mode sync`
3. Restart: `aws ecs update-service --cluster aws-staging-api-cluster --service aws-staging-api-service --force-new-deployment --region us-east-1`
4. Wait 2-3 minutes for new tasks to stabilize
### Cloudflare Workers Builds failing on PRs
**Symptom:** GitHub checks show 4 failing "Workers Builds" checks.
**Root cause:** Legacy Cloudflare integration hooks still active post-migration.
**Fix:** Remove the Cloudflare Workers build hooks from GitHub repo settings or Cloudflare dashboard.
### ECS tasks crash-looping
**Check logs:**
```bash
aws logs tail /aws/ecs/aws-staging-api/api-service --region us-east-1 --since 10m
```
Common causes: missing secrets, bad Docker image, health check timeout.
With circuit breaker enabled, ECS will auto-rollback after repeated failures instead of
retrying forever. Check the deployment rollout state:
```bash
aws ecs describe-services --cluster aws-staging-api-cluster --services aws-staging-api-service \
--query 'services[0].deployments[*].{status:status,rolloutState:rolloutState}' --region us-east-1
```
If `rolloutState` is `FAILED`, the circuit breaker tripped. Check stopped task reasons:
```bash
TASK=$(aws ecs list-tasks --cluster aws-staging-api-cluster --service-name aws-staging-api-service \
--desired-status STOPPED --query 'taskArns[0]' --output text --region us-east-1)
aws ecs describe-tasks --cluster aws-staging-api-cluster --tasks "$TASK" \
--query 'tasks[0].{stoppedReason:stoppedReason,containers:containers[0].{exitCode:exitCode,reason:reason}}' \
--region us-east-1
```
### CloudFormation stack stuck in UPDATE_IN_PROGRESS
**Symptom:** `cdk deploy` fails with "Stack is in UPDATE_IN_PROGRESS state and can not be updated."
**Root cause:** A previous CDK deploy submitted a changeset that CloudFormation is still
processing. Common when ECS service rollouts hang (especially before circuit breaker was enabled).
**Fix:**
1. Cancel the in-progress update: `aws cloudformation cancel-update-stack --stack-name CedarAwsStagingEnvironmentStack --region us-east-1`
2. Wait for rollback: `aws cloudformation wait stack-update-complete --stack-name CedarAwsStagingEnvironmentStack --region us-east-1`
3. If stuck in `UPDATE_ROLLBACK_IN_PROGRESS` for >10 min, force the ECS services to stabilize:
```bash
for svc in api-service chat-service worker-service; do
aws ecs update-service --cluster aws-staging-api-cluster --service "aws-staging-$svc" \
--force-new-deployment --region us-east-1
done
```
4. Once stack reaches `UPDATE_ROLLBACK_COMPLETE`, retry the deploy.
**Prevention:** Circuit breaker is now enabled on all services, so failed deploys trigger
auto-rollback in ~2 minutes instead of hanging CloudFormation for 30+ minutes.
### CloudFront returning 403 on custom domain
**Symptom:** `mail-staging.cedarcopilot.com` returns 403, but `d2p1ksd83o7plh.cloudfront.net` works.
**Root cause:** CloudFront distribution lost its alternate domain names (aliases) and ACM
certificate. This can happen when a CDK deploy rolls back or when running CDK manually without
the custom domain env vars (`CEDAR_AWS_STAGING_APP_DOMAIN`, etc.).
**Check:**
```bash
DIST_ID=$(aws cloudformation describe-stacks --stack-name CedarAwsStagingEnvironmentStack \
--query "Stacks[0].Outputs[?contains(OutputKey,'FrontendDistributionId')].OutputValue | [0]" \
--output text --region us-east-1)
aws cloudfront get-distribution-config --id "$DIST_ID" --query 'DistributionConfig.Aliases' --region us-east-1
```
If `Quantity: 0`, aliases are missing.
**Fix:** Run a CDK deploy (via CI/CD or manually with all domain env vars set). The CI/CD
pipeline always passes these vars from GitHub Actions environment settings. If deploying
manually, use `deploy-surface.sh` which resolves them automatically.
### Manual CDK deploy missing custom domains
**Symptom:** After running `cdk deploy` manually, custom domains stop working.
**Root cause:** CDK reads `CEDAR_AWS_{STAGING|PROD}_APP_DOMAIN`, `_API_DOMAIN`, and
`_CERTIFICATE_ARN` from env vars. These are set in GitHub Actions but not in your local shell.
Without them, CDK creates the CloudFront distribution without aliases.
**Prevention:** Always use `deploy-surface.sh` instead of `cdk deploy` directly. It resolves
the domain vars via `cedar_export_custom_domain_env` in `deploy-utils.sh`. Or set them
explicitly:
```bash
export CEDAR_AWS_STAGING_APP_DOMAIN=mail-staging.cedarcopilot.com
export CEDAR_AWS_STAGING_API_DOMAIN=api.mail-staging.cedarcopilot.com
export CEDAR_AWS_STAGING_CERTIFICATE_ARN=$(aws acm list-certificates --certificate-statuses ISSUED \
--query "CertificateSummaryList[?DomainName=='mail-staging.cedarcopilot.com'].CertificateArn | [0]" \
--output text --region us-east-1)
```