AWS_MIGRATION.md41.5 KBView on GitHub # Cedar AWS Migration Backlog
This file tracks the work that remains after the AWS cutover is functional. It is the place
for deferred fixes, cleanup, and follow-up validation.
## Current State
As of `2026-04-06`, the AWS migration is complete for both staging and prod:
- Both staging and prod custom domains now resolve to their respective AWS CloudFront distributions
- AWS staging custom domains:
- `https://mail-staging.cedarcopilot.com`
- `https://api.mail-staging.cedarcopilot.com`
- AWS prod custom domains:
- `https://mail.cedarcopilot.com`
- `https://api.mail.cedarcopilot.com`
- Cloudflare Workers builds still trigger on PRs but are vestigial and should be removed
- The dedicated migration dashboard is live.
- Some operational follow-up items still remain even though the main AWS surfaces are up.
- AWS staging and prod now run with ECS target-tracking autoscaling for api-service,
chat-service, and worker-service.
- AWS branch-driven AWS CI/CD is now split into readiness checks plus real deploys:
- pull requests run `AWS Builds` readiness checks for:
- `cedar-app-staging`
- `cedar-api-staging`
- `cedar-app`
- `cedar-api`
- push to `staging`
- GitHub Actions publishes the changed Cedar AWS staging surfaces
- frontend publishes go straight to the staging S3 bucket + CloudFront distribution
- service / infra changes redeploy `CedarAwsStagingEnvironmentStack`
- push to `main`
- GitHub Actions deploys the changed raw AWS prod surfaces
- AWS staging task env and browser-side runtime URL handling are now correct for the active AWS
surfaces, and the same guardrails must remain in place for the prod domain flip:
- browser bundles must not hardcode Cloudflare staging hosts through
`import.meta.env.VITE_PUBLIC_BACKEND_URL` or `import.meta.env.VITE_PUBLIC_APP_URL`
- AWS-hosted pages must resolve app and backend requests from the current browser origin at
runtime
- All 6 Cedar workflow classes now run through dedicated per-workflow Step Functions state
machines where each `step.do()` is a separate SF state:
- `SYNC_THREADS_WORKFLOW`, `SYNC_THREADS_COORDINATOR_WORKFLOW`, `CRON_SYNC_WORKFLOW`,
`CONVERSATION_SYNC_WORKFLOW`, `CONVERSATION_REFRESH_WORKFLOW`, `CALENDAR_SYNC_WORKFLOW`
- Routing is controlled by `AWS_SFN_WF_*` ARN env vars in `apps/server/src/env.ts`
- If the ARN is set, dispatch goes to the per-workflow SM; if unset, falls back to in-process
- See `aws/lib/stacks/workflow-state-machines.ts` for the full step chain definitions
- A real Chrome-based prod AWS smoke pass was completed on `2026-04-01`.
- That pass found one live parity gap in the AWS-primary zero-agent shim:
calendar routes were calling methods that existed on the Cloudflare path but were missing from
the AWS path.
- The missing AWS calendar surface was implemented in
`apps/server/src/lib/aws-primary-mail-runtime.ts` and deployed as:
- staging api image: `calendarfix-20260401-1`
- prod api image: `calendarfix-20260401-1`
- Live AWS staging validation on `2026-04-02` found another AWS-primary zero-agent parity gap:
labels routes were calling label methods that existed on the Cloudflare path but were missing
from the AWS path.
- The missing AWS labels surface was implemented in
`apps/server/src/lib/aws-primary-mail-runtime.ts` and deployed to staging as:
- staging api image: `staging-labels-fix-20260402-1`
- Post-fix CloudWatch validation on AWS staging now shows:
- `labels.list` succeeds on the live API service
- no fresh `agent.getUserLabels is not a function` errors after the rollout
- Post-fix validation on AWS prod now passes for:
- inbox load
- thread open
- CRM tab
- Timeline tab
- Sent folder
- Calendar route
- Live autoscaling posture (as defined in `aws/lib/config.ts`):
- staging:
- `api-service`: desired `2`, range `2..6`
- `chat-service`: desired `2`, range `2..4`
- `worker-service`: desired `2`, range `2..6`
- prod:
- `api-service`: desired `2`, range `2..8`
- `chat-service`: desired `2`, range `2..6`
- `worker-service`: desired `2`, range `2..6`
- Frontend performance follow-up shipped on `2026-04-02`:
- thread body prefetch now warms `mail.get` on row hover, thread open, draft hover/click, and
keyboard focus windows around the active row
- speculative prefetches now set silent query metadata so cache warm-up misses do not spam the
global query error handler
- rollout completed on both AWS frontend surfaces:
- staging CloudFront invalidation: `IC3MT2Q07IN6ZP7YBK4CYIQF4W`
- prod CloudFront invalidation: `I94ODJYZDKP7VNA9U9OI7MPWIY`
- post-deploy smoke checks passed on both:
- staging:
- `http://aws-staging-api-alb-1395590711.us-east-1.elb.amazonaws.com/health`
- `https://d2p1ksd83o7plh.cloudfront.net/health`
- `https://d2p1ksd83o7plh.cloudfront.net/`
- prod:
- `http://aws-prod-api-alb-1448513427.us-east-1.elb.amazonaws.com/health`
- `https://d28vdqmgxberim.cloudfront.net/health`
- `https://d28vdqmgxberim.cloudfront.net/`
- On `2026-04-03`, the current `origin/staging` state was synced into `aws-migration` and then
repaired for the AWS surfaces before promotion.
- The merge fallout fixed before rollout was:
- duplicate merged helper/symbol definitions in server routes and services
- Step Functions workflow-runner env wiring in `aws/lib/stacks/app-stack.ts` that created a CDK
circular dependency
- a missing `adminDocumentsRouter` import that crashed the staging API after rollout
- AWS staging rollout record on `2026-04-03`:
- image tag: `staging-merge-sync-20260403-2`
- CloudFront invalidation: `I3Z2QNAPAIDSEXEA6AG7BWIK0T`
- smoke checks passed on ALB and CloudFront
- AWS staging custom-domain cutover record on `2026-04-03` / `2026-04-04`:
- ACM certificate ARN:
`arn:aws:acm:us-east-1:597088032164:certificate/5e90385d-642a-4da3-a7df-d052a99c0ff6`
- certificate status: `ISSUED`
- legacy Cloudflare Workers custom domains were detached from:
- `cedar-app-staging`
- `cedar-api-staging`
- Cloudflare DNS now points both staging aliases at:
- `d2p1ksd83o7plh.cloudfront.net`
- confirmed healthy on the custom domains:
- `GET /`
- `GET /login`
- `GET /api/auth/get-session`
- unauthenticated batched tRPC with the correct `Origin` header
- prod lesson from staging:
- ordinary auth and tRPC probes are not enough
- `agentExecutions.stream` must be validated explicitly because gateway failures can look like
browser CORS
- AWS staging chat parity hotfix on `2026-04-03`:
- deployed as an API-only ECS task-definition rollout
- ECS service:
- `aws-staging-api-service`
- task definition revision:
- `CedarAwsStagingEnvironmentStackCedarAwsStagingAppStackApiTaskDefinition165EFDED:88`
- root cause removed:
- the live `/trpc/mastra.chatStream` path was executing in-process on the API service
- the API ECS task definition did not inject the model-provider runtime keys required by the
chat workflow
- runtime fix:
- the AWS runtime contract now injects the AI provider secrets into `api-service` as well as
`chat-service`
- post-deploy CloudWatch validation no longer shows the live
`Could not find API key process.env.ANTHROPIC_API_KEY` failure after the `:88` rollout
- AWS staging API parity hotfix on `2026-04-03` / `2026-04-04`:
- deployed as a service-only API rollout, not a stack redeploy
- ECS service:
- `aws-staging-api-service`
- task definition revision:
- `CedarAwsStagingEnvironmentStackCedarAwsStagingAppStackApiTaskDefinition165EFDED:86`
- root cause removed:
- in `staging`, thread snapshot misses could still fall back into the legacy bridge
object-read path
- that produced `r2.get` timeout errors and `mail.listThreads` failures on the custom domain
- runtime fix:
- AWS-primary object reads now stay AWS-only on snapshot miss
- `mail.listThreads` now builds thread previews from CRM / Postgres first and only touches
object storage for the remaining misses
- operational lesson:
- this failure was not fixed by changing CloudFormation or generic capacity knobs
- the correct fix path was a code deploy on the API service
- AWS prod rollout record on `2026-04-03`:
- image tag: `prod-merge-sync-20260403-1`
- CloudFront invalidation: `IZHAU82E4AJXQ8YE4VZD289NC`
- smoke checks passed on ALB and CloudFront
- Operator promotion rule remains:
- sync `staging` into `aws-migration`
- validate AWS staging first
- only then promote the same merged state to raw AWS prod
- keep the canonical domain flip as a separate change
- AWS staging runtime parity pass on `2026-04-04`:
- deployed as API / worker service-only ECS rollouts
- commit / image tag:
- `32d169b65`
- ECS revisions:
- API:
`CedarAwsStagingEnvironmentStackCedarAwsStagingAppStackApiTaskDefinition165EFDED:90`
- Chat:
`CedarAwsStagingEnvironmentStackCedarAwsStagingAppStackChatTaskDefinitionDB94FCCB:47`
- Worker:
`CedarAwsStagingEnvironmentStackCedarAwsStagingAppStackWorkerTaskDefinitionA3E4A222:57`
- runtime change:
- AWS-primary execution methods were added to
`apps/server/src/lib/aws-primary-mail-runtime.ts`
- staging request/workflow paths now route through `getMailRuntime(...).stub`
instead of directly calling `ZERO_DRIVER` for:
- agent execution refresh / task-now / scheduled execution
- CRM thread hydration / recreate-sync execution
- scheduled execution pipeline fan-out
- CRM integrity-heal refresh
- conversation repair refresh
- post-rollout validation:
- `https://mail-staging.cedarcopilot.com/health` returns `200`
- ECS services are back to `2/2` running
- CloudWatch shows recent success for:
- `mail.listThreads`
- `mail.get`
- `crm.searchConversationsMinimal`
- `crm.getConversation`
- `mastra.chatWarm`
- no fresh `agent.getUserLabels is not a function` errors were seen after the rollout
- no fresh `Could not find API key process.env.ANTHROPIC_API_KEY` errors were seen after the
rollout
- still open:
- one-time AWS object / snapshot backfill is not done yet
- full removal of Cloudflare-era code from the entire repo is not done yet
- the fix here was to remove more of it from the active AWS-primary correctness path first
## Retro: Staging Domain Flip
### What went right
- the custom-domain cutover itself landed correctly:
- staging aliases now point at the AWS staging CloudFront distribution
- the raw "redirect to CloudFront hostname" behavior is no longer the intended staging path
- the rollout model was good:
- raw AWS staging was validated before the alias flip
- the custom-domain flip stayed separate from the earlier raw AWS rollout
- once the real root cause was identified, the fix shipped through the right deploy path:
- code-only backend bug
- API-only ECS rollout
- no unnecessary stack mutation
### What went wrong
- two different incident classes showed up during the same migration window:
- surface leakage / stale host handling
- AWS-primary parity gaps after the alias flip
- the second class was easy to misread as generic infra trouble:
- browser CORS-looking errors
- intermittent `502` / `503`
- route-level `500`s
- repeated execution stream reconnects
- the key inbox incident was not a domain mapping problem after the flip:
- `mail.listThreads` could still drop into the legacy bridge object-read path when an AWS thread
snapshot was missing
- that created `r2.get` timeout errors even though staging was supposed to be AWS-primary
### What we learned
- after the staging alias flip, correctness must be AWS-only:
- no Cloudflare object dependency
- no migration-era bridge fallback for core read paths
- primary mode should fail fast if code tries to cross back into legacy bridge infrastructure
- chat-path lesson from the same window:
- "chat runs on chat-service" was not a safe operational assumption by itself
- as long as `/trpc/mastra.chatStream` can execute in-process on the API service, the API task
definition also needs the model-provider runtime keys
- otherwise staging can look healthy while chat fails only at stream-start time
- browser-reported CORS during cutover is a symptom, not a diagnosis
- CloudFormation is the wrong lever for code-only route regressions:
- it is slower
- it broadens the blast radius
- it obscures whether the real problem is infra or runtime logic
- the fast, correct deploy hierarchy is:
- frontend-only publish for app bundle changes
- service-only ECS rollout for code-only backend fixes
- stack deploy only for real infra / env / alias changes
### What still remains
- complete the AWS object/snapshot backfill so migrated thread artifacts exist in the AWS buckets
instead of being available only on the old source side
- continue deleting the remaining Cloudflare-era code paths and migration tooling outside the live
AWS-primary staging request path
- investigate odd post-cutover parity issues that are not blocking the merge but were observed
during staging validation
- current example: read/unread state looks wrong in the extra Conversations email tab on the
AWS side
- observed behavior: emails are being marked read in Gmail correctly, but the Cedar
conversation email tab can still render them as unread
- explicitly validate or gate `agentExecutions.stream` for users who do not have that feature so
staging does not produce noisy reconnect churn
- remove temporary migration-only allowlists after staging is stable
- keep prod deploy automation separate until the staging CI/CD path has enough stable history
## Open Items
### 1. Carry the staging custom-domain cutover lessons into prod
Status: staging complete, prod checklist still open
What is now true on staging:
- the staging stack has both aliases attached in CloudFormation:
- `CedarAwsStagingAppStackFrontendDistributionAliases840CAD29`
- value:
`mail-staging.cedarcopilot.com,api.mail-staging.cedarcopilot.com`
- the staging custom domains now resolve to the AWS staging distribution:
- `https://mail-staging.cedarcopilot.com`
- `https://api.mail-staging.cedarcopilot.com`
- CloudFront distribution:
`d2p1ksd83o7plh.cloudfront.net`
- the staging ACM certificate is issued:
- ARN:
`arn:aws:acm:us-east-1:597088032164:certificate/5e90385d-642a-4da3-a7df-d052a99c0ff6`
- status: `ISSUED`
- the legacy Cloudflare Workers custom domains were detached before the DNS cutover:
- `cedar-app-staging`
- `cedar-api-staging`
- current direct custom-domain checks now pass:
- `GET https://mail-staging.cedarcopilot.com/`
- `GET https://mail-staging.cedarcopilot.com/login`
- `GET https://mail-staging.cedarcopilot.com/api/auth/get-session`
- current AWS-side verification also passes:
- `curl -I https://mail-staging.cedarcopilot.com/health`
returns through CloudFront
- `curl -I https://mail-staging.cedarcopilot.com/mail/inbox`
returns the app shell from S3 through CloudFront
- ECS cluster `aws-staging-api-cluster` is active with:
- `aws-staging-api-service`
- `aws-staging-chat-service`
- `aws-staging-worker-service`
- staging ECS revisions after the April 3, 2026 fix:
- API:
`CedarAwsStagingEnvironmentStackCedarAwsStagingAppStackApiTaskDefinition165EFDED:88`
- Chat:
`CedarAwsStagingEnvironmentStackCedarAwsStagingAppStackChatTaskDefinitionDB94FCCB:47`
- Worker:
`CedarAwsStagingEnvironmentStackCedarAwsStagingAppStackWorkerTaskDefinitionA3E4A222:55`
- all three staging ECS services are back to `2/2` running tasks after the secret/task
definition rollout
- CloudWatch log group `/aws/ecs/aws-staging-api/api-service` shows recent successful
`mail.listThreads` and `mail.get` traffic
- AWS runtime secrets were sanitized on `2026-04-04`:
- removed `BRIDGE_SECRET`
- removed `WORKER_URL`
- removed `CLOUDFLARE_ACCOUNT_ID`
- removed `CLOUDFLARE_API_TOKEN`
- `GET https://api.mail-staging.cedarcopilot.com/api/auth/get-session`
- unauthenticated
`POST https://api.mail-staging.cedarcopilot.com/api/trpc/user.getUserOrganization?batch=1`
with `Origin: https://mail-staging.cedarcopilot.com`
What staging taught us:
- the original staging problem was real browser-side surface leakage:
- build-time public URLs and legacy aliases could still pull the browser back onto Cloudflare
- that produced the old-bundle / wrong-host / wrong-auth-context failure mode
- after the alias cutover, the failure class changed:
- hostname mapping can be correct while specific upstream routes still fail
- the browser may report this as CORS because the `502` / `503` gateway page has no Cedar CORS
headers
- secret cleanup has to be deployed as a contract change, not just a secret edit:
- removing `BRIDGE_SECRET` from Secrets Manager without rolling fresh ECS task definitions left
staging hard-down because the old task definitions still injected that key
- the actual recovery step was to register new ECS task definition revisions and roll all three
services onto them
- `agentExecutions.stream` must be treated as an explicit cutover validation item:
- ordinary auth/session probes can be healthy
- ordinary batched tRPC probes can be healthy
- the stream route can still return a gateway failure and create repeated SSE reconnect churn
Why it matters:
- prod should not repeat the staging mistake of treating homepage/login checks as sufficient
- the prod cutover needs separate validation for:
- static app shell
- normal API/tRPC
- long-lived execution stream behavior
Exit criteria for carrying this into prod:
- prod keeps runtime-origin helpers for browser-side app/backend URLs
- the prod cutover runbook explicitly validates:
- `GET /`
- `GET /login`
- `GET /api/auth/get-session` on both app and api hosts
- unauthenticated batched tRPC with an explicit `Origin` header
- `POST /api/trpc/agentExecutions.stream`
- authenticated `/mail/inbox` navigation on the canonical prod host
- when the browser reports CORS during cutover, operators first reproduce the route with direct
`curl` before changing DNS or trusted origins
Post-cutover cleanup that still remains:
- complete the AWS object/snapshot backfill so migrated thread artifacts are present in AWS buckets
and no AWS-primary read path needs migration-era source data for correctness
- explicitly validate or access-gate `agentExecutions.stream` so accounts without that feature do
not create misleading retry noise during staging validation
- remove the temporary `<email>` staging allowlist after the staging migration is verified
stable:
- this special-case was added only to keep staging usable during migration debugging
- it must not become a permanent replacement for normal Cedar-member/internal-user checks
### 2. Complete prod custom-domain cutover
Status: **complete** as of `2026-04-06`
- AWS prod custom domains now resolve to the AWS prod CloudFront distribution:
- `https://mail.cedarcopilot.com`
- `https://api.mail.cedarcopilot.com`
- prod CloudFront distribution: `EKWAET474CHUG` / `d28vdqmgxberim.cloudfront.net`
- both `mail.cedarcopilot.com` and `api.mail.cedarcopilot.com` serve from AWS
Historical context (preserved for reference):
- the prod ACM certificate was previously `PENDING_VALIDATION`
- Cloudflare DNS write access was the original blocker
- the staging cutover lessons (documented below) were applied to the prod cutover
Exit criteria (all met):
- ACM validation CNAMEs were added in Cloudflare
- the prod CloudFront distribution was updated with the Cedar aliases and ACM cert
- `mail.cedarcopilot.com` and `api.mail.cedarcopilot.com` point to the AWS distribution
- login, inbox, and dashboard flows are healthy on the canonical Cedar domains
- the legacy Cloudflare Workers custom domains were detached before the DNS cutover
### 3. Datadog telemetry is degraded in the live dashboard
Status: open
Observed behavior:
- live dashboard payload currently reports `datadog-health = degraded`
- live telemetry rows are `unavailable`
- the dedicated dashboard host is backed by the staging AWS API, and that staging API was using
a bad Datadog application key in `/cedar/aws-staging/shared/observability`
- observed live Datadog failures include:
- `Datadog aggregate failed (401)` on the dedicated dashboard host when the staging key is bad
- `Datadog aggregate failed (429)` on some higher-fanout prod reads
Why it matters:
- the dashboard has Datadog support in code, but operators cannot rely on it yet
- CloudWatch is still the practical source of truth for AWS incidents
Exit criteria:
- dashboard telemetry queries return live counts instead of `unavailable`
- `datadog-health` reports `healthy`
- the staging shared observability secret carries a valid Datadog application key
- the same credential/query behavior works from both the AWS app surface and the dedicated
dashboard host without getting rate-limited back into failure
### 4. Clean up stale dashboard rollout language
Status: complete on this branch
What changed:
- the migration dashboard now reports AWS-only surfaces and AWS-only error signals
- the control-room copy no longer frames Cloudflare as an active staging runtime dependency
- the dashboard now matches the actual promotion path:
- validate AWS staging
- merge `aws-migration -> main`
- keep the prod domain flip as a final separate operational step
Why it matters:
- operators should not need to mentally translate old shadow terms into the current AWS plan
Result:
- surface deployment labels reflect current AWS validation reality
- checklist language no longer implies the old shadow-forwarding rollout is the active plan
- dashboard copy matches the current AWS-primary validation path
### 5. Investigate transient Cloudflare prod conversation-load failure
Status: deferred, non-blocking for AWS migration
Observed behavior:
- `POST https://api.mail.cedarcopilot.com/api/trpc/crm.loadConversation?batch=1` briefly
returned `500` / `503`
- UI showed `Failed to load conversation`
- console also showed `upsert mode requires content`
- refresh recovered the flow
Why it is deferred:
- the failure recovered on retry
- AWS prod CloudWatch did not show matching API or worker errors
- the observed issue happened on the legacy Cloudflare prod surface, not the AWS prod URL
Exit criteria:
- capture a live failing Cloudflare request with correlated server logs
- identify the exact server-side root cause
- fix it or explicitly retire that legacy surface
### 6. Automate AWS deploys
Status: PR readiness enabled, staging enabled, prod enabled
Current state:
- the branch model is now explicit:
- pull requests run AWS staging/prod build and synth readiness checks
- merge to `staging` deploys AWS staging automatically
- merge to `main` deploys raw AWS prod automatically
- the canonical prod domain flip remains a final separate cutover step
- AWS deploys still need the image publish/CDK/runtime rollout sequence, but that sequence is now
tied to the branch model instead of Cloudflare-era promotion rules
- the PR readiness workflow now lives in:
- `.github/workflows/aws-builds.yml`
- the live staging workflow is now:
- `.github/workflows/staging-deploy.yml`
- `aws/scripts/deploy-surface.sh`
- `aws/scripts/publish-frontend-assets.sh`
- staging deploy scope is change-aware:
- frontend-only publish for `apps/mail/**`
- service image publish + stack deploy for `apps/server/**` and shared runtime changes
- stack deploy for AWS CDK / deploy-script changes
- staging and prod ECS services now target rolling replacement rather than drain-first deploys:
- `minHealthyPercent = 100`
- `maxHealthyPercent = 200`
- with the current two-task floor, deploys keep healthy capacity online while replacement tasks
come up
Exit criteria:
- automated build, image push, and staging rollout pipeline for:
- `api-service`
- `chat-service`
- `worker-service`
- frontend S3 / CloudFront publish
- GitHub staging environment configured with:
- `AWS_STAGING_DEPLOY_ROLE_ARN`
- `CEDAR_AWS_STAGING_APP_DOMAIN`
- `CEDAR_AWS_STAGING_API_DOMAIN`
- `CEDAR_AWS_STAGING_CERTIFICATE_ARN`
- prod remains manual until staging automation has enough clean deploy history
- `worker-service`
- `chat-service`
- workflow is enabled after the prod domain flip
- branch mapping and rollback behavior are documented for operators
### 7. Finalize Cloudflare retirement scope
Status: in progress
Current state:
- AWS primary mode now hard-fails on legacy bridge access instead of silently proxying back
- the migration dashboard/runtime branch no longer treats Cloudflare as an active primary surface
- the remaining Cloudflare-related work is historical migration material and source-side backfill,
not a runtime fallback path for `staging` / `prod`
Exit criteria:
- clear statement of what remains historical migration source versus removed runtime dependency
- stale Cloudflare-primary documentation removed
- legacy runtime/config pieces deleted when no longer needed
### 8. Load-test and tune ECS autoscaling thresholds
Status: open
Observed behavior:
- live `us-east-1` AWS Application Auto Scaling is now configured for all three ECS services in
staging and prod
- live task sizes are currently (same for staging and prod):
- `api-service`: `0.5 vCPU / 1 GB`
- `chat-service`: `1 vCPU / 2 GB`
- `worker-service`: `1 vCPU / 4 GB`
- staging target-tracking policy:
- CPU target `50%`
- memory target `60%`
- scale out cooldown `60s`
- scale in cooldown `180s`
- prod target-tracking policy:
- CPU target `60%`
- memory target `70%`
- scale out cooldown `60s`
- scale in cooldown `180s`
- DB pool sizes: `5` per task for all services (staging and prod identical)
- reduced from previous 50/80 after discovering Supabase Supavisor (port 6543)
already handles connection multiplexing at the database level
- see AWS_DOCUMENTATION.md "Pool sizing" section for full rationale
Why it matters:
- autoscaling exists now, but the thresholds are still first-pass operational defaults
- Cedar still needs real manual validation and load observation to confirm the chosen floors,
ceilings, and cooldowns are correct
- request mix matters: read-heavy inbox traffic, chat streams, and worker queue bursts do not
stress the stack in the same way
- user-visible latency still depends heavily on ECS/Fargate task warmness and app-level data
loading, so cold starts are a real part of the observed response time on first request after a
restart or scale-out
Exit criteria:
- manual validation confirms the new floors are stable during real Cedar usage
- CloudWatch confirms scale-out and scale-in behavior looks reasonable under load
- any threshold changes discovered during validation are pushed back into the CDK config
### 9. Investigate AWS prod execution-stream poll-loop errors
Status: open
Observed behavior:
- during the live Chrome prod AWS smoke pass, the UI path itself worked, but CloudWatch still
showed recurring API log lines from `handleExecutionStream`
- repeated error:
- `Error in poll loop`
- `Database connection not initialized`
- this did not block the inbox/thread/calendar pass, but it is still a backend correctness risk
for streaming execution paths
Why it matters:
- the core AWS UI now works for the tested read/navigation flows, but this log pattern suggests a
still-fragile runtime path for agent execution streaming
- it may explain intermittent retry churn or aborted stream requests even when the main page loads
are successful
Exit criteria:
- reproduce the failing stream path intentionally on AWS staging first
- identify whether the missing initialization is request-scoped, worker-scoped, or race-related
- remove the recurring `Database connection not initialized` error from AWS prod API logs
### 10. Tune slow first-load routes seen in the live Chrome AWS prod pass
Status: open
Observed behavior:
- real Chrome timings from the `2026-04-01` AWS prod pass were acceptable for a smoke test but
still slower than ideal:
- inbox load: `~4.9s`
- thread open: `~3.1s`
- CRM tab: `~3.0s`
- Timeline tab: `~2.5s`
- Sent folder: `~3.1s`
- Calendar route: `~3.1s`
- notable slow API calls observed during the same pass included:
- `crm.listCompanies`: `~4.7s`
- `labels.list`: `~4.3s`
- `settings.get`: `~4.1s`
- `connections.list`: `~3.4s`
- `crm.searchConversationsMinimal`: `~3.2s`
- `agentExecutions.getAgentExecutions`: `~3.1s`
- `canvas.getCanvases`: `~3.1s`
- `chat.getMessages`: `~3.0s`
- `mail.listThreads`: `~2.6s`
Why it matters:
- the AWS stack is functional, but these are the routes users feel immediately on first load and
navigation
- the next stage of validation should not just be pass/fail; it should move these user-visible
latencies down
Progress on `2026-04-02`:
- the frontend now prefetches `mail.get` for likely-next thread opens:
- thread row hover
- thread click/open
- draft hover/click
- keyboard-focused row plus adjacent rows
- the inbox now also warms the first three visible threads shortly after list paint to reduce
first-open latency near the top of the mailbox
- BIMI avatar lookup was tightened after staging validation:
- malformed display-email strings no longer call `bimi.getByEmail`
- avatar BIMI resolution now uses the validated domain path instead
- known benign `Conversation not found or access denied` misses are no longer emitted through the
global frontend query logger when they happen in already-handled states
- this is intentionally an app-path optimization, not an AWS infra change:
- it reduces avoidable wait after the user signals intent
- it does not replace the need to keep `mail.listThreads`, labels, CRM fanout, and backend
latency under control
- staging was deployed first and passed live smoke checks before the same frontend bundle pattern
was promoted to prod
- a fresh authenticated timing pass is still required to measure the actual delta on inbox and
thread-open latency
Exit criteria:
- identify the worst offenders by route and backing dependency
- reduce the highest first-load paths materially from the current `~3-5s` range
- rerun the same Chrome AWS prod smoke pass and record improved timings
## Reversion & Rollback Procedures
This section documents how to revert at each layer of the AWS deployment. Procedures are ordered
from least disruptive (bad container deploy) to most disruptive (full DNS revert to Cloudflare).
### Level 1: Bad Container Deploy (ECS service rollback)
**Trigger**: A new container image crashes, fails health checks, or causes errors after deploy.
**Automatic recovery**: ECS deployment circuit breaker is enabled on all three services
(api, chat, worker) with `rollback: true`. If new tasks cannot reach a steady state, ECS
automatically reverts to the previous task definition revision. No manual action needed.
**Manual rollback** (if circuit breaker did not trigger or you want to force revert):
```bash
# 1. Find the previous healthy task definition revision
aws ecs describe-services \
--cluster aws-prod-api-cluster \
--services aws-prod-api-service \
--query 'services[0].deployments[?status==`ACTIVE`].taskDefinition' \
--output text
# 2. Force the service back to the previous revision
aws ecs update-service \
--cluster aws-prod-api-cluster \
--service aws-prod-api-service \
--task-definition <previous-task-definition-arn> \
--force-new-deployment
# 3. Wait for stability
aws ecs wait services-stable \
--cluster aws-prod-api-cluster \
--services aws-prod-api-service
# Repeat for chat-service and worker-service if affected.
```
**Validation after rollback**:
- `curl https://d28vdqmgxberim.cloudfront.net/health` → `{"status":"ok"}`
- Check CloudWatch logs for the service: no crash loops or repeated restarts
- If custom domain is live: `curl https://mail.cedarcopilot.com/health` → `{"status":"ok"}`
**Customer impact**: None if circuit breaker catches it. Minimal (seconds of 502s) if manual
rollback is needed. The `minHealthyPercent: 100%` setting ensures old tasks are never killed
before new ones are healthy.
---
### Level 2: Bad Frontend Deploy (S3/CloudFront rollback)
**Trigger**: Broken frontend bundle deployed to S3. App fails to load or shows white screen.
**Rollback steps**:
```bash
# 1. Identify the last good commit
git log --oneline apps/mail/ | head -5
# 2. Check out the last good commit and rebuild
git checkout <good-commit> -- apps/mail/
pnpm --filter @zero/mail build
# 3. Re-publish frontend assets
CEDAR_AWS_ENV=prod CEDAR_AWS_REGION=us-east-1 \
bash ./aws/scripts/publish-frontend-assets.sh prod
```
Alternatively, if S3 versioning is enabled:
```bash
# List versions of index.html
aws s3api list-object-versions \
--bucket <frontend-bucket-name> \
--prefix index.html \
--query 'Versions[0:3].[VersionId,LastModified]' \
--output table
# Restore previous version by copying it over
aws s3api copy-object \
--bucket <frontend-bucket-name> \
--key index.html \
--copy-source "<frontend-bucket-name>/index.html?versionId=<previous-version-id>"
# Invalidate CloudFront
aws cloudfront create-invalidation \
--distribution-id <distribution-id> \
--paths '/*'
```
**Customer impact**: Users on cached version continue working. New page loads see the broken
bundle until CloudFront invalidation propagates (~30s). Hashed asset files are immutable, so
only `index.html` and non-hashed files are affected.
---
### Level 3: Bad CDK Stack Deploy (CloudFormation rollback)
**Trigger**: CDK deploy changes infrastructure (ALB rules, security groups, task definitions)
in a way that breaks the service.
**Automatic recovery**: CloudFormation rolls back automatically if a stack update fails.
**Manual rollback** (if the stack update succeeded but the result is broken):
```bash
# Option A: Redeploy the previous commit
git checkout <previous-good-commit>
CEDAR_AWS_ENV=prod CEDAR_AWS_REGION=us-east-1 \
CEDAR_AWS_DEPLOY_STACK=true \
CEDAR_AWS_DEPLOY_FRONTEND=false \
CEDAR_AWS_PUBLISH_API=false \
CEDAR_AWS_PUBLISH_CHAT=false \
CEDAR_AWS_PUBLISH_WORKER=false \
bash ./aws/scripts/deploy-surface.sh prod
# Option B: Roll back via CloudFormation console
# AWS Console → CloudFormation → CedarAwsProdEnvironmentStack → Roll back
```
**Customer impact**: Depends on what changed. ALB rule changes can cause immediate routing
failures. Security group changes can block traffic. Rolling back the stack typically takes
2-5 minutes.
---
### Level 4: DNS Revert to Cloudflare (full cutover rollback)
**Trigger**: AWS prod is fundamentally broken and cannot serve traffic. Need to route customers
back to Cloudflare immediately.
**Prerequisites**:
- Cloudflare Workers (`cedar-app`, `cedar-api`) must still be deployed and functional
- Cloudflare Workers custom domains must not have been deleted yet
- If Workers custom domains were detached, they must be re-attached first
**Rollback steps**:
1. **Re-attach Cloudflare Workers custom domains** (if previously detached):
- Cloudflare Dashboard → Workers & Pages → `cedar-app` → Settings → Domains & Routes
- Add custom domain: `mail.cedarcopilot.com`
- Cloudflare Dashboard → Workers & Pages → `cedar-api` → Settings → Domains & Routes
- Add custom domain: `api.mail.cedarcopilot.com`
2. **Update DNS records in Cloudflare** (if they were changed to point to AWS):
- Change `mail.cedarcopilot.com` CNAME back to Cloudflare-proxied (orange cloud)
- Change `api.mail.cedarcopilot.com` CNAME back to Cloudflare-proxied (orange cloud)
- Or: if using Workers custom domains, Cloudflare manages DNS automatically
3. **Remove custom domain aliases from AWS CloudFront** (to avoid certificate conflicts):
```bash
# Get current config
aws cloudfront get-distribution-config --id EKWAET474CHUG > /tmp/cf-prod.json
ETAG=$(jq -r '.ETag' /tmp/cf-prod.json)
# Remove aliases
jq '.DistributionConfig | .Aliases = {"Quantity": 0, "Items": []}' /tmp/cf-prod.json > /tmp/cf-prod-update.json
aws cloudfront update-distribution \
--id EKWAET474CHUG \
--if-match "$ETAG" \
--distribution-config file:///tmp/cf-prod-update.json
```
4. **Validate Cloudflare is serving**:
```bash
curl -sI https://mail.cedarcopilot.com/ | grep server
# Expected: server: cloudflare
curl -s https://api.mail.cedarcopilot.com/api/auth/get-session | head -1
# Expected: JSON response (not HTML)
```
**Customer impact**: DNS propagation takes 30s-5min depending on TTL. During propagation,
some users hit AWS (broken) and some hit Cloudflare (working). After propagation, all traffic
goes to Cloudflare.
**Important caveats**:
- Better Auth sessions are shared (same Supabase DB) — users do NOT need to re-login
- Any data written to AWS-only stores (S3 thread artifacts, DynamoDB) during the AWS period
will not be visible on Cloudflare unless the bridge fallback code is still active
- Cloudflare Workers must be on a recent deploy that is compatible with the current database
schema. If the DB schema has changed since the last Cloudflare deploy, reverting DNS alone
may not be sufficient — the Workers code may also need updating
---
### Level 5: Point of No Return
After the following actions, reverting to Cloudflare is no longer possible without significant
effort:
- [ ] Cloudflare Workers (`cedar-app`, `cedar-api`) are deleted
- [ ] Cloudflare KV namespaces are purged
- [ ] Cloudflare R2 buckets are emptied (thread artifact source of truth)
- [ ] Database schema migrations are deployed that are incompatible with Cloudflare Worker code
- [ ] Cloudflare account billing/plan is downgraded
**Recommendation**: Keep Cloudflare Workers deployed and functional for at least 2 weeks after
the prod DNS cutover. Do not delete any Cloudflare resources until the AWS prod surface has been
stable under real customer traffic for that period.
---
### Rollback Decision Matrix
| Symptom | Level | Action |
| -------------------------------------- | ----------- | ----------------------------------------------- |
| New container crashing after deploy | 1 | Wait for circuit breaker or manual ECS rollback |
| App loads but shows white screen | 2 | Re-publish previous frontend to S3 |
| API routes returning 503/502 | 1 or 3 | Check ECS task health first, then CDK stack |
| All requests failing after CDK deploy | 3 | CloudFormation rollback |
| Sustained failures, unknown root cause | 4 | Revert DNS to Cloudflare |
| Auth/sessions broken | Check DB | Likely schema or secret mismatch, not infra |
| Thread data missing | Check S3/R2 | Bridge fallback may need re-enabling |
## Verification
When closing an item:
1. Validate on AWS staging first.
2. Confirm prod only after staging is clean.
3. Use CloudWatch for AWS-runtime verification.
4. Update the live dashboard if operator-visible state changes.
## Step Functions Evolution
This section documents the evolution of Cedar's Step Functions architecture for historical
context. The authoritative current-state documentation lives in
[AWS_DOCUMENTATION.md](./AWS_DOCUMENTATION.md).
### Phase 1: workflow-runner + cron-orchestrator (2 state machines)
The initial Step Functions integration used two state machines:
1. **workflow-runner**: dispatched workflow execution to worker-service via SQS with
`waitForTaskToken`. The worker ran the entire workflow in-process as a single unit.
All `step.do()` calls were plain function calls within the worker process.
2. **cron-orchestrator**: replaced the monolithic hourly cron handler with 14 parallel
branches, each using the SQS + waitForTaskToken pattern via `cron-task-queue`.
### Phase 2: per-workflow multi-step state machines (8 state machines)
Added in PR #1061 (`c032c64f1`, `effc51080`). Each of the 6 Cedar workflow classes now has
its own dedicated Step Functions state machine where every `step.do()` is a separate SF state:
- `{prefix}-wf-calendar-sync` (5 steps)
- `{prefix}-wf-cron-sync` (2 steps)
- `{prefix}-wf-sync-threads` (2 steps)
- `{prefix}-wf-sync-threads-coordinator` (3 steps + pagination loop)
- `{prefix}-wf-conversation-refresh` (conditional AOP + search branches with Map)
- `{prefix}-wf-conversation-sync` (12+ steps with conditional phases and Map iterators)
Key architectural decisions:
- Each step sends an SQS message to `workflow-step-queue` with a `taskToken`
- The worker polls the queue, executes the step logic, and calls `SendTaskSuccess`/`SendTaskFailure`
- Map states (label sync, CRM batch, etc.) use bounded `maxConcurrency` (3-10)
- Routing is per-workflow via `AWS_SFN_WF_*` env vars in `apps/server/src/env.ts`
- Removing an ARN env var reverts that single workflow to in-process execution
- The old workflow-runner SM still exists for backward compatibility
Fixes applied post-merge (PR #1074):
- `f6eec4bca`: SF states read `$.workflowId` not `$.workflowRunId`
- `b8d5c1689`: workflow-step-queue poller blocked by shared `cronPollerStarted` flag
- `081cfbc37`: Map iterator steps receive item data via `itemSelector`
## Prod Cutover Record
As of `2026-04-06`, the AWS migration is complete for both staging and prod.
- Prod custom domains now resolve to the AWS prod CloudFront distribution (`EKWAET474CHUG`)
- Both `mail.cedarcopilot.com` and `api.mail.cedarcopilot.com` serve from AWS
- Cloudflare Workers builds still trigger on PRs but are vestigial
### What remains after full cutover
- Remove vestigial Cloudflare Workers build triggers from GitHub
- Complete removal of Cloudflare-era code paths from the repo
- Complete the AWS object/snapshot backfill for older thread artifacts
- Remove temporary migration-only allowlists
- Continue performance tuning on slow first-load routes