AWS_ARCHITECTURE.md29.0 KBView on GitHub # Cedar AWS Architecture
Infrastructure reference for the AWS deployment. For mail flow walkthroughs, see `AWS_MAIL_WORKFLOWS.md`.
---
## Deployment Overview
```
CLIENTS
Browser / Electron Desktop App
│
HTTPS
│
┌───────────▼────────────┐
│ AWS CLOUDFRONT │
│ mail.cedarcopilot.com │
│ api.cedarcopilot.com │
└───────────┬────────────┘
│
┌───────────┴────────────┐
│ │
▼ ▼
┌────────────────┐ ┌─────────────────┐
│ S3 (Frontend) │ │ AWS ALB │
│ index.html │ │ (port 80/443) │
│ hashed assets │ └────────┬─────────┘
└────────────────┘ │
┌──────────┼───────────┐
│ │ │
▼ ▼ ▼
┌──────────────┐ ┌──────────┐ ┌──────────────┐
│ API Service │ │ Chat │ │ Worker │
│ ECS Fargate │ │ Service │ │ Service │
│ port 8787 │ │ Fargate │ │ Fargate │
│ Hono + tRPC │ │ port │ │ No HTTP │
└──────┬───────┘ │ 8789 │ └──────┬───────┘
│ └──────────┘ │
│ │ polls
│ ┌────────▼────────┐
│ │ AWS SQS │
│ │ thread-queue │
│ │ subscribe-queue │
│ │ send-email-queue│
│ │ send-remind- │
│ │ queue │
│ │ agent-action- │
│ │ queue │
│ └─────────────────┘
│
┌──────────────┼─────────────────────────┐
│ │ │
▼ ▼ ▼
┌──────────────┐ ┌──────────────────┐ ┌────────────────────┐
│ PostgreSQL │ │ AWS DynamoDB │ │ AWS S3 │
│ (Supabase) │ │ Generic KV │ │ │
└──────────────┘ └──────────────────┘ └────────────────────┘
EMAIL PROVIDERS
Gmail API / Microsoft Graph
```
---
## ECS Services
Three Fargate services run in every environment (staging + prod). All are in the same ECS cluster.
| Service | Port | Role |
| -------------- | ---- | --------------------------------------- |
| API Service | 8787 | Handles all HTTP — tRPC, auth, webhooks |
| Chat Service | 8789 | Streams AI responses via SSE |
| Worker Service | none | Polls SQS queues, runs scheduled jobs |
**Scaling:**
- API: 2 min → 8 max tasks
- Chat: 2 min → 6 max tasks
- Worker: 2 min → 6 max tasks
**Deploys** use rolling updates — `minHealthyPercent: 100`, `maxHealthyPercent: 200` — so tasks are never fully replaced at once.
**Health checks:** Each service exposes `/health` and `/ready` endpoints checked by the ALB target group.
**Entry points:**
- API: `apps/server/src/runtime/api-entry.ts`
- Worker: `apps/server/src/runtime/worker-entry.ts`
---
## CloudFront + S3 (Frontend)
The React frontend is served from S3 behind CloudFront.
**Deploy order matters:**
1. Upload new hashed asset files to S3 first
2. Upload `index.html` only after assets are uploaded
3. Invalidate CloudFront so new HTML is served promptlyh
Why: old `index.html` references old hashed files that still exist. New `index.html` references new hashed files that must already exist before it goes live. This prevents a window where the page loads but its JS/CSS return 404.
Hashed assets are cached indefinitely. `index.html` is aggressively revalidated on deploy boundaries.
---
## Storage
### PostgreSQL
Primary relational store. Accessed via `DATABASE_URL` (Supabase-hosted).
Connection pool per ECS task:
- 16–24 connections max
- 600s connection TTL
- Configurable via `CEDAR_DB_POOL_MAX`, `CEDAR_DB_POOL_IDLE_TIMEOUT_SEC`
Holds:
- User accounts, auth sessions
- Email connections (credentials, token state)
- Thread metadata: labels, read/unread, timestamps, draft flags
- CRM records: conversations, contacts, companies, email events
- Email tracking records (opens, clicks)
- Agent execution history
Schemas: `apps/server/src/db/`
### S3 (Object Storage)
Replaces Cloudflare R2. Stores large JSON blobs — primarily full thread snapshots.
| Binding name | Key format | Contents |
| -------------------- | --------------------------------- | ---------------------------------------- |
| `THREADS_BUCKET` | `{connectionId}/{threadId}.json` | Full thread — all messages, decoded HTML |
| `MEETINGS_BUCKET` | `{connectionId}/{meetingId}.json` | Calendar meetings + transcripts |
| `CRM_CONTENT_BUCKET` | varies | External CRM data snapshots |
| `SLACK_BUCKET` | varies | Slack conversation data |
Adapter: `createR2BucketProxy()` in `apps/server/src/container/aws-runtime-bindings.ts`. Wraps the AWS S3 SDK behind the old Cloudflare `R2Bucket` interface so call sites didn't need to change. Includes a write-through cache so a `put()` followed by `get()` in the same request doesn't round-trip to S3.
### DynamoDB (Key-Value Store)
#### What is DynamoDB?
Amazon DynamoDB is a fully managed NoSQL database. Unlike a relational database (rows and columns with a schema), DynamoDB stores **items** — schemaless JSON-like documents — in **tables**. Every item must have a primary key; everything else is optional.
**Key concepts:**
- **Partition key** — determines which physical shard holds the item. Queries must include the exact partition key (no range scans across partitions).
- **Sort key** — optional second key component. Enables prefix queries and range scans within a partition (`begins_with`, `between`).
- **TTL** — a native expiry field (`expiresAt` epoch timestamp). DynamoDB deletes expired items automatically, typically within 48 hours.
- **Conditional writes** — `PutItem` / `UpdateItem` with a `ConditionExpression` atomically succeeds only if the condition holds. Cedar uses this for its distributed cron lock (`attribute_not_exists(key)`).
- **On-demand capacity** — DynamoDB can scale read/write throughput automatically with no pre-provisioned capacity units.
**Pros:**
- Fully managed, serverless — no cluster to operate
- Single-digit millisecond latency at any scale
- Native TTL removes stale data without application logic
- Conditional writes enable atomic operations without transactions
- On-demand mode means no capacity planning for low-traffic tables
**Cons:**
- No joins; relational queries require multiple round-trips or denormalized data
- 400 KB item size limit
- Partition key design matters — a hot partition key can throttle the entire table
- No schema enforcement — application must validate its own data shape
- Pricing model can surprise at very high read/write volumes
Replaces Cloudflare KV. Used for ephemeral, per-connection state that doesn't need relational queries — sync cursors, in-flight markers, scheduled job payloads, and cached counts.
#### How it differs from Cloudflare KV
Cloudflare KV was a flat `string → string` store globally replicated across Cloudflare's edge network. DynamoDB is a NoSQL document database hosted in a single AWS region. The practical differences for Cedar:
| | Cloudflare KV | AWS DynamoDB |
| ------------------- | ------------------------------------------------ | ------------------------------------------------ |
| **Data model** | Flat string→string | Structured items with multiple typed fields |
| **Keys** | Single string key | Composite key=[redacted] key + sort key |
| **Consistency** | Eventually consistent globally (edge-replicated) | Eventually or strongly consistent, single region |
| **TTL** | Native per-key TTL | Native via `expiresAt` epoch field |
| **Prefix queries** | `list({ prefix })` supported | `begins_with()` on sort key supported |
| **Throughput** | Unlimited (billed per request) | Provisioned or on-demand capacity units |
| **Latency** | ~1ms at edge (read from nearest PoP) | ~1–5ms from same AWS region |
| **Item size limit** | 25 MB per value | 400 KB per item |
For Cedar's use cases (storing a historyId string, a snooze timestamp, a sync cursor), none of these differences matter in practice. The values are small, low-frequency reads/writes, and single-region consistency is fine.
#### Why Cedar still looks like it's using KV
The adapter (`createKVProxy()` in `aws-runtime-bindings.ts`) wraps `DynamoDBDocumentClient` behind the exact same `KVNamespace` interface that Cloudflare KV used — exposing only `get`, `put`, `delete`, and `list`. None of the DynamoDB-specific capabilities (secondary indexes, transactions, streams, conditional writes) are used through this adapter. The application code never knows it's talking to DynamoDB.
The one place DynamoDB's extra capabilities _are_ used directly — bypassing the adapter — is the Worker's distributed lock (`worker-entry.ts:226`), which uses a raw `PutItem` with a `ConditionExpression: attribute_not_exists(key)` to atomically claim the scheduled cron slot. That conditional write is something Cloudflare KV couldn't do.
#### Table structure
Single table per environment, named `{prefix}-generic-kv` (e.g. `aws-staging-generic-kv`, `aws-prod-generic-kv`). Composite key: `{ namespace (partition), key (sort) }`.
19 namespaces are active. Grouped by domain:
---
##### Email Scheduling
**`pending_emails_status`**
- Key=[redacted]
- Value: `"pending"` or `"cancelled"`
- Purpose: Tracks whether a scheduled email is still waiting to send or was cancelled (idempotency guard)
- TTL: delay + 1 hour buffer
**`pending_emails_payload`**
- Key=[redacted]
- Value: Full outgoing email — to/cc/bcc, subject, HTML body, attachments, tracking options
- Purpose: Stores the actual email content so the Worker can send it when the scheduled time arrives
- TTL: Same as status
**`scheduled_emails`**
- Key=[redacted]
- Value: `{ messageId, connectionId, sendAt }`
- Purpose: Index of **long-term** scheduled emails (≥12 hours out). Short-term emails go directly into SQS.
- TTL: Up to 1 year
---
##### Snooze & Reminders
**`snoozed_emails`**
- Key=[redacted]
- Value: ISO wake-up timestamp
- Purpose: Keeps snoozed threads suppressed until wake time. Mail sync coordinator checks this before surfacing a thread.
**`remind_emails`**
- Key=[redacted]
- Value: `{ remindId, remindAt, threadId, connectionId }`
- Purpose: Reminder index — cron promotes long-term reminders to SQS when their time comes
- TTL: ~30 days
**`remind_email_payload`**
- Key=[redacted]
- Value: Full reminder context (thread metadata, user, connection, timing)
- Purpose: What to send when the reminder fires
**`remind_email_status`**
- Key=[redacted]
- Value: `"pending"` | `"cancelled"` | `"sent"`
- Purpose: Idempotency guard — prevents double-execution if a Worker retries
- TTL: 30 days (60s for cancelled)
---
##### Agent Actions
**`scheduled_agent_actions`**
- Key=[redacted]
- Value: `{ runId, userId, triggerAt }`
- Purpose: Index of long-term (≥12 hours) scheduled agent executions. PostgreSQL is source of truth; this is the cron-promotable index.
- TTL: Up to 1 year
---
##### Initial Sync State
**`initial_sync_state`**
- Key 1: `run:${workflowRunId}` — full `InitialSyncProgress` object (phase 1/2 status, stats, errors)
- Key 2: `phase2:${userId}:${conversationId}` — `{ aopId, finalizedAt, eventCount }`
- Purpose: Tracks two-phase initial sync workflow progress; the phase2 keys prevent duplicate finalization
- TTL: 7 days (progress), 30 days (phase2 idempotency)
---
##### Gmail
**`gmail_history_id`**
- Key=[redacted]
- Value: Gmail `historyId` string
- Purpose: Incremental sync checkpoint — tells Gmail to send only changes since this point
**`gmail_processing_threads`**
- Key=[redacted]
- Value: Timestamp (ms as string)
- Purpose: 120s rate-limiting flag per label — prevents thrashing the same label during rapid changes
- TTL: 120 seconds
**`gmail_sub_age`**
- Key=[redacted]
- Value: ISO timestamp (when the subscription was created)
- Purpose: Tracks age of Gmail push subscriptions for renewal rotation logic
**`subscribed_accounts`**
- Key=[redacted]
- Value: ISO timestamp
- Purpose: Registry of connections with active Google pub/sub subscriptions
---
##### Labels
**`connection_labels`**
- Key=[redacted]
- Value: JSON array of label definitions
- Purpose: Cached default/custom labels per connection. Populated on first subscription setup.
---
##### Calendar
**`calendar_sync_tokens`**
- Key=[redacted]
- Value: Google Calendar `syncToken` (opaque string)
- Purpose: Incremental calendar sync checkpoint. Present → fetch only changes. Absent or 410 expired → full sync (7 days back, 30 days forward).
**`calendar_watch_channels`**
- Key 1: `${channelId}` → connection metadata
- Key 2: `connection:${connectionId}` → channelId
- Purpose: Bidirectional mapping between Google Calendar push channels and Cedar connections. Forward lookup for incoming webhooks; reverse lookup for renewal and cleanup.
---
##### Email Tracking
**`email_tracking`**
- Key=[redacted] (UUID embedded in the tracking pixel URL)
- Value: `{ userId, connectionId, gmailMessageId, threadId, subject, toEmails, conversationId, sentAt, senderIp }`
- Purpose: Fast sub-5ms lookup when a tracking pixel fires — avoids hitting PostgreSQL on every open/click event
- TTL: 90 days
---
##### Chat Monitor
**`chat_monitor_buckets`**
- Key=[redacted]
- Value: `{ threadTs }` — Slack thread timestamp
- Purpose: Groups chat notifications into 2-hour Slack thread buckets. If bucket exists → reply to existing thread. After TTL expires → next message starts a new thread.
- TTL: 2 hours
---
##### Unused
**`prompts_storage`** — defined in `env.ts` but not referenced in the codebase. Placeholder for future prompt template caching.
---
#### Two patterns worth knowing
**Two-tier scheduling:** Events <12 hours out go directly to SQS. Events ≥12 hours out sit in KV (`scheduled_emails`, `scheduled_agent_actions`, `remind_emails`) and a cron job promotes them to SQS when their time comes.
**Idempotency via status keys:** `pending_emails_status`, `remind_email_status`, and `phase2:*` keys in `initial_sync_state` exist purely to prevent double-execution — if a Worker crashes and retries, the status key stops it from firing twice.
---
## SQS Queues
### What is SQS?
Amazon Simple Queue Service (SQS) is a fully managed message queuing service. It lets components of a distributed system communicate asynchronously: a **producer** sends a message to a queue, and a **consumer** reads and processes it independently — without either side needing to know about the other or be online at the same time.
**Two queue types:**
| Type | Ordering | Delivery | Throughput |
| -------- | ---------------- | ------------------------- | ---------------- |
| Standard | Best-effort | At-least-once (may duplicate) | Nearly unlimited |
| FIFO | Strict per group | Exactly-once | Up to 3,000 msg/s with batching |
Cedar uses **FIFO queues** for email connections so that messages for the same connection are processed in order and never duplicated (identified by `MessageGroupId = connectionId`).
**How polling works:**
1. Consumer calls `ReceiveMessage` — SQS returns up to 10 messages
2. Messages become **invisible** to other consumers for the visibility timeout window (30s in Cedar)
3. Consumer processes the messages
4. Consumer calls `DeleteMessage` to permanently remove them
5. If processing fails or times out, messages reappear for retry
**Long polling** (Cedar uses 20s) reduces empty responses and cost vs. short polling — the request stays open until messages arrive or the wait expires.
**Pros:**
- Decouples slow background work from the request path
- Built-in retry with configurable visibility timeout
- Dead-letter queues automatically catch repeatedly-failing messages
- Scales horizontally — add more Worker tasks without changing producers
- Fully managed (no infrastructure to maintain)
**Cons:**
- At-least-once delivery on standard queues requires idempotent handlers
- Maximum message size is 256 KB (Cedar stores large payloads in S3/DynamoDB, enqueues only a reference)
- Polling latency: not truly push (though long polling gets close)
- FIFO throughput cap can be a bottleneck at very high scale
### Cedar's Queues
Five queues. The Worker Service polls all of them in parallel, receiving up to 10 messages per poll with a 20s long-poll wait and 30s visibility timeout.
| Queue env var | Purpose |
| ------------------------------------------ | ---------------------------------- |
| `AWS_SQS_THREAD_QUEUE_URL` | Background thread sync jobs |
| `AWS_SQS_SUBSCRIBE_QUEUE_URL` | Email subscription changes |
| `AWS_SQS_SEND_EMAIL_QUEUE_URL` | Outbound email delivery + tracking |
| `AWS_SQS_SEND_REMIND_QUEUE_URL` | Reminder emails |
| `AWS_SQS_SCHEDULED_AGENT_ACTION_QUEUE_URL` | AI agent step execution |
| `AWS_SQS_CHANNEL_SYNC_QUEUE_URL` | Slack/LinkedIn/WhatsApp buffer drain wake-up (FIFO, group = `container_key`). **Optional** — when unset the binding is absent and the drain falls back to the reconcile cron. |
FIFO queues are auto-detected by `.fifo` suffix. The adapter adds `MessageGroupId` and `MessageDeduplicationId` automatically.
Adapter: `createQueueProxy()` in `aws-runtime-bindings.ts`.
---
## AWS Step Functions
### What is Step Functions?
AWS Step Functions is a serverless workflow orchestration service. You define a **state machine** — a directed graph of states (tasks, choices, waits, parallel branches, map iterations) — and Step Functions manages execution: tracking where in the workflow you are, retrying on failure, timing out stuck steps, and persisting state across long-running executions.
Workflows survive infrastructure restarts. A state machine execution that pauses for 4 hours waiting on a task doesn't consume a thread or process — Step Functions simply resumes it when the signal arrives.
**Key state types used in Cedar:**
| State type | What it does |
| -------------- | ---------------------------------------------------------- |
| `Task` | Sends work somewhere and waits for completion |
| `Choice` | Conditional branch — like an `if/else` |
| `Map` | Fan-out — runs the same step over a list, with concurrency |
| `Pass` | No-op join/reshape state (used for wiring branches) |
**Pros:**
- Long-running workflows without holding a thread open
- Built-in retry, timeout, and error handling per step
- Visual execution history in the AWS console
- Map state handles parallelism without custom worker pooling code
- No infrastructure to manage
**Cons:**
- Each state transition costs money (Standard Workflows price per transition)
- State machine definitions can become verbose for complex flows
- Max execution history is 25,000 events — very long or deeply nested workflows must be broken up
- Debugging failed executions requires the AWS console or CloudWatch
### How Cedar Uses Step Functions
Cedar uses Step Functions to orchestrate **multi-step sync workflows** — sequences of operations that would otherwise require complex coordination between queue messages, retry logic, and pagination loops.
The core pattern is **WAIT_FOR_TASK_TOKEN over SQS**:
```
Step Functions SQS Worker Service
────────────── ─── ──────────────
Execute step ──sends msg──▶ step-queue ──Worker polls──▶ Execute handler
(with taskToken embedded) ↓
◀──SendTaskSuccess(taskToken, result)───────────
Advance to next state
```
1. Step Functions sends a message to the SQS `step-queue` containing the `taskToken`, step name, workflow ID, and parameters
2. The Worker Service polls the step-queue and executes the appropriate handler
3. When the handler finishes, it calls `SendTaskSuccess` (or `SendTaskFailure`) with the task token
4. Step Functions receives the callback and advances to the next state, passing the result as output
This means the Worker Service executes the actual logic; Step Functions provides the orchestration shell (ordering, branching, parallel fan-out, timeouts, retries).
### State Machines
Six state machines defined in `aws/lib/stacks/workflow-state-machines.ts`:
| State Machine | Steps | Timeout |
| -------------------------- | ------------------------------------------------------------------------ | ------- |
| `wf-calendar-sync` | setup → get-sync-token → fetch-events → process-events → save-sync-token | 2h |
| `wf-cron-sync` | setup → checkAndSync | 1h |
| `wf-sync-threads` | setup → process-page | 1h |
| `wf-sync-threads-coordinator` | setup → process-page (loop on `nextPageToken`) → update-history-id | 4h |
| `wf-conversation-refresh` | conditional AOP branch + conditional search-terms branch, each with a parallel Map | 2h |
| `wf-conversation-sync` | setup → sent → inbox → labels (Map) → search (Map) → meetings (Map) → Slack → CRM (Map) → cleanup → finalize (Map) | 4h |
**Map states** fan out over lists with a configurable concurrency limit (e.g., 3–10 parallel branches). This replaces what would have been manual worker pool management.
**Choice states** make each sync phase optional at runtime — if a feature flag is off, that branch is skipped via a `Pass` state and the workflow continues.
**Pagination loop** (SyncThreadsCoordinator): after `process-page`, if the step result contains a `nextPageToken`, a `Choice` state loops back to `process-page` with the token injected into `params`. Step Functions handles the loop without any recursion or re-enqueueing.
### Cloudflare Workers → ECS + Step Functions
In the original Cloudflare architecture, Workers handled both compute and coordination:
- A Worker received a queue message and ran the full sync inline
- Pagination was done with recursive queue messages or internal loops
- There was no persistent execution state between steps
In AWS, these responsibilities are split:
| Concern | Cloudflare | AWS |
| ------------------------ | --------------------------------------- | ------------------------------------ |
| Compute (step execution) | Cloudflare Worker | ECS Fargate Worker Service |
| Orchestration / ordering | Queue messages + application logic | AWS Step Functions state machine |
| Parallel fan-out | Multiple queue messages | Step Functions Map state |
| Conditional branching | Application `if/else` | Step Functions Choice state |
| Pagination loop | Re-enqueue with cursor | Step Functions loop via Choice |
| Execution visibility | None | Step Functions console + CloudWatch |
---
## Worker Service Internals
**File:** `apps/server/src/runtime/worker-entry.ts`
The Worker Service runs two loops:
### Queue pollers (`startSqsPollers()`)
One async polling loop per queue. Each loop:
1. `ReceiveMessageCommand` — up to 10 messages, 20s wait
2. Converts SQS messages to the legacy `MessageBatch` format
3. Calls `handlers.queue(batch, env, ctx)` — the shared queue handler from `main.ts`
4. `DeleteMessageBatchCommand` — removes processed messages
5. Retries with 5s backoff on error
### Scheduled handler (`runScheduledCycle()`)
Runs every 60 seconds. Replaces Cloudflare's `scheduled()` entrypoint.
Because multiple Worker tasks run in parallel, a **DynamoDB distributed lock** prevents duplicate execution:
- Attempt `PutItem` with condition `attribute_not_exists(key)`
- If the put succeeds: this instance runs the scheduled handler
- If the put fails: another instance already holds the lock — skip this cycle
- Lock TTL: 600s
Calls `handlers.scheduled(controller, env)` with `cron: '* * * * *'`.
---
## Runtime Permission Guards
`apps/server/src/lib/runtime-permissions.ts` controls which side effects are allowed per environment:
| Flag | Controls |
| ------------------------------------- | ------------------------- |
| `CEDAR_DATABASE_WRITE_ALLOWED` | PostgreSQL mutations |
| `CEDAR_AUTH_DATABASE_WRITE_ALLOWED` | Auth table mutations |
| `CEDAR_PROVIDER_SIDE_EFFECTS_ALLOWED` | Gmail / Outlook API calls |
| `CEDAR_NOTIFICATIONS_ALLOWED` | Push notifications |
| `CEDAR_OBJECT_STORE_WRITE_ALLOWED` | S3 writes |
| `CEDAR_QUEUE_PUBLISH_ALLOWED` | SQS enqueues |
Used to prevent unintended mutations in read-only or batch scenarios.
---
## Local Development Shims
When `CEDAR_ENV === 'local'` or `NODE_ENV === 'test'`, AWS adapters fall back to in-memory implementations:
- KV → `Map` (in-process)
- Queues → `Array` (in-process)
- S3 → in-memory cache Map
This allows local dev and tests without an AWS account.
---
## Cloudflare → AWS Mapping
| Cloudflare | AWS equivalent |
| --------------------------------------- | ---------------------------------------------------- |
| Durable Object (compute) | ECS Fargate Worker Service |
| Durable Object (SQLite) | PostgreSQL + DynamoDB |
| R2 | S3 |
| KV Namespace | DynamoDB generic KV table |
| Cloudflare Queue | SQS |
| Cron Triggers | Worker internal ticker + DynamoDB lock |
| Workers (edge compute) | ECS Fargate (API + Chat + Worker) |
| Multi-step workflow orchestration | AWS Step Functions state machines |
| Recursive queue messages for pagination | Step Functions Choice state loop |
| Manual fan-out via multiple enqueues | Step Functions Map state with concurrency limit |
| Wrangler local dev | In-memory Map/Array shims |