CedarCopilot/cedar-mail

# Libra Runtime Setup

This repository is configured for Libra runtime analysis through OpenTelemetry traces.

## What Libra reads

- OTEL traces from `cedar-api`, `cedar-worker`, and `cedar-chat`
- Auto-instrumented spans for every HTTP request in/out (`node:http` + `undici`),
  Postgres query (`pg`), Redis call (`ioredis`), AWS SDK call (`@aws-sdk/*`),
  and GraphQL/gRPC surface — produced by `@opentelemetry/auto-instrumentations-node`
  registered at module load in `otel-init.ts`
- Postgres.js (`postgres` package) query spans via `traceSql()` wrapper in
  `lib/logging/pg-tracing.ts` — no community OTel package exists for postgres.js
- Every tRPC procedure as its own `trpc.<path>` span with `rpc.method`,
  `rpc.service`, and user context attributes (`lib/logging/libra-context.ts`)
- Mastra agent / workflow / LLM-call spans via `@mastra/otel-bridge`
  (configured in `mastra/index.ts`) with `gen_ai.*` attributes
- SQS messages carry W3C `traceparent` in `MessageAttributes` — api → worker
  flows stay on a single trace
- Existing Cedar spans from `initTracing()` and `safeStartSpan()` across mail, CRM,
  agent execution, queue, and provider code paths (nest under the auto-instrumented
  root spans via OTel context propagation)
- Resource attributes that identify repo, service, cloud region / provider /
  account id, container id, deployment environment, PR number, and commit SHA
- Business-context attributes on root spans: `user.id`, `user.email_domain`,
  `tenant.id`, `cedar.trace_id`, `cedar.request_id`
- Structured JSON error logs to stdout (→ CloudWatch) with `error.type`,
  `error.code`, `error.stack`, `trace_id`, `span_id`, and `libra.pr_number` —
  Libra's CloudWatch Insights queries can cluster without regex-matching
  free-text messages

## Runtime env contract

### Required

| Env var | Purpose |
|---|---|
| `LIBRA_API_KEY` | Bearer token for the OTLP export to Libra. Setting just this one variable is enough — Cedar uses the default endpoint, auto-detects PR number and commit SHA from the CI environment, and derives deployment metadata from the runtime. |

### Optional overrides

Use these only when the defaults do not fit.

| Env var | Default | Purpose |
|---|---|---|
| `LIBRA_OTEL_TRACES_ENDPOINT` | `https://api.libra.exla.ai/v1/traces` | Override the Libra ingest URL (self-hosted Libra, region pinning, staging targets). |
| `LIBRA_PR_NUMBER` | auto-detected | Override the PR number `otel-init.ts` discovers from `GITHUB_PR_NUMBER`, `VERCEL_GIT_PULL_REQUEST_ID`, `CIRCLE_PULL_REQUEST`, `CI_MERGE_REQUEST_IID`, `BITBUCKET_PR_ID`. |
| `LIBRA_COMMIT_SHA` | auto-detected | Override the commit SHA discovered from `GITHUB_SHA`, `VERCEL_GIT_COMMIT_SHA`, `CIRCLE_SHA1`, `CI_COMMIT_SHA`, `BITBUCKET_COMMIT`, `SOURCE_VERSION`. |
| `LIBRA_SERVICE_VERSION` | *(omitted)* | Libra-scoped version tag — attached as `libra.service_version` resource attribute. Does not override the shared `service.version` used by other tracing destinations (e.g. Axiom). |
| `CEDAR_SERVICE_VERSION` | `1.0.0` | Shared `service.version` resource attribute applied to every destination. |

### How to enable Libra for a new customer

1. Add `LIBRA_API_KEY=[redacted] to the runtime secret store.
2. Redeploy. That is the entire integration.

### Firehose model

Cedar emits **100 % of traces** to Libra. Sampling, filtering, aggregation,
replay-corpus construction, error grouping, and PR attribution all happen
on Libra's receive side using the trace firehose plus the CloudWatch log
stream that Libra already reads. Cedar intentionally does not expose
per-route sampling, fixture-capture, or kill-switch knobs — every extra
knob on Cedar would couple Libra's product iteration to a Cedar deploy,
which defeats the point.

### How to add the key to ECS containers

Cedar services receive secrets from a single AWS Secrets Manager JSON
whose name is `config.secretName` (printed as `RuntimeSecretName` on
`cdk deploy`). Add one key=[redacted]
{
  "LIBRA_API_KEY": "<your-libra-api-key>"
}
```

The CDK runtime contract already allowlists `LIBRA_API_KEY`, so the next
`cdk deploy` wires it into all three ECS task definitions — no IAM
changes required beyond the existing Secrets Manager `GetSecretValue`
grant the task roles already hold.

## No new runtime package

Cedar already has a bespoke OTEL bootstrap in
`apps/server/src/runtime/otel-init.ts` that supports both Axiom and a second
OTLP destination. Libra is wired in as that second destination. The `libra-exla`
runtime package is **not** needed and is **not** installed.

## Build and typecheck commands

```sh
# Typecheck the server (fast; no output files)
pnpm --filter @zero/server types

# Run server unit tests
pnpm --filter @zero/server test
```

## Replay safety contract

This PR does not add a replay command (`replayCommand` is null in `.libra/config.json`).
Libra can use the traces for diagnosis, severity, Slack context, and future sandbox
input selection.

Any future replay command must run only against fixture infrastructure. Expected guards:

| Env var | Replay value | Effect |
|---|---|---|
| `CEDAR_DATABASE_WRITE_ALLOWED` | `false` | Blocks all DB writes |
| `CEDAR_AUTH_DATABASE_WRITE_ALLOWED` | `false` | Blocks auth-table writes |
| `CEDAR_PROVIDER_SIDE_EFFECTS_ALLOWED` | `false` | Blocks Gmail/Calendar/Outlook API calls |
| `CEDAR_NOTIFICATIONS_ALLOWED` | `false` | Blocks email/push delivery |
| `CEDAR_OBJECT_STORE_WRITE_ALLOWED` | `false` | Blocks S3 writes |
| `CEDAR_QUEUE_PUBLISH_ALLOWED` | `false` | Blocks SQS publishes |
| `LIBRA_REPLAY_DATABASE_URL` | *(fixture URL)* | Points to an isolated read-only fixture DB |

The `.libra/config.json` `safety.blockedProductionMarkers` list prevents any
replay from running against a database URL that contains `prod`, `production`,
or `cedar-mail-prod`.