CedarCopilot/cedar-mail

---
name: read-cloudwatch-logs
description: Read Cedar production and staging logs from AWS CloudWatch. Use when asked to debug errors, investigate what happened in prod/staging, trace a specific request, find logs for a user or run ID, check service health, or investigate an exception. Covers api-service, chat-service, and worker-service across both prod and staging environments.
---

# Read CloudWatch Logs

Use the Shell tool to run `aws logs` CLI commands. Credentials come from the local AWS config (`~/.aws/credentials`) — region is `us-east-1`.

---

## Log groups

| Service | Environment | Log group |
|---|---|---|
| API service | prod | `/aws/ecs/aws-prod-api/api-service` |
| Chat service | prod | `/aws/ecs/aws-prod-api/chat-service` |
| Worker service | prod | `/aws/ecs/aws-prod-api/worker-service` |
| API service | staging | `/aws/ecs/aws-staging-api/api-service` |
| Chat service | staging | `/aws/ecs/aws-staging-api/chat-service` |
| Worker service | staging | `/aws/ecs/aws-staging-api/worker-service` |

---

## Base command pattern

```bash
aws logs filter-log-events \
  --log-group-name "LOG_GROUP" \
  --start-time $(python3 -c "import time; print(int((time.time() - SECONDS) * 1000))") \
  --filter-pattern "PATTERN" \
  --limit 50 \
  --query 'events[*].message' \
  --output text
```

- `SECONDS`: lookback window (e.g. `3600` = last hour, `86400` = last day)
- `--filter-pattern`: CloudWatch filter syntax. Omit for all logs. Use `"ERROR"` for errors, `'"userId"'` to match a JSON field, etc.
- `--limit`: max events returned (default 10000, cap at 10000)

---

## Common queries

### Recent errors across all prod services

```bash
for GROUP in /aws/ecs/aws-prod-api/api-service /aws/ecs/aws-prod-api/chat-service /aws/ecs/aws-prod-api/worker-service; do
  echo "=== $GROUP ==="
  aws logs filter-log-events \
    --log-group-name "$GROUP" \
    --start-time $(python3 -c "import time; print(int((time.time() - 3600) * 1000))") \
    --filter-pattern "ERROR" \
    --limit 10 \
    --query 'events[*].message' \
    --output text
done
```

### Find logs for a specific user ID

```bash
aws logs filter-log-events \
  --log-group-name "/aws/ecs/aws-prod-api/worker-service" \
  --start-time $(python3 -c "import time; print(int((time.time() - 86400) * 1000))") \
  --filter-pattern '"USER_ID"' \
  --limit 50 \
  --query 'events[*].message' \
  --output text
```

### Find logs for a specific run ID / execution

```bash
aws logs filter-log-events \
  --log-group-name "/aws/ecs/aws-prod-api/worker-service" \
  --start-time $(python3 -c "import time; print(int((time.time() - 86400) * 1000))") \
  --filter-pattern '"RUN_ID"' \
  --limit 100 \
  --query 'events[*].message' \
  --output text
```

### Tail recent logs from chat service (last 5 minutes)

```bash
aws logs filter-log-events \
  --log-group-name "/aws/ecs/aws-prod-api/chat-service" \
  --start-time $(python3 -c "import time; print(int((time.time() - 300) * 1000))") \
  --limit 50 \
  --query 'events[*].[timestamp,message]' \
  --output text
```

### Filter by log level (structured JSON logs)

```bash
# Errors only
aws logs filter-log-events \
  --log-group-name "/aws/ecs/aws-prod-api/api-service" \
  --start-time $(python3 -c "import time; print(int((time.time() - 3600) * 1000))") \
  --filter-pattern '{ $.level = "error" }' \
  --limit 20 \
  --query 'events[*].message' \
  --output text
```

### Specific time window (absolute)

```bash
# Use epoch milliseconds for precise windows
aws logs filter-log-events \
  --log-group-name "/aws/ecs/aws-prod-api/worker-service" \
  --start-time 1712800000000 \
  --end-time   1712803600000 \
  --filter-pattern "PATTERN" \
  --limit 100 \
  --query 'events[*].message' \
  --output text
```

---

## Timestamps

Log events include a `timestamp` field (epoch milliseconds). To convert:

```bash
python3 -c "import datetime; print(datetime.datetime.fromtimestamp(EPOCH_MS / 1000))"
```

To get epoch ms for a specific time:

```bash
python3 -c "import datetime; print(int(datetime.datetime(2026, 4, 11, 20, 0, 0).timestamp() * 1000))"
```

---

## CloudWatch filter pattern syntax

| Goal | Pattern |
|---|---|
| Text match | `"some string"` |
| JSON field equals | `{ $.level = "error" }` |
| JSON field contains | `{ $.message = "*exception*" }` |
| Multiple terms | `"term1" "term2"` |
| Either term | `?"term1" ?"term2"` |
| Exclude term | `-"term"` |

---

## Notes

- Log retention varies by group — check with `aws logs describe-log-groups --log-group-name-prefix "/aws/ecs/aws-prod-api"` if old logs seem missing.
- Logs are structured JSON (Cedar uses `createStructuredLog`): fields include `level`, `message`, `trace_id`, `span_id`, `user_id`, `run_id`.
- `trace_id` / `span_id` in logs correlate directly to OTel spans in Axiom — use both together to get full request context.
- The `--filter-pattern` is case-sensitive.
- If a query returns nothing, widen the `--start-time` window or remove the filter pattern to confirm logs exist.