Libra CodeHub

CedarCopilot/cedar-mail

Branch: staging

chore(mastra): upgrade @mastra/core beta.19 -> 1.67.0

merged#2815CedarCopilot

CedarCopilot wants to merge 3 commits into staging from chore/mastra-v1-upgrade

Live on prod, 1 of 11 surfaces degradedTimeline and evidence
  1. Opened
    Sep 17, 2026, 6:00 PM
  2. Sep 17, 2026, 9:23 PM
  3. Merged
    Sep 17, 2026, 9:28 PM
  4. Live on staging
    Sep 17, 2026, 9:28 PM
  5. Live on prod
    Sep 17, 2026, 9:28 PM
  6. Observed 19 hours, 11 surfaces, 8,622 requests
    Sep 17, 2026, 9:28 PM
  7. Pipelines steady after this deploy
    Sep 17, 2026, 9:28 PM
  8. Pipelines steady after this deploy
    Sep 17, 2026, 9:28 PM
  9. Mixed signals

    Live on prod, 1 of 11 surfaces degraded

    Sep 18, 2026, 4:18 PM

Behaviors Libra is checking

Mastra observability and the OTEL bridge remain compatible with the upgraded core runtime so execution traces continue to reach the existing telemetry backend.Degradedlow confidence

Verifier did not provide tied failure evidence for degraded, so Libra is keeping it inconclusive instead of moving the card. Original assessment: Post-deploy staging telemetry confirms Mastra execution spans are still emitted: 89 STATUS_CODE_OK spans across 6 traces, plus 14.

staging, checked Sep 20, 2026, 9:07 PM
Client-send event execution failures are emitted as structured logs containing the conversation ID, optional run ID, and normalized error text.Inconclusivelow confidence

Verification reached its 130s outer budget before the verifier returned. Libra is keeping this intent verifying until the next check.

prod, checked Sep 18, 2026, 2:36 PM
Nested and family-tool delegations invoke child tools through the Mastra-compatible callTool path so current execute signatures, request context, and validation behavior are preserved.Inconclusivelow confidence

Verification reached its 120s wall-clock budget before a tied verdict. Libra is keeping this intent verifying until the next check.

prod, checked Sep 18, 2026, 2:38 PM
Mastra agent, workflow, and tool executions continue exporting traces through the existing observability and OTel bridge after the package upgrade.Inconclusivelow confidence

Unable to verify the intended behavior: the available CloudWatch queries returned no concrete post-deploy Mastra agent, workflow, or tool execution operations, and OTEL/Axiom spans are unavailable or stale. Positive runtime evidence required by the contract is therefore absent.

prod, checked Sep 18, 2026, 3:17 PM
The server runs its agents, workflows, and tools on the aligned stable Mastra 1.x packages instead of the pre-1.0 beta APIs.Inconclusivelow confidence

Verification reached its 120s wall-clock budget before a tied verdict. Libra is keeping this intent verifying until the next check.

prod, checked Sep 18, 2026, 3:19 PM
Tool results that do not match their declared outputSchema fail with a Mastra ValidationError instead of being silently accepted.Inconclusivemedium confidence

Post-deploy CloudWatch telemetry shows substantial Mastra tool activity: 33,126 logged tool calls in worker-service and 2,600 in api-service, plus 69 structured operations including 8,940 agent-execution.slot.acquire events. However, there were 0 exact ValidationError logs and.

prod, checked Sep 18, 2026, 4:17 PM

Libra measured 9 of 11 production surfaces on prod. 2 surfaces had no requests at all. Libra checks hourly for 3 days after each deploy.

Summary

Upgrades Mastra from 1.0.0-beta.19 to 1.67.0 (see apps/server/docs/design/mastra-v1-upgrade.md for the full scoping/risk assessment written before starting this).

  • Bumps @mastra/core1.67.0, @mastra/observability1.17.8, @mastra/otel-bridge1.5.8, root mastra CLI → 1.30.0, in lockstep.
  • Drops @mastra/memory and @mastra/pg entirely , both confirmed to have zero import sites anywhere in the repo.
  • Ran npx @mastra/codemod@latest v1 , zero changes needed; this codebase's tool/agent/observability shapes were already ahead of the beta→1.0 API transition.

Real breaks found and fixed

  • Tool.execute became optional and its return type gained void (Output | ValidationError | void) , added a shared callTool() helper (apps/server/src/mastra/utils/call-tool.ts) for the ~110 call sites across the codebase that invoke another tool's execute directly (family-tool facades, dispatch/workflow code), rather than through Mastra's own agent-loop.
  • Tools with no outputSchema now type their result as unknown instead of any , cast to each tool's real return shape, read directly off its implementation.
  • RequestContext<T>'s generic became invariant (a new generic set()), so a specifically-typed RequestContext<Shape> and a bare RequestContext are no longer mutually assignable in either direction. Added eraseRequestContext() (apps/server/src/mastra/context/values.ts) for the specific→bare direction; switched a couple of reverse-direction sites to getRaw/setRaw.
  • Agent.stream()'s top-level threadId/resourceId were removed in favor of a nested memory: { thread, resource } option.
  • Dropped a dead resourceId param on roadmapAgent.generate() , that agent has no memory configured, so it was already inert before this upgrade.
  • Confirmed Agent.__registerMastra() (an undocumented private method this codebase relies on for tracing registration, flagged as the top risk in the scoping doc) is unchanged at 1.67.

Verification

  • Typecheck: clean (pnpm --filter @zero/server run types)
  • Tests: clean , full sweep across src/mastra/{tools,utils,workflows,skills,mcp,context} + src/services/playbook: 237 test files / 3,621 tests passing (only pre-existing skips)
  • Manual dev smoke test: ran the full local dev stack, exercised /roadmap-chat and /chat (multi-turn, multi-tool-call CRM query) against a real dev DB , correct results, no errors , and confirmed via Axiom that Mastra spans (invoke_agent, chat claude-sonnet-4-6, etc.) land correctly nested under the app's existing OTel spans with real gen_ai.* attributes (token usage, model), matching the pre-upgrade observability design.

Test plan

  • Scoped typecheck clean
  • Full mastra subtree test sweep clean
  • Manual dev smoke test: roadmap chat + multi-tool chat conversation against real data
  • Confirmed traces land correctly in Axiom post-upgrade
  • Deploy to staging and watch trace volume/error rates before merging further (per the design doc's verification plan , the dominant failure mode this investigation surfaced is "runs without erroring, silently wrong" rather than a crash, so staging observation matters more than usual here)

🤖 Generated with Claude Code

RetriggerConfidence Score: 4/5

The behavioral migration appears sound, but the explicit repository requirement against casts must be satisfied before merging;

Show production surfaces and changed-file mapping

Production surfaces

SurfaceRequestsErrorsp95UsersVerdict
slack.webhook.bufferMessage931 → 7460 → 0 (0%)38563 ms → 255259 ms0Degraded
Error rate is clean (0 of 746), but p95 latency more than doubled since the deploy. p95 255259 ms, was 38563 ms.
/api/trpc/crm.getConversation46 → 750 → 0 (0%)7920 ms → 21851 ms0Degraded
Error rate is clean (0 of 75), but p95 latency more than doubled since the deploy. p95 21851 ms, was 7920 ms.
process_calendar_event_sync_only4431 → 31860 → 0 (0%)92 ms → 87 ms27Working
3,186 requests since the deploy with 0 errors (0 errors in the 4,431-request baseline before it). p95 87 ms, was 92 ms.
/slack/events1767 → 20520 → 0 (0%)28 ms → 28 ms0Working
2,052 requests since the deploy with 0 errors (0 errors in the 1,767-request baseline before it). p95 28 ms, was 28 ms.
process_calendar_event_sync_only3014 → 17040 → 0 (0%)146 ms → 164 ms3Working
1,704 requests since the deploy with 0 errors (0 errors in the 3,014-request baseline before it). p95 164 ms, was 146 ms.
POST /slack/events615 → 10260 → 0 (0%)27 ms → 28 ms0Working
1,026 requests since the deploy with 0 errors (0 errors in the 615-request baseline before it). p95 28 ms, was 27 ms.
slack.link.channel_only914 → 9300 → 0 (0%)6122 ms → 6663 ms0Working
930 requests since the deploy with 0 errors (0 errors in the 914-request baseline before it). p95 6663 ms, was 6122 ms.
/a8n/notify/google970 → 8010 → 0 (0%)226 ms → 183 ms0Working
801 requests since the deploy with 0 errors (0 errors in the 970-request baseline before it). p95 183 ms, was 226 ms.
db.getConversationIdForThread344 → 4500 → 0 (0%)65 ms → 99 ms0Working
450 requests since the deploy with 0 errors (0 errors in the 344-request baseline before it). p95 99 ms, was 65 ms.
slack.link.channel_only227 → 2630 → 0 (0%)6727 ms → 6924 ms0Working
263 requests since the deploy with 0 errors (0 errors in the 227-request baseline before it). p95 6924 ms, was 6727 ms.
/a8n/notify/google-calendar141 → 570 → 0 (0%)420 ms → 58 ms0Working
57 requests since the deploy with 0 errors (0 errors in the 141-request baseline before it). p95 58 ms, was 420 ms.
/a8n/notify/google36 → 420 → 0 (0%)254 ms → 250 ms0Working
42 requests since the deploy with 0 errors (0 errors in the 36-request baseline before it). p95 250 ms, was 254 ms.
/api/trpc/crm.getConversation24 → 410 → 0 (0%)12120 ms → 9519 ms0Working
41 requests since the deploy with 0 errors (0 errors in the 24-request baseline before it). p95 9519 ms, was 12120 ms.
db.getConversationIdForThread32 → 230 → 0 (0%)96 ms → 34 ms0Working
23 requests since the deploy with 0 errors (0 errors in the 32-request baseline before it). p95 34 ms, was 96 ms.
/api/trpc/calendar.listEvents1 → 210 → 0 (0%)2 ms → 846 ms0Working
21 requests since the deploy with 0 errors (0 errors in the 1-request baseline before it). p95 846 ms, was 2 ms.
/a8n/notify/google-calendar15 → 180 → 0 (0%)63 ms → 53 ms0Insufficient traffic
18 requests, under the 20 Libra needs
roadmapAgent.generate0 → 00 → 0 (0%)not measured0No traffic
No requests recorded since this deploy.
roadmapAgent.generate0 → 00 → 0 (0%)not measured0No traffic
No requests recorded since this deploy.
/api/trpc/calendar.listEvents3 → 00 → 0 (0%)2088 ms → not measured0No traffic
No requests recorded since this deploy.
slack.webhook.bufferMessage0 → 00 → 0 (0%)not measured0No traffic
No requests recorded since this deploy.
/slack/events0 → 00 → 0 (0%)not measured0No traffic
No requests recorded since this deploy.
POST /slack/events0 → 00 → 0 (0%)not measured0No traffic
No requests recorded since this deploy.

Changed files → surfaces

  • apps/server/src/mastra/tools/calendar/calendarTool.tsprocess_calendar_event_sync_onlyprocess_calendar_event_sync_only/a8n/notify/google-calendar/api/trpc/calendar.listEvents/a8n/notify/google-calendar/api/trpc/calendar.listEvents
  • apps/server/src/mastra/tools/conversation/conversationTool.tsdb.getConversationIdForThread/api/trpc/crm.getConversation/api/trpc/crm.getConversationdb.getConversationIdForThread
  • apps/server/src/mastra/tools/notify/notifyTool.ts/a8n/notify/google/a8n/notify/google-calendar/a8n/notify/google/a8n/notify/google-calendar
  • apps/server/src/mastra/tools/slack/slackTool.ts/slack/eventsPOST /slack/eventsslack.link.channel_onlyslack.webhook.bufferMessageslack.link.channel_onlyslack.webhook.bufferMessage+2
  • apps/mail/package.jsonno production surface mapped
  • apps/server/docs/design/mastra-v1-upgrade.mdno production surface mapped
  • apps/server/package.jsonno production surface mapped
  • apps/server/src/mastra/context/values.tsno production surface mapped
  • apps/server/src/mastra/mcp/external/load-brain-skills-tool.tsno production surface mapped
  • apps/server/src/mastra/mcp/external/load-skill-tool.tsno production surface mapped
  • apps/server/src/mastra/mcp/external/server.tsno production surface mapped
  • apps/server/src/mastra/routeHandlers/event-execution/handleExecuteFromClientSend.tsno production surface mapped
  • apps/server/src/mastra/routeHandlers/roadmap/roadmap-chat.tsno production surface mapped
  • apps/server/src/mastra/skills/account-config/tools/diagnoseConversationTool.tsno production surface mapped
  • apps/server/src/mastra/tools/chat/chatContextTool.tsno production surface mapped
  • apps/server/src/mastra/tools/config/config-read-tool.tsno production surface mapped
  • apps/server/src/mastra/tools/config/config-write-tool.tsno production surface mapped
  • apps/server/src/mastra/tools/config/configureStrategistTool.tsno production surface mapped
  • apps/server/src/mastra/tools/config/connection-write-tool.tsno production surface mapped
  • apps/server/src/mastra/tools/config/systems-write-tool.tsno production surface mapped
  • apps/server/src/mastra/tools/config/workspace-write-tool.tsno production surface mapped
  • apps/server/src/mastra/tools/debug/debugTool.tsno production surface mapped
  • apps/server/src/mastra/tools/document/documentTool.tsno production surface mapped
  • apps/server/src/mastra/tools/draft-comms/draftCommsTool.tsno production surface mapped
  • apps/server/src/mastra/tools/event-execution/onEventExecutionDraftEmailTool.tsno production surface mapped
  • apps/server/src/mastra/tools/event-execution/onEventExecutionDraftSlackTool.tsno production surface mapped
  • apps/server/src/mastra/tools/event-execution/runSubagentTool.tsno production surface mapped
  • apps/server/src/mastra/tools/integrations/externalMcpTool.tsno production surface mapped
  • apps/server/src/mastra/tools/mail/mailTool.tsno production surface mapped
  • apps/server/src/mastra/tools/meetings/meetingsTool.tsno production surface mapped
  • apps/server/src/mastra/tools/research/researchTool.tsno production surface mapped
  • apps/server/src/mastra/tools/research/webSearchTool.tsno production surface mapped
  • apps/server/src/mastra/tools/skills/delegateTool.tsno production surface mapped
  • apps/server/src/mastra/tools/task/taskTool.tsno production surface mapped
  • apps/server/src/mastra/tools/task/updateNextStepsAndTasksTool.tsno production surface mapped
  • apps/server/src/mastra/tools/ui-config/__tests__/uiConfigTool.test.tsno production surface mapped
  • apps/server/src/mastra/tools/ui-config/uiConfigTool.tsno production surface mapped
  • apps/server/src/mastra/utils/call-tool.tsno production surface mapped
  • apps/server/src/mastra/utils/conversation-field-updates.tsno production surface mapped
  • apps/server/src/mastra/utils/slack-message-generation.tsno production surface mapped
  • apps/server/src/mastra/utils/web-citations.tsno production surface mapped
  • apps/server/src/mastra/workflows/chat/chat-workflow.tsno production surface mapped
  • apps/server/src/mastra/workflows/chat/harness/mcp-server.tsno production surface mapped
  • apps/server/src/mastra/workflows/event-execution/on-event-agent-execution-workflow.tsno production surface mapped
  • apps/server/src/mastra/workflows/sync/initial-sync-conversation-update-workflow.tsno production surface mapped
  • apps/server/src/services/playbook/playbook-execution-triggers.tsno production surface mapped
  • apps/server/src/trpc/routes/admin-next-steps.tsno production surface mapped
  • package.jsonno production surface mapped
  • pnpm-lock.yamlno production surface mapped