CedarCopilot wants to merge 3 commits into staging from chore/mastra-v1-upgrade
Live on prod, 1 of 11 surfaces degraded
Verifier did not provide tied failure evidence for degraded, so Libra is keeping it inconclusive instead of moving the card. Original assessment: Post-deploy staging telemetry confirms Mastra execution spans are still emitted: 89 STATUS_CODE_OK spans across 6 traces, plus 14.
Verification reached its 130s outer budget before the verifier returned. Libra is keeping this intent verifying until the next check.
Verification reached its 120s wall-clock budget before a tied verdict. Libra is keeping this intent verifying until the next check.
Unable to verify the intended behavior: the available CloudWatch queries returned no concrete post-deploy Mastra agent, workflow, or tool execution operations, and OTEL/Axiom spans are unavailable or stale. Positive runtime evidence required by the contract is therefore absent.
Verification reached its 120s wall-clock budget before a tied verdict. Libra is keeping this intent verifying until the next check.
Post-deploy CloudWatch telemetry shows substantial Mastra tool activity: 33,126 logged tool calls in worker-service and 2,600 in api-service, plus 69 structured operations including 8,940 agent-execution.slot.acquire events. However, there were 0 exact ValidationError logs and.
Libra measured 9 of 11 production surfaces on prod. 2 surfaces had no requests at all. Libra checks hourly for 3 days after each deploy.
Upgrades Mastra from 1.0.0-beta.19 to 1.67.0 (see apps/server/docs/design/mastra-v1-upgrade.md for the full scoping/risk assessment written before starting this).
@mastra/core → 1.67.0, @mastra/observability → 1.17.8, @mastra/otel-bridge → 1.5.8, root mastra CLI → 1.30.0, in lockstep.@mastra/memory and @mastra/pg entirely , both confirmed to have zero import sites anywhere in the repo.npx @mastra/codemod@latest v1 , zero changes needed; this codebase's tool/agent/observability shapes were already ahead of the beta→1.0 API transition.Tool.execute became optional and its return type gained void (Output | ValidationError | void) , added a shared callTool() helper (apps/server/src/mastra/utils/call-tool.ts) for the ~110 call sites across the codebase that invoke another tool's execute directly (family-tool facades, dispatch/workflow code), rather than through Mastra's own agent-loop.outputSchema now type their result as unknown instead of any , cast to each tool's real return shape, read directly off its implementation.RequestContext<T>'s generic became invariant (a new generic set()), so a specifically-typed RequestContext<Shape> and a bare RequestContext are no longer mutually assignable in either direction. Added eraseRequestContext() (apps/server/src/mastra/context/values.ts) for the specific→bare direction; switched a couple of reverse-direction sites to getRaw/setRaw.Agent.stream()'s top-level threadId/resourceId were removed in favor of a nested memory: { thread, resource } option.resourceId param on roadmapAgent.generate() , that agent has no memory configured, so it was already inert before this upgrade.Agent.__registerMastra() (an undocumented private method this codebase relies on for tracing registration, flagged as the top risk in the scoping doc) is unchanged at 1.67.pnpm --filter @zero/server run types)src/mastra/{tools,utils,workflows,skills,mcp,context} + src/services/playbook: 237 test files / 3,621 tests passing (only pre-existing skips)/roadmap-chat and /chat (multi-turn, multi-tool-call CRM query) against a real dev DB , correct results, no errors , and confirmed via Axiom that Mastra spans (invoke_agent, chat claude-sonnet-4-6, etc.) land correctly nested under the app's existing OTel spans with real gen_ai.* attributes (token usage, model), matching the pre-upgrade observability design.mastra subtree test sweep clean🤖 Generated with Claude Code
The behavioral migration appears sound, but the explicit repository requirement against casts must be satisfied before merging;
| Surface | Requests | Errors | p95 | Users | Verdict |
|---|---|---|---|---|---|
| slack.webhook.bufferMessage | 931 → 746 | 0 → 0 (0%) | 38563 ms → 255259 ms | 0 | Degraded Error rate is clean (0 of 746), but p95 latency more than doubled since the deploy. p95 255259 ms, was 38563 ms. |
| /api/trpc/crm.getConversation | 46 → 75 | 0 → 0 (0%) | 7920 ms → 21851 ms | 0 | Degraded Error rate is clean (0 of 75), but p95 latency more than doubled since the deploy. p95 21851 ms, was 7920 ms. |
| process_calendar_event_sync_only | 4431 → 3186 | 0 → 0 (0%) | 92 ms → 87 ms | 27 | Working 3,186 requests since the deploy with 0 errors (0 errors in the 4,431-request baseline before it). p95 87 ms, was 92 ms. |
| /slack/events | 1767 → 2052 | 0 → 0 (0%) | 28 ms → 28 ms | 0 | Working 2,052 requests since the deploy with 0 errors (0 errors in the 1,767-request baseline before it). p95 28 ms, was 28 ms. |
| process_calendar_event_sync_only | 3014 → 1704 | 0 → 0 (0%) | 146 ms → 164 ms | 3 | Working 1,704 requests since the deploy with 0 errors (0 errors in the 3,014-request baseline before it). p95 164 ms, was 146 ms. |
| POST /slack/events | 615 → 1026 | 0 → 0 (0%) | 27 ms → 28 ms | 0 | Working 1,026 requests since the deploy with 0 errors (0 errors in the 615-request baseline before it). p95 28 ms, was 27 ms. |
| slack.link.channel_only | 914 → 930 | 0 → 0 (0%) | 6122 ms → 6663 ms | 0 | Working 930 requests since the deploy with 0 errors (0 errors in the 914-request baseline before it). p95 6663 ms, was 6122 ms. |
| /a8n/notify/google | 970 → 801 | 0 → 0 (0%) | 226 ms → 183 ms | 0 | Working 801 requests since the deploy with 0 errors (0 errors in the 970-request baseline before it). p95 183 ms, was 226 ms. |
| db.getConversationIdForThread | 344 → 450 | 0 → 0 (0%) | 65 ms → 99 ms | 0 | Working 450 requests since the deploy with 0 errors (0 errors in the 344-request baseline before it). p95 99 ms, was 65 ms. |
| slack.link.channel_only | 227 → 263 | 0 → 0 (0%) | 6727 ms → 6924 ms | 0 | Working 263 requests since the deploy with 0 errors (0 errors in the 227-request baseline before it). p95 6924 ms, was 6727 ms. |
| /a8n/notify/google-calendar | 141 → 57 | 0 → 0 (0%) | 420 ms → 58 ms | 0 | Working 57 requests since the deploy with 0 errors (0 errors in the 141-request baseline before it). p95 58 ms, was 420 ms. |
| /a8n/notify/google | 36 → 42 | 0 → 0 (0%) | 254 ms → 250 ms | 0 | Working 42 requests since the deploy with 0 errors (0 errors in the 36-request baseline before it). p95 250 ms, was 254 ms. |
| /api/trpc/crm.getConversation | 24 → 41 | 0 → 0 (0%) | 12120 ms → 9519 ms | 0 | Working 41 requests since the deploy with 0 errors (0 errors in the 24-request baseline before it). p95 9519 ms, was 12120 ms. |
| db.getConversationIdForThread | 32 → 23 | 0 → 0 (0%) | 96 ms → 34 ms | 0 | Working 23 requests since the deploy with 0 errors (0 errors in the 32-request baseline before it). p95 34 ms, was 96 ms. |
| /api/trpc/calendar.listEvents | 1 → 21 | 0 → 0 (0%) | 2 ms → 846 ms | 0 | Working 21 requests since the deploy with 0 errors (0 errors in the 1-request baseline before it). p95 846 ms, was 2 ms. |
| /a8n/notify/google-calendar | 15 → 18 | 0 → 0 (0%) | 63 ms → 53 ms | 0 | Insufficient traffic 18 requests, under the 20 Libra needs |
| roadmapAgent.generate | 0 → 0 | 0 → 0 (0%) | not measured | 0 | No traffic No requests recorded since this deploy. |
| roadmapAgent.generate | 0 → 0 | 0 → 0 (0%) | not measured | 0 | No traffic No requests recorded since this deploy. |
| /api/trpc/calendar.listEvents | 3 → 0 | 0 → 0 (0%) | 2088 ms → not measured | 0 | No traffic No requests recorded since this deploy. |
| slack.webhook.bufferMessage | 0 → 0 | 0 → 0 (0%) | not measured | 0 | No traffic No requests recorded since this deploy. |
| /slack/events | 0 → 0 | 0 → 0 (0%) | not measured | 0 | No traffic No requests recorded since this deploy. |
| POST /slack/events | 0 → 0 | 0 → 0 (0%) | not measured | 0 | No traffic No requests recorded since this deploy. |