CedarCopilot wants to merge 1 commit into staging from fix/understudy-tag-key-sanitization
Live on prod, 0 of 5 surfaces working, 1 day left
Libra found 5 production surfaces on prod but could not judge any of them yet. 4 surfaces had under 20 requests, so Libra has not judged them; 1 surface had no requests at all. Libra checks hourly for 3 days after each deploy.
Production incident, 2026-09-22T14:44 UTC: the orchestrator workload (Cedar's general-purpose orchestrator subagent, mastra/agents/sub-agents/orchestrator-agent.ts) started 400ing 100% of Understudy-routed requests:
x-understudy-tags key 'userId' must match /^[a-z0-9][a-z0-9-]{0,63}$/
Confirmed via understudy reporting workload-status (workload degraded, 5/5 requests failed) and Axiom's cedar-prod-logs llm_request_failed events , production only, nothing in cedar-staging-logs. Aamir flagged it in Slack and is making a platform-side change on Understudy to tolerate this case; this PR fixes the root cause on Cedar's side too.
understudyHeaders() (services/llm/understudy.ts) already sanitizes tag values , its own doc comment explains this was fixed once before for exactly this reason (Better Auth userIds are mixed-case). It never sanitized tag keys. orchestrator-agent.ts builds its tags with object-shorthand:
tags: { userId, ...(agentId ? { agentId } : {}), ...(orgId ? { orgId } : {}) }
The key userId is the literal camelCase JS property name, passed straight through to x-understudy-tags unsanitized , and it independently fails the exact same regex Understudy validates values against.
understudy.ts: understudyHeaders() now runs every tag key through the same sanitizer as values (renamed sanitizeUnderstudyTagValue ā sanitizeUnderstudyTagToken since it now does both). This protects every current and future call site automatically , there are ~20 across the codebase, and any camelCase tag key anywhere would have hit this same bug.orchestrator-agent.ts: renamed the three tags: {...} call sites to kebab-case (user-id/agent-id/org-id) for clarity at the source, even though the central fix makes this redundant.understudyHeaders() unsanitized.pnpm --filter @zero/server run types , cleanpnpm deps:check , no violationsresolve-model.test.ts (updated existing assertions for the new sanitized-key contract, added a dedicated regression test) + 3 other affected suites , 58/58 passedx-understudy-tags key ... failures recurš¤ Generated with Claude Code
https://claude.ai/code/session_012WwyQEcVnYQzkB4Fxsfpx6
The request-rejection fix appears safe to merge, with a non-blocking hardening issue around silent collisions between normalized tag keys.
### Issue 1
apps/server/src/services/llm/understudy.ts:45-48
This normalization is many-to-one: for example, `user_id` and `user-id` both become `user-id`, while `ENV` becomes `env`. Because the resulting pairs are passed to `Object.fromEntries`, one tag silently replaces the other and a caller can also replace the built-in environment tag. Since this API accepts arbitrary `Record<string, ...>` keys, please detect collisions after normalization rather than silently losing telemetry dimensions.
---
For ea
| Surface | Requests | Errors | p95 | Users | Verdict |
|---|---|---|---|---|---|
| model_chunk Automation Agent (Understudy) | 2 ā 12 | 0 ā 0 (0%) | 1 ms ā 1 ms | 0 | Insufficient traffic 12 requests, under the 20 Libra needs |
| model_chunk On-Event Execution Agent (Understudy) | 8 ā 8 | 0 ā 0 (0%) | 1 ms ā 3 ms | 0 | Insufficient traffic 8 requests, under the 20 Libra needs |
| model_step Automation Agent (Understudy) | 0 ā 3 | 0 ā 0 (0%) | not measured ā 10275 ms | 0 | Insufficient traffic 3 requests, under the 20 Libra needs |
| model_step On-Event Execution Agent (Understudy) | 2 ā 1 | 0 ā 0 (0%) | 9527 ms ā 7557 ms | 0 | Insufficient traffic 1 request, under the 20 Libra needs |
| model_chunk On-Event Orchestrator Agent (Meeting, Understudy) | 2 ā 0 | 0 ā 0 (0%) | 1 ms ā not measured | 0 | No traffic No requests recorded since this deploy. |