Libra CodeHub

CedarCopilot/cedar-mail

Branch: staging

Sanitize Understudy tag KEYS too, not just values

merged#2877CedarCopilot

CedarCopilot wants to merge 1 commit into staging from fix/understudy-tag-key-sanitization

Live on prod, 0 of 5 surfaces working, 1 day leftTimeline and evidence
  1. Opened
    Sep 22, 2026, 8:43 AM
  2. Sep 22, 2026, 8:56 AM
  3. Merged
    Sep 22, 2026, 10:41 AM
  4. Live on prod
    Sep 22, 2026, 10:41 AM
  5. Observed 6 hours, 5 surfaces, 24 requests
    Sep 22, 2026, 10:41 AM
  6. Watching

    Live on prod, 0 of 5 surfaces working, 1 day left

    Sep 22, 2026, 10:41 AM
  7. Pipelines steady after this deploy
    Sep 22, 2026, 10:41 AM

Behaviors Libra is checking

All orchestrator agent variants emit `user-id`, `agent-id`, and `org-id` tag keys instead of camelCase keys when constructing provider requests.Not checked
prod
Understudy request tags sanitize caller-supplied keys as well as values so uppercase, camelCase, or otherwise invalid tag names cannot cause the gateway to reject the request.Not checked
prod

Libra found 5 production surfaces on prod but could not judge any of them yet. 4 surfaces had under 20 requests, so Libra has not judged them; 1 surface had no requests at all. Libra checks hourly for 3 days after each deploy.

Summary

Production incident, 2026-09-22T14:44 UTC: the orchestrator workload (Cedar's general-purpose orchestrator subagent, mastra/agents/sub-agents/orchestrator-agent.ts) started 400ing 100% of Understudy-routed requests:

x-understudy-tags key 'userId' must match /^[a-z0-9][a-z0-9-]{0,63}$/

Confirmed via understudy reporting workload-status (workload degraded, 5/5 requests failed) and Axiom's cedar-prod-logs llm_request_failed events , production only, nothing in cedar-staging-logs. Aamir flagged it in Slack and is making a platform-side change on Understudy to tolerate this case; this PR fixes the root cause on Cedar's side too.

Root cause

understudyHeaders() (services/llm/understudy.ts) already sanitizes tag values , its own doc comment explains this was fixed once before for exactly this reason (Better Auth userIds are mixed-case). It never sanitized tag keys. orchestrator-agent.ts builds its tags with object-shorthand:

tags: { userId, ...(agentId ? { agentId } : {}), ...(orgId ? { orgId } : {}) }

The key userId is the literal camelCase JS property name, passed straight through to x-understudy-tags unsanitized , and it independently fails the exact same regex Understudy validates values against.

Fix

  • understudy.ts: understudyHeaders() now runs every tag key through the same sanitizer as values (renamed sanitizeUnderstudyTagValue → sanitizeUnderstudyTagToken since it now does both). This protects every current and future call site automatically , there are ~20 across the codebase, and any camelCase tag key anywhere would have hit this same bug.
  • orchestrator-agent.ts: renamed the three tags: {...} call sites to kebab-case (user-id/agent-id/org-id) for clarity at the source, even though the central fix makes this redundant.
  • Added a regression test asserting the exact incident: a camelCase tag key must never survive understudyHeaders() unsanitized.

Test plan

  • pnpm --filter @zero/server run types , clean
  • pnpm deps:check , no violations
  • resolve-model.test.ts (updated existing assertions for the new sanitized-key contract, added a dedicated regression test) + 3 other affected suites , 58/58 passed
  • Staging traffic spot-check once deployed , confirm no x-understudy-tags key ... failures recur

šŸ¤– Generated with Claude Code

https://claude.ai/code/session_012WwyQEcVnYQzkB4Fxsfpx6

RetriggerConfidence Score: 4/5

The request-rejection fix appears safe to merge, with a non-blocking hardening issue around silent collisions between normalized tag keys.

Findings

  1. P2Ā Normalized keys can collide ā–¶
Fix with agent prompt
### Issue 1
apps/server/src/services/llm/understudy.ts:45-48
This normalization is many-to-one: for example, `user_id` and `user-id` both become `user-id`, while `ENV` becomes `env`. Because the resulting pairs are passed to `Object.fromEntries`, one tag silently replaces the other and a caller can also replace the built-in environment tag. Since this API accepts arbitrary `Record<string, ...>` keys, please detect collisions after normalization rather than silently losing telemetry dimensions.

---

For ea
Show production surfaces and changed-file mapping

Production surfaces

SurfaceRequestsErrorsp95UsersVerdict
model_chunk Automation Agent (Understudy)2 → 120 → 0 (0%)1 ms → 1 ms0Insufficient traffic
12 requests, under the 20 Libra needs
model_chunk On-Event Execution Agent (Understudy)8 → 80 → 0 (0%)1 ms → 3 ms0Insufficient traffic
8 requests, under the 20 Libra needs
model_step Automation Agent (Understudy)0 → 30 → 0 (0%)not measured → 10275 ms0Insufficient traffic
3 requests, under the 20 Libra needs
model_step On-Event Execution Agent (Understudy)2 → 10 → 0 (0%)9527 ms → 7557 ms0Insufficient traffic
1 request, under the 20 Libra needs
model_chunk On-Event Orchestrator Agent (Meeting, Understudy)2 → 00 → 0 (0%)1 ms → not measured0No traffic
No requests recorded since this deploy.

Changed files → surfaces

  • apps/server/src/services/llm/understudy.tsmodel_chunk Automation Agent (Understudy)model_chunk On-Event Execution Agent (Understudy)model_step Automation Agent (Understudy)model_step On-Event Execution Agent (Understudy)model_chunk On-Event Orchestrator Agent (Meeting, Understudy)
  • apps/server/src/mastra/agents/sub-agents/orchestrator-agent.tsno production surface mapped
  • apps/server/src/services/llm/__tests__/resolve-model.test.tsno production surface mapped