CedarCopilot wants to merge 0 commits into staging from feat/automation-sol-experiment
Live on prod, no production signal yet
Strict CloudWatch fallback found 2 prod failure log lines for [run-chat-agent-sdk] query() failed (Claude Code process aborted by user) in the last 24h while checking /deepseek, runautomationinprocess, deepseek, automation-agent, automations-service.test.
Libra has verdicts on 0 of 2 tracked behaviors on prod; 2 are still being checked. Libra has 1 low-confidence degraded verdict it is still confirming. Libra checks hourly for 3 days after each deploy.
strategist/pipeline-review/meeting-prep → Kimi K3, daily-agenda → DeepSeek V4 Pro) with a single new candidate, Sol (gpt-5.6-sol), for all four patterns , that experiment's real-world savings were modest and inconsistent (see the Sep 2026 staging A/B in #understudy-cedar).automation-sol) instead of sharing the automation workload with a tags.experiment split , so the unflagged path's request shape is completely untouched by this change.MODEL_EXPERIMENT_AGENT_BY_PATTERN (a Record<pattern, agent>) collapses to MODEL_EXPERIMENT_PATTERNS (a Set<pattern>) since every experiment pattern now maps to the same agent.cedar-automation-model-experiment PostHog flag (default off).The automation-sol Understudy workload does not pre-exist and must be registered before this can route without a workload_not_found 400. I registered it via the understudy CLI before opening this PR:
understudy workloads create automation-sol --project cedar --capture
(id usp_01a0838d02027051b4729b40)
pnpm --filter @zero/server run types , passespnpm --filter @zero/server exec vitest run src/services/aop/__tests__/automations-service.test.ts , 20 passedpnpm deps:check , no dependency-direction violationscedar-automation-model-experiment on for an internal test user and confirm a strategist/pipeline-review/meeting-prep/daily-agenda run actually routes to automation-agent-sol and shows up under the automation-sol workload in Understudy's cost dashboard🤖 Generated with Claude Code
https://claude.ai/code/session_017pZCUbjXJgwakzPsbtNbip
This PR replaces the Kimi K3/DeepSeek automation experiment with a single Sol agent and isolates flagged traffic on the new automation-sol Understudy workload.
automationAgentSol using gpt-5.6-sol.The implementation appears safe to merge, with a non-blocking test-coverage gap around the new experiment routing.
The routing logic is internally consistent and uses canonical pattern names, but the tests never exercise the experiment flag, Sol selection, fallback behavior, or flag precedence.
Files Needing Attention: apps/server/src/services/aop/tests/automations-service.test.ts
| Filename | Overview |
|---|---|
| apps/server/src/mastra/agents/automation-agent.ts | Replaces the two previous experiment agents with a Sol agent on a dedicated Understudy workload. |
| apps/server/src/services/aop/automations.ts | Consolidates the pattern mapping into a set and gives eligible flagged runs precedence for Sol routing. |
| apps/server/src/services/aop/tests/automations-service.test.ts | Updates the agent mock export but leaves the newly introduced experiment-routing branches untested. |
| apps/server/src/services/llm/models.ts | Replaces the prior experimental model constants with the Sol model identifier. |
| apps/server/src/mastra/index.ts | Registers the new Sol agent and removes the superseded experiment agents. |
Libra has not measured any production surfaces for this change yet.