Libra CodeHub

CedarCopilot/cedar-mail

Branch: staging

feat(eval): add playbook-instruction-eval suite

merged#2812CedarCopilot

CedarCopilot wants to merge 2 commits into staging from feat/playbook-instruction-eval-suite

Live on prod, no production signal yetTimeline and evidence
  1. Opened
    Sep 17, 2026, 3:32 PM
  2. No findings
    Sep 17, 2026, 3:54 PM
  3. Merged
    Sep 17, 2026, 5:37 PM
  4. Live on prod
    Sep 17, 2026, 5:37 PM
  5. Observed 20 hours
    Sep 17, 2026, 5:37 PM
  6. Pipelines steady after this deploy
    Sep 17, 2026, 5:37 PM
  7. Unobserved

    Live on prod, no production signal yet

    Sep 18, 2026, 1:41 PM

Behaviors Libra is checking

AOP context behavior from playbook-instruction-eval/aop-classification-eval.ts keeps working in prod.Inconclusivelow confidence

Verification cited only 10 matching post-deploy events, below the 20-event floor for calling a change verified. Nothing is failing; there is not yet enough traffic to confirm it.

prod, checked Sep 18, 2026, 1:01 PM
CRM integration behavior from playbook-instruction-eval/deal-close-task-cancel-eval.ts keeps working in prod.Inconclusivelow confidence

Verification cited only 9 matching post-deploy events, below the 20-event floor for calling a change verified. Nothing is failing; there is not yet enough traffic to confirm it.

prod, checked Sep 18, 2026, 1:41 PM
Slack integration behavior from playbook-instruction-eval/stage-channel-override-eval.ts keeps working in prod.Inconclusivelow confidence

Strict CloudWatch fallback saw 1,782 success-shaped log lines matching slack, channel, channels, workspace, workspaces, but no tied operation was present, so Libra is not calling this working.

prod, checked Sep 18, 2026, 1:41 PM

Libra has verdicts on 0 of 3 tracked behaviors on prod; 3 are still being checked. Libra checks hourly for 3 days after each deploy.

Summary

  • Adds the playbook-instruction-eval skill's eval suite: AOP classification, approval/task parity, deal-close task cancellation, domain-return-shape, notify gating, orchestrator/sequencer, and stage-channel-override evals, plus fixtures/baselines each depends on.
  • Expands eval-instructions.ts with the instructions for the new evals.
  • Adds docs/cedar-configuration-knowledge.md, the configuration reference these evals were authored against.

Test plan

  • Run the eval suite locally and confirm each new eval executes against its fixtures

RetriggerConfidence Score: 0/5

This PR is not safe to merge because multiple advertised evaluators cannot run, several core measurements use non-production prompt contracts, and production-derived customer data is exposed.

Findings

  1. P1 Required fixtures are missing ▶
  2. P1 Evaluator depends on local files ▶
  3. P1 Prompt extraction stops early ▶
  4. P1 Prompt layout differs from production ▶
  5. P1 Notification schema mismatches production ▶
  6. P1 Security Production data is exposed ▶
  7. P2 Errors count as safe outcomes ▶
  8. P2 Relative imports violate guidance ▶
  9. P2 Type assertion violates guidance ▶
Fix with agent prompt 
### Issue 1
apps/server/.claude/skills/playbook-instruction-eval/eval-instructions.ts:83-100
This evaluator reads four meeting fixtures that are not tracked or generated by a setup step, so the documented command fails with `ENOENT` before any trial executes. The same omission affects `scout-pilot-stage-event.txt` in `stage-channel-override-eval.ts` and both sequencer th
Show production surfaces and changed-file mapping

Production surfaces

Libra has not measured any production surfaces for this change yet.

Changed files → surfaces

  • apps/server/.claude/skills/playbook-instruction-eval/aop-classification-eval.tsno production surface mapped
  • apps/server/.claude/skills/playbook-instruction-eval/approval-task-parity-eval.tsno production surface mapped
  • apps/server/.claude/skills/playbook-instruction-eval/deal-close-task-cancel-eval.tsno production surface mapped
  • apps/server/.claude/skills/playbook-instruction-eval/domain-return-shape-eval.tsno production surface mapped
  • apps/server/.claude/skills/playbook-instruction-eval/eval-instructions.tsno production surface mapped
  • apps/server/.claude/skills/playbook-instruction-eval/fixtures/aop-classification-cases.tsno production surface mapped
  • apps/server/.claude/skills/playbook-instruction-eval/fixtures/aop-selection-baseline.mdno production surface mapped
  • apps/server/.claude/skills/playbook-instruction-eval/fixtures/deal-close-next-steps-baseline.mdno production surface mapped
  • apps/server/.claude/skills/playbook-instruction-eval/fixtures/deal-close-task-cancel-cases.tsno production surface mapped
  • apps/server/.claude/skills/playbook-instruction-eval/fixtures/on-event-execution-agent-instructions.mdno production surface mapped
  • apps/server/.claude/skills/playbook-instruction-eval/fixtures/scout-noah-crm-updater.mdno production surface mapped
  • apps/server/.claude/skills/playbook-instruction-eval/fixtures/willem-crm-updater.mdno production surface mapped
  • apps/server/.claude/skills/playbook-instruction-eval/notify-gating-eval.tsno production surface mapped
  • apps/server/.claude/skills/playbook-instruction-eval/orchestrator-sequencer-eval.tsno production surface mapped
  • apps/server/.claude/skills/playbook-instruction-eval/sequencer-suppression-eval.tsno production surface mapped
  • apps/server/.claude/skills/playbook-instruction-eval/stage-channel-override-eval.tsno production surface mapped
  • docs/cedar-configuration-knowledge.mdno production surface mapped