CedarCopilot wants to merge 4 commits into staging from fix/task-rationale-header
Not deployed
Libra has no production signal for this change yet because it has not deployed. Libra checks hourly for 3 days after each deploy.
Note on the evidence, added after a second look at the run data. An earlier version of this description claimed this change lifts retirement recall 81% → 90%. That claim does not hold , it rests on 7 cases of which 3 moved (sign test p = 0.25) and it did not replicate. The change still ships, on narrower and better-stated grounds: it is never worse than the baseline on any axis in either run, it carries the experiment's only sub-0.05 signal (less sibling over-cancelling, p = 0.038, caveated below), and it costs one string. See Every recall difference here is within noise.
One string. <existing_tasks> headed a task's stored rationale as Notes:; it now reads
Why this exists / satisfied when:. Same text, same field, no schema change, no new capture.
notes is the column's name; it is not what the column carries. 133 of one rep's 137 open
non-approval tasks hold ~340 characters there stating why the task exists and what would settle
it , "Do not send if they've already responded", "only execute if the Sep 8 meeting is
cancelled", "do not send until after the event reply goes out". Under Notes: that reads as trailing detail.
The header is now one helper shared by the two renderers of this block , listTasksTool for the
real run and existing-tasks-block.ts for the read-only preview , which previously wrote the
string out twice. field-approval and crm-opportunity tasks store a machine payload in notes
rather than a rationale, and keep the neutral Notes: label.
apps/server/.claude/skills/playbook-instruction-eval/task-retirement-eval.ts, a new
confusion-matrix harness. 14 labelled cases built from real production rows on one rep's account
(7 must-retire, 7 must-keep), 5 variants, 6 trials each = 420 model calls, 2126s wall clock at 8
in flight, 0 errored trials. Record-only tool stubs; zero writes anywhere. The committed fixture
is pseudonymised , invented names on .example domains, same prose and structure. The two
variants that matter were re-run on the pseudonymised fixture, and that replication is what
the analysis below turns on.
"Retired" = cancel-task on the task under test or update-task {status: "done"} , the
system prompt deliberately splits those and a variant that picks the right one has not made a
mistake.
| Variant | TP | FP | FN | TN | precision | recall | F0.5 | chandler | merrick grove | collateral/run |
|---|---|---|---|---|---|---|---|---|---|---|
V0 today's Notes: (baseline) | 5.67 | 0.00 | 1.33 | 7.00 | 100% | 81% | 0.955 | 0% ✅ | 0% ✅ | 12.33 |
V1 Why this exists / satisfied when: | 6.33 | 0.00 | 0.67 | 7.00 | 100% | 90% | 0.979 | 0% ✅ | 0% ✅ | 9.17 |
| V2 forced per-task KEEP/DONE/CANCEL verdict | 5.67 | 0.17 | 1.33 | 6.83 | 97% | 81% | 0.934 | 0% ✅ | 0% ✅ | 10.00 |
V3 explicit synthesised satisfiedWhen line | 5.00 | 0.00 | 2.00 | 7.00 | 100% | 71% | 0.926 | 0% ✅ | 0% ✅ | 9.67 |
| V4 V1 + V2 + a clock-starter guard | 5.50 | 0.00 | 1.50 | 7.00 | 100% | 79% | 0.948 | 0% ✅ | 0% ✅ | 9.17 |
Do not read the recall column on its own , every difference in it is inside the noise. The analysis below is the honest version, and it changes the argument for this PR.
The unit of analysis is the case, not the trial. Six trials on one case are repeats of one prompt: they measure the model's consistency on that case, not six independent observations. That leaves 7 independent units per side, not 42.
Per-case retire counts (out of 6) for the two variants that matter:
| Case | V0 | V1 | Δ | pseudonymised V0 | V1 | Δ | |
|---|---|---|---|---|---|---|---|
| harlowfinch | 6 | 6 | 0 | 6 | 5 | −1 | |
| bramley | 5 | 6 | +1 | 6 | 6 | 0 | |
| calder | 4 | 5 | +1 | 3 | 6 | +3 | |
| trelawney | 4 | 4 | 0 | 5 | 2 | −3 | |
| vela | 6 | 6 | 0 |
Libra has not measured any production surfaces for this change yet.