Libra CodeHub

CedarCopilot/cedar-mail

Branch: staging

fix(deploy): wait for CloudFormation stack to stabilize before cdk deploy

merged#2728CedarCopilot

CedarCopilot wants to merge 0 commits into staging from fix/staging-deploy-stack-collision

Live on prod, no production signal yetTimeline and evidence
  1. Opened
    Sep 8, 2026, 11:49 AM
  2. Merged
    Sep 8, 2026, 2:32 PM
  3. Live on prod
    Sep 8, 2026, 2:32 PM
  4. Observed 2 days
    Sep 8, 2026, 2:32 PM
  5. Pipelines steady after this deploy
    Sep 8, 2026, 2:32 PM
  6. Unobserved

    Live on prod, no production signal yet

    Sep 10, 2026, 11:43 PM

Behaviors Libra is checking

A stack that does not yet exist is treated as ready so first-time or bootstrap deployments can proceed to cdk deploy.Inconclusivelow confidence

Verification reached its budget before a verdict.

prod, checked Sep 10, 2026, 10:43 PM
Transient failures while reading CloudFormation stack status are retried instead of being treated as proof that the stack is stable.Inconclusivelow confidence

Unable to complete the production verification from the available telemetry: the expected retry log was absent, and the stack-name search matched unrelated container-performance telemetry rather than tied deploy executions. No positive success signal or tied failure evidence.

prod, checked Sep 10, 2026, 10:44 PM
The deployment waits for an existing CloudFormation stack to leave any *_IN_PROGRESS state before invoking cdk deploy, preventing collisions with updates left running by cancelled workflows.Inconclusivelow confidence

Unable to verify the intended behavior: no matching production CloudWatch log entries were found after the deployment for stack stabilization waits, cdk deploy execution, or UPDATE_IN_PROGRESS collision errors. Positive runtime evidence is absent, so the result is inconclusive.

prod, checked Sep 10, 2026, 11:43 PM
The deployment aborts with a nonzero status when CloudFormation cannot be read or remains in progress beyond the configured maximum wait, defaulting to 900 seconds.Inconclusivehigh confidence

Inconclusive: after the 2026-09-08T21:32:05Z deploy, exact production CloudWatch searches returned 0 tied cedar_wait_for_stack_stable timeout/read-retry/in-progress messages, while OTEL returned 0 deploy/CloudFormation/CDK-specific spans. The available repository-wide OTEL.

prod, checked Sep 10, 2026, 11:43 PM

Libra has verdicts on 0 of 4 tracked behaviors on prod; 4 are still being checked. Libra checks hourly for 3 days after each deploy.

Summary

  • The staging deploy workflow's cancel-in-progress: true concurrency group kills the GitHub Actions job on a new push, but does not cancel the cdk deploy CloudFormation update already underway , CFN keeps applying it until it reaches a terminal state on its own.
  • The next queued deploy run then starts immediately and its cdk deploy collides with the still-running update: ValidationError: Stack:...CedarAwsStagingEnvironmentStack... is in UPDATE_IN_PROGRESS state and can not be updated.
  • This is exactly what happened today: PR #2723's deploy was cancelled mid-update at 18:32 UTC by PR #2726/#2727's push, CloudFormation kept applying that update until 18:37:46, and PR #2727's Deploy AWS Staging run (started ~18:43) failed on the collision. The stack itself reached UPDATE_COMPLETE fine , only the next workflow run failed.
  • Adds cedar_wait_for_stack_stable to aws/scripts/deploy-utils.sh, which polls the stack status and waits for any *_IN_PROGRESS state to clear (up to 15 min) before proceeding. Wired into deploy_stack() in aws/scripts/deploy-surface.sh, which is the single call site used by both the bootstrap and main deploy paths.

Test plan

  • bash -n syntax check on both edited scripts
  • Verified live CedarAwsStagingEnvironmentStack is currently UPDATE_COMPLETE (no residual stuck state)
  • Next staging deploy run exercises the new wait , confirm no regression via Actions logs

🤖 Generated with Claude Code

https://claude.ai/code/session_01BWvDYfrsL4PAK3iqd3AzQU

Greptile Summary

The PR adds a CloudFormation stabilization gate before both bootstrap and normal CDK deployments.

  • Polls the target environment stack while its status ends in _IN_PROGRESS.
  • Waits up to 15 minutes by default, polling every 15 seconds.
  • Covers both deploy_stack call paths through the shared helper.
  • The new poll currently misclassifies AWS API failures as successful stability checks.

Confidence Score: 4/5

The PR should not merge until the stability check distinguishes a genuinely absent stack from AWS API failures that leave stability unconfirmed.

A failed status query currently returns success, allowing CDK deployment to begin while the target stack may still be updating and thereby preserving the collision path this change is intended to prevent.

Files Needing Attention: aws/scripts/deploy-utils.sh

Important Files Changed

FilenameOverview
aws/scripts/deploy-surface.shAdds the stabilization gate to the shared function used by both bootstrap and normal CDK deployment paths.
aws/scripts/deploy-utils.shImplements bounded CloudFormation polling, but suppresses and accepts operational describe-stacks failures.

Flowchart

rendering diagram…
Prompt To Fix All With AI
### Issue 1
aws/scripts/deploy-utils.sh:67-70
**AWS Errors Bypass Waiting**

If `describe-stacks` fails because of a transient AWS API or network error while the stack is still updating, `|| true` turns the failure into an empty status and the next condition treats it as stable. The deployment then starts without confirming stability and can hit the same CloudFormation update collision this guard is intended to prevent. Distinguish an expected stack-not-found response from operational failures, and fail or retry the latter.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Reviews (1): Last reviewed commit: ["fix(deploy): wait for CloudFor

Show production surfaces and changed-file mapping

Production surfaces

Libra has not measured any production surfaces for this change yet.

Changed files → surfaces

  • aws/scripts/deploy-surface.shno production surface mapped
  • aws/scripts/deploy-utils.shno production surface mapped