Release 4.2 · Blocker
Re: Postmark 4.2 — go/no-go for Thursday freeze
Nadia — writing this before standup so the council has something concrete to react to. My recommendation is amber, not green. I don't think we should call it off, but I don't think Thursday's freeze survives contact with the two items below.
- Delivery worker retries. The exponential backoff change is correct in isolation, but under the new queue shape it produces a thundering herd at the 4-minute mark. We saw it in the eu-west-2 incident this morning.
- Migration rehearsal. We haven't re-run it since Friday's schema change landed. The last clean rehearsal is now six commits stale, which is not a rehearsal.
For context, here's where the release criteria actually stand as of this morning. Everything not listed is green and has been for a week.
| Criterion | Owner | Status | Open |
|---|---|---|---|
| Delivery worker retries | Priya R. | Amber | 2 |
| Migration rehearsal | Marcus B. | Amber | 1 |
| Load test p99 < 800ms | Priya R. | Green | 0 |
| Docs & changelog | Ines D. | Green | 0 |
| Rollback drill | Marcus B. | Green | 0 |
If the council would rather hold the date regardless, say so explicitly and I'll write down what we're accepting. I'd just like it accepted on purpose rather than by drift.
Lisbon (WEST, UTC+1) · usually offline after 18:00
Nadia Ferrante Thanks — can you put a number on "amber"? Wed 21:14
Thanks for flagging early. Can you put a number on “amber”? The council responds much better to “72 hours of rehearsal time” than to a colour.
Also: if we slip, does anything downstream slip with it, or is the December window unaffected?
Priya Raghunathan First read on the retry change… Wed 16:02
First read on the retry change: it's doing what we asked, just not where we expected. Backoff is correct per-worker; the problem is that all workers agree on the same schedule, so they wake together.
Marcus is adding jitter. Small patch, but it needs a full load cycle to prove.
Marcus Bell Rehearsal environment is rebuilt, snapshot is 3 days old… Wed 10:37
Rehearsal environment is rebuilt. The snapshot it runs against is three days old, which was fine until the schema change. I can pull a fresh one Monday morning — it takes about six hours to restore.
Rollback drill passed cleanly on Tuesday, for whatever comfort that's worth.
Release Council Freeze date confirmed for Thursday 26 September… Mon 08:00
Freeze date confirmed for Thursday 26 September at 12:00 WEST. Owners: please confirm criteria status in this thread no later than Thursday morning.
Standing reminder: an unconfirmed criterion is treated as red.