Incident history
all times UTC
Elevated build queue times in the CI cache layer
DegradedBuild PipelineMonitoring
- Monitoring10:40 UTC
Cache hit rates are back above 92%. We are leaving the incident open while the warm pool refills.
- Identified09:38 UTC
A bad eviction policy shipped with runner image 4.19 caused the shared layer cache to thrash. Rolling back to 4.18.
- Investigating09:12 UTC
Builds are queuing 3–6× longer than baseline in us-east-1. Investigating.
Scheduled reindex of the documentation search cluster
MaintenanceDocs & SearchCompleted
Object Store returned 503s for multipart uploads in ap-southeast-1
OutageObject Storeap-southeast-1Resolved
- Resolved17:29 UTC
All upload paths healthy for 20 consecutive minutes. A full write-up is published in our postmortem archive.
- Identified16:58 UTC
A metadata shard failed to rejoin quorum after a routine host replacement. Draining traffic to the remaining two shards.
- Investigating16:44 UTC
Multipart uploads over 8 MB are failing in Singapore. Single-part uploads and all reads are unaffected.
Increased error rate on Compute Runners autoscale events
DegradedCompute RunnersResolved
No incidents reported between 02 May and 29 May
All clear27 days