Website: automatic library update rolled back four times this morning, no visitor impact found

On the morning of September 25, 2026, the automatic update process for gablvm.org applied a new release of a page rendering library that turned out to break every page. The process detected this within seconds through its own test requests and restored the previous version each time, in about thirty seconds, but it repeated the attempt at 8:30, 8:45, 9:00 and 9:15 AM Central before it was stopped. Our server logs show that every error response in those windows went to the update process itself, and none to a visitor, a search engine or the GABLVM app. No data was affected. The upstream fix was released on September 26, and the site now runs the updated library normally. The update process was also changed so a version that fails its test is not tried again.

• Posted: Sep 25, 2026, 13:30 UTC
• Resolved: Sep 25, 2026, 14:21 UTC
• Duration: 51m

Timeline

1. update

   Sep 26, 2026, 22:08 UTC

   Follow-up: the upstream fix was released in Drupal 11.4.8 on September 26. Our update process applied it, the hold on the library was removed, and gablvm.org has been running the new library version normally since 5:08 PM Central.

2. resolved

   Sep 25, 2026, 14:21 UTC

   This incident has been resolved. The update process now remembers a release that fails its test and leaves it alone until a newer one appears. Log review found no visitor requests among the error responses; every error went to the update process itself. The library stays held at its previous release until the upstream fix is published.

3. monitoring

   14:18 UTC

   A fix has been implemented and we are monitoring the results. The library release has been blocked from being installed, and the update process was watched skipping it while continuing with other updates.

4. identified

   14:13 UTC

   The issue has been identified. A new release of the page rendering library used by the site fails inside every page template, and the update process had retried it three more times, rolling back each time. A fix is being implemented.

5. investigating

   13:31 UTC

   We are currently investigating an alert from our automated update process. A library update applied to gablvm.org failed its post-update page test and was rolled back automatically. The site is serving pages normally while we investigate.

Permalink: /incident/90