Service Disruption: All Services

All GABLVM services experienced an unplanned disruption due to a hosting infrastructure issue. The root cause has been identified and resolved.

• Posted: Mar 29, 2026, 19:30 UTC
• Resolved: Mar 30, 2026, 18:16 UTC
• Duration: 22h 46m

Timeline

1. resolved

   Mar 30, 2026, 18:16 UTC

   This incident has been permanently resolved. Our hosting provider confirmed the root cause was disk I/O errors on the physical host server caused by orphaned virtual disk images from other customers. Six virtual machines were affected, not just ours. Our server has been migrated to a new, healthier physical host in a different data center with significantly lower density (6 servers vs 52 on the previous host). All services have been verified and are operating normally on the new infrastructure.

2. monitoring

   11:15 UTC

   Our hosting provider has confirmed the root cause: disk I/O errors on the physical host affected multiple servers, not just ours. All services are currently running normally. We are working with the provider on a permanent infrastructure resolution and will update when that is complete.

3. monitoring

   Mar 29, 2026, 21:20 UTC

   All services have been restored following a server restart. We are working with our hosting provider to determine the root cause and prevent recurrence. This is the second infrastructure-related disruption in 7 days.

4. identified

   20:30 UTC

   The issue has been identified as a hosting infrastructure problem external to our server. Our server logs show no internal errors, crashes, or resource exhaustion. We have contacted our hosting provider for investigation.

5. investigating

   19:40 UTC

   All services became unresponsive. Our monitoring systems detected the disruption.

Permalink: /incident/41