Distributed storage unavailable
Incident Report for Makersphere
Resolved
We've implemented a new, fully distributed storage layer based on Portworx.
The service will be rolled-out in the coming days. 💪
Posted Aug 05, 2017 - 18:08 CEST
Identified
We've decided to switch from GlusterFS as our storage backend to Portworx, which provides support for Docker Volumes natively.
Secondly, in these nearly two weeks of investigation, we've already deployed a fix to make sure systemd waits for ZeroTier One (VPN-like service) to become available. We'll publish this improved unit file soon on our GitHub.
And last but not least, we'll take parts of our cluster down for maintenance until March 4th. 👨‍🔧
Posted Feb 26, 2017 - 01:27 CET
Investigating
We discovered that our GlusterFS based distributed storage becomes unavailable after rebooting storage nodes.
It looks like mounting volumes using systemd doesn't wait for the Gluster service to be ready. 🕵️
Posted Feb 14, 2017 - 21:57 CET
This incident affected: Backend (LDAP Directory, GitLab).