Heisenbug at network level

Incident Report for Makersphere

Resolved

This incident has been resolved.

Posted Nov 23, 2017 - 04:41 CET

Monitoring

Good news! Our ISP has finally configured our CPE correctly. 👏
Most backend services will be online again in just a few hours.
We still have to wait for some DNS & PTR records to propagate, this can take up to 48h. 👨🏻‍💻

Posted Aug 16, 2017 - 21:29 CEST

Update

Our provider informed us that their internal ticket/management system has scheduled the deployment of RIPv2 for our account at sometime today, but no specific time was given. 😒
We should finally receive our /29 subnet tomorrow, if everything works out well. 👀

Posted Aug 15, 2017 - 19:01 CEST

Update

After multiple calls with our ISP, we're still waiting for them to enable the new CPE with RIPv2 routing.
Apparently they are experiencing some issues with their internal ticket & provisioning systems.
But those should be fixed on Thursday and then, we finally can finish the setup. 😬

Posted Aug 15, 2017 - 00:42 CEST

Update

Unfortunately we're not able to finish the setup because our ISP needs to enable RIPv2 mode on the new CPE.
This will probably happen on Monday. So no internet for us over the weekend. 😳

Posted Aug 13, 2017 - 03:30 CEST

Update

We finally received the new CPE from our ISP. 🙏
It will take us some time to get everything connected & configured but we'll keep you posted. 👩🏻‍🔧

Posted Aug 12, 2017 - 14:45 CEST

Update

We initiated the transition to a static IPv4 configuration.
In the coming days we will receive new CPE that supports the required protocols/speeds.
During this transition to our own IPv4 subnet, degraded performance is expected. 👨🏻‍🔧

Posted Aug 07, 2017 - 17:57 CEST

Identified

We're currently experiencing immensely degraded performance & reliability at our DE-RFN node.
After several hours of investigation, we have concluded that wrong routing policies on our side are part of the issue.
Based on our current understanding of the problem at hand, we believe that the ISP's DHCP setup (in combination with our routing policies) to be the root cause of this Heisenbug.
We work with our provider to replace the current (DHCP-based setup) with a "real" static configuration.
This transition (starting today) will take about a three to four day and also includes the assignment of a new IPv4 subnet. 👩🏼‍💻

Posted Aug 07, 2017 - 02:50 CEST