Heisenbug at network level
Incident Report for Makersphere
Resolved
This incident has been resolved.
Posted over 1 year ago. Nov 23, 2017 - 04:41 CET
Monitoring
Good news! Our ISP has finally configured our CPE correctly. 👏
Most backend services will be online again in just a few hours.
We still have to wait for some DNS & PTR records to propagate, this can take up to 48h. 👨🏻‍💻
Posted over 1 year ago. Aug 16, 2017 - 21:29 CEST
Update
Our provider informed us that their internal ticket/management system has scheduled the deployment of RIPv2 for our account at sometime today, but no specific time was given. 😒
We should finally receive our /29 subnet tomorrow, if everything works out well. 👀
Posted over 1 year ago. Aug 15, 2017 - 19:01 CEST
Update
After multiple calls with our ISP, we're still waiting for them to enable the new CPE with RIPv2 routing.
Apparently they are experiencing some issues with their internal ticket & provisioning systems.
But those should be fixed on Thursday and then, we finally can finish the setup. 😬
Posted over 1 year ago. Aug 15, 2017 - 00:42 CEST
Update
Unfortunately we're not able to finish the setup because our ISP needs to enable RIPv2 mode on the new CPE.
This will probably happen on Monday. So no internet for us over the weekend. 😳
Posted over 1 year ago. Aug 13, 2017 - 03:30 CEST
Update
We finally received the new CPE from our ISP. 🙏
It will take us some time to get everything connected & configured but we'll keep you posted. 👩🏻‍🔧
Posted over 1 year ago. Aug 12, 2017 - 14:45 CEST
Update
We initiated the transition to a static IPv4 configuration.
In the coming days we will receive new CPE that supports the required protocols/speeds.
During this transition to our own IPv4 subnet, degraded performance is expected. 👨🏻‍🔧
Posted over 1 year ago. Aug 07, 2017 - 17:57 CEST
Identified
We're currently experiencing immensely degraded performance & reliability at our DE-RFN node.
After several hours of investigation, we have concluded that wrong routing policies on our side are part of the issue.
Based on our current understanding of the problem at hand, we believe that the ISP's DHCP setup (in combination with our routing policies) to be the root cause of this Heisenbug.
We work with our provider to replace the current (DHCP-based setup) with a "real" static configuration.
This transition (starting today) will take about a three to four day and also includes the assignment of a new IPv4 subnet. 👩🏼‍💻
Posted over 1 year ago. Aug 07, 2017 - 02:50 CEST