Level3 huge packet loss around AMS
Anyway else seeing this? Huge loss in Amsterdam on level3 1. 10.15.252.1 0.0% 50 0.4 0.3 0.2 0.6 0.1 2. 69.55.63.193 0.0% 50 0.6 2.4 0.4 11.0 3.2 3. 198.211.111.78 0.0% 50 0.6 0.6 0.4 1.3 0.2 4. nyk-b6-link.telia.net 0.0% 50 0.8 1.0 0.7 9.7 1.3 5. nyk-b5-link.telia.net 0.0% 50 0.8 1.1 0.7 9.8 1.4 6. ??? 100.0 50 0.0 0.0 0.0 0.0 0.0 7. ae-237-3613.edge6.Amsterdam1.Level3.net 88.0% 50 142.2 141.2 139.3 142.2 1.2 8. te5-8.bg1.am3.eu.equinix.net 88.0% 50 133.8 137.3 133.8 140.1 2.6 9. v116.ed1.am3.eu.equinix.net 90.0% 50 138.7 136.7 131.1 139.1 3.4 10. 5.175.83.162 90.0% 50 137.2 138.4 137.2 139.2 0.9 11. 5.175.83.163 90.0% 50 133.4 134.5 133.2 135.4 1.1 Luca
Confirming. Level3 seems to be affected mostly... but other carriers still have higher latency/packetloss, maybe due to more load? On 04.09.2015 at 23:52 Luca Salvatore via Outages wrote:
Anyway else seeing this? Huge loss in Amsterdam on level3
1. 10.15.252.1 0.0% 50 0.4 0.3 0.2 0.6 0.1 2. 69.55.63.193 0.0% 50 0.6 2.4 0.4 11.0 3.2 3. 198.211.111.78 0.0% 50 0.6 0.6 0.4 1.3 0.2 4. nyk-b6-link.telia.net <http://nyk-b6-link.telia.net> 0.0% 50 0.8 1.0 0.7 9.7 1.3 5. nyk-b5-link.telia.net <http://nyk-b5-link.telia.net> 0.0% 50 0.8 1.1 0.7 9.8 1.4 6. ??? 100.0 50 0.0 0.0 0.0 0.0 0.0 7. ae-237-3613.edge6.Amsterdam1.Level3.net <http://ae-237-3613.edge6.Amsterdam1.Level3.net> 88.0% 50 142.2 141.2 139.3 142.2 1.2 8. te5-8.bg1.am3.eu.equinix.net <http://te5-8.bg1.am3.eu.equinix.net> 88.0% 50 133.8 137.3 133.8 140.1 2.6 9. v116.ed1.am3.eu.equinix.net <http://v116.ed1.am3.eu.equinix.net> 90.0% 50 138.7 136.7 131.1 139.1 3.4 10. 5.175.83.162 90.0% 50 137.2 138.4 137.2 139.2 0.9 11. 5.175.83.163 90.0% 50 133.4 134.5 133.2 135.4 1.1
Luca
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Hi - Yes, we are seeing inbound connectivity checks from Europe failing to us (polyvore.com), and are unable to hit AWS EU-CENTRAL-1 endpoints from parts of our infrastructure. I do not have other information at this time specific to the incident. - Stu On Friday, September 4, 2015 02:52 PM PDT, Luca Salvatore via Outages <outages@outages.org> wrote:
Anyway else seeing this? Huge loss in Amsterdam on level3
1. 10.15.252.1 0.0% 50 0.4 0.3 0.2 0.6 0.1 2. 69.55.63.193 0.0% 50 0.6 2.4 0.4 11.0 3.2 3. 198.211.111.78 0.0% 50 0.6 0.6 0.4 1.3 0.2 4. nyk-b6-link.telia.net 0.0% 50 0.8 1.0 0.7 9.7 1.3 5. nyk-b5-link.telia.net 0.0% 50 0.8 1.1 0.7 9.8 1.4 6. ??? 100.0 50 0.0 0.0 0.0 0.0 0.0 7. ae-237-3613.edge6.Amsterdam1.Level3.net 88.0% 50 142.2 141.2 139.3 142.2 1.2 8. te5-8.bg1.am3.eu.equinix.net 88.0% 50 133.8 137.3 133.8 140.1 2.6 9. v116.ed1.am3.eu.equinix.net 90.0% 50 138.7 136.7 131.1 139.1 3.4 10. 5.175.83.162 90.0% 50 137.2 138.4 137.2 139.2 0.9 11. 5.175.83.163 90.0% 50 133.4 134.5 133.2 135.4 1.1
Luca
Getting better *right now*. Close to 0% packet-loss again. On 04.09.2015 at 23:59 Stuart Cianos via Outages wrote:
Hi -
Yes, we are seeing inbound connectivity checks from Europe failing to us (polyvore.com), and are unable to hit AWS EU-CENTRAL-1 endpoints from parts of our infrastructure.
I do not have other information at this time specific to the incident.
- Stu
On Friday, September 4, 2015 02:52 PM PDT, Luca Salvatore via Outages <outages@outages.org> wrote:
Anyway else seeing this? Huge loss in Amsterdam on level3
1. 10.15.252.1 0.0% 50 0.4 0.3 0.2 0.6 0.1 2. 69.55.63.193 0.0% 50 0.6 2.4 0.4 11.0 3.2 3. 198.211.111.78 0.0% 50 0.6 0.6 0.4 1.3 0.2 4. nyk-b6-link.telia.net 0.0% 50 0.8 1.0 0.7 9.7 1.3 5. nyk-b5-link.telia.net 0.0% 50 0.8 1.1 0.7 9.8 1.4 6. ??? 100.0 50 0.0 0.0 0.0 0.0 0.0 7. ae-237-3613.edge6.Amsterdam1.Level3.net 88.0% 50 142.2 141.2 139.3 142.2 1.2 8. te5-8.bg1.am3.eu.equinix.net 88.0% 50 133.8 137.3 133.8 140.1 2.6 9. v116.ed1.am3.eu.equinix.net 90.0% 50 138.7 136.7 131.1 139.1 3.4 10. 5.175.83.162 90.0% 50 137.2 138.4 137.2 139.2 0.9 11. 5.175.83.163 90.0% 50 133.4 134.5 133.2 135.4 1.1
Luca
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Yup lots of packet loss (up to 100%) between US and EU over level3. First packet loss detected at 21:04 UTC, things got better at around 21:55 UTC .-- My secret spy satellite informs me that at 2015-09-04 2:52 PM Luca Salvatore via Outages wrote:
Anyway else seeing this? Huge loss in Amsterdam on level3
1. 10.15.252.1 0.0% 50 0.4 0.3 0.2 0.6 0.1 2. 69.55.63.193 0.0% 50 0.6 2.4 0.4 11.0 3.2 3. 198.211.111.78 0.0% 50 0.6 0.6 0.4 1.3 0.2 4. nyk-b6-link.telia.net <http://nyk-b6-link.telia.net> 0.0% 50 0.8 1.0 0.7 9.7 1.3 5. nyk-b5-link.telia.net <http://nyk-b5-link.telia.net> 0.0% 50 0.8 1.1 0.7 9.8 1.4 6. ??? 100.0 50 0.0 0.0 0.0 0.0 0.0 7. ae-237-3613.edge6.Amsterdam1.Level3.net <http://ae-237-3613.edge6.Amsterdam1.Level3.net> 88.0% 50 142.2 141.2 139.3 142.2 1.2 8. te5-8.bg1.am3.eu.equinix.net <http://te5-8.bg1.am3.eu.equinix.net> 88.0% 50 133.8 137.3 133.8 140.1 2.6 9. v116.ed1.am3.eu.equinix.net <http://v116.ed1.am3.eu.equinix.net> 90.0% 50 138.7 136.7 131.1 139.1 3.4 10. 5.175.83.162 90.0% 50 137.2 138.4 137.2 139.2 0.9 11. 5.175.83.163 90.0% 50 133.4 134.5 133.2 135.4 1.1
Luca
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
In case anybody was wondering what happened.. *** CASCADED EXTERNAL NOTES 04-Sep-2015 23:10:04 GMT From CASE: 9674475 - Event Event Conclusion Summary Outage Start: September 04, 2015 21:05 GMT Outage Stop: September 04, 2015 22:002 GMT Root Cause: A planned maintenance on a local providers network in London, England impacted Transport Unprotected, CDN and IP services. Fix Action: The IP NOC migrated traffic and services have been restored. Reason for Outage (RFO) Summary: A planned maintenance on a local providers network in London, England impacted Transport Unprotected, CDN and IP services. The IP NOC migrated traffic and services have been restored. If additional issues are experienced, please contact the Level 3 Technical Service Center.
participants (5)
-
Andree Toonk -
Fred Hollis -
Luca Salvatore -
Ryan Gelobter -
Stuart Cianos