Level3 routing in DC area
It seems traffic attempting to pass through Level3's network in the Washington, DC area is getting lost in the abyss. Here's a trace from VZ residential FIOS to www.level3.com: Tracing route to www.level3.com [4.68.95.11] over a maximum of 30 hops: 1 <1 ms <1 ms <1 ms Wireless_Broadband_Router.home [192.168.3.254] 2 7 ms 6 ms 6 ms L100.BLTMMD-VFTTP-40.verizon-gni.net [96.244.79. 1] 3 11 ms 8 ms 9 ms G10-0-1-440.BLTMMD-LCR-04.verizon-gni.net [130.8 1.110.158] 4 13 ms 11 ms 14 ms so-2-0-0-0.PHIL-BB-RTR2.verizon-gni.net [130.81. 28.82] 5 21 ms 19 ms 19 ms so-7-1-0-0.RES-BB-RTR2.verizon-gni.net [130.81.1 9.106] 6 20 ms 19 ms 19 ms 0.ae2.BR2.IAD8.ALTER.NET [152.63.34.73] 7 16 ms 16 ms 18 ms ae7.edge1.washingtondc4.level3.net [4.68.62.137] 8 26 ms 19 ms 17 ms vlan80.csw3.Washington1.Level3.net [4.69.149.190 ] 9 22 ms 24 ms 26 ms ae-92-92.ebr2.Washington1.Level3.net [4.69.134.1 57] 10 * * * Request timed out. Anyone else?
I show here a traffic dropoff starting around 10(ish) minutes ago, and Internetpulse shows red on Level(3) <-> Savvis and Verizon <-> NTT. Might be peering related? -Bill
-----Original Message----- From: Robert Johnson [mailto:fasterfourier@gmail.com] Sent: Thursday, December 16, 2010 5:07 PM To: outages@outages.org Subject: [outages] Level3 routing in DC area
It seems traffic attempting to pass through Level3's network in the Washington, DC area is getting lost in the abyss. Here's a trace from VZ residential FIOS to www.level3.com:
Tracing route to www.level3.com [4.68.95.11] over a maximum of 30 hops:
1 <1 ms <1 ms <1 ms Wireless_Broadband_Router.home [192.168.3.254] 2 7 ms 6 ms 6 ms L100.BLTMMD-VFTTP-40.verizon-gni.net [96.244.79. 1] 3 11 ms 8 ms 9 ms G10-0-1-440.BLTMMD-LCR-04.verizon-gni.net [130.8 1.110.158] 4 13 ms 11 ms 14 ms so-2-0-0-0.PHIL-BB-RTR2.verizon-gni.net [130.81. 28.82] 5 21 ms 19 ms 19 ms so-7-1-0-0.RES-BB-RTR2.verizon-gni.net [130.81.1 9.106] 6 20 ms 19 ms 19 ms 0.ae2.BR2.IAD8.ALTER.NET [152.63.34.73] 7 16 ms 16 ms 18 ms ae7.edge1.washingtondc4.level3.net [4.68.62.137] 8 26 ms 19 ms 17 ms vlan80.csw3.Washington1.Level3.net [4.69.149.190 ] 9 22 ms 24 ms 26 ms ae-92-92.ebr2.Washington1.Level3.net [4.69.134.1 57] 10 * * * Request timed out.
Anyone else?
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
I should have added it has been working for us again for the past five minutes or so.
Seeing issues with packet loss on Level3 at 60 hudson. -----Original Message----- From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On Behalf Of Tim Sent: Thursday, December 16, 2010 4:24 PM To: 'Robert Johnson'; outages@outages.org Subject: Re: [outages] Level3 routing in DC area I should have added it has been working for us again for the past five minutes or so. _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
According to Level3 they had a router failure in the WDC area. Master ticket is 4521074. They report the issue is isolated and services are being routed around the failed equipment. -----Original Message----- From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On Behalf Of Tim Sent: Thursday, December 16, 2010 5:24 PM To: 'Robert Johnson'; outages@outages.org Subject: Re: [outages] Level3 routing in DC area I should have added it has been working for us again for the past five minutes or so. _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
I'm seeing similar from residential FiOS in DC:: Jason-Sparkss-MacBook-Pro:~ jlsparks$ traceroute www.level3.com traceroute to www.level3.com (4.68.95.11), 64 hops max, 52 byte packets 1 192.168.1.1 (192.168.1.1) 3.706 ms 0.796 ms 0.854 ms 2 l100.washdc-vfttp-47.verizon-gni.net (96.255.98.1) 2.191 ms 1.793 ms 1.779 ms 3 g4-0-1-747.washdc-lcr-07.verizon-gni.net (130.81.59.152) 2.397 ms 2.125 ms 2.186 ms 4 so-3-0-0-0.lcc1-res-bb-rtr1-re1.verizon-gni.net (130.81.29.0) 3.974 ms 3.023 ms 2.965 ms 5 0.ae1.br1.iad8.alter.net (152.63.32.141) 5.651 ms 3.790 ms 0.ae1.br2.iad8.alter.net (152.63.34.21) 3.579 ms 6 ae6.edge1.washingtondc4.level3.net (4.68.62.133) 6.286 ms * * 7 vlan90.csw4.washington1.level3.net (4.69.149.254) 4.479 ms vlan70.csw2.washington1.level3.net (4.69.149.126) 10.478 ms vlan90.csw4.washington1.level3.net (4.69.149.254) 7.878 ms 8 ae-71-71.ebr1.washington1.level3.net (4.69.134.133) 4.918 ms 4.653 ms ae-61-61.ebr1.washington1.level3.net (4.69.134.129) 4.540 ms 9 ae-8-8.ebr1.washington12.level3.net (4.69.143.218) 5.132 ms 5.234 ms 5.787 ms 10 ae-1-100.ebr2.washington12.level3.net (4.69.143.214) 5.241 ms 5.075 ms 87.833 ms 11 ae-6-6.ebr2.chicago2.level3.net (4.69.148.146) 25.148 ms 24.995 ms 23.701 ms 12 ae-1-100.ebr1.chicago2.level3.net (4.69.132.113) 24.656 ms 24.834 ms 24.737 ms 13 ae-3-3.ebr2.denver1.level3.net (4.69.132.61) 50.896 ms 53.748 ms 54.740 ms 14 ge-9-1.hsa1.denver1.level3.net (4.68.107.99) 50.641 ms 53.752 ms 50.221 ms 15 4.68.94.27 (4.68.94.27) 51.479 ms 53.397 ms 51.894 ms 16 4.68.94.33 (4.68.94.33) 52.956 ms 52.233 ms 52.237 ms 17 * * * 18 * * * On Thu, Dec 16, 2010 at 5:06 PM, Robert Johnson <fasterfourier@gmail.com>wrote:
It seems traffic attempting to pass through Level3's network in the Washington, DC area is getting lost in the abyss. Here's a trace from VZ residential FIOS to www.level3.com:
Tracing route to www.level3.com [4.68.95.11] over a maximum of 30 hops:
1 <1 ms <1 ms <1 ms Wireless_Broadband_Router.home [192.168.3.254] 2 7 ms 6 ms 6 ms L100.BLTMMD-VFTTP-40.verizon-gni.net[96.244.79. 1] 3 11 ms 8 ms 9 ms G10-0-1-440.BLTMMD-LCR-04.verizon-gni.net[130.8 1.110.158] 4 13 ms 11 ms 14 ms so-2-0-0-0.PHIL-BB-RTR2.verizon-gni.net[130.81. 28.82] 5 21 ms 19 ms 19 ms so-7-1-0-0.RES-BB-RTR2.verizon-gni.net[130.81.1 9.106] 6 20 ms 19 ms 19 ms 0.ae2.BR2.IAD8.ALTER.NET [152.63.34.73] 7 16 ms 16 ms 18 ms ae7.edge1.washingtondc4.level3.net[4.68.62.137] 8 26 ms 19 ms 17 ms vlan80.csw3.Washington1.Level3.net[4.69.149.190 ] 9 22 ms 24 ms 26 ms ae-92-92.ebr2.Washington1.Level3.net[4.69.134.1 57] 10 * * * Request timed out.
Anyone else?
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
participants (6)
-
Bill Wichers -
Jason L. Sparks -
Joseph Jackson -
Pugh, James -
Robert Johnson -
Tim