Using road runner in Brooklyn NY. Cant access certain sites ( Facebook, twitter, citibank) . All of them have one thing in common, they go thru level3. (FYI full name of hop 7 is xe-4-2-0.edge4.frankfurt1.level3.net ) sajal@sajal-laptop:~$ mtr --report www.citi.com HOST: sajal-laptop Loss% Snt Last Avg Best Wrst StDev 1.|-- 192.168.1.1 0.0% 10 1.2 110.3 1.0 603.9 230.5 2.|-- cpe-24-90-104-1.nyc.res.r 0.0% 10 37.9 207.9 17.0 1418. 444.2 3.|-- tge-0-0-0-1.nycmnyr-rtr2. 0.0% 10 9.8 93.3 9.4 471.9 170.3 4.|-- agg23.nyquny9101r.nyc.rr. 0.0% 10 12.6 78.3 10.1 402.2 142.2 5.|-- 107.14.19.22 0.0% 10 13.8 64.6 11.5 331.0 112.6 6.|-- ae-0-0.pr0.nyc30.tbone.rr 0.0% 10 13.8 50.9 11.4 257.6 82.3 7.|-- xe-4-2-0.edge4.frankfurt1 0.0% 10 13.4 35.0 9.1 186.5 55.8 8.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0 sajal@sajal-laptop:~$ sajal@sajal-laptop:~$ mtr --report www.twitter.com HOST: sajal-laptop Loss% Snt Last Avg Best Wrst StDev 1.|-- 192.168.1.1 0.0% 10 1.2 2.9 1.0 9.5 2.8 2.|-- cpe-24-90-104-1.nyc.res.r 0.0% 10 151.6 184.3 75.8 529.8 137.6 3.|-- tge-0-0-0-1.nycmnyr-rtr2. 0.0% 10 252.6 160.8 64.7 299.7 87.2 4.|-- agg23.nyquny9101r.nyc.rr. 0.0% 10 243.5 166.4 64.1 345.8 88.3 5.|-- ae-3-0.cr0.nyc20.tbone.rr 0.0% 10 246.7 189.2 88.6 309.1 75.7 6.|-- 107.14.17.169 0.0% 10 256.6 175.3 102.6 256.6 60.5 7.|-- xe-4-2-0.edge4.frankfurt1 0.0% 10 258.6 179.6 95.4 302.2 69.1 8.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0 sajal@sajal-laptop:~$ sajal@sajal-laptop:~$ mtr --report www.facebook.com HOST: sajal-laptop Loss% Snt Last Avg Best Wrst StDev 1.|-- 192.168.1.1 0.0% 10 4.0 3.1 1.1 8.5 2.8 2.|-- cpe-24-90-104-1.nyc.res.r 0.0% 10 225.9 163.7 97.8 252.8 57.0 3.|-- tge-0-0-0-1.nycmnyr-rtr2. 0.0% 10 298.0 168.6 69.3 298.0 79.0 4.|-- agg23.nyquny9101r.nyc.rr. 0.0% 10 292.5 155.0 90.8 292.5 71.0 5.|-- ae-3-0.cr0.nyc20.tbone.rr 0.0% 10 239.0 148.6 68.5 248.6 63.2 6.|-- 107.14.17.169 0.0% 10 268.1 148.7 73.0 268.1 64.9 7.|-- xe-4-2-0.edge4.frankfurt1 10.0% 10 235.1 130.2 66.2 235.1 49.2 8.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0 sajal@sajal-laptop:~$ -Sajal
I just opened a ticket with L3. I've been having packet loss since 9:07AM today over my L3 circuit. I am in Philly and L3's customer access router is experiencing high packet loss. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Timothy Glen Sr. Network Engineer Advertising Specialty Institute | Promoting Success tglen@asicentral.com<mailto:tglen@asicentral.com> tel: (215) 953-3692 fax: (215) 953-3759 Think before you print. _____ From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Sajal Kayan Sent: Saturday, October 19, 2013 9:52 AM To: OUTAGES Mailing List Subject: [outages] Road Runner/Level 3 broked? Using road runner in Brooklyn NY. Cant access certain sites ( Facebook, twitter, citibank) . All of them have one thing in common, they go thru level3. (FYI full name of hop 7 is xe-4-2-0.edge4.frankfurt1.level3.net<http://xe-4-2-0.edge4.frankfurt1.level3.net> ) sajal@sajal-laptop:~$ mtr --report www.citi.com<http://www.citi.com> HOST: sajal-laptop Loss% Snt Last Avg Best Wrst StDev 1.|-- 192.168.1.1 0.0% 10 1.2 110.3 1.0 603.9 230.5 2.|-- cpe-24-90-104-1.nyc.res.r 0.0% 10 37.9 207.9 17.0 1418. 444.2 3.|-- tge-0-0-0-1.nycmnyr-rtr2. 0.0% 10 9.8 93.3 9.4 471.9 170.3 4.|-- agg23.nyquny9101r.nyc.rr. 0.0% 10 12.6 78.3 10.1 402.2 142.2 5.|-- 107.14.19.22 0.0% 10 13.8 64.6 11.5 331.0 112.6 6.|-- ae-0-0.pr0.nyc30.tbone.rr 0.0% 10 13.8 50.9 11.4 257.6 82.3 7.|-- xe-4-2-0.edge4.frankfurt1 0.0% 10 13.4 35.0 9.1 186.5 55.8 8.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0 sajal@sajal-laptop:~$ sajal@sajal-laptop:~$ mtr --report www.twitter.com<http://www.twitter.com> HOST: sajal-laptop Loss% Snt Last Avg Best Wrst StDev 1.|-- 192.168.1.1 0.0% 10 1.2 2.9 1.0 9.5 2.8 2.|-- cpe-24-90-104-1.nyc.res.r 0.0% 10 151.6 184.3 75.8 529.8 137.6 3.|-- tge-0-0-0-1.nycmnyr-rtr2. 0.0% 10 252.6 160.8 64.7 299.7 87.2 4.|-- agg23.nyquny9101r.nyc.rr. 0.0% 10 243.5 166.4 64.1 345.8 88.3 5.|-- ae-3-0.cr0.nyc20.tbone.rr 0.0% 10 246.7 189.2 88.6 309.1 75.7 6.|-- 107.14.17.169 0.0% 10 256.6 175.3 102.6 256.6 60.5 7.|-- xe-4-2-0.edge4.frankfurt1 0.0% 10 258.6 179.6 95.4 302.2 69.1 8.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0 sajal@sajal-laptop:~$ sajal@sajal-laptop:~$ mtr --report www.facebook.com<http://www.facebook.com> HOST: sajal-laptop Loss% Snt Last Avg Best Wrst StDev 1.|-- 192.168.1.1 0.0% 10 4.0 3.1 1.1 8.5 2.8 2.|-- cpe-24-90-104-1.nyc.res.r 0.0% 10 225.9 163.7 97.8 252.8 57.0 3.|-- tge-0-0-0-1.nycmnyr-rtr2. 0.0% 10 298.0 168.6 69.3 298.0 79.0 4.|-- agg23.nyquny9101r.nyc.rr. 0.0% 10 292.5 155.0 90.8 292.5 71.0 5.|-- ae-3-0.cr0.nyc20.tbone.rr 0.0% 10 239.0 148.6 68.5 248.6 63.2 6.|-- 107.14.17.169 0.0% 10 268.1 148.7 73.0 268.1 64.9 7.|-- xe-4-2-0.edge4.frankfurt1 10.0% 10 235.1 130.2 66.2 235.1 49.2 8.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0 sajal@sajal-laptop:~$ -Sajal
I can confirm the same from our datacenter in NY. Tracepath goes in a loop… From: Tim Glen <tglen@asicentral.com<mailto:tglen@asicentral.com>> Date: Saturday, October 19, 2013 6:59 AM To: 'Sajal Kayan' <sajal83@gmail.com<mailto:sajal83@gmail.com>>, OUTAGES Mailing List <Outages@outages.org<mailto:Outages@outages.org>> Subject: Re: [outages] Road Runner/Level 3 broked? I just opened a ticket with L3. I've been having packet loss since 9:07AM today over my L3 circuit. I am in Philly and L3's customer access router is experiencing high packet loss. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Timothy Glen Sr. Network Engineer Advertising Specialty Institute | Promoting Success tglen@asicentral.com<mailto:tglen@asicentral.com> tel: (215) 953-3692 fax: (215) 953-3759 Think before you print. ________________________________ From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Sajal Kayan Sent: Saturday, October 19, 2013 9:52 AM To: OUTAGES Mailing List Subject: [outages] Road Runner/Level 3 broked? Using road runner in Brooklyn NY. Cant access certain sites ( Facebook, twitter, citibank) . All of them have one thing in common, they go thru level3. (FYI full name of hop 7 is xe-4-2-0.edge4.frankfurt1.level3.net<http://xe-4-2-0.edge4.frankfurt1.level3.net> ) sajal@sajal-laptop:~$ mtr --report www.citi.com<http://www.citi.com> HOST: sajal-laptop Loss% Snt Last Avg Best Wrst StDev 1.|-- 192.168.1.1 0.0% 10 1.2 110.3 1.0 603.9 230.5 2.|-- cpe-24-90-104-1.nyc.res.r 0.0% 10 37.9 207.9 17.0 1418. 444.2 3.|-- tge-0-0-0-1.nycmnyr-rtr2. 0.0% 10 9.8 93.3 9.4 471.9 170.3 4.|-- agg23.nyquny9101r.nyc.rr. 0.0% 10 12.6 78.3 10.1 402.2 142.2 5.|-- 107.14.19.22 0.0% 10 13.8 64.6 11.5 331.0 112.6 6.|-- ae-0-0.pr0.nyc30.tbone.rr 0.0% 10 13.8 50.9 11.4 257.6 82.3 7.|-- xe-4-2-0.edge4.frankfurt1 0.0% 10 13.4 35.0 9.1 186.5 55.8 8.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0 sajal@sajal-laptop:~$ sajal@sajal-laptop:~$ mtr --report www.twitter.com<http://www.twitter.com> HOST: sajal-laptop Loss% Snt Last Avg Best Wrst StDev 1.|-- 192.168.1.1 0.0% 10 1.2 2.9 1.0 9.5 2.8 2.|-- cpe-24-90-104-1.nyc.res.r 0.0% 10 151.6 184.3 75.8 529.8 137.6 3.|-- tge-0-0-0-1.nycmnyr-rtr2. 0.0% 10 252.6 160.8 64.7 299.7 87.2 4.|-- agg23.nyquny9101r.nyc.rr. 0.0% 10 243.5 166.4 64.1 345.8 88.3 5.|-- ae-3-0.cr0.nyc20.tbone.rr 0.0% 10 246.7 189.2 88.6 309.1 75.7 6.|-- 107.14.17.169 0.0% 10 256.6 175.3 102.6 256.6 60.5 7.|-- xe-4-2-0.edge4.frankfurt1 0.0% 10 258.6 179.6 95.4 302.2 69.1 8.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0 sajal@sajal-laptop:~$ sajal@sajal-laptop:~$ mtr --report www.facebook.com<http://www.facebook.com> HOST: sajal-laptop Loss% Snt Last Avg Best Wrst StDev 1.|-- 192.168.1.1 0.0% 10 4.0 3.1 1.1 8.5 2.8 2.|-- cpe-24-90-104-1.nyc.res.r 0.0% 10 225.9 163.7 97.8 252.8 57.0 3.|-- tge-0-0-0-1.nycmnyr-rtr2. 0.0% 10 298.0 168.6 69.3 298.0 79.0 4.|-- agg23.nyquny9101r.nyc.rr. 0.0% 10 292.5 155.0 90.8 292.5 71.0 5.|-- ae-3-0.cr0.nyc20.tbone.rr 0.0% 10 239.0 148.6 68.5 248.6 63.2 6.|-- 107.14.17.169 0.0% 10 268.1 148.7 73.0 268.1 64.9 7.|-- xe-4-2-0.edge4.frankfurt1 10.0% 10 235.1 130.2 66.2 235.1 49.2 8.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0 sajal@sajal-laptop:~$ -Sajal
Same here. Our White Plains NY POP starting having issues at about 9:05am. Had to shut down circuit. Ticket has been opened with Level3. From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Darren Schreiber Sent: Saturday, October 19, 2013 10:04 AM To: Tim Glen; 'Sajal Kayan'; OUTAGES Mailing List Subject: Re: [outages] Road Runner/Level 3 broked? I can confirm the same from our datacenter in NY. Tracepath goes in a loop. From: Tim Glen <tglen@asicentral.com <mailto:tglen@asicentral.com> > Date: Saturday, October 19, 2013 6:59 AM To: 'Sajal Kayan' <sajal83@gmail.com <mailto:sajal83@gmail.com> >, OUTAGES Mailing List <Outages@outages.org <mailto:Outages@outages.org> > Subject: Re: [outages] Road Runner/Level 3 broked? I just opened a ticket with L3. I've been having packet loss since 9:07AM today over my L3 circuit. I am in Philly and L3's customer access router is experiencing high packet loss. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Timothy Glen Sr. Network Engineer Advertising Specialty Institute | Promoting Success tglen@asicentral.com <mailto:tglen@asicentral.com> tel: (215) 953-3692 fax: (215) 953-3759 Think before you print. _____ From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Sajal Kayan Sent: Saturday, October 19, 2013 9:52 AM To: OUTAGES Mailing List Subject: [outages] Road Runner/Level 3 broked? Using road runner in Brooklyn NY. Cant access certain sites ( Facebook, twitter, citibank) . All of them have one thing in common, they go thru level3. (FYI full name of hop 7 is xe-4-2-0.edge4.frankfurt1.level3.net <http://xe-4-2-0.edge4.frankfurt1.level3.net> ) sajal@sajal-laptop:~$ mtr --report www.citi.com <http://www.citi.com> HOST: sajal-laptop Loss% Snt Last Avg Best Wrst StDev 1.|-- 192.168.1.1 0.0% 10 1.2 110.3 1.0 603.9 230.5 2.|-- cpe-24-90-104-1.nyc.res.r 0.0% 10 37.9 207.9 17.0 1418. 444.2 3.|-- tge-0-0-0-1.nycmnyr-rtr2. 0.0% 10 9.8 93.3 9.4 471.9 170.3 4.|-- agg23.nyquny9101r.nyc.rr. 0.0% 10 12.6 78.3 10.1 402.2 142.2 5.|-- 107.14.19.22 0.0% 10 13.8 64.6 11.5 331.0 112.6 6.|-- ae-0-0.pr0.nyc30.tbone.rr 0.0% 10 13.8 50.9 11.4 257.6 82.3 7.|-- xe-4-2-0.edge4.frankfurt1 0.0% 10 13.4 35.0 9.1 186.5 55.8 8.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0 sajal@sajal-laptop:~$ sajal@sajal-laptop:~$ mtr --report www.twitter.com <http://www.twitter.com> HOST: sajal-laptop Loss% Snt Last Avg Best Wrst StDev 1.|-- 192.168.1.1 0.0% 10 1.2 2.9 1.0 9.5 2.8 2.|-- cpe-24-90-104-1.nyc.res.r 0.0% 10 151.6 184.3 75.8 529.8 137.6 3.|-- tge-0-0-0-1.nycmnyr-rtr2. 0.0% 10 252.6 160.8 64.7 299.7 87.2 4.|-- agg23.nyquny9101r.nyc.rr. 0.0% 10 243.5 166.4 64.1 345.8 88.3 5.|-- ae-3-0.cr0.nyc20.tbone.rr 0.0% 10 246.7 189.2 88.6 309.1 75.7 6.|-- 107.14.17.169 0.0% 10 256.6 175.3 102.6 256.6 60.5 7.|-- xe-4-2-0.edge4.frankfurt1 0.0% 10 258.6 179.6 95.4 302.2 69.1 8.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0 sajal@sajal-laptop:~$ sajal@sajal-laptop:~$ mtr --report www.facebook.com <http://www.facebook.com> HOST: sajal-laptop Loss% Snt Last Avg Best Wrst StDev 1.|-- 192.168.1.1 0.0% 10 4.0 3.1 1.1 8.5 2.8 2.|-- cpe-24-90-104-1.nyc.res.r 0.0% 10 225.9 163.7 97.8 252.8 57.0 3.|-- tge-0-0-0-1.nycmnyr-rtr2. 0.0% 10 298.0 168.6 69.3 298.0 79.0 4.|-- agg23.nyquny9101r.nyc.rr. 0.0% 10 292.5 155.0 90.8 292.5 71.0 5.|-- ae-3-0.cr0.nyc20.tbone.rr 0.0% 10 239.0 148.6 68.5 248.6 63.2 6.|-- 107.14.17.169 0.0% 10 268.1 148.7 73.0 268.1 64.9 7.|-- xe-4-2-0.edge4.frankfurt1 10.0% 10 235.1 130.2 66.2 235.1 49.2 8.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0 sajal@sajal-laptop:~$ -Sajal
Same issue seen from Cablevision's network this morning. Traceroutes seem to die after reaching edge2.newark1.level3.net for me. This is to www.level3.net: 3 19 ms 9 ms 10 ms 67.59.240.101 4 12 ms 10 ms 10 ms rtr1-ge6-1.mhe.prnynj.cv.net [67.83.249.129] 5 17 ms 18 ms 12 ms 65.19.119.209 6 * * * Request timed out. 7 26 ms 11 ms 18 ms xe-11-3-2.edge2.Newark1.Level3.net [4.28.5.117] 8 * * * Request timed out. 9 * * * Request timed out. 10 * * * Request timed out. 11 * * * Request timed out. 12 * * * Request timed out. 13 * * * Request timed out. 14 * * * Request timed out. 15 * * * Request timed out. I was seeing similar things going through Cogent, but I suspect it's asymmetrical routing with the return traffic coming in via Level 3. -Vinny From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Darren Schreiber Sent: Saturday, October 19, 2013 10:04 AM To: Tim Glen; 'Sajal Kayan'; OUTAGES Mailing List Subject: Re: [outages] Road Runner/Level 3 broked? I can confirm the same from our datacenter in NY. Tracepath goes in a loop... From: Tim Glen <tglen@asicentral.com<mailto:tglen@asicentral.com>> Date: Saturday, October 19, 2013 6:59 AM To: 'Sajal Kayan' <sajal83@gmail.com<mailto:sajal83@gmail.com>>, OUTAGES Mailing List <Outages@outages.org<mailto:Outages@outages.org>> Subject: Re: [outages] Road Runner/Level 3 broked? I just opened a ticket with L3. I've been having packet loss since 9:07AM today over my L3 circuit. I am in Philly and L3's customer access router is experiencing high packet loss. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Timothy Glen Sr. Network Engineer Advertising Specialty Institute | Promoting Success tglen@asicentral.com<mailto:tglen@asicentral.com> tel: (215) 953-3692 fax: (215) 953-3759 Think before you print. ________________________________ From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Sajal Kayan Sent: Saturday, October 19, 2013 9:52 AM To: OUTAGES Mailing List Subject: [outages] Road Runner/Level 3 broked? Using road runner in Brooklyn NY. Cant access certain sites ( Facebook, twitter, citibank) . All of them have one thing in common, they go thru level3. (FYI full name of hop 7 is xe-4-2-0.edge4.frankfurt1.level3.net<http://xe-4-2-0.edge4.frankfurt1.level3.net> ) sajal@sajal-laptop:~$ mtr --report www.citi.com<http://www.citi.com> HOST: sajal-laptop Loss% Snt Last Avg Best Wrst StDev 1.|-- 192.168.1.1 0.0% 10 1.2 110.3 1.0 603.9 230.5 2.|-- cpe-24-90-104-1.nyc.res.r 0.0% 10 37.9 207.9 17.0 1418. 444.2 3.|-- tge-0-0-0-1.nycmnyr-rtr2. 0.0% 10 9.8 93.3 9.4 471.9 170.3 4.|-- agg23.nyquny9101r.nyc.rr. 0.0% 10 12.6 78.3 10.1 402.2 142.2 5.|-- 107.14.19.22 0.0% 10 13.8 64.6 11.5 331.0 112.6 6.|-- ae-0-0.pr0.nyc30.tbone.rr 0.0% 10 13.8 50.9 11.4 257.6 82.3 7.|-- xe-4-2-0.edge4.frankfurt1 0.0% 10 13.4 35.0 9.1 186.5 55.8 8.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0 sajal@sajal-laptop:~$ sajal@sajal-laptop:~$ mtr --report www.twitter.com<http://www.twitter.com> HOST: sajal-laptop Loss% Snt Last Avg Best Wrst StDev 1.|-- 192.168.1.1 0.0% 10 1.2 2.9 1.0 9.5 2.8 2.|-- cpe-24-90-104-1.nyc.res.r 0.0% 10 151.6 184.3 75.8 529.8 137.6 3.|-- tge-0-0-0-1.nycmnyr-rtr2. 0.0% 10 252.6 160.8 64.7 299.7 87.2 4.|-- agg23.nyquny9101r.nyc.rr. 0.0% 10 243.5 166.4 64.1 345.8 88.3 5.|-- ae-3-0.cr0.nyc20.tbone.rr 0.0% 10 246.7 189.2 88.6 309.1 75.7 6.|-- 107.14.17.169 0.0% 10 256.6 175.3 102.6 256.6 60.5 7.|-- xe-4-2-0.edge4.frankfurt1 0.0% 10 258.6 179.6 95.4 302.2 69.1 8.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0 sajal@sajal-laptop:~$ sajal@sajal-laptop:~$ mtr --report www.facebook.com<http://www.facebook.com> HOST: sajal-laptop Loss% Snt Last Avg Best Wrst StDev 1.|-- 192.168.1.1 0.0% 10 4.0 3.1 1.1 8.5 2.8 2.|-- cpe-24-90-104-1.nyc.res.r 0.0% 10 225.9 163.7 97.8 252.8 57.0 3.|-- tge-0-0-0-1.nycmnyr-rtr2. 0.0% 10 298.0 168.6 69.3 298.0 79.0 4.|-- agg23.nyquny9101r.nyc.rr. 0.0% 10 292.5 155.0 90.8 292.5 71.0 5.|-- ae-3-0.cr0.nyc20.tbone.rr 0.0% 10 239.0 148.6 68.5 248.6 63.2 6.|-- 107.14.17.169 0.0% 10 268.1 148.7 73.0 268.1 64.9 7.|-- xe-4-2-0.edge4.frankfurt1 10.0% 10 235.1 130.2 66.2 235.1 49.2 8.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0 sajal@sajal-laptop:~$ -Sajal
Newark, huh? C'mon guys, we know you're crazily celebrating being the most recent state in the union to have marriage equality, but get it together, okay? On Sat, Oct 19, 2013 at 10:31 AM, <Vinny_Abello@dell.com> wrote:
Same issue seen from Cablevision’s network this morning. Traceroutes seem to die after reaching edge2.newark1.level3.net for me.****
** **
This is to www.level3.net:****
** **
3 19 ms 9 ms 10 ms 67.59.240.101****
4 12 ms 10 ms 10 ms rtr1-ge6-1.mhe.prnynj.cv.net [67.83.249.129]****
5 17 ms 18 ms 12 ms 65.19.119.209****
6 * * * Request timed out.****
7 26 ms 11 ms 18 ms xe-11-3-2.edge2.Newark1.Level3.net[4.28.5.117] ****
8 * * * Request timed out.****
9 * * * Request timed out.****
10 * * * Request timed out.****
11 * * * Request timed out.****
12 * * * Request timed out.****
13 * * * Request timed out.****
14 * * * Request timed out.****
15 * * * Request timed out.****
** **
I was seeing similar things going through Cogent, but I suspect it’s asymmetrical routing with the return traffic coming in via Level 3.****
** **
-Vinny****
** **
*From:* Outages [mailto:outages-bounces@outages.org] *On Behalf Of *Darren Schreiber *Sent:* Saturday, October 19, 2013 10:04 AM *To:* Tim Glen; 'Sajal Kayan'; OUTAGES Mailing List *Subject:* Re: [outages] Road Runner/Level 3 broked?****
** **
I can confirm the same from our datacenter in NY. Tracepath goes in a loop… ****
** **
*From: *Tim Glen <tglen@asicentral.com> *Date: *Saturday, October 19, 2013 6:59 AM *To: *'Sajal Kayan' <sajal83@gmail.com>, OUTAGES Mailing List < Outages@outages.org> *Subject: *Re: [outages] Road Runner/Level 3 broked?****
** **
I just opened a ticket with L3. ****
****
I've been having packet loss since 9:07AM today over my L3 circuit. ****
****
I am in Philly and L3's customer access router is experiencing high packet loss.****
****
****
****
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ****
Timothy Glen****
Sr. Network Engineer****
Advertising Specialty Institute | Promoting Success****
****
tglen@asicentral.com****
tel: (215) 953-3692****
fax: (215) 953-3759****
Think before you print.****
****
****
** ** ------------------------------
*From:* Outages [mailto:outages-bounces@outages.org<outages-bounces@outages.org>] *On Behalf Of *Sajal Kayan *Sent:* Saturday, October 19, 2013 9:52 AM *To:* OUTAGES Mailing List *Subject:* [outages] Road Runner/Level 3 broked?****
Using road runner in Brooklyn NY. ****
** **
Cant access certain sites ( Facebook, twitter, citibank) . All of them have one thing in common, they go thru level3.****
** **
(FYI full name of hop 7 is xe-4-2-0.edge4.frankfurt1.level3.net )****
** **
sajal@sajal-laptop:~$ mtr --report www.citi.com****
HOST: sajal-laptop Loss% Snt Last Avg Best Wrst StDev****
1.|-- 192.168.1.1 0.0% 10 1.2 110.3 1.0 603.9 230.5****
2.|-- cpe-24-90-104-1.nyc.res.r 0.0% 10 37.9 207.9 17.0 1418. 444.2****
3.|-- tge-0-0-0-1.nycmnyr-rtr2. 0.0% 10 9.8 93.3 9.4 471.9 170.3****
4.|-- agg23.nyquny9101r.nyc.rr. 0.0% 10 12.6 78.3 10.1 402.2 142.2****
5.|-- 107.14.19.22 0.0% 10 13.8 64.6 11.5 331.0 112.6****
6.|-- ae-0-0.pr0.nyc30.tbone.rr 0.0% 10 13.8 50.9 11.4 257.6 82.3****
7.|-- xe-4-2-0.edge4.frankfurt1 0.0% 10 13.4 35.0 9.1 186.5 55.8****
8.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0****
sajal@sajal-laptop:~$ ****
** **
** **
sajal@sajal-laptop:~$ mtr --report www.twitter.com****
HOST: sajal-laptop Loss% Snt Last Avg Best Wrst StDev****
1.|-- 192.168.1.1 0.0% 10 1.2 2.9 1.0 9.5 2.8****
2.|-- cpe-24-90-104-1.nyc.res.r 0.0% 10 151.6 184.3 75.8 529.8 137.6****
3.|-- tge-0-0-0-1.nycmnyr-rtr2. 0.0% 10 252.6 160.8 64.7 299.7 87.2****
4.|-- agg23.nyquny9101r.nyc.rr. 0.0% 10 243.5 166.4 64.1 345.8 88.3****
5.|-- ae-3-0.cr0.nyc20.tbone.rr 0.0% 10 246.7 189.2 88.6 309.1 75.7****
6.|-- 107.14.17.169 0.0% 10 256.6 175.3 102.6 256.6 60.5****
7.|-- xe-4-2-0.edge4.frankfurt1 0.0% 10 258.6 179.6 95.4 302.2 69.1****
8.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0****
sajal@sajal-laptop:~$ ****
** **
sajal@sajal-laptop:~$ mtr --report www.facebook.com****
HOST: sajal-laptop Loss% Snt Last Avg Best Wrst StDev****
1.|-- 192.168.1.1 0.0% 10 4.0 3.1 1.1 8.5 2.8****
2.|-- cpe-24-90-104-1.nyc.res.r 0.0% 10 225.9 163.7 97.8 252.8 57.0****
3.|-- tge-0-0-0-1.nycmnyr-rtr2. 0.0% 10 298.0 168.6 69.3 298.0 79.0****
4.|-- agg23.nyquny9101r.nyc.rr. 0.0% 10 292.5 155.0 90.8 292.5 71.0****
5.|-- ae-3-0.cr0.nyc20.tbone.rr 0.0% 10 239.0 148.6 68.5 248.6 63.2****
6.|-- 107.14.17.169 0.0% 10 268.1 148.7 73.0 268.1 64.9****
7.|-- xe-4-2-0.edge4.frankfurt1 10.0% 10 235.1 130.2 66.2 235.1 49.2****
8.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0****
sajal@sajal-laptop:~$ ****
** **
-Sajal ****
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Level3 just called me back for the ticket I filed. Tech was very knowledgable. Is aware of the issue (obviously) and has attached my ticket to the master event ticket so that I can see realtime updates. Appears the ETA is within the next 2 hours. From: Darren Schreiber <darren@2600hz.com<mailto:darren@2600hz.com>> Date: Saturday, October 19, 2013 7:03 AM To: Tim Glen <tglen@asicentral.com<mailto:tglen@asicentral.com>>, 'Sajal Kayan' <sajal83@gmail.com<mailto:sajal83@gmail.com>>, OUTAGES Mailing List <Outages@outages.org<mailto:Outages@outages.org>> Subject: Re: [outages] Road Runner/Level 3 broked? I can confirm the same from our datacenter in NY. Tracepath goes in a loop… From: Tim Glen <tglen@asicentral.com<mailto:tglen@asicentral.com>> Date: Saturday, October 19, 2013 6:59 AM To: 'Sajal Kayan' <sajal83@gmail.com<mailto:sajal83@gmail.com>>, OUTAGES Mailing List <Outages@outages.org<mailto:Outages@outages.org>> Subject: Re: [outages] Road Runner/Level 3 broked? I just opened a ticket with L3. I've been having packet loss since 9:07AM today over my L3 circuit. I am in Philly and L3's customer access router is experiencing high packet loss. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Timothy Glen Sr. Network Engineer Advertising Specialty Institute | Promoting Success tglen@asicentral.com<mailto:tglen@asicentral.com> tel: (215) 953-3692 fax: (215) 953-3759 Think before you print. ________________________________ From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Sajal Kayan Sent: Saturday, October 19, 2013 9:52 AM To: OUTAGES Mailing List Subject: [outages] Road Runner/Level 3 broked? Using road runner in Brooklyn NY. Cant access certain sites ( Facebook, twitter, citibank) . All of them have one thing in common, they go thru level3. (FYI full name of hop 7 is xe-4-2-0.edge4.frankfurt1.level3.net<http://xe-4-2-0.edge4.frankfurt1.level3.net> ) sajal@sajal-laptop:~$ mtr --report www.citi.com<http://www.citi.com> HOST: sajal-laptop Loss% Snt Last Avg Best Wrst StDev 1.|-- 192.168.1.1 0.0% 10 1.2 110.3 1.0 603.9 230.5 2.|-- cpe-24-90-104-1.nyc.res.r 0.0% 10 37.9 207.9 17.0 1418. 444.2 3.|-- tge-0-0-0-1.nycmnyr-rtr2. 0.0% 10 9.8 93.3 9.4 471.9 170.3 4.|-- agg23.nyquny9101r.nyc.rr. 0.0% 10 12.6 78.3 10.1 402.2 142.2 5.|-- 107.14.19.22 0.0% 10 13.8 64.6 11.5 331.0 112.6 6.|-- ae-0-0.pr0.nyc30.tbone.rr 0.0% 10 13.8 50.9 11.4 257.6 82.3 7.|-- xe-4-2-0.edge4.frankfurt1 0.0% 10 13.4 35.0 9.1 186.5 55.8 8.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0 sajal@sajal-laptop:~$ sajal@sajal-laptop:~$ mtr --report www.twitter.com<http://www.twitter.com> HOST: sajal-laptop Loss% Snt Last Avg Best Wrst StDev 1.|-- 192.168.1.1 0.0% 10 1.2 2.9 1.0 9.5 2.8 2.|-- cpe-24-90-104-1.nyc.res.r 0.0% 10 151.6 184.3 75.8 529.8 137.6 3.|-- tge-0-0-0-1.nycmnyr-rtr2. 0.0% 10 252.6 160.8 64.7 299.7 87.2 4.|-- agg23.nyquny9101r.nyc.rr. 0.0% 10 243.5 166.4 64.1 345.8 88.3 5.|-- ae-3-0.cr0.nyc20.tbone.rr 0.0% 10 246.7 189.2 88.6 309.1 75.7 6.|-- 107.14.17.169 0.0% 10 256.6 175.3 102.6 256.6 60.5 7.|-- xe-4-2-0.edge4.frankfurt1 0.0% 10 258.6 179.6 95.4 302.2 69.1 8.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0 sajal@sajal-laptop:~$ sajal@sajal-laptop:~$ mtr --report www.facebook.com<http://www.facebook.com> HOST: sajal-laptop Loss% Snt Last Avg Best Wrst StDev 1.|-- 192.168.1.1 0.0% 10 4.0 3.1 1.1 8.5 2.8 2.|-- cpe-24-90-104-1.nyc.res.r 0.0% 10 225.9 163.7 97.8 252.8 57.0 3.|-- tge-0-0-0-1.nycmnyr-rtr2. 0.0% 10 298.0 168.6 69.3 298.0 79.0 4.|-- agg23.nyquny9101r.nyc.rr. 0.0% 10 292.5 155.0 90.8 292.5 71.0 5.|-- ae-3-0.cr0.nyc20.tbone.rr 0.0% 10 239.0 148.6 68.5 248.6 63.2 6.|-- 107.14.17.169 0.0% 10 268.1 148.7 73.0 268.1 64.9 7.|-- xe-4-2-0.edge4.frankfurt1 10.0% 10 235.1 130.2 66.2 235.1 49.2 8.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0 sajal@sajal-laptop:~$ -Sajal
Were they able to provide any detail with respect to root cause? ________________________________________ From: Outages [outages-bounces@outages.org] On Behalf Of Darren Schreiber [darren@2600hz.com] Sent: Saturday, October 19, 2013 12:01 PM To: Tim Glen; 'Sajal Kayan'; OUTAGES Mailing List Subject: Re: [outages] Road Runner/Level 3 broked? Level3 just called me back for the ticket I filed. Tech was very knowledgable. Is aware of the issue (obviously) and has attached my ticket to the master event ticket so that I can see realtime updates. Appears the ETA is within the next 2 hours. From: Darren Schreiber <darren@2600hz.com<mailto:darren@2600hz.com>> Date: Saturday, October 19, 2013 7:03 AM To: Tim Glen <tglen@asicentral.com<mailto:tglen@asicentral.com>>, 'Sajal Kayan' <sajal83@gmail.com<mailto:sajal83@gmail.com>>, OUTAGES Mailing List <Outages@outages.org<mailto:Outages@outages.org>> Subject: Re: [outages] Road Runner/Level 3 broked? I can confirm the same from our datacenter in NY. Tracepath goes in a loop… From: Tim Glen <tglen@asicentral.com<mailto:tglen@asicentral.com>> Date: Saturday, October 19, 2013 6:59 AM To: 'Sajal Kayan' <sajal83@gmail.com<mailto:sajal83@gmail.com>>, OUTAGES Mailing List <Outages@outages.org<mailto:Outages@outages.org>> Subject: Re: [outages] Road Runner/Level 3 broked? I just opened a ticket with L3. I've been having packet loss since 9:07AM today over my L3 circuit. I am in Philly and L3's customer access router is experiencing high packet loss. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Timothy Glen Sr. Network Engineer Advertising Specialty Institute | Promoting Success tglen@asicentral.com<mailto:tglen@asicentral.com> tel: (215) 953-3692 fax: (215) 953-3759 Think before you print. ________________________________ From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Sajal Kayan Sent: Saturday, October 19, 2013 9:52 AM To: OUTAGES Mailing List Subject: [outages] Road Runner/Level 3 broked? Using road runner in Brooklyn NY. Cant access certain sites ( Facebook, twitter, citibank) . All of them have one thing in common, they go thru level3. (FYI full name of hop 7 is xe-4-2-0.edge4.frankfurt1.level3.net<http://xe-4-2-0.edge4.frankfurt1.level3.net> ) sajal@sajal-laptop:~$ mtr --report www.citi.com<http://www.citi.com> HOST: sajal-laptop Loss% Snt Last Avg Best Wrst StDev 1.|-- 192.168.1.1 0.0% 10 1.2 110.3 1.0 603.9 230.5 2.|-- cpe-24-90-104-1.nyc.res.r 0.0% 10 37.9 207.9 17.0 1418. 444.2 3.|-- tge-0-0-0-1.nycmnyr-rtr2. 0.0% 10 9.8 93.3 9.4 471.9 170.3 4.|-- agg23.nyquny9101r.nyc.rr. 0.0% 10 12.6 78.3 10.1 402.2 142.2 5.|-- 107.14.19.22 0.0% 10 13.8 64.6 11.5 331.0 112.6 6.|-- ae-0-0.pr0.nyc30.tbone.rr 0.0% 10 13.8 50.9 11.4 257.6 82.3 7.|-- xe-4-2-0.edge4.frankfurt1 0.0% 10 13.4 35.0 9.1 186.5 55.8 8.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0 sajal@sajal-laptop:~$ sajal@sajal-laptop:~$ mtr --report www.twitter.com<http://www.twitter.com> HOST: sajal-laptop Loss% Snt Last Avg Best Wrst StDev 1.|-- 192.168.1.1 0.0% 10 1.2 2.9 1.0 9.5 2.8 2.|-- cpe-24-90-104-1.nyc.res.r 0.0% 10 151.6 184.3 75.8 529.8 137.6 3.|-- tge-0-0-0-1.nycmnyr-rtr2. 0.0% 10 252.6 160.8 64.7 299.7 87.2 4.|-- agg23.nyquny9101r.nyc.rr. 0.0% 10 243.5 166.4 64.1 345.8 88.3 5.|-- ae-3-0.cr0.nyc20.tbone.rr 0.0% 10 246.7 189.2 88.6 309.1 75.7 6.|-- 107.14.17.169 0.0% 10 256.6 175.3 102.6 256.6 60.5 7.|-- xe-4-2-0.edge4.frankfurt1 0.0% 10 258.6 179.6 95.4 302.2 69.1 8.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0 sajal@sajal-laptop:~$ sajal@sajal-laptop:~$ mtr --report www.facebook.com<http://www.facebook.com> HOST: sajal-laptop Loss% Snt Last Avg Best Wrst StDev 1.|-- 192.168.1.1 0.0% 10 4.0 3.1 1.1 8.5 2.8 2.|-- cpe-24-90-104-1.nyc.res.r 0.0% 10 225.9 163.7 97.8 252.8 57.0 3.|-- tge-0-0-0-1.nycmnyr-rtr2. 0.0% 10 298.0 168.6 69.3 298.0 79.0 4.|-- agg23.nyquny9101r.nyc.rr. 0.0% 10 292.5 155.0 90.8 292.5 71.0 5.|-- ae-3-0.cr0.nyc20.tbone.rr 0.0% 10 239.0 148.6 68.5 248.6 63.2 6.|-- 107.14.17.169 0.0% 10 268.1 148.7 73.0 268.1 64.9 7.|-- xe-4-2-0.edge4.frankfurt1 10.0% 10 235.1 130.2 66.2 235.1 49.2 8.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0 sajal@sajal-laptop:~$ -Sajal
Hardware failure and apparently the part that was sent to fix was also broke... This is from the most recent update to the ticket. The Transport NOC has reported that the field technician completed rerouting traffic; however, some services failed to restore. The replacement circuit pack is still in transit to the site, and an emergency maintenance window has been scheduled for 04:00 GMT<x-apple-data-detectors://3> to perform the replacement. Once the circuit pack is replaced, customers will be reverted to the original route. All services will be impacted while traffic is being routed back. The next update will be sent once it has been confirmed that the maintenance activity has commenced, or should any additional information become available. On Oct 19, 2013, at 12:55 PM, "Darrell Hyde" <DHyde@hosting.com<mailto:DHyde@hosting.com>> wrote: Were they able to provide any detail with respect to root cause? ________________________________________ From: Outages [outages-bounces@outages.org<mailto:outages-bounces@outages.org>] On Behalf Of Darren Schreiber [darren@2600hz.com<mailto:darren@2600hz.com>] Sent: Saturday, October 19, 2013 12:01 PM To: Tim Glen; 'Sajal Kayan'; OUTAGES Mailing List Subject: Re: [outages] Road Runner/Level 3 broked? Level3 just called me back for the ticket I filed. Tech was very knowledgable. Is aware of the issue (obviously) and has attached my ticket to the master event ticket so that I can see realtime updates. Appears the ETA is within the next 2 hours. From: Darren Schreiber <darren@2600hz.com<mailto:darren@2600hz.com><mailto:darren@2600hz.com>> Date: Saturday, October 19, 2013 7:03 AM To: Tim Glen <tglen@asicentral.com<mailto:tglen@asicentral.com><mailto:tglen@asicentral.com>>, 'Sajal Kayan' <sajal83@gmail.com<mailto:sajal83@gmail.com><mailto:sajal83@gmail.com>>, OUTAGES Mailing List <Outages@outages.org<mailto:Outages@outages.org><mailto:Outages@outages.org>> Subject: Re: [outages] Road Runner/Level 3 broked? I can confirm the same from our datacenter in NY. Tracepath goes in a loop… From: Tim Glen <tglen@asicentral.com<mailto:tglen@asicentral.com><mailto:tglen@asicentral.com>> Date: Saturday, October 19, 2013 6:59 AM To: 'Sajal Kayan' <sajal83@gmail.com<mailto:sajal83@gmail.com><mailto:sajal83@gmail.com>>, OUTAGES Mailing List <Outages@outages.org<mailto:Outages@outages.org><mailto:Outages@outages.org>> Subject: Re: [outages] Road Runner/Level 3 broked? I just opened a ticket with L3. I've been having packet loss since 9:07AM today over my L3 circuit. I am in Philly and L3's customer access router is experiencing high packet loss. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Timothy Glen Sr. Network Engineer Advertising Specialty Institute | Promoting Success tglen@asicentral.com<mailto:tglen@asicentral.com><mailto:tglen@asicentral.com> tel: (215) 953-3692 fax: (215) 953-3759 Think before you print. ________________________________ From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Sajal Kayan Sent: Saturday, October 19, 2013 9:52 AM To: OUTAGES Mailing List Subject: [outages] Road Runner/Level 3 broked? Using road runner in Brooklyn NY. Cant access certain sites ( Facebook, twitter, citibank) . All of them have one thing in common, they go thru level3. (FYI full name of hop 7 is xe-4-2-0.edge4.frankfurt1.level3.net<http://edge4.frankfurt1.level3.net><http://xe-4-2-0.edge4.frankfurt1.level3.net> ) sajal@sajal-laptop:~$ mtr --report www.citi.com<http://www.citi.com><http://www.citi.com> HOST: sajal-laptop Loss% Snt Last Avg Best Wrst StDev 1.|-- 192.168.1.1 0.0% 10 1.2 110.3 1.0 603.9 230.5 2.|-- cpe-24-90-104-1.nyc.res.r 0.0% 10 37.9 207.9 17.0 1418. 444.2 3.|-- tge-0-0-0-1.nycmnyr-rtr2. 0.0% 10 9.8 93.3 9.4 471.9 170.3 4.|-- agg23.nyquny9101r.nyc.rr. 0.0% 10 12.6 78.3 10.1 402.2 142.2 5.|-- 107.14.19.22 0.0% 10 13.8 64.6 11.5 331.0 112.6 6.|-- ae-0-0.pr0.nyc30.tbone.rr 0.0% 10 13.8 50.9 11.4 257.6 82.3 7.|-- xe-4-2-0.edge4.frankfurt1 0.0% 10 13.4 35.0 9.1 186.5 55.8 8.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0 sajal@sajal-laptop:~$ sajal@sajal-laptop:~$ mtr --report www.twitter.com<http://www.twitter.com><http://www.twitter.com> HOST: sajal-laptop Loss% Snt Last Avg Best Wrst StDev 1.|-- 192.168.1.1 0.0% 10 1.2 2.9 1.0 9.5 2.8 2.|-- cpe-24-90-104-1.nyc.res.r 0.0% 10 151.6 184.3 75.8 529.8 137.6 3.|-- tge-0-0-0-1.nycmnyr-rtr2. 0.0% 10 252.6 160.8 64.7 299.7 87.2 4.|-- agg23.nyquny9101r.nyc.rr. 0.0% 10 243.5 166.4 64.1 345.8 88.3 5.|-- ae-3-0.cr0.nyc20.tbone.rr 0.0% 10 246.7 189.2 88.6 309.1 75.7 6.|-- 107.14.17.169 0.0% 10 256.6 175.3 102.6 256.6 60.5 7.|-- xe-4-2-0.edge4.frankfurt1 0.0% 10 258.6 179.6 95.4 302.2 69.1 8.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0 sajal@sajal-laptop:~$ sajal@sajal-laptop:~$ mtr --report www.facebook.com<http://www.facebook.com><http://www.facebook.com> HOST: sajal-laptop Loss% Snt Last Avg Best Wrst StDev 1.|-- 192.168.1.1 0.0% 10 4.0 3.1 1.1 8.5 2.8 2.|-- cpe-24-90-104-1.nyc.res.r 0.0% 10 225.9 163.7 97.8 252.8 57.0 3.|-- tge-0-0-0-1.nycmnyr-rtr2. 0.0% 10 298.0 168.6 69.3 298.0 79.0 4.|-- agg23.nyquny9101r.nyc.rr. 0.0% 10 292.5 155.0 90.8 292.5 71.0 5.|-- ae-3-0.cr0.nyc20.tbone.rr 0.0% 10 239.0 148.6 68.5 248.6 63.2 6.|-- 107.14.17.169 0.0% 10 268.1 148.7 73.0 268.1 64.9 7.|-- xe-4-2-0.edge4.frankfurt1 10.0% 10 235.1 130.2 66.2 235.1 49.2 8.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0 sajal@sajal-laptop:~$ -Sajal
Thanks Tim and Darrell, these updates are priceless for us, since the provider we use to connect to Level3 still seems to be totally clueless about the issue. Would you mind sharing the Level3 master ticket # so we can raise their awareness? If you receive any further updates, especially confirmation of issue resolution, it would be wonderful if you post that to this list, too. Thanks again. From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Tim Glen Sent: Saturday, October 19, 2013 7:20 PM To: Darrell Hyde Cc: OUTAGES Mailing List Subject: Re: [outages] Road Runner/Level 3 broked? Hardware failure and apparently the part that was sent to fix was also broke... This is from the most recent update to the ticket. The Transport NOC has reported that the field technician completed rerouting traffic; however, some services failed to restore. The replacement circuit pack is still in transit to the site, and an emergency maintenance window has been scheduled for 04:00 GMT <x-apple-data-detectors://3> to perform the replacement. Once the circuit pack is replaced, customers will be reverted to the original route. All services will be impacted while traffic is being routed back. The next update will be sent once it has been confirmed that the maintenance activity has commenced, or should any additional information become available. On Oct 19, 2013, at 12:55 PM, "Darrell Hyde" <DHyde@hosting.com <mailto:DHyde@hosting.com> > wrote: Were they able to provide any detail with respect to root cause? ________________________________________ From: Outages [outages-bounces@outages.org <mailto:outages-bounces@outages.org> ] On Behalf Of Darren Schreiber [darren@2600hz.com <mailto:darren@2600hz.com> ] Sent: Saturday, October 19, 2013 12:01 PM To: Tim Glen; 'Sajal Kayan'; OUTAGES Mailing List Subject: Re: [outages] Road Runner/Level 3 broked? Level3 just called me back for the ticket I filed. Tech was very knowledgable. Is aware of the issue (obviously) and has attached my ticket to the master event ticket so that I can see realtime updates. Appears the ETA is within the next 2 hours. From: Darren Schreiber <darren@2600hz.com <mailto:darren@2600hz.com> <mailto:darren@2600hz.com>> Date: Saturday, October 19, 2013 7:03 AM To: Tim Glen <tglen@asicentral.com <mailto:tglen@asicentral.com> <mailto:tglen@asicentral.com>>, 'Sajal Kayan' <sajal83@gmail.com <mailto:sajal83@gmail.com> <mailto:sajal83@gmail.com>>, OUTAGES Mailing List <Outages@outages.org <mailto:Outages@outages.org> <mailto:Outages@outages.org>> Subject: Re: [outages] Road Runner/Level 3 broked? I can confirm the same from our datacenter in NY. Tracepath goes in a loop… From: Tim Glen <tglen@asicentral.com <mailto:tglen@asicentral.com> <mailto:tglen@asicentral.com>> Date: Saturday, October 19, 2013 6:59 AM To: 'Sajal Kayan' <sajal83@gmail.com <mailto:sajal83@gmail.com> <mailto:sajal83@gmail.com>>, OUTAGES Mailing List <Outages@outages.org <mailto:Outages@outages.org> <mailto:Outages@outages.org>> Subject: Re: [outages] Road Runner/Level 3 broked? I just opened a ticket with L3. I've been having packet loss since 9:07AM today over my L3 circuit. I am in Philly and L3's customer access router is experiencing high packet loss. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Timothy Glen Sr. Network Engineer Advertising Specialty Institute | Promoting Success tglen@asicentral.com <mailto:tglen@asicentral.com> <mailto:tglen@asicentral.com> tel: (215) 953-3692 fax: (215) 953-3759 Think before you print. ________________________________ From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Sajal Kayan Sent: Saturday, October 19, 2013 9:52 AM To: OUTAGES Mailing List Subject: [outages] Road Runner/Level 3 broked? Using road runner in Brooklyn NY. Cant access certain sites ( Facebook, twitter, citibank) . All of them have one thing in common, they go thru level3. (FYI full name of hop 7 is xe-4-2-0.edge4.frankfurt1.level3.net <http://edge4.frankfurt1.level3.net> <http://xe-4-2-0.edge4.frankfurt1.level3.net> ) sajal@sajal-laptop:~$ mtr --report www.citi.com <http://www.citi.com> <http://www.citi.com> HOST: sajal-laptop Loss% Snt Last Avg Best Wrst StDev 1.|-- 192.168.1.1 0.0% 10 1.2 110.3 1.0 603.9 230.5 2.|-- cpe-24-90-104-1.nyc.res.r 0.0% 10 37.9 207.9 17.0 1418. 444.2 3.|-- tge-0-0-0-1.nycmnyr-rtr2. 0.0% 10 9.8 93.3 9.4 471.9 170.3 4.|-- agg23.nyquny9101r.nyc.rr. 0.0% 10 12.6 78.3 10.1 402.2 142.2 5.|-- 107.14.19.22 0.0% 10 13.8 64.6 11.5 331.0 112.6 6.|-- ae-0-0.pr0.nyc30.tbone.rr 0.0% 10 13.8 50.9 11.4 257.6 82.3 7.|-- xe-4-2-0.edge4.frankfurt1 0.0% 10 13.4 35.0 9.1 186.5 55.8 8.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0 sajal@sajal-laptop:~$ sajal@sajal-laptop:~$ mtr --report www.twitter.com <http://www.twitter.com> <http://www.twitter.com> HOST: sajal-laptop Loss% Snt Last Avg Best Wrst StDev 1.|-- 192.168.1.1 0.0% 10 1.2 2.9 1.0 9.5 2.8 2.|-- cpe-24-90-104-1.nyc.res.r 0.0% 10 151.6 184.3 75.8 529.8 137.6 3.|-- tge-0-0-0-1.nycmnyr-rtr2. 0.0% 10 252.6 160.8 64.7 299.7 87.2 4.|-- agg23.nyquny9101r.nyc.rr. 0.0% 10 243.5 166.4 64.1 345.8 88.3 5.|-- ae-3-0.cr0.nyc20.tbone.rr 0.0% 10 246.7 189.2 88.6 309.1 75.7 6.|-- 107.14.17.169 0.0% 10 256.6 175.3 102.6 256.6 60.5 7.|-- xe-4-2-0.edge4.frankfurt1 0.0% 10 258.6 179.6 95.4 302.2 69.1 8.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0 sajal@sajal-laptop:~$ sajal@sajal-laptop:~$ mtr --report www.facebook.com <http://www.facebook.com> <http://www.facebook.com> HOST: sajal-laptop Loss% Snt Last Avg Best Wrst StDev 1.|-- 192.168.1.1 0.0% 10 4.0 3.1 1.1 8.5 2.8 2.|-- cpe-24-90-104-1.nyc.res.r 0.0% 10 225.9 163.7 97.8 252.8 57.0 3.|-- tge-0-0-0-1.nycmnyr-rtr2. 0.0% 10 298.0 168.6 69.3 298.0 79.0 4.|-- agg23.nyquny9101r.nyc.rr. 0.0% 10 292.5 155.0 90.8 292.5 71.0 5.|-- ae-3-0.cr0.nyc20.tbone.rr 0.0% 10 239.0 148.6 68.5 248.6 63.2 6.|-- 107.14.17.169 0.0% 10 268.1 148.7 73.0 268.1 64.9 7.|-- xe-4-2-0.edge4.frankfurt1 10.0% 10 235.1 130.2 66.2 235.1 49.2 8.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0 sajal@sajal-laptop:~$ -Sajal
Most recent update we have which includes the "external ticket" which i think is the master *** CASCADED EXTERNAL NOTES 20-Oct-2013 03:42:42 GMT From CASE: 7126134 - Event The Transport NOC has advised that the replacement circuit pack has arrived on site, and in cooperation the field technician will replace the faulty circuit pack. Once the stability of the circuit pack is confirmed, efforts will begin to move all services back to their intended paths. An update will be sent prior to the commencement of that demand maintenance activity. On Sat, Oct 19, 2013 at 7:53 PM, Adam Greene <maillist@webjogger.net> wrote:
Thanks Tim and Darrell, these updates are priceless for us, since the provider we use to connect to Level3 still seems to be totally clueless about the issue.
Would you mind sharing the Level3 master ticket # so we can raise their awareness?
If you receive any further updates, especially confirmation of issue resolution, it would be wonderful if you post that to this list, too.
Thanks again.
From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Tim Glen Sent: Saturday, October 19, 2013 7:20 PM To: Darrell Hyde Cc: OUTAGES Mailing List
Subject: Re: [outages] Road Runner/Level 3 broked?
Hardware failure and apparently the part that was sent to fix was also broke... This is from the most recent update to the ticket.
The Transport NOC has reported that the field technician completed rerouting traffic; however, some services failed to restore. The replacement circuit pack is still in transit to the site, and an emergency maintenance window has been scheduled for 04:00 GMT to perform the replacement. Once the circuit pack is replaced, customers will be reverted to the original route. All services will be impacted while traffic is being routed back. The next update will be sent once it has been confirmed that the maintenance activity has commenced, or should any additional information become available.
On Oct 19, 2013, at 12:55 PM, "Darrell Hyde" <DHyde@hosting.com> wrote:
Were they able to provide any detail with respect to root cause? ________________________________________ From: Outages [outages-bounces@outages.org] On Behalf Of Darren Schreiber [darren@2600hz.com] Sent: Saturday, October 19, 2013 12:01 PM To: Tim Glen; 'Sajal Kayan'; OUTAGES Mailing List Subject: Re: [outages] Road Runner/Level 3 broked?
Level3 just called me back for the ticket I filed. Tech was very knowledgable. Is aware of the issue (obviously) and has attached my ticket to the master event ticket so that I can see realtime updates.
Appears the ETA is within the next 2 hours.
From: Darren Schreiber <darren@2600hz.com<mailto:darren@2600hz.com>> Date: Saturday, October 19, 2013 7:03 AM To: Tim Glen <tglen@asicentral.com<mailto:tglen@asicentral.com>>, 'Sajal Kayan' <sajal83@gmail.com<mailto:sajal83@gmail.com>>, OUTAGES Mailing List <Outages@outages.org<mailto:Outages@outages.org>> Subject: Re: [outages] Road Runner/Level 3 broked?
I can confirm the same from our datacenter in NY. Tracepath goes in a loop…
From: Tim Glen <tglen@asicentral.com<mailto:tglen@asicentral.com>> Date: Saturday, October 19, 2013 6:59 AM To: 'Sajal Kayan' <sajal83@gmail.com<mailto:sajal83@gmail.com>>, OUTAGES Mailing List <Outages@outages.org<mailto:Outages@outages.org>> Subject: Re: [outages] Road Runner/Level 3 broked?
I just opened a ticket with L3.
I've been having packet loss since 9:07AM today over my L3 circuit.
I am in Philly and L3's customer access router is experiencing high packet loss.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Timothy Glen Sr. Network Engineer Advertising Specialty Institute | Promoting Success
tglen@asicentral.com<mailto:tglen@asicentral.com> tel: (215) 953-3692 fax: (215) 953-3759 Think before you print.
________________________________ From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Sajal Kayan Sent: Saturday, October 19, 2013 9:52 AM To: OUTAGES Mailing List Subject: [outages] Road Runner/Level 3 broked?
Using road runner in Brooklyn NY.
Cant access certain sites ( Facebook, twitter, citibank) . All of them have one thing in common, they go thru level3.
(FYI full name of hop 7 is xe-4-2-0.edge4.frankfurt1.level3.net<http://xe-4-2-0.edge4.frankfurt1.level3.net> )
sajal@sajal-laptop:~$ mtr --report www.citi.com<http://www.citi.com> HOST: sajal-laptop Loss% Snt Last Avg Best Wrst StDev 1.|-- 192.168.1.1 0.0% 10 1.2 110.3 1.0 603.9 230.5 2.|-- cpe-24-90-104-1.nyc.res.r 0.0% 10 37.9 207.9 17.0 1418. 444.2 3.|-- tge-0-0-0-1.nycmnyr-rtr2. 0.0% 10 9.8 93.3 9.4 471.9 170.3 4.|-- agg23.nyquny9101r.nyc.rr. 0.0% 10 12.6 78.3 10.1 402.2 142.2 5.|-- 107.14.19.22 0.0% 10 13.8 64.6 11.5 331.0 112.6 6.|-- ae-0-0.pr0.nyc30.tbone.rr 0.0% 10 13.8 50.9 11.4 257.6 82.3 7.|-- xe-4-2-0.edge4.frankfurt1 0.0% 10 13.4 35.0 9.1 186.5 55.8 8.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0 sajal@sajal-laptop:~$
sajal@sajal-laptop:~$ mtr --report www.twitter.com<http://www.twitter.com> HOST: sajal-laptop Loss% Snt Last Avg Best Wrst StDev 1.|-- 192.168.1.1 0.0% 10 1.2 2.9 1.0 9.5 2.8 2.|-- cpe-24-90-104-1.nyc.res.r 0.0% 10 151.6 184.3 75.8 529.8 137.6 3.|-- tge-0-0-0-1.nycmnyr-rtr2. 0.0% 10 252.6 160.8 64.7 299.7 87.2 4.|-- agg23.nyquny9101r.nyc.rr. 0.0% 10 243.5 166.4 64.1 345.8 88.3 5.|-- ae-3-0.cr0.nyc20.tbone.rr 0.0% 10 246.7 189.2 88.6 309.1 75.7 6.|-- 107.14.17.169 0.0% 10 256.6 175.3 102.6 256.6 60.5 7.|-- xe-4-2-0.edge4.frankfurt1 0.0% 10 258.6 179.6 95.4 302.2 69.1 8.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0 sajal@sajal-laptop:~$
sajal@sajal-laptop:~$ mtr --report www.facebook.com<http://www.facebook.com> HOST: sajal-laptop Loss% Snt Last Avg Best Wrst StDev 1.|-- 192.168.1.1 0.0% 10 4.0 3.1 1.1 8.5 2.8 2.|-- cpe-24-90-104-1.nyc.res.r 0.0% 10 225.9 163.7 97.8 252.8 57.0 3.|-- tge-0-0-0-1.nycmnyr-rtr2. 0.0% 10 298.0 168.6 69.3 298.0 79.0 4.|-- agg23.nyquny9101r.nyc.rr. 0.0% 10 292.5 155.0 90.8 292.5 71.0 5.|-- ae-3-0.cr0.nyc20.tbone.rr 0.0% 10 239.0 148.6 68.5 248.6 63.2 6.|-- 107.14.17.169 0.0% 10 268.1 148.7 73.0 268.1 64.9 7.|-- xe-4-2-0.edge4.frankfurt1 10.0% 10 235.1 130.2 66.2 235.1 49.2 8.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0 sajal@sajal-laptop:~$
-Sajal
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Latest *** CASCADED EXTERNAL NOTES 20-Oct-2013 05:15:03 GMT From CASE: 7126134 - Event The Transport NOC, the field technician, and Tier III Technical Support have confirmed the circuit pack has been successfully installed and have begun the demand maintenance to restore customers to their original path. On Sat, Oct 19, 2013 at 11:54 PM, Mitch <mitpatterson@gmail.com> wrote:
Most recent update we have which includes the "external ticket" which i think is the master
*** CASCADED EXTERNAL NOTES 20-Oct-2013 03:42:42 GMT From CASE: 7126134 - Event The Transport NOC has advised that the replacement circuit pack has arrived on site, and in cooperation the field technician will replace the faulty circuit pack. Once the stability of the circuit pack is confirmed, efforts will begin to move all services back to their intended paths. An update will be sent prior to the commencement of that demand maintenance activity.
On Sat, Oct 19, 2013 at 7:53 PM, Adam Greene <maillist@webjogger.net> wrote:
Thanks Tim and Darrell, these updates are priceless for us, since the provider we use to connect to Level3 still seems to be totally clueless about the issue.
Would you mind sharing the Level3 master ticket # so we can raise their awareness?
If you receive any further updates, especially confirmation of issue resolution, it would be wonderful if you post that to this list, too.
Thanks again.
From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Tim Glen Sent: Saturday, October 19, 2013 7:20 PM To: Darrell Hyde Cc: OUTAGES Mailing List
Subject: Re: [outages] Road Runner/Level 3 broked?
Hardware failure and apparently the part that was sent to fix was also broke... This is from the most recent update to the ticket.
The Transport NOC has reported that the field technician completed rerouting traffic; however, some services failed to restore. The replacement circuit pack is still in transit to the site, and an emergency maintenance window has been scheduled for 04:00 GMT to perform the replacement. Once the circuit pack is replaced, customers will be reverted to the original route. All services will be impacted while traffic is being routed back. The next update will be sent once it has been confirmed that the maintenance activity has commenced, or should any additional information become available.
On Oct 19, 2013, at 12:55 PM, "Darrell Hyde" <DHyde@hosting.com> wrote:
Were they able to provide any detail with respect to root cause? ________________________________________ From: Outages [outages-bounces@outages.org] On Behalf Of Darren Schreiber [darren@2600hz.com] Sent: Saturday, October 19, 2013 12:01 PM To: Tim Glen; 'Sajal Kayan'; OUTAGES Mailing List Subject: Re: [outages] Road Runner/Level 3 broked?
Level3 just called me back for the ticket I filed. Tech was very knowledgable. Is aware of the issue (obviously) and has attached my ticket to the master event ticket so that I can see realtime updates.
Appears the ETA is within the next 2 hours.
From: Darren Schreiber <darren@2600hz.com<mailto:darren@2600hz.com>> Date: Saturday, October 19, 2013 7:03 AM To: Tim Glen <tglen@asicentral.com<mailto:tglen@asicentral.com>>, 'Sajal Kayan' <sajal83@gmail.com<mailto:sajal83@gmail.com>>, OUTAGES Mailing List <Outages@outages.org<mailto:Outages@outages.org>> Subject: Re: [outages] Road Runner/Level 3 broked?
I can confirm the same from our datacenter in NY. Tracepath goes in a loop…
From: Tim Glen <tglen@asicentral.com<mailto:tglen@asicentral.com>> Date: Saturday, October 19, 2013 6:59 AM To: 'Sajal Kayan' <sajal83@gmail.com<mailto:sajal83@gmail.com>>, OUTAGES Mailing List <Outages@outages.org<mailto:Outages@outages.org>> Subject: Re: [outages] Road Runner/Level 3 broked?
I just opened a ticket with L3.
I've been having packet loss since 9:07AM today over my L3 circuit.
I am in Philly and L3's customer access router is experiencing high packet loss.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Timothy Glen Sr. Network Engineer Advertising Specialty Institute | Promoting Success
tglen@asicentral.com<mailto:tglen@asicentral.com> tel: (215) 953-3692 fax: (215) 953-3759 Think before you print.
________________________________ From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Sajal Kayan Sent: Saturday, October 19, 2013 9:52 AM To: OUTAGES Mailing List Subject: [outages] Road Runner/Level 3 broked?
Using road runner in Brooklyn NY.
Cant access certain sites ( Facebook, twitter, citibank) . All of them have one thing in common, they go thru level3.
(FYI full name of hop 7 is xe-4-2-0.edge4.frankfurt1.level3.net<http://xe-4-2-0.edge4.frankfurt1.level3.net> )
sajal@sajal-laptop:~$ mtr --report www.citi.com<http://www.citi.com> HOST: sajal-laptop Loss% Snt Last Avg Best Wrst StDev 1.|-- 192.168.1.1 0.0% 10 1.2 110.3 1.0 603.9 230.5 2.|-- cpe-24-90-104-1.nyc.res.r 0.0% 10 37.9 207.9 17.0 1418. 444.2 3.|-- tge-0-0-0-1.nycmnyr-rtr2. 0.0% 10 9.8 93.3 9.4 471.9 170.3 4.|-- agg23.nyquny9101r.nyc.rr. 0.0% 10 12.6 78.3 10.1 402.2 142.2 5.|-- 107.14.19.22 0.0% 10 13.8 64.6 11.5 331.0 112.6 6.|-- ae-0-0.pr0.nyc30.tbone.rr 0.0% 10 13.8 50.9 11.4 257.6 82.3 7.|-- xe-4-2-0.edge4.frankfurt1 0.0% 10 13.4 35.0 9.1 186.5 55.8 8.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0 sajal@sajal-laptop:~$
sajal@sajal-laptop:~$ mtr --report www.twitter.com<http://www.twitter.com> HOST: sajal-laptop Loss% Snt Last Avg Best Wrst StDev 1.|-- 192.168.1.1 0.0% 10 1.2 2.9 1.0 9.5 2.8 2.|-- cpe-24-90-104-1.nyc.res.r 0.0% 10 151.6 184.3 75.8 529.8 137.6 3.|-- tge-0-0-0-1.nycmnyr-rtr2. 0.0% 10 252.6 160.8 64.7 299.7 87.2 4.|-- agg23.nyquny9101r.nyc.rr. 0.0% 10 243.5 166.4 64.1 345.8 88.3 5.|-- ae-3-0.cr0.nyc20.tbone.rr 0.0% 10 246.7 189.2 88.6 309.1 75.7 6.|-- 107.14.17.169 0.0% 10 256.6 175.3 102.6 256.6 60.5 7.|-- xe-4-2-0.edge4.frankfurt1 0.0% 10 258.6 179.6 95.4 302.2 69.1 8.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0 sajal@sajal-laptop:~$
sajal@sajal-laptop:~$ mtr --report www.facebook.com<http://www.facebook.com> HOST: sajal-laptop Loss% Snt Last Avg Best Wrst StDev 1.|-- 192.168.1.1 0.0% 10 4.0 3.1 1.1 8.5 2.8 2.|-- cpe-24-90-104-1.nyc.res.r 0.0% 10 225.9 163.7 97.8 252.8 57.0 3.|-- tge-0-0-0-1.nycmnyr-rtr2. 0.0% 10 298.0 168.6 69.3 298.0 79.0 4.|-- agg23.nyquny9101r.nyc.rr. 0.0% 10 292.5 155.0 90.8 292.5 71.0 5.|-- ae-3-0.cr0.nyc20.tbone.rr 0.0% 10 239.0 148.6 68.5 248.6 63.2 6.|-- 107.14.17.169 0.0% 10 268.1 148.7 73.0 268.1 64.9 7.|-- xe-4-2-0.edge4.frankfurt1 10.0% 10 235.1 130.2 66.2 235.1 49.2 8.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0 sajal@sajal-laptop:~$
-Sajal
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
The latest from about 4AM. *** CASCADED EXTERNAL NOTES 20-Oct-2013 07:47:55 GMT From CASE: 7126134 - Event The Transport NOC has reported network alarms have cleared. This was a permanent restoration. If additional issues are experienced, please contact the Level 3 Technical Service Center at 1-877-4LEVEL3. -----Original Message----- From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Mitch Sent: Sunday, October 20, 2013 1:32 AM To: Adam Greene Cc: OUTAGES Mailing List Subject: Re: [outages] Road Runner/Level 3 broked? Latest *** CASCADED EXTERNAL NOTES 20-Oct-2013 05:15:03 GMT From CASE: 7126134 - Event The Transport NOC, the field technician, and Tier III Technical Support have confirmed the circuit pack has been successfully installed and have begun the demand maintenance to restore customers to their original path. On Sat, Oct 19, 2013 at 11:54 PM, Mitch <mitpatterson@gmail.com> wrote:
Most recent update we have which includes the "external ticket" which i think is the master
*** CASCADED EXTERNAL NOTES 20-Oct-2013 03:42:42 GMT From CASE: 7126134 - Event The Transport NOC has advised that the replacement circuit pack has arrived on site, and in cooperation the field technician will replace the faulty circuit pack. Once the stability of the circuit pack is confirmed, efforts will begin to move all services back to their intended paths. An update will be sent prior to the commencement of that demand maintenance activity.
On Sat, Oct 19, 2013 at 7:53 PM, Adam Greene <maillist@webjogger.net> wrote:
Thanks Tim and Darrell, these updates are priceless for us, since the provider we use to connect to Level3 still seems to be totally clueless about the issue.
Would you mind sharing the Level3 master ticket # so we can raise their awareness?
If you receive any further updates, especially confirmation of issue resolution, it would be wonderful if you post that to this list, too.
Thanks again.
From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Tim Glen Sent: Saturday, October 19, 2013 7:20 PM To: Darrell Hyde Cc: OUTAGES Mailing List
Subject: Re: [outages] Road Runner/Level 3 broked?
Hardware failure and apparently the part that was sent to fix was also broke... This is from the most recent update to the ticket.
The Transport NOC has reported that the field technician completed rerouting traffic; however, some services failed to restore. The replacement circuit pack is still in transit to the site, and an emergency maintenance window has been scheduled for 04:00 GMT to perform the replacement. Once the circuit pack is replaced, customers will be reverted to the original route. All services will be impacted while traffic is being routed back. The next update will be sent once it has been confirmed that the maintenance activity has commenced, or should any additional information become available.
On Oct 19, 2013, at 12:55 PM, "Darrell Hyde" <DHyde@hosting.com> wrote:
Were they able to provide any detail with respect to root cause? ________________________________________ From: Outages [outages-bounces@outages.org] On Behalf Of Darren Schreiber [darren@2600hz.com] Sent: Saturday, October 19, 2013 12:01 PM To: Tim Glen; 'Sajal Kayan'; OUTAGES Mailing List Subject: Re: [outages] Road Runner/Level 3 broked?
Level3 just called me back for the ticket I filed. Tech was very knowledgable. Is aware of the issue (obviously) and has attached my ticket to the master event ticket so that I can see realtime updates.
Appears the ETA is within the next 2 hours.
From: Darren Schreiber <darren@2600hz.com<mailto:darren@2600hz.com>> Date: Saturday, October 19, 2013 7:03 AM To: Tim Glen <tglen@asicentral.com<mailto:tglen@asicentral.com>>, 'Sajal Kayan' <sajal83@gmail.com<mailto:sajal83@gmail.com>>, OUTAGES Mailing List <Outages@outages.org<mailto:Outages@outages.org>> Subject: Re: [outages] Road Runner/Level 3 broked?
I can confirm the same from our datacenter in NY. Tracepath goes in a loop...
From: Tim Glen <tglen@asicentral.com<mailto:tglen@asicentral.com>> Date: Saturday, October 19, 2013 6:59 AM To: 'Sajal Kayan' <sajal83@gmail.com<mailto:sajal83@gmail.com>>, OUTAGES Mailing List <Outages@outages.org<mailto:Outages@outages.org>> Subject: Re: [outages] Road Runner/Level 3 broked?
I just opened a ticket with L3.
I've been having packet loss since 9:07AM today over my L3 circuit.
I am in Philly and L3's customer access router is experiencing high packet loss.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Timothy Glen Sr. Network Engineer Advertising Specialty Institute | Promoting Success
tglen@asicentral.com<mailto:tglen@asicentral.com> tel: (215) 953-3692 fax: (215) 953-3759 Think before you print.
________________________________ From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Sajal Kayan Sent: Saturday, October 19, 2013 9:52 AM To: OUTAGES Mailing List Subject: [outages] Road Runner/Level 3 broked?
Using road runner in Brooklyn NY.
Cant access certain sites ( Facebook, twitter, citibank) . All of them have one thing in common, they go thru level3.
(FYI full name of hop 7 is xe-4-2-0.edge4.frankfurt1.level3.net<http://xe-4-2-0.edge4.frankfurt1 .level3.net> )
sajal@sajal-laptop:~$ mtr --report www.citi.com<http://www.citi.com> HOST: sajal-laptop Loss% Snt Last Avg Best Wrst StDev 1.|-- 192.168.1.1 0.0% 10 1.2 110.3 1.0 603.9 230.5 2.|-- cpe-24-90-104-1.nyc.res.r 0.0% 10 37.9 207.9 17.0 1418. 444.2 3.|-- tge-0-0-0-1.nycmnyr-rtr2. 0.0% 10 9.8 93.3 9.4 471.9 170.3 4.|-- agg23.nyquny9101r.nyc.rr. 0.0% 10 12.6 78.3 10.1 402.2 142.2 5.|-- 107.14.19.22 0.0% 10 13.8 64.6 11.5 331.0 112.6 6.|-- ae-0-0.pr0.nyc30.tbone.rr 0.0% 10 13.8 50.9 11.4 257.6 82.3 7.|-- xe-4-2-0.edge4.frankfurt1 0.0% 10 13.4 35.0 9.1 186.5 55.8 8.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0 sajal@sajal-laptop:~$
sajal@sajal-laptop:~$ mtr --report www.twitter.com<http://www.twitter.com> HOST: sajal-laptop Loss% Snt Last Avg Best Wrst StDev 1.|-- 192.168.1.1 0.0% 10 1.2 2.9 1.0 9.5 2.8 2.|-- cpe-24-90-104-1.nyc.res.r 0.0% 10 151.6 184.3 75.8 529.8 137.6 3.|-- tge-0-0-0-1.nycmnyr-rtr2. 0.0% 10 252.6 160.8 64.7 299.7 87.2 4.|-- agg23.nyquny9101r.nyc.rr. 0.0% 10 243.5 166.4 64.1 345.8 88.3 5.|-- ae-3-0.cr0.nyc20.tbone.rr 0.0% 10 246.7 189.2 88.6 309.1 75.7 6.|-- 107.14.17.169 0.0% 10 256.6 175.3 102.6 256.6 60.5 7.|-- xe-4-2-0.edge4.frankfurt1 0.0% 10 258.6 179.6 95.4 302.2 69.1 8.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0 sajal@sajal-laptop:~$
sajal@sajal-laptop:~$ mtr --report www.facebook.com<http://www.facebook.com> HOST: sajal-laptop Loss% Snt Last Avg Best Wrst StDev 1.|-- 192.168.1.1 0.0% 10 4.0 3.1 1.1 8.5 2.8 2.|-- cpe-24-90-104-1.nyc.res.r 0.0% 10 225.9 163.7 97.8 252.8 57.0 3.|-- tge-0-0-0-1.nycmnyr-rtr2. 0.0% 10 298.0 168.6 69.3 298.0 79.0 4.|-- agg23.nyquny9101r.nyc.rr. 0.0% 10 292.5 155.0 90.8 292.5 71.0 5.|-- ae-3-0.cr0.nyc20.tbone.rr 0.0% 10 239.0 148.6 68.5 248.6 63.2 6.|-- 107.14.17.169 0.0% 10 268.1 148.7 73.0 268.1 64.9 7.|-- xe-4-2-0.edge4.frankfurt1 10.0% 10 235.1 130.2 66.2 235.1 49.2 8.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0 sajal@sajal-laptop:~$
-Sajal
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Thank you very much, very helpful! We have reactivated our L3 circuit, and are seeing nice clean traffic & routing. Thanks for all the info. -----Original Message----- From: Tim Glen [mailto:tglen@asicentral.com] Sent: Sunday, October 20, 2013 9:45 AM To: 'Mitch'; Adam Greene Cc: OUTAGES Mailing List Subject: RE: [outages] Road Runner/Level 3 broked? The latest from about 4AM. *** CASCADED EXTERNAL NOTES 20-Oct-2013 07:47:55 GMT From CASE: 7126134 - Event The Transport NOC has reported network alarms have cleared. This was a permanent restoration. If additional issues are experienced, please contact the Level 3 Technical Service Center at 1-877-4LEVEL3. -----Original Message----- From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Mitch Sent: Sunday, October 20, 2013 1:32 AM To: Adam Greene Cc: OUTAGES Mailing List Subject: Re: [outages] Road Runner/Level 3 broked? Latest *** CASCADED EXTERNAL NOTES 20-Oct-2013 05:15:03 GMT From CASE: 7126134 - Event The Transport NOC, the field technician, and Tier III Technical Support have confirmed the circuit pack has been successfully installed and have begun the demand maintenance to restore customers to their original path. On Sat, Oct 19, 2013 at 11:54 PM, Mitch <mitpatterson@gmail.com> wrote:
Most recent update we have which includes the "external ticket" which i think is the master
*** CASCADED EXTERNAL NOTES 20-Oct-2013 03:42:42 GMT From CASE: 7126134 - Event The Transport NOC has advised that the replacement circuit pack has arrived on site, and in cooperation the field technician will replace the faulty circuit pack. Once the stability of the circuit pack is confirmed, efforts will begin to move all services back to their intended paths. An update will be sent prior to the commencement of that demand maintenance activity.
On Sat, Oct 19, 2013 at 7:53 PM, Adam Greene <maillist@webjogger.net> wrote:
Thanks Tim and Darrell, these updates are priceless for us, since the provider we use to connect to Level3 still seems to be totally clueless about the issue.
Would you mind sharing the Level3 master ticket # so we can raise their awareness?
If you receive any further updates, especially confirmation of issue resolution, it would be wonderful if you post that to this list, too.
Thanks again.
From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Tim Glen Sent: Saturday, October 19, 2013 7:20 PM To: Darrell Hyde Cc: OUTAGES Mailing List
Subject: Re: [outages] Road Runner/Level 3 broked?
Hardware failure and apparently the part that was sent to fix was also broke... This is from the most recent update to the ticket.
The Transport NOC has reported that the field technician completed rerouting traffic; however, some services failed to restore. The replacement circuit pack is still in transit to the site, and an emergency maintenance window has been scheduled for 04:00 GMT to perform the replacement. Once the circuit pack is replaced, customers will be reverted to the original route. All services will be impacted while traffic is being routed back. The next update will be sent once it has been confirmed that the maintenance activity has commenced, or should any additional information become available.
On Oct 19, 2013, at 12:55 PM, "Darrell Hyde" <DHyde@hosting.com> wrote:
Were they able to provide any detail with respect to root cause? ________________________________________ From: Outages [outages-bounces@outages.org] On Behalf Of Darren Schreiber [darren@2600hz.com] Sent: Saturday, October 19, 2013 12:01 PM To: Tim Glen; 'Sajal Kayan'; OUTAGES Mailing List Subject: Re: [outages] Road Runner/Level 3 broked?
Level3 just called me back for the ticket I filed. Tech was very knowledgable. Is aware of the issue (obviously) and has attached my ticket to the master event ticket so that I can see realtime updates.
Appears the ETA is within the next 2 hours.
From: Darren Schreiber <darren@2600hz.com<mailto:darren@2600hz.com>> Date: Saturday, October 19, 2013 7:03 AM To: Tim Glen <tglen@asicentral.com<mailto:tglen@asicentral.com>>, 'Sajal Kayan' <sajal83@gmail.com<mailto:sajal83@gmail.com>>, OUTAGES Mailing List <Outages@outages.org<mailto:Outages@outages.org>> Subject: Re: [outages] Road Runner/Level 3 broked?
I can confirm the same from our datacenter in NY. Tracepath goes in a loop...
From: Tim Glen <tglen@asicentral.com<mailto:tglen@asicentral.com>> Date: Saturday, October 19, 2013 6:59 AM To: 'Sajal Kayan' <sajal83@gmail.com<mailto:sajal83@gmail.com>>, OUTAGES Mailing List <Outages@outages.org<mailto:Outages@outages.org>> Subject: Re: [outages] Road Runner/Level 3 broked?
I just opened a ticket with L3.
I've been having packet loss since 9:07AM today over my L3 circuit.
I am in Philly and L3's customer access router is experiencing high packet loss.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Timothy Glen Sr. Network Engineer Advertising Specialty Institute | Promoting Success
tglen@asicentral.com<mailto:tglen@asicentral.com> tel: (215) 953-3692 fax: (215) 953-3759 Think before you print.
________________________________ From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Sajal Kayan Sent: Saturday, October 19, 2013 9:52 AM To: OUTAGES Mailing List Subject: [outages] Road Runner/Level 3 broked?
Using road runner in Brooklyn NY.
Cant access certain sites ( Facebook, twitter, citibank) . All of them have one thing in common, they go thru level3.
(FYI full name of hop 7 is xe-4-2-0.edge4.frankfurt1.level3.net<http://xe-4-2-0.edge4.frankfurt1 .level3.net> )
sajal@sajal-laptop:~$ mtr --report www.citi.com<http://www.citi.com> HOST: sajal-laptop Loss% Snt Last Avg Best Wrst StDev 1.|-- 192.168.1.1 0.0% 10 1.2 110.3 1.0 603.9 230.5 2.|-- cpe-24-90-104-1.nyc.res.r 0.0% 10 37.9 207.9 17.0 1418. 444.2 3.|-- tge-0-0-0-1.nycmnyr-rtr2. 0.0% 10 9.8 93.3 9.4 471.9 170.3 4.|-- agg23.nyquny9101r.nyc.rr. 0.0% 10 12.6 78.3 10.1 402.2 142.2 5.|-- 107.14.19.22 0.0% 10 13.8 64.6 11.5 331.0 112.6 6.|-- ae-0-0.pr0.nyc30.tbone.rr 0.0% 10 13.8 50.9 11.4 257.6 82.3 7.|-- xe-4-2-0.edge4.frankfurt1 0.0% 10 13.4 35.0 9.1 186.5 55.8 8.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0 sajal@sajal-laptop:~$
sajal@sajal-laptop:~$ mtr --report www.twitter.com<http://www.twitter.com> HOST: sajal-laptop Loss% Snt Last Avg Best Wrst StDev 1.|-- 192.168.1.1 0.0% 10 1.2 2.9 1.0 9.5 2.8 2.|-- cpe-24-90-104-1.nyc.res.r 0.0% 10 151.6 184.3 75.8 529.8 137.6 3.|-- tge-0-0-0-1.nycmnyr-rtr2. 0.0% 10 252.6 160.8 64.7 299.7 87.2 4.|-- agg23.nyquny9101r.nyc.rr. 0.0% 10 243.5 166.4 64.1 345.8 88.3 5.|-- ae-3-0.cr0.nyc20.tbone.rr 0.0% 10 246.7 189.2 88.6 309.1 75.7 6.|-- 107.14.17.169 0.0% 10 256.6 175.3 102.6 256.6 60.5 7.|-- xe-4-2-0.edge4.frankfurt1 0.0% 10 258.6 179.6 95.4 302.2 69.1 8.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0 sajal@sajal-laptop:~$
sajal@sajal-laptop:~$ mtr --report www.facebook.com<http://www.facebook.com> HOST: sajal-laptop Loss% Snt Last Avg Best Wrst StDev 1.|-- 192.168.1.1 0.0% 10 4.0 3.1 1.1 8.5 2.8 2.|-- cpe-24-90-104-1.nyc.res.r 0.0% 10 225.9 163.7 97.8 252.8 57.0 3.|-- tge-0-0-0-1.nycmnyr-rtr2. 0.0% 10 298.0 168.6 69.3 298.0 79.0 4.|-- agg23.nyquny9101r.nyc.rr. 0.0% 10 292.5 155.0 90.8 292.5 71.0 5.|-- ae-3-0.cr0.nyc20.tbone.rr 0.0% 10 239.0 148.6 68.5 248.6 63.2 6.|-- 107.14.17.169 0.0% 10 268.1 148.7 73.0 268.1 64.9 7.|-- xe-4-2-0.edge4.frankfurt1 10.0% 10 235.1 130.2 66.2 235.1 49.2 8.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0 sajal@sajal-laptop:~$
-Sajal
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Yup [mike@orion ~]$ sudo mtr --report facebook.com Start: Sat Oct 19 10:25:15 2013 HOST: orion.lieman.net Loss% Snt Last Avg Best Wrst StDev 1.|-- 172.16.32.1 0.0% 10 0.8 0.8 0.7 1.0 0.0 2.|-- cable-mac1.ldvlnyae-ar400 0.0% 10 29.6 36.5 19.5 97.2 23.8 3.|-- gig1-19.troynydv-rtr003.a 0.0% 10 26.6 14.8 10.9 26.6 5.8 4.|-- rdc-74-76-243-54.alb.east 0.0% 10 11.5 13.7 11.5 23.8 3.6 5.|-- rdc-74-76-241-193.alb.nor 0.0% 10 21.2 25.4 20.1 35.9 5.6 6.|-- 107.14.19.26 0.0% 10 48.1 31.7 21.3 50.9 12.4 7.|-- 107.14.17.169 0.0% 10 21.3 26.0 20.0 65.7 14.1 8.|-- ae9.edge3.Newark1.Level3. 0.0% 10 47.4 30.4 20.2 47.4 10.6 9.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0
participants (8)
-
Adam Greene -
Darrell Hyde -
Darren Schreiber -
Mike Lieman -
Mitch -
Sajal Kayan -
Tim Glen -
Vinny_Abello＠Dell.com