We had a report from a customer earlier this morning, but they advised the problem went away. We connect to Level3 at Southfield MI. At 09:35 AM 25/02/2014, *Network wrote:
Content-Language: en-US Content-Type: multipart/alternative;
boundary="_000_391AAB48E84BB24288BED210F379E1DC4DEA4C02NOCMAILDAG2Inte_"
Latency issue with Level 3 here in Detroit area
Anton
From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Luke Rockwell Sent: Tuesday, February 18, 2014 12:24 PM To: Petter Bruland Cc: Outages@outages.org Subject: Re: [outages] Level3 Routing...
nothing on the level 3 network events page I have access to
Luke
On Tue, Feb 18, 2014 at 9:17 AM, Petter Bruland <<mailto:Petter.Bruland@allegiantair.com>Petter.Bruland@allegiantair.com> wrote: Not seeing any issues with Level3 in Las Vegas.
-Petter
From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Nick Olsen Sent: Tuesday, February 18, 2014 6:04 AM To: Morgan Miskell; <mailto:Outages@outages.org>Outages@outages.org
Subject: Re: [outages] Level3 Routing...
Not seeing anything homed off of Level 3 Orlando.
Traceroutes to a few common places look fine.
~474815 Routes from 3356. Nick Olsen Network Operations (855) FLSPEED x106
---------- From: "Morgan Miskell" <<mailto:morgan.miskell@caro.net>morgan.miskell@caro.net> Sent: Tuesday, February 18, 2014 9:00 AM To: "<mailto:Outages@outages.org>Outages@outages.org" <<mailto:outages@outages.org>outages@outages.org> Subject: [outages] Level3 Routing...
Anyone seeing Level3 routing issues? We just observed a massive drop in routes in the BGP feeds and seeing some strange ICMP issues along their backbone in terms of traceroutes along their backbone. They might be adding some filters based on the recent DDOS attacks, I can ping interfaces but traceroutes go to hell... -- Morgan A. Miskell CaroNet Data Centers 704-643-8330 x206 ---------------------------------------------------------------------------- The information contained in this e-mail is confidential and is intended only for the named recipient(s). If you are not the intended recipient you must not copy, distribute, or take any action or reliance on it. If you have received this e-mail in error, please notify the sender. Any unauthorized disclosure of the information contained in this e-mail is strictly prohibited. ----------------------------------------------------------------------------
_______________________________________________ Outages mailing list <mailto:Outages@outages.org>Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list <mailto:Outages@outages.org>Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
--- Clayton Zekelman Managed Network Systems Inc. (MNSi) 3363 Tecumseh Rd. E Windsor, Ontario N8W 1H4 tel. 519-985-8410 fax. 519-985-8409
As always, trace routes are helpful. We've been having intermittent issue with Level 3 routes for the past couple weeks, most issue for us seem to be when going from Level 3 to Global Crossing routes. From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Clayton Zekelman Sent: Tuesday, February 25, 2014 6:39 AM To: *Network; Luke Rockwell; Petter Bruland Cc: Outages@outages.org Subject: Re: [outages] Level3 Routing... We had a report from a customer earlier this morning, but they advised the problem went away. We connect to Level3 at Southfield MI. At 09:35 AM 25/02/2014, *Network wrote: Content-Language: en-US Content-Type: multipart/alternative; boundary="_000_391AAB48E84BB24288BED210F379E1DC4DEA4C02NOCMAILDAG2Inte_" Latency issue with Level 3 here in Detroit area Anton From: Outages [ mailto:outages-bounces@outages.org] On Behalf Of Luke Rockwell Sent: Tuesday, February 18, 2014 12:24 PM To: Petter Bruland Cc: Outages@outages.org<mailto:Outages@outages.org> Subject: Re: [outages] Level3 Routing... nothing on the level 3 network events page I have access to Luke On Tue, Feb 18, 2014 at 9:17 AM, Petter Bruland < Petter.Bruland@allegiantair.com<mailto:Petter.Bruland@allegiantair.com>> wrote: Not seeing any issues with Level3 in Las Vegas. -Petter From: Outages [ mailto:outages-bounces@outages.org] On Behalf Of Nick Olsen Sent: Tuesday, February 18, 2014 6:04 AM To: Morgan Miskell; Outages@outages.org<mailto:Outages@outages.org> Subject: Re: [outages] Level3 Routing... Not seeing anything homed off of Level 3 Orlando. Traceroutes to a few common places look fine. ~474815 Routes from 3356. Nick Olsen Network Operations (855) FLSPEED x106 ________________________________ From: "Morgan Miskell" <morgan.miskell@caro.net<mailto:morgan.miskell@caro.net> > Sent: Tuesday, February 18, 2014 9:00 AM To: "Outages@outages.org<mailto:Outages@outages.org>" <outages@outages.org<mailto:outages@outages.org>> Subject: [outages] Level3 Routing... Anyone seeing Level3 routing issues? We just observed a massive drop in routes in the BGP feeds and seeing some strange ICMP issues along their backbone in terms of traceroutes along their backbone. They might be adding some filters based on the recent DDOS attacks, I can ping interfaces but traceroutes go to hell... -- Morgan A. Miskell CaroNet Data Centers 704-643-8330 x206 ---------------------------------------------------------------------------- The information contained in this e-mail is confidential and is intended only for the named recipient(s). If you are not the intended recipient you must not copy, distribute, or take any action or reliance on it. If you have received this e-mail in error, please notify the sender. Any unauthorized disclosure of the information contained in this e-mail is strictly prohibited. ---------------------------------------------------------------------------- _______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages _______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages _______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages --- Clayton Zekelman Managed Network Systems Inc. (MNSi) 3363 Tecumseh Rd. E Windsor, Ontario N8W 1H4 tel. 519-985-8410 fax. 519-985-8409
That's what we told our customer... by the time they replied, the issue had gone away. At 09:57 AM 25/02/2014, Michael Morrison wrote:
Content-Language: en-US Content-Type: multipart/alternative; boundary="_000_0693cfd728744391a68dbd0e93123a64EX01wcilocal_"
As always, trace routes are helpful. We've been having intermittent issue with Level 3 routes for the past couple weeks, most issue for us seem to be when going from Level 3 to Global Crossing routes.
From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Clayton Zekelman Sent: Tuesday, February 25, 2014 6:39 AM To: *Network; Luke Rockwell; Petter Bruland Cc: Outages@outages.org Subject: Re: [outages] Level3 Routing...
We had a report from a customer earlier this morning, but they advised the problem went away.
We connect to Level3 at Southfield MI.
At 09:35 AM 25/02/2014, *Network wrote:
Content-Language: en-US Content-Type: multipart/alternative;
boundary="_000_391AAB48E84BB24288BED210F379E1DC4DEA4C02NOCMAILDAG2Inte_"
Latency issue with Level 3 here in Detroit area
Anton
From: Outages [ mailto:outages-bounces@outages.org] On Behalf Of Luke Rockwell Sent: Tuesday, February 18, 2014 12:24 PM To: Petter Bruland Cc: <mailto:Outages@outages.org>Outages@outages.org Subject: Re: [outages] Level3 Routing...
nothing on the level 3 network events page I have access to
Luke
On Tue, Feb 18, 2014 at 9:17 AM, Petter Bruland <<mailto:Petter.Bruland@allegiantair.com> Petter.Bruland@allegiantair.com> wrote: Not seeing any issues with Level3 in Las Vegas.
-Petter
From: Outages [ mailto:outages-bounces@outages.org] On Behalf Of Nick Olsen Sent: Tuesday, February 18, 2014 6:04 AM To: Morgan Miskell; <mailto:Outages@outages.org>Outages@outages.org
Subject: Re: [outages] Level3 Routing...
Not seeing anything homed off of Level 3 Orlando.
Traceroutes to a few common places look fine.
~474815 Routes from 3356. Nick Olsen Network Operations (855) FLSPEED x106
---------- From: "Morgan Miskell" <<mailto:morgan.miskell@caro.net>morgan.miskell@caro.net > Sent: Tuesday, February 18, 2014 9:00 AM To: "<mailto:Outages@outages.org>Outages@outages.org" <<mailto:outages@outages.org>outages@outages.org> Subject: [outages] Level3 Routing...
Anyone seeing Level3 routing issues? We just observed a massive drop in routes in the BGP feeds and seeing some strange ICMP issues along their backbone in terms of traceroutes along their backbone. They might be adding some filters based on the recent DDOS attacks, I can ping interfaces but traceroutes go to hell... -- Morgan A. Miskell CaroNet Data Centers 704-643-8330 x206 ---------------------------------------------------------------------------- The information contained in this e-mail is confidential and is intended only for the named recipient(s). If you are not the intended recipient you must not copy, distribute, or take any action or reliance on it. If you have received this e-mail in error, please notify the sender. Any unauthorized disclosure of the information contained in this e-mail is strictly prohibited. ----------------------------------------------------------------------------
_______________________________________________ Outages mailing list <mailto:Outages@outages.org>Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list <mailto:Outages@outages.org>Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list <mailto:Outages@outages.org>Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
---
Clayton Zekelman Managed Network Systems Inc. (MNSi) 3363 Tecumseh Rd. E Windsor, Ontario N8W 1H4
tel. 519-985-8410 fax. 519-985-8409 _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
--- Clayton Zekelman Managed Network Systems Inc. (MNSi) 3363 Tecumseh Rd. E Windsor, Ontario N8W 1H4 tel. 519-985-8410 fax. 519-985-8409
We've been having issues in San Jose the past two days on both L3 MPLS and Internet circuits. MPLS was resolved yesterday with these notes: "Event The IP NOC advised that troubleshooting is ongoing with the equipment vendor to further isolate the point of failure. Additionally, it has been advised that voice services are impacted and calls are failing network wide. Collaborative troubleshooting efforts between the IP and Voice NOC and the equipment vendor will continue in an effort to expedite service restoral." However, we're still having internet issues. Alex From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Clayton Zekelman Sent: Tuesday, February 25, 2014 7:04 AM To: Michael Morrison; Outages@outages.org Subject: Re: [outages] Level3 Routing... That's what we told our customer... by the time they replied, the issue had gone away. At 09:57 AM 25/02/2014, Michael Morrison wrote: Content-Language: en-US Content-Type: multipart/alternative; boundary="_000_0693cfd728744391a68dbd0e93123a64EX01wcilocal_" As always, trace routes are helpful. We've been having intermittent issue with Level 3 routes for the past couple weeks, most issue for us seem to be when going from Level 3 to Global Crossing routes. From: Outages [ mailto:outages-bounces@outages.org <mailto:outages-bounces@outages.org> ] On Behalf Of Clayton Zekelman Sent: Tuesday, February 25, 2014 6:39 AM To: *Network; Luke Rockwell; Petter Bruland Cc: Outages@outages.org Subject: Re: [outages] Level3 Routing... We had a report from a customer earlier this morning, but they advised the problem went away. We connect to Level3 at Southfield MI. At 09:35 AM 25/02/2014, *Network wrote: Content-Language: en-US Content-Type: multipart/alternative; boundary="_000_391AAB48E84BB24288BED210F379E1DC4DEA4C02NOCMAILDAG2Inte_" Latency issue with Level 3 here in Detroit area Anton From: Outages [ <mailto:outages-bounces@outages.org> mailto:outages-bounces@outages.org] On Behalf Of Luke Rockwell Sent: Tuesday, February 18, 2014 12:24 PM To: Petter Bruland Cc: Outages@outages.org Subject: Re: [outages] Level3 Routing... nothing on the level 3 network events page I have access to Luke On Tue, Feb 18, 2014 at 9:17 AM, Petter Bruland < Petter.Bruland@allegiantair.com <mailto:Petter.Bruland@allegiantair.com> > wrote: Not seeing any issues with Level3 in Las Vegas. -Petter From: Outages [ <mailto:outages-bounces@outages.org> mailto:outages-bounces@outages.org] On Behalf Of Nick Olsen Sent: Tuesday, February 18, 2014 6:04 AM To: Morgan Miskell; Outages@outages.org Subject: Re: [outages] Level3 Routing... Not seeing anything homed off of Level 3 Orlando. Traceroutes to a few common places look fine. ~474815 Routes from 3356. Nick Olsen Network Operations (855) FLSPEED x106 _____ From: "Morgan Miskell" <morgan.miskell@caro.net > Sent: Tuesday, February 18, 2014 9:00 AM To: "Outages@outages.org" <outages@outages.org> Subject: [outages] Level3 Routing... Anyone seeing Level3 routing issues? We just observed a massive drop in routes in the BGP feeds and seeing some strange ICMP issues along their backbone in terms of traceroutes along their backbone. They might be adding some filters based on the recent DDOS attacks, I can ping interfaces but traceroutes go to hell... -- Morgan A. Miskell CaroNet Data Centers 704-643-8330 x206 ---------------------------------------------------------------------------- The information contained in this e-mail is confidential and is intended only for the named recipient(s). If you are not the intended recipient you must not copy, distribute, or take any action or reliance on it. If you have received this e-mail in error, please notify the sender. Any unauthorized disclosure of the information contained in this e-mail is strictly prohibited. ---------------------------------------------------------------------------- _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages --- Clayton Zekelman Managed Network Systems Inc. (MNSi) 3363 Tecumseh Rd. E Windsor, Ontario N8W 1H4 tel. 519-985-8410 fax. 519-985-8409 _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages --- Clayton Zekelman Managed Network Systems Inc. (MNSi) 3363 Tecumseh Rd. E Windsor, Ontario N8W 1H4 tel. 519-985-8410 fax. 519-985-8409
We've also see increased packet loss and latency over the past week or so from our London Level3 office to our Verizon offices in NY and LA. Routes to our ATT office in Chicago have been okay for the most part. Here's a current trace from London to LAX. Heavy packet loss in London. It was in Paris before that :) My traceroute [v0.80] lon-nagios (0.0.0.0) Fri Feb 28 21:52:35 2014 Resolver: Received error response 2. (server failure)er of fields quit Packets Pings Host Loss% Snt Last Avg Best Wrst StDev 1. xxxxxxxxxxxxxxxxxxxxxxxxxxx 0.0% 468 0.3 0.3 0.2 4.0 0.3 2. xxxxxxxxxxxxxxx 0.0% 468 0.9 2.3 0.8 79.1 7.7 3. ae-0-11.edge3.london2.level3.net 61.8% 468 1.1 1.6 1.0 18.0 2.5 4. ae-3-3.ebr1.paris1.level3.net 0.0% 468 109.9 171.8 103.0 246.0 26.9 5. ae-91-91.csw4.paris1.level3.net 0.2% 468 111.7 170.4 99.6 241.8 27.0 6. ae-92-92.ebr2.paris1.level3.net 0.0% 468 121.8 170.1 95.8 236.3 27.0 7. ae-41-41.ebr2.washington1.level3.net 0.0% 467 118.5 171.4 95.7 241.1 27.1 8. ae-45-45.ebr2.washington12.level3.net 0.0% 467 113.4 171.8 94.9 243.5 27.0 9. ae-204-3604.edge1.washington12.level3.net 0.0% 467 107.7 171.7 95.4 250.7 27.8 ae-201-3601.edge1.washington12.level3.net ae-203-3603.edge1.washington12.level3.net ae-202-3602.edge1.washington12.level3.net 10. 208.51.134.89 0.0% 467 144.0 233.5 124.8 454.1 47.8 11. po1-20g.ar6.dca3.gblx.net 0.0% 467 148.8 226.8 125.8 448.9 37.7 12. uunet-2.ar3.dca3.gblx.net 0.0% 467 146.1 225.9 126.5 319.5 32.9 13. ??? 14. tengige0-6-0-2.gw5.lax7.alter 0.0% 467 203.1 290.3 203.1 370.6 31.4 15. whitehousepost-gw.customer.alter.net 0.0% 467 210.6 289.2 200.1 368.3 30.6 From: Outages [mailto:outages-bounces@outages.org] On Behalf Of alexzarris@yahoo.com Sent: Tuesday, February 25, 2014 11:00 AM To: 'Clayton Zekelman'; 'Michael Morrison'; 'Outages@outages.org' Subject: Re: [outages] Level3 Routing... We've been having issues in San Jose the past two days on both L3 MPLS and Internet circuits. MPLS was resolved yesterday with these notes: "Event The IP NOC advised that troubleshooting is ongoing with the equipment vendor to further isolate the point of failure. Additionally, it has been advised that voice services are impacted and calls are failing network wide. Collaborative troubleshooting efforts between the IP and Voice NOC and the equipment vendor will continue in an effort to expedite service restoral." However, we're still having internet issues. Alex From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Clayton Zekelman Sent: Tuesday, February 25, 2014 7:04 AM To: Michael Morrison; Outages@outages.org<mailto:Outages@outages.org> Subject: Re: [outages] Level3 Routing... That's what we told our customer... by the time they replied, the issue had gone away. At 09:57 AM 25/02/2014, Michael Morrison wrote: Content-Language: en-US Content-Type: multipart/alternative; boundary="_000_0693cfd728744391a68dbd0e93123a64EX01wcilocal_" As always, trace routes are helpful. We've been having intermittent issue with Level 3 routes for the past couple weeks, most issue for us seem to be when going from Level 3 to Global Crossing routes. From: Outages [ mailto:outages-bounces@outages.org] On Behalf Of Clayton Zekelman Sent: Tuesday, February 25, 2014 6:39 AM To: *Network; Luke Rockwell; Petter Bruland Cc: Outages@outages.org<mailto:Outages@outages.org> Subject: Re: [outages] Level3 Routing... We had a report from a customer earlier this morning, but they advised the problem went away. We connect to Level3 at Southfield MI. At 09:35 AM 25/02/2014, *Network wrote: Content-Language: en-US Content-Type: multipart/alternative; boundary="_000_391AAB48E84BB24288BED210F379E1DC4DEA4C02NOCMAILDAG2Inte_" Latency issue with Level 3 here in Detroit area Anton From: Outages [ <mailto:outages-bounces@outages.org> mailto:outages-bounces@outages.org] On Behalf Of Luke Rockwell Sent: Tuesday, February 18, 2014 12:24 PM To: Petter Bruland Cc: Outages@outages.org<mailto:Outages@outages.org> Subject: Re: [outages] Level3 Routing... nothing on the level 3 network events page I have access to Luke On Tue, Feb 18, 2014 at 9:17 AM, Petter Bruland < Petter.Bruland@allegiantair.com<mailto:Petter.Bruland@allegiantair.com>> wrote: Not seeing any issues with Level3 in Las Vegas. -Petter From: Outages [ <mailto:outages-bounces@outages.org> mailto:outages-bounces@outages.org] On Behalf Of Nick Olsen Sent: Tuesday, February 18, 2014 6:04 AM To: Morgan Miskell; Outages@outages.org<mailto:Outages@outages.org> Subject: Re: [outages] Level3 Routing... Not seeing anything homed off of Level 3 Orlando. Traceroutes to a few common places look fine. ~474815 Routes from 3356. Nick Olsen Network Operations (855) FLSPEED x106 ________________________________ From: "Morgan Miskell" <morgan.miskell@caro.net<mailto:morgan.miskell@caro.net> > Sent: Tuesday, February 18, 2014 9:00 AM To: "Outages@outages.org<mailto:Outages@outages.org>" <outages@outages.org<mailto:outages@outages.org>> Subject: [outages] Level3 Routing... Anyone seeing Level3 routing issues? We just observed a massive drop in routes in the BGP feeds and seeing some strange ICMP issues along their backbone in terms of traceroutes along their backbone. They might be adding some filters based on the recent DDOS attacks, I can ping interfaces but traceroutes go to hell... -- Morgan A. Miskell CaroNet Data Centers 704-643-8330 x206 ---------------------------------------------------------------------------- The information contained in this e-mail is confidential and is intended only for the named recipient(s). If you are not the intended recipient you must not copy, distribute, or take any action or reliance on it. If you have received this e-mail in error, please notify the sender. Any unauthorized disclosure of the information contained in this e-mail is strictly prohibited. ---------------------------------------------------------------------------- _______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages _______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages _______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages --- Clayton Zekelman Managed Network Systems Inc. (MNSi) 3363 Tecumseh Rd. E Windsor, Ontario N8W 1H4 tel. 519-985-8410 fax. 519-985-8409 _______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages --- Clayton Zekelman Managed Network Systems Inc. (MNSi) 3363 Tecumseh Rd. E Windsor, Ontario N8W 1H4 tel. 519-985-8410 fax. 519-985-8409
Nothing on the L3 trouble ticket system ..................................... *Luke Rockwell* Systems and Support Analyst Information Technology ..................................... Cal Alumni Association | UC Berkeley 1 Alumni House, Berkeley, CA 94720 *T* 510.900.8196 *F* 510.642.6252 ..................................... *Commitment, Support, Passion* _____________________________________ *alumni.berkeley.edu <http://alumni.berkeley.edu/>* _____________________________________ *Facebook <https://www.facebook.com/CalAlumni> | LinkedIn <http://www.linkedin.com/groups?gid=70245>* On Fri, Feb 28, 2014 at 2:04 PM, jay lark <jay@jaylark.com> wrote:
We've also see increased packet loss and latency over the past week or so from our London Level3 office to our Verizon offices in NY and LA. Routes to our ATT office in Chicago have been okay for the most part.
Here's a current trace from London to LAX. Heavy packet loss in London. It was in Paris before that J
My traceroute [v0.80]
lon-nagios (0.0.0.0) Fri Feb 28 21:52:35 2014
Resolver: Received error response 2. (server failure)er of fields quit
Packets Pings
Host Loss% Snt Last Avg Best Wrst StDev
1. xxxxxxxxxxxxxxxxxxxxxxxxxxx 0.0% 468 0.3 0.3 0.2 4.0 0.3
2. xxxxxxxxxxxxxxx 0.0% 468 0.9 2.3 0.8 79.1 7.7
3. ae-0-11.edge3.london2.level3.net 61.8% 468 1.1 1.6 1.0 18.0 2.5
4. ae-3-3.ebr1.paris1.level3.net 0.0% 468 109.9 171.8 103.0 246.0 26.9
5. ae-91-91.csw4.paris1.level3.net 0.2% 468 111.7 170.4 99.6 241.8 27.0
6. ae-92-92.ebr2.paris1.level3.net 0.0% 468 121.8 170.1 95.8 236.3 27.0
7. ae-41-41.ebr2.washington1.level3.net 0.0% 467 118.5 171.4 95.7 241.1 27.1
8. ae-45-45.ebr2.washington12.level3.net 0.0% 467 113.4 171.8 94.9 243.5 27.0
9. ae-204-3604.edge1.washington12.level3.net 0.0% 467 107.7 171.7 95.4 250.7 27.8
ae-201-3601.edge1.washington12.level3.net
ae-203-3603.edge1.washington12.level3.net
ae-202-3602.edge1.washington12.level3.net
10. 208.51.134.89 0.0% 467 144.0 233.5 124.8 454.1 47.8
11. po1-20g.ar6.dca3.gblx.net 0.0% 467 148.8 226.8 125.8 448.9 37.7
12. uunet-2.ar3.dca3.gblx.net 0.0% 467 146.1 225.9 126.5 319.5 32.9
13. ???
14. tengige0-6-0-2.gw5.lax7.alter 0.0% 467 203.1 290.3 203.1 370.6 31.4
15. whitehousepost-gw.customer.alter.net 0.0% 467 210.6 289.2 200.1 368.3 30.6
*From:* Outages [mailto:outages-bounces@outages.org] *On Behalf Of * alexzarris@yahoo.com *Sent:* Tuesday, February 25, 2014 11:00 AM *To:* 'Clayton Zekelman'; 'Michael Morrison'; 'Outages@outages.org'
*Subject:* Re: [outages] Level3 Routing...
We've been having issues in San Jose the past two days on both L3 MPLS and Internet circuits. MPLS was resolved yesterday with these notes:
"Event The IP NOC advised that troubleshooting is ongoing with the equipment vendor to further isolate the point of failure. Additionally, it has been advised that voice services are impacted and calls are failing network wide. Collaborative troubleshooting efforts between the IP and Voice NOC and the equipment vendor will continue in an effort to expedite service restoral."
However, we're still having internet issues.
Alex
*From:* Outages [mailto:outages-bounces@outages.org<outages-bounces@outages.org>] *On Behalf Of *Clayton Zekelman *Sent:* Tuesday, February 25, 2014 7:04 AM *To:* Michael Morrison; Outages@outages.org *Subject:* Re: [outages] Level3 Routing...
That's what we told our customer... by the time they replied, the issue had gone away.
At 09:57 AM 25/02/2014, Michael Morrison wrote:
Content-Language: en-US Content-Type: multipart/alternative; boundary="_000_0693cfd728744391a68dbd0e93123a64EX01wcilocal_"
As always, trace routes are helpful. We've been having intermittent issue with Level 3 routes for the past couple weeks, most issue for us seem to be when going from Level 3 to Global Crossing routes.
*From:* Outages [ mailto:outages-bounces@outages.org<outages-bounces@outages.org>] *On Behalf Of *Clayton Zekelman *Sent:* Tuesday, February 25, 2014 6:39 AM *To:* *Network; Luke Rockwell; Petter Bruland *Cc:* Outages@outages.org *Subject:* Re: [outages] Level3 Routing...
We had a report from a customer earlier this morning, but they advised the problem went away.
We connect to Level3 at Southfield MI.
At 09:35 AM 25/02/2014, *Network wrote:
Content-Language: en-US
Content-Type: multipart/alternative;
boundary="_000_391AAB48E84BB24288BED210F379E1DC4DEA4C02NOCMAILDAG2Inte_"
Latency issue with Level 3 here in Detroit area
Anton
From: Outages [ <outages-bounces@outages.org> mailto:outages-bounces@outages.org <outages-bounces@outages.org>] On Behalf Of Luke Rockwell
Sent: Tuesday, February 18, 2014 12:24 PM
To: Petter Bruland
Cc: Outages@outages.org
Subject: Re: [outages] Level3 Routing...
nothing on the level 3 network events page I have access to
Luke
On Tue, Feb 18, 2014 at 9:17 AM, Petter Bruland <Petter.Bruland@allegiantair.com> wrote:
Not seeing any issues with Level3 in Las Vegas.
-Petter
From: Outages [ <outages-bounces@outages.org> mailto:outages-bounces@outages.org <outages-bounces@outages.org>] On Behalf Of Nick Olsen
Sent: Tuesday, February 18, 2014 6:04 AM
To: Morgan Miskell; Outages@outages.org
Subject: Re: [outages] Level3 Routing...
Not seeing anything homed off of Level 3 Orlando.
Traceroutes to a few common places look fine.
~474815 Routes from 3356.
Nick Olsen
Network Operations
(855) FLSPEED x106
------------------------------
From: "Morgan Miskell" <morgan.miskell@caro.net >
Sent: Tuesday, February 18, 2014 9:00 AM
To: "Outages@outages.org" <outages@outages.org>
Subject: [outages] Level3 Routing...
Anyone seeing Level3 routing issues? We just observed a massive drop in
routes in the BGP feeds and seeing some strange ICMP issues along their
backbone in terms of traceroutes along their backbone. They might be
adding some filters based on the recent DDOS attacks, I can ping
interfaces but traceroutes go to hell...
--
Morgan A. Miskell
CaroNet Data Centers
704-643-8330 x206
----------------------------------------------------------------------------
The information contained in this e-mail is confidential and is intended
only for the named recipient(s). If you are not the intended recipient
you must not copy, distribute, or take any action or reliance on it. If
you have received this e-mail in error, please notify the sender. Any
unauthorized disclosure of the information contained in this e-mail is
strictly prohibited.
----------------------------------------------------------------------------
_______________________________________________
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages
_______________________________________________
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages
_______________________________________________
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages
---
Clayton Zekelman Managed Network Systems Inc. (MNSi) 3363 Tecumseh Rd. E Windsor, Ontario N8W 1H4
tel. 519-985-8410 fax. 519-985-8409 _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
---
Clayton Zekelman Managed Network Systems Inc. (MNSi) 3363 Tecumseh Rd. E Windsor, Ontario N8W 1H4
tel. 519-985-8410 fax. 519-985-8409
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Remember (and maybe someone can link to the trace route interpretation PDF from Nanog) that if the packet loss doesn't continue end to end then its not likely a real issue as the forwarding plane isn't loosing packets. The loss on a single hop either points to limits on ICMP responses on the control plane, or congestion on the forwarding plane. On Feb 28, 2014 5:09 PM, "jay lark" <jay@jaylark.com> wrote:
We've also see increased packet loss and latency over the past week or so from our London Level3 office to our Verizon offices in NY and LA. Routes to our ATT office in Chicago have been okay for the most part.
Here's a current trace from London to LAX. Heavy packet loss in London. It was in Paris before that J
My traceroute [v0.80]
lon-nagios (0.0.0.0) Fri Feb 28 21:52:35 2014
Resolver: Received error response 2. (server failure)er of fields quit
Packets Pings
Host Loss% Snt Last Avg Best Wrst StDev
1. xxxxxxxxxxxxxxxxxxxxxxxxxxx 0.0% 468 0.3 0.3 0.2 4.0 0.3
2. xxxxxxxxxxxxxxx 0.0% 468 0.9 2.3 0.8 79.1 7.7
3. ae-0-11.edge3.london2.level3.net 61.8% 468 1.1 1.6 1.0 18.0 2.5
4. ae-3-3.ebr1.paris1.level3.net 0.0% 468 109.9 171.8 103.0 246.0 26.9
5. ae-91-91.csw4.paris1.level3.net 0.2% 468 111.7 170.4 99.6 241.8 27.0
6. ae-92-92.ebr2.paris1.level3.net 0.0% 468 121.8 170.1 95.8 236.3 27.0
7. ae-41-41.ebr2.washington1.level3.net 0.0% 467 118.5 171.4 95.7 241.1 27.1
8. ae-45-45.ebr2.washington12.level3.net 0.0% 467 113.4 171.8 94.9 243.5 27.0
9. ae-204-3604.edge1.washington12.level3.net 0.0% 467 107.7 171.7 95.4 250.7 27.8
ae-201-3601.edge1.washington12.level3.net
ae-203-3603.edge1.washington12.level3.net
ae-202-3602.edge1.washington12.level3.net
10. 208.51.134.89 0.0% 467 144.0 233.5 124.8 454.1 47.8
11. po1-20g.ar6.dca3.gblx.net 0.0% 467 148.8 226.8 125.8 448.9 37.7
12. uunet-2.ar3.dca3.gblx.net 0.0% 467 146.1 225.9 126.5 319.5 32.9
13. ???
14. tengige0-6-0-2.gw5.lax7.alter 0.0% 467 203.1 290.3 203.1 370.6 31.4
15. whitehousepost-gw.customer.alter.net 0.0% 467 210.6 289.2 200.1 368.3 30.6
*From:* Outages [mailto:outages-bounces@outages.org] *On Behalf Of * alexzarris@yahoo.com *Sent:* Tuesday, February 25, 2014 11:00 AM *To:* 'Clayton Zekelman'; 'Michael Morrison'; 'Outages@outages.org' *Subject:* Re: [outages] Level3 Routing...
We've been having issues in San Jose the past two days on both L3 MPLS and Internet circuits. MPLS was resolved yesterday with these notes:
"Event The IP NOC advised that troubleshooting is ongoing with the equipment vendor to further isolate the point of failure. Additionally, it has been advised that voice services are impacted and calls are failing network wide. Collaborative troubleshooting efforts between the IP and Voice NOC and the equipment vendor will continue in an effort to expedite service restoral."
However, we're still having internet issues.
Alex
*From:* Outages [mailto:outages-bounces@outages.org<outages-bounces@outages.org>] *On Behalf Of *Clayton Zekelman *Sent:* Tuesday, February 25, 2014 7:04 AM *To:* Michael Morrison; Outages@outages.org *Subject:* Re: [outages] Level3 Routing...
That's what we told our customer... by the time they replied, the issue had gone away.
At 09:57 AM 25/02/2014, Michael Morrison wrote:
Content-Language: en-US Content-Type: multipart/alternative; boundary="_000_0693cfd728744391a68dbd0e93123a64EX01wcilocal_"
As always, trace routes are helpful. We've been having intermittent issue with Level 3 routes for the past couple weeks, most issue for us seem to be when going from Level 3 to Global Crossing routes.
*From:* Outages [ mailto:outages-bounces@outages.org<outages-bounces@outages.org>] *On Behalf Of *Clayton Zekelman *Sent:* Tuesday, February 25, 2014 6:39 AM *To:* *Network; Luke Rockwell; Petter Bruland *Cc:* Outages@outages.org *Subject:* Re: [outages] Level3 Routing...
We had a report from a customer earlier this morning, but they advised the problem went away.
We connect to Level3 at Southfield MI.
At 09:35 AM 25/02/2014, *Network wrote:
Content-Language: en-US
Content-Type: multipart/alternative;
boundary="_000_391AAB48E84BB24288BED210F379E1DC4DEA4C02NOCMAILDAG2Inte_"
Latency issue with Level 3 here in Detroit area
Anton
From: Outages [ <outages-bounces@outages.org> mailto:outages-bounces@outages.org <outages-bounces@outages.org>] On Behalf Of Luke Rockwell
Sent: Tuesday, February 18, 2014 12:24 PM
To: Petter Bruland
Cc: Outages@outages.org
Subject: Re: [outages] Level3 Routing...
nothing on the level 3 network events page I have access to
Luke
On Tue, Feb 18, 2014 at 9:17 AM, Petter Bruland <Petter.Bruland@allegiantair.com> wrote:
Not seeing any issues with Level3 in Las Vegas.
-Petter
From: Outages [ <outages-bounces@outages.org> mailto:outages-bounces@outages.org <outages-bounces@outages.org>] On Behalf Of Nick Olsen
Sent: Tuesday, February 18, 2014 6:04 AM
To: Morgan Miskell; Outages@outages.org
Subject: Re: [outages] Level3 Routing...
Not seeing anything homed off of Level 3 Orlando.
Traceroutes to a few common places look fine.
~474815 Routes from 3356.
Nick Olsen
Network Operations
(855) FLSPEED x106
------------------------------
From: "Morgan Miskell" <morgan.miskell@caro.net >
Sent: Tuesday, February 18, 2014 9:00 AM
To: "Outages@outages.org" <outages@outages.org>
Subject: [outages] Level3 Routing...
Anyone seeing Level3 routing issues? We just observed a massive drop in
routes in the BGP feeds and seeing some strange ICMP issues along their
backbone in terms of traceroutes along their backbone. They might be
adding some filters based on the recent DDOS attacks, I can ping
interfaces but traceroutes go to hell...
--
Morgan A. Miskell
CaroNet Data Centers
704-643-8330 x206
----------------------------------------------------------------------------
The information contained in this e-mail is confidential and is intended
only for the named recipient(s). If you are not the intended recipient
you must not copy, distribute, or take any action or reliance on it. If
you have received this e-mail in error, please notify the sender. Any
unauthorized disclosure of the information contained in this e-mail is
strictly prohibited.
----------------------------------------------------------------------------
_______________________________________________
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages
_______________________________________________
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages
_______________________________________________
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages
---
Clayton Zekelman Managed Network Systems Inc. (MNSi) 3363 Tecumseh Rd. E Windsor, Ontario N8W 1H4
tel. 519-985-8410 fax. 519-985-8409 _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
---
Clayton Zekelman Managed Network Systems Inc. (MNSi) 3363 Tecumseh Rd. E Windsor, Ontario N8W 1H4
tel. 519-985-8410 fax. 519-985-8409
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
I see no indication of packet loss in this mtr; others have already explained how to properly read output. Loss at hop #3 is not indicative of a universal problem; it would be different if that latency "trickled down" through all subsequent hops, all the way to the destination. I do see indication in this mtr of generally high latency (between London<->Paris), reaching up to 246ms and averaging around 171ms. I do not know if that's normal or not, due to my lack of familiarity with said region/paths. However, as I've stated repeatedly [like a broken record], a traceroute/mtr from point A to B is only half the picture -- you need the same from point B to A, due to asymmetrical routing being commonplace. -- | Jeremy Chadwick jdc@koitsu.org | | UNIX Systems Administrator http://jdc.koitsu.org/ | | Making life hard for others since 1977. PGP 4BD6C0CB | On Fri, Feb 28, 2014 at 10:04:18PM +0000, jay lark wrote:
We've also see increased packet loss and latency over the past week or so from our London Level3 office to our Verizon offices in NY and LA. Routes to our ATT office in Chicago have been okay for the most part.
Here's a current trace from London to LAX. Heavy packet loss in London. It was in Paris before that :)
My traceroute [v0.80] lon-nagios (0.0.0.0) Fri Feb 28 21:52:35 2014 Resolver: Received error response 2. (server failure)er of fields quit Packets Pings Host Loss% Snt Last Avg Best Wrst StDev 1. xxxxxxxxxxxxxxxxxxxxxxxxxxx 0.0% 468 0.3 0.3 0.2 4.0 0.3 2. xxxxxxxxxxxxxxx 0.0% 468 0.9 2.3 0.8 79.1 7.7 3. ae-0-11.edge3.london2.level3.net 61.8% 468 1.1 1.6 1.0 18.0 2.5 4. ae-3-3.ebr1.paris1.level3.net 0.0% 468 109.9 171.8 103.0 246.0 26.9 5. ae-91-91.csw4.paris1.level3.net 0.2% 468 111.7 170.4 99.6 241.8 27.0 6. ae-92-92.ebr2.paris1.level3.net 0.0% 468 121.8 170.1 95.8 236.3 27.0 7. ae-41-41.ebr2.washington1.level3.net 0.0% 467 118.5 171.4 95.7 241.1 27.1 8. ae-45-45.ebr2.washington12.level3.net 0.0% 467 113.4 171.8 94.9 243.5 27.0 9. ae-204-3604.edge1.washington12.level3.net 0.0% 467 107.7 171.7 95.4 250.7 27.8 ae-201-3601.edge1.washington12.level3.net ae-203-3603.edge1.washington12.level3.net ae-202-3602.edge1.washington12.level3.net 10. 208.51.134.89 0.0% 467 144.0 233.5 124.8 454.1 47.8 11. po1-20g.ar6.dca3.gblx.net 0.0% 467 148.8 226.8 125.8 448.9 37.7 12. uunet-2.ar3.dca3.gblx.net 0.0% 467 146.1 225.9 126.5 319.5 32.9 13. ??? 14. tengige0-6-0-2.gw5.lax7.alter 0.0% 467 203.1 290.3 203.1 370.6 31.4 15. whitehousepost-gw.customer.alter.net 0.0% 467 210.6 289.2 200.1 368.3 30.6
From: Outages [mailto:outages-bounces@outages.org] On Behalf Of alexzarris@yahoo.com Sent: Tuesday, February 25, 2014 11:00 AM To: 'Clayton Zekelman'; 'Michael Morrison'; 'Outages@outages.org' Subject: Re: [outages] Level3 Routing...
We've been having issues in San Jose the past two days on both L3 MPLS and Internet circuits. MPLS was resolved yesterday with these notes:
"Event The IP NOC advised that troubleshooting is ongoing with the equipment vendor to further isolate the point of failure. Additionally, it has been advised that voice services are impacted and calls are failing network wide. Collaborative troubleshooting efforts between the IP and Voice NOC and the equipment vendor will continue in an effort to expedite service restoral."
However, we're still having internet issues.
Alex
From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Clayton Zekelman Sent: Tuesday, February 25, 2014 7:04 AM To: Michael Morrison; Outages@outages.org<mailto:Outages@outages.org> Subject: Re: [outages] Level3 Routing...
That's what we told our customer... by the time they replied, the issue had gone away.
At 09:57 AM 25/02/2014, Michael Morrison wrote: Content-Language: en-US Content-Type: multipart/alternative; boundary="_000_0693cfd728744391a68dbd0e93123a64EX01wcilocal_"
As always, trace routes are helpful. We've been having intermittent issue with Level 3 routes for the past couple weeks, most issue for us seem to be when going from Level 3 to Global Crossing routes.
From: Outages [ mailto:outages-bounces@outages.org] On Behalf Of Clayton Zekelman Sent: Tuesday, February 25, 2014 6:39 AM To: *Network; Luke Rockwell; Petter Bruland Cc: Outages@outages.org<mailto:Outages@outages.org> Subject: Re: [outages] Level3 Routing...
We had a report from a customer earlier this morning, but they advised the problem went away.
We connect to Level3 at Southfield MI.
At 09:35 AM 25/02/2014, *Network wrote: Content-Language: en-US Content-Type: multipart/alternative; boundary="_000_391AAB48E84BB24288BED210F379E1DC4DEA4C02NOCMAILDAG2Inte_" Latency issue with Level 3 here in Detroit area
Anton
From: Outages [ <mailto:outages-bounces@outages.org> mailto:outages-bounces@outages.org] On Behalf Of Luke Rockwell Sent: Tuesday, February 18, 2014 12:24 PM To: Petter Bruland Cc: Outages@outages.org<mailto:Outages@outages.org> Subject: Re: [outages] Level3 Routing...
nothing on the level 3 network events page I have access to Luke
On Tue, Feb 18, 2014 at 9:17 AM, Petter Bruland < Petter.Bruland@allegiantair.com<mailto:Petter.Bruland@allegiantair.com>> wrote: Not seeing any issues with Level3 in Las Vegas.
-Petter
From: Outages [ <mailto:outages-bounces@outages.org> mailto:outages-bounces@outages.org] On Behalf Of Nick Olsen Sent: Tuesday, February 18, 2014 6:04 AM To: Morgan Miskell; Outages@outages.org<mailto:Outages@outages.org> Subject: Re: [outages] Level3 Routing...
Not seeing anything homed off of Level 3 Orlando.
Traceroutes to a few common places look fine.
~474815 Routes from 3356. Nick Olsen Network Operations (855) FLSPEED x106
________________________________ From: "Morgan Miskell" <morgan.miskell@caro.net<mailto:morgan.miskell@caro.net> > Sent: Tuesday, February 18, 2014 9:00 AM To: "Outages@outages.org<mailto:Outages@outages.org>" <outages@outages.org<mailto:outages@outages.org>> Subject: [outages] Level3 Routing... Anyone seeing Level3 routing issues? We just observed a massive drop in routes in the BGP feeds and seeing some strange ICMP issues along their backbone in terms of traceroutes along their backbone. They might be adding some filters based on the recent DDOS attacks, I can ping interfaces but traceroutes go to hell... -- Morgan A. Miskell CaroNet Data Centers 704-643-8330 x206 ---------------------------------------------------------------------------- The information contained in this e-mail is confidential and is intended only for the named recipient(s). If you are not the intended recipient you must not copy, distribute, or take any action or reliance on it. If you have received this e-mail in error, please notify the sender. Any unauthorized disclosure of the information contained in this e-mail is strictly prohibited. ---------------------------------------------------------------------------- _______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages _______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages ---
Clayton Zekelman Managed Network Systems Inc. (MNSi) 3363 Tecumseh Rd. E Windsor, Ontario N8W 1H4
tel. 519-985-8410 fax. 519-985-8409 _______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages
---
Clayton Zekelman Managed Network Systems Inc. (MNSi) 3363 Tecumseh Rd. E Windsor, Ontario N8W 1H4
tel. 519-985-8410 fax. 519-985-8409
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Correct, no packet loss in that mtr but packet loss is occurring from time to time. The main point of that particular trace was to show the latency which is not normal. That same trace now is reporting times around 165ms. The reverse trace which appears to go back along the same path is around 160ms. Again the problem has been off and on for a while now as others here have said. I'm just adding my data point and have a ticket with level3 as well. -----Original Message----- From: Jeremy Chadwick [mailto:jdc@koitsu.org] Sent: Friday, February 28, 2014 5:55 PM To: jay lark Cc: 'Outages@outages.org' Subject: Re: [outages] Level3 Routing... I see no indication of packet loss in this mtr; others have already explained how to properly read output. Loss at hop #3 is not indicative of a universal problem; it would be different if that latency "trickled down" through all subsequent hops, all the way to the destination. I do see indication in this mtr of generally high latency (between London<->Paris), reaching up to 246ms and averaging around 171ms. I do not know if that's normal or not, due to my lack of familiarity with said region/paths. However, as I've stated repeatedly [like a broken record], a traceroute/mtr from point A to B is only half the picture -- you need the same from point B to A, due to asymmetrical routing being commonplace. -- | Jeremy Chadwick jdc@koitsu.org | | UNIX Systems Administrator http://jdc.koitsu.org/ | | Making life hard for others since 1977. PGP 4BD6C0CB | On Fri, Feb 28, 2014 at 10:04:18PM +0000, jay lark wrote:
We've also see increased packet loss and latency over the past week or so from our London Level3 office to our Verizon offices in NY and LA. Routes to our ATT office in Chicago have been okay for the most part.
Here's a current trace from London to LAX. Heavy packet loss in London. It was in Paris before that :)
My traceroute [v0.80] lon-nagios (0.0.0.0) Fri Feb 28 21:52:35 2014 Resolver: Received error response 2. (server failure)er of fields quit Packets Pings Host Loss% Snt Last Avg Best Wrst StDev 1. xxxxxxxxxxxxxxxxxxxxxxxxxxx 0.0% 468 0.3 0.3 0.2 4.0 0.3 2. xxxxxxxxxxxxxxx 0.0% 468 0.9 2.3 0.8 79.1 7.7 3. ae-0-11.edge3.london2.level3.net 61.8% 468 1.1 1.6 1.0 18.0 2.5 4. ae-3-3.ebr1.paris1.level3.net 0.0% 468 109.9 171.8 103.0 246.0 26.9 5. ae-91-91.csw4.paris1.level3.net 0.2% 468 111.7 170.4 99.6 241.8 27.0 6. ae-92-92.ebr2.paris1.level3.net 0.0% 468 121.8 170.1 95.8 236.3 27.0 7. ae-41-41.ebr2.washington1.level3.net 0.0% 467 118.5 171.4 95.7 241.1 27.1 8. ae-45-45.ebr2.washington12.level3.net 0.0% 467 113.4 171.8 94.9 243.5 27.0 9. ae-204-3604.edge1.washington12.level3.net 0.0% 467 107.7 171.7 95.4 250.7 27.8 ae-201-3601.edge1.washington12.level3.net ae-203-3603.edge1.washington12.level3.net ae-202-3602.edge1.washington12.level3.net 10. 208.51.134.89 0.0% 467 144.0 233.5 124.8 454.1 47.8 11. po1-20g.ar6.dca3.gblx.net 0.0% 467 148.8 226.8 125.8 448.9 37.7 12. uunet-2.ar3.dca3.gblx.net 0.0% 467 146.1 225.9 126.5 319.5 32.9 13. ??? 14. tengige0-6-0-2.gw5.lax7.alter 0.0% 467 203.1 290.3 203.1 370.6 31.4 15. whitehousepost-gw.customer.alter.net 0.0% 467 210.6 289.2 200.1 368.3 30.6
From: Outages [mailto:outages-bounces@outages.org] On Behalf Of alexzarris@yahoo.com Sent: Tuesday, February 25, 2014 11:00 AM To: 'Clayton Zekelman'; 'Michael Morrison'; 'Outages@outages.org' Subject: Re: [outages] Level3 Routing...
We've been having issues in San Jose the past two days on both L3 MPLS and Internet circuits. MPLS was resolved yesterday with these notes:
"Event The IP NOC advised that troubleshooting is ongoing with the equipment vendor to further isolate the point of failure. Additionally, it has been advised that voice services are impacted and calls are failing network wide. Collaborative troubleshooting efforts between the IP and Voice NOC and the equipment vendor will continue in an effort to expedite service restoral."
However, we're still having internet issues.
Alex
From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Clayton Zekelman Sent: Tuesday, February 25, 2014 7:04 AM To: Michael Morrison; Outages@outages.org<mailto:Outages@outages.org> Subject: Re: [outages] Level3 Routing...
That's what we told our customer... by the time they replied, the issue had gone away.
At 09:57 AM 25/02/2014, Michael Morrison wrote: Content-Language: en-US Content-Type: multipart/alternative; boundary="_000_0693cfd728744391a68dbd0e93123a64EX01wcilocal_"
As always, trace routes are helpful. We've been having intermittent issue with Level 3 routes for the past couple weeks, most issue for us seem to be when going from Level 3 to Global Crossing routes.
From: Outages [ mailto:outages-bounces@outages.org] On Behalf Of Clayton Zekelman Sent: Tuesday, February 25, 2014 6:39 AM To: *Network; Luke Rockwell; Petter Bruland Cc: Outages@outages.org<mailto:Outages@outages.org> Subject: Re: [outages] Level3 Routing...
We had a report from a customer earlier this morning, but they advised the problem went away.
We connect to Level3 at Southfield MI.
At 09:35 AM 25/02/2014, *Network wrote: Content-Language: en-US Content-Type: multipart/alternative; boundary="_000_391AAB48E84BB24288BED210F379E1DC4DEA4C02NOCMAILDAG2Inte_" Latency issue with Level 3 here in Detroit area
Anton
From: Outages [ <mailto:outages-bounces@outages.org> mailto:outages-bounces@outages.org] On Behalf Of Luke Rockwell Sent: Tuesday, February 18, 2014 12:24 PM To: Petter Bruland Cc: Outages@outages.org<mailto:Outages@outages.org> Subject: Re: [outages] Level3 Routing...
nothing on the level 3 network events page I have access to Luke
On Tue, Feb 18, 2014 at 9:17 AM, Petter Bruland < Petter.Bruland@allegiantair.com<mailto:Petter.Bruland@allegiantair.com>> wrote: Not seeing any issues with Level3 in Las Vegas.
-Petter
From: Outages [ <mailto:outages-bounces@outages.org> mailto:outages-bounces@outages.org] On Behalf Of Nick Olsen Sent: Tuesday, February 18, 2014 6:04 AM To: Morgan Miskell; Outages@outages.org<mailto:Outages@outages.org> Subject: Re: [outages] Level3 Routing...
Not seeing anything homed off of Level 3 Orlando.
Traceroutes to a few common places look fine.
~474815 Routes from 3356. Nick Olsen Network Operations (855) FLSPEED x106
________________________________ From: "Morgan Miskell" <morgan.miskell@caro.net<mailto:morgan.miskell@caro.net> > Sent: Tuesday, February 18, 2014 9:00 AM To: "Outages@outages.org<mailto:Outages@outages.org>" <outages@outages.org<mailto:outages@outages.org>> Subject: [outages] Level3 Routing... Anyone seeing Level3 routing issues? We just observed a massive drop in routes in the BGP feeds and seeing some strange ICMP issues along their backbone in terms of traceroutes along their backbone. They might be adding some filters based on the recent DDOS attacks, I can ping interfaces but traceroutes go to hell... -- Morgan A. Miskell CaroNet Data Centers 704-643-8330 x206 ---------------------------------------------------------------------- ------ The information contained in this e-mail is confidential and is intended only for the named recipient(s). If you are not the intended recipient you must not copy, distribute, or take any action or reliance on it. If you have received this e-mail in error, please notify the sender. Any unauthorized disclosure of the information contained in this e-mail is strictly prohibited. ---------------------------------------------------------------------- ------ _______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages _______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages ---
Clayton Zekelman Managed Network Systems Inc. (MNSi) 3363 Tecumseh Rd. E Windsor, Ontario N8W 1H4
tel. 519-985-8410 fax. 519-985-8409 _______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages
---
Clayton Zekelman Managed Network Systems Inc. (MNSi) 3363 Tecumseh Rd. E Windsor, Ontario N8W 1H4
tel. 519-985-8410 fax. 519-985-8409
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
I'd suggest setting up a cron job that logs the mtr data on two systems (system A, going from A->B, and system B, going from B->A), so that you can review it if/when it happens again. I have a small shell script I wrote, which uses a dorky config file of sorts, that can do this for you if you want me to provide it. I use it on my home FreeBSD system, and my FreeBSD VPS system, checking connectivity between them (and to some other places as well). -- | Jeremy Chadwick jdc@koitsu.org | | UNIX Systems Administrator http://jdc.koitsu.org/ | | Making life hard for others since 1977. PGP 4BD6C0CB | On Sat, Mar 01, 2014 at 12:25:43AM +0000, jay lark wrote:
Correct, no packet loss in that mtr but packet loss is occurring from time to time. The main point of that particular trace was to show the latency which is not normal. That same trace now is reporting times around 165ms. The reverse trace which appears to go back along the same path is around 160ms.
Again the problem has been off and on for a while now as others here have said. I'm just adding my data point and have a ticket with level3 as well.
-----Original Message----- From: Jeremy Chadwick [mailto:jdc@koitsu.org] Sent: Friday, February 28, 2014 5:55 PM To: jay lark Cc: 'Outages@outages.org' Subject: Re: [outages] Level3 Routing...
I see no indication of packet loss in this mtr; others have already explained how to properly read output. Loss at hop #3 is not indicative of a universal problem; it would be different if that latency "trickled down" through all subsequent hops, all the way to the destination.
I do see indication in this mtr of generally high latency (between London<->Paris), reaching up to 246ms and averaging around 171ms. I do not know if that's normal or not, due to my lack of familiarity with said region/paths.
However, as I've stated repeatedly [like a broken record], a traceroute/mtr from point A to B is only half the picture -- you need the same from point B to A, due to asymmetrical routing being commonplace.
-- | Jeremy Chadwick jdc@koitsu.org | | UNIX Systems Administrator http://jdc.koitsu.org/ | | Making life hard for others since 1977. PGP 4BD6C0CB |
On Fri, Feb 28, 2014 at 10:04:18PM +0000, jay lark wrote:
We've also see increased packet loss and latency over the past week or so from our London Level3 office to our Verizon offices in NY and LA. Routes to our ATT office in Chicago have been okay for the most part.
Here's a current trace from London to LAX. Heavy packet loss in London. It was in Paris before that :)
My traceroute [v0.80] lon-nagios (0.0.0.0) Fri Feb 28 21:52:35 2014 Resolver: Received error response 2. (server failure)er of fields quit Packets Pings Host Loss% Snt Last Avg Best Wrst StDev 1. xxxxxxxxxxxxxxxxxxxxxxxxxxx 0.0% 468 0.3 0.3 0.2 4.0 0.3 2. xxxxxxxxxxxxxxx 0.0% 468 0.9 2.3 0.8 79.1 7.7 3. ae-0-11.edge3.london2.level3.net 61.8% 468 1.1 1.6 1.0 18.0 2.5 4. ae-3-3.ebr1.paris1.level3.net 0.0% 468 109.9 171.8 103.0 246.0 26.9 5. ae-91-91.csw4.paris1.level3.net 0.2% 468 111.7 170.4 99.6 241.8 27.0 6. ae-92-92.ebr2.paris1.level3.net 0.0% 468 121.8 170.1 95.8 236.3 27.0 7. ae-41-41.ebr2.washington1.level3.net 0.0% 467 118.5 171.4 95.7 241.1 27.1 8. ae-45-45.ebr2.washington12.level3.net 0.0% 467 113.4 171.8 94.9 243.5 27.0 9. ae-204-3604.edge1.washington12.level3.net 0.0% 467 107.7 171.7 95.4 250.7 27.8 ae-201-3601.edge1.washington12.level3.net ae-203-3603.edge1.washington12.level3.net ae-202-3602.edge1.washington12.level3.net 10. 208.51.134.89 0.0% 467 144.0 233.5 124.8 454.1 47.8 11. po1-20g.ar6.dca3.gblx.net 0.0% 467 148.8 226.8 125.8 448.9 37.7 12. uunet-2.ar3.dca3.gblx.net 0.0% 467 146.1 225.9 126.5 319.5 32.9 13. ??? 14. tengige0-6-0-2.gw5.lax7.alter 0.0% 467 203.1 290.3 203.1 370.6 31.4 15. whitehousepost-gw.customer.alter.net 0.0% 467 210.6 289.2 200.1 368.3 30.6
From: Outages [mailto:outages-bounces@outages.org] On Behalf Of alexzarris@yahoo.com Sent: Tuesday, February 25, 2014 11:00 AM To: 'Clayton Zekelman'; 'Michael Morrison'; 'Outages@outages.org' Subject: Re: [outages] Level3 Routing...
We've been having issues in San Jose the past two days on both L3 MPLS and Internet circuits. MPLS was resolved yesterday with these notes:
"Event The IP NOC advised that troubleshooting is ongoing with the equipment vendor to further isolate the point of failure. Additionally, it has been advised that voice services are impacted and calls are failing network wide. Collaborative troubleshooting efforts between the IP and Voice NOC and the equipment vendor will continue in an effort to expedite service restoral."
However, we're still having internet issues.
Alex
From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Clayton Zekelman Sent: Tuesday, February 25, 2014 7:04 AM To: Michael Morrison; Outages@outages.org<mailto:Outages@outages.org> Subject: Re: [outages] Level3 Routing...
That's what we told our customer... by the time they replied, the issue had gone away.
At 09:57 AM 25/02/2014, Michael Morrison wrote: Content-Language: en-US Content-Type: multipart/alternative; boundary="_000_0693cfd728744391a68dbd0e93123a64EX01wcilocal_"
As always, trace routes are helpful. We've been having intermittent issue with Level 3 routes for the past couple weeks, most issue for us seem to be when going from Level 3 to Global Crossing routes.
From: Outages [ mailto:outages-bounces@outages.org] On Behalf Of Clayton Zekelman Sent: Tuesday, February 25, 2014 6:39 AM To: *Network; Luke Rockwell; Petter Bruland Cc: Outages@outages.org<mailto:Outages@outages.org> Subject: Re: [outages] Level3 Routing...
We had a report from a customer earlier this morning, but they advised the problem went away.
We connect to Level3 at Southfield MI.
At 09:35 AM 25/02/2014, *Network wrote: Content-Language: en-US Content-Type: multipart/alternative; boundary="_000_391AAB48E84BB24288BED210F379E1DC4DEA4C02NOCMAILDAG2Inte_" Latency issue with Level 3 here in Detroit area
Anton
From: Outages [ <mailto:outages-bounces@outages.org> mailto:outages-bounces@outages.org] On Behalf Of Luke Rockwell Sent: Tuesday, February 18, 2014 12:24 PM To: Petter Bruland Cc: Outages@outages.org<mailto:Outages@outages.org> Subject: Re: [outages] Level3 Routing...
nothing on the level 3 network events page I have access to Luke
On Tue, Feb 18, 2014 at 9:17 AM, Petter Bruland < Petter.Bruland@allegiantair.com<mailto:Petter.Bruland@allegiantair.com>> wrote: Not seeing any issues with Level3 in Las Vegas.
-Petter
From: Outages [ <mailto:outages-bounces@outages.org> mailto:outages-bounces@outages.org] On Behalf Of Nick Olsen Sent: Tuesday, February 18, 2014 6:04 AM To: Morgan Miskell; Outages@outages.org<mailto:Outages@outages.org> Subject: Re: [outages] Level3 Routing...
Not seeing anything homed off of Level 3 Orlando.
Traceroutes to a few common places look fine.
~474815 Routes from 3356. Nick Olsen Network Operations (855) FLSPEED x106
________________________________ From: "Morgan Miskell" <morgan.miskell@caro.net<mailto:morgan.miskell@caro.net> > Sent: Tuesday, February 18, 2014 9:00 AM To: "Outages@outages.org<mailto:Outages@outages.org>" <outages@outages.org<mailto:outages@outages.org>> Subject: [outages] Level3 Routing... Anyone seeing Level3 routing issues? We just observed a massive drop in routes in the BGP feeds and seeing some strange ICMP issues along their backbone in terms of traceroutes along their backbone. They might be adding some filters based on the recent DDOS attacks, I can ping interfaces but traceroutes go to hell... -- Morgan A. Miskell CaroNet Data Centers 704-643-8330 x206 ---------------------------------------------------------------------- ------ The information contained in this e-mail is confidential and is intended only for the named recipient(s). If you are not the intended recipient you must not copy, distribute, or take any action or reliance on it. If you have received this e-mail in error, please notify the sender. Any unauthorized disclosure of the information contained in this e-mail is strictly prohibited. ---------------------------------------------------------------------- ------ _______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages _______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages ---
Clayton Zekelman Managed Network Systems Inc. (MNSi) 3363 Tecumseh Rd. E Windsor, Ontario N8W 1H4
tel. 519-985-8410 fax. 519-985-8409 _______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages
---
Clayton Zekelman Managed Network Systems Inc. (MNSi) 3363 Tecumseh Rd. E Windsor, Ontario N8W 1H4
tel. 519-985-8410 fax. 519-985-8409
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
On 2/28/2014 20:49, Jeremy Chadwick wrote:
I'd suggest setting up a cron job that logs the mtr data on two systems (system A, going from A->B, and system B, going from B->A), so that you can review it if/when it happens again.
I have a small shell script I wrote, which uses a dorky config file of sorts, that can do this for you if you want me to provide it. I use it on my home FreeBSD system, and my FreeBSD VPS system, checking connectivity between them (and to some other places as well).
Something like this would be much appreciated. I was thinking of writing a similar script to mail mtr output on an hourly basis. Thanks, -- staticsafe
On Fri, Feb 28, 2014 at 09:23:06PM -0500, staticsafe wrote:
On 2/28/2014 20:49, Jeremy Chadwick wrote:
I'd suggest setting up a cron job that logs the mtr data on two systems (system A, going from A->B, and system B, going from B->A), so that you can review it if/when it happens again.
I have a small shell script I wrote, which uses a dorky config file of sorts, that can do this for you if you want me to provide it. I use it on my home FreeBSD system, and my FreeBSD VPS system, checking connectivity between them (and to some other places as well).
Something like this would be much appreciated. I was thinking of writing a similar script to mail mtr output on an hourly basis.
I take no responsibility for support or misuse or misunderstanding of how this works. Normally anything I release publicly I tend to provide some degree of support for; this is an exception, i.e. "this works for me" and was thrown together one day without the usual amount of care or caution I normally apply when writing code. I expect people using this to take the time to figure out how it works (it's not hard); paths/etc. should become fairly obvious once said examination is done. People may also need to update to mtr 0.85 to get all the benefits. Attachments: mtr.sh = shell script that calls mtr with specific arguments, and reads a pseudo configuration file from stdin config = configuration file cronjob itself: ## # Periodic mtrs to numerous locations. # # EXECUTION: Every 1 minute # */1 * * * * cd /var/log/captures && cat config | ./mtr.sh How relevant log files are rotated, how many are kept, etc. -- done quite simply through FreeBSD newsyslog(8). That entry: /var/log/captures/*.log root:staff 664 40 * @T00 GB -- | Jeremy Chadwick jdc@koitsu.org | | UNIX Systems Administrator http://jdc.koitsu.org/ | | Making life hard for others since 1977. PGP 4BD6C0CB |
On Mar 1, 2014, at 2:23 AM, staticsafe wrote:
On 2/28/2014 20:49, Jeremy Chadwick wrote:
I'd suggest setting up a cron job that logs the mtr data on two systems (system A, going from A->B, and system B, going from B->A), so that you can review it if/when it happens again.
I have a small shell script I wrote, which uses a dorky config file of sorts, that can do this for you if you want me to provide it. I use it on my home FreeBSD system, and my FreeBSD VPS system, checking connectivity between them (and to some other places as well).
Something like this would be much appreciated. I was thinking of writing a similar script to mail mtr output on an hourly basis.
Slightly OT, but a nice tool to have running that provides a history of packet loss to various destinations is smokeping: http://oss.oetiker.ch/smokeping/ It won't show you the path, but the historic data is useful. Charles
Thanks, -- staticsafe _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
----- Original Message -----
From: "Jeremy Chadwick" <jdc@koitsu.org>
However, as I've stated repeatedly [like a broken record], a traceroute/mtr from point A to B is only half the picture -- you need the same from point B to A, due to asymmetrical routing being commonplace.
We say this a lot, but no one's ever explained to me how you're supposed to break into the apache server at the other end to run the reverse trace. 1/2 :-) Cheers, -- jra -- Jay R. Ashworth Baylink jra@baylink.com Designer The Things I Think RFC 2100 Ashworth & Associates http://www.bcp38.info 2000 Land Rover DII St Petersburg FL USA BCP38: Ask For It By Name! +1 727 647 1274
On Fri, Feb 28, 2014 at 11:51:41PM -0500, Jay Ashworth wrote:
----- Original Message -----
From: "Jeremy Chadwick" <jdc@koitsu.org>
However, as I've stated repeatedly [like a broken record], a traceroute/mtr from point A to B is only half the picture -- you need the same from point B to A, due to asymmetrical routing being commonplace.
We say this a lot, but no one's ever explained to me how you're supposed to break into the apache server at the other end to run the reverse trace.
1/2 :-)
You must have access to both ends (A and B). If you don't, correct, all you can provide is half the picture. Transport/peering providers will often stall you (or flat out reject you) if you've only half. It's just another excuse in the long list of things network providers can use to avoid investigating a problem. -- | Jeremy Chadwick jdc@koitsu.org | | UNIX Systems Administrator http://jdc.koitsu.org/ | | Making life hard for others since 1977. PGP 4BD6C0CB |
participants (10)
-
alexzarris＠yahoo.com -
Charles Sprickman -
Clayton Zekelman -
Jay Ashworth -
jay lark -
Jeremy Chadwick -
Luke Rockwell -
Michael Morrison -
Mitch Patterson -
staticsafe