widespread level3 issue amsterdam/western europe?
is anyone else seeing routing implosion on l3 or first symtoms were in western europe. I had packets bound for l3 ips in ams making lazy circles. started circa 0430 utc joel
Yes. Major impact in the US. Still trying to figure out how widespread. Bob Hancock Sr Network Engineer Momentum | www.gomomentum.com O: 205.978.6027 On 09/09/2014 12:31 AM, joel jaeggli via Outages wrote:
is anyone else seeing routing implosion on l3 or first symtoms were in western europe.
I had packets bound for l3 ips in ams making lazy circles. started circa 0430 utc
joel
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
I'm having trouble completing traceroutes, but the packets seem to be getting to their final destinations between San Jose and Frankfurt across Level(3)'s network. --Bill
Similar results here from Las Vegas to Atlanta. Bob Hancock Sr Network Engineer Momentum | www.gomomentum.com O: 205.978.6027 On 09/09/2014 01:07 AM, Bill Fehring wrote:
I'm having trouble completing traceroutes, but the packets seem to be getting to their final destinations between San Jose and Frankfurt across Level(3)'s network.
--Bill
Similar issues outbound from Seattle...issue started as serious inbound (mostly EU) issues...inbound seemed OK for a bit but still having outbound issues it looks like so I've removed them from my mix for now. On Mon, Sep 8, 2014 at 11:18 PM, Bob Hancock via Outages <outages@outages.org> wrote:
Similar results here from Las Vegas to Atlanta.
Bob Hancock Sr Network Engineer Momentum | www.gomomentum.com O: 205.978.6027
On 09/09/2014 01:07 AM, Bill Fehring wrote:
I'm having trouble completing traceroutes, but the packets seem to be getting to their final destinations between San Jose and Frankfurt across Level(3)'s network.
--Bill
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
-- "Genius might be described as a supreme capacity for getting its possessors into trouble of all kinds." -- Samuel Butler
Has anyone received an RFO for this incident they are willing to share? Thanks, Rick On Mon, Sep 8, 2014 at 11:30 PM, Michael Loftis via Outages < outages@outages.org> wrote:
Similar issues outbound from Seattle...issue started as serious inbound (mostly EU) issues...inbound seemed OK for a bit but still having outbound issues it looks like so I've removed them from my mix for now.
On Mon, Sep 8, 2014 at 11:18 PM, Bob Hancock via Outages <outages@outages.org> wrote:
Similar results here from Las Vegas to Atlanta.
Bob Hancock Sr Network Engineer Momentum | www.gomomentum.com O: 205.978.6027
On 09/09/2014 01:07 AM, Bill Fehring wrote:
I'm having trouble completing traceroutes, but the packets seem to be getting to their final destinations between San Jose and Frankfurt across Level(3)'s network.
--Bill
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
--
"Genius might be described as a supreme capacity for getting its possessors into trouble of all kinds." -- Samuel Butler _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
This is what I received from Level3 this morning. 09/09/2014 10:30:16 GMT - *** CASCADED EXTERNAL NOTES 09-Sep-2014 10:29:58 GMT From CASE: 8335069 - Event The planned maintenance activity has been completed for this evening and traffic will now be moved back to its original path. The planned maintenance activity is scheduled to continue on September 10, 2014 from 04:00 GMT to 10:00 GMT. The IP NOC will reroute services during the maintenance window to ensure service stability and no further issues impacts are expected to occur. If additional issues are experienced, please contact the Level 3 Technical Service Center at 1-877-4LEVEL3. 09/09/2014 08:02:05 GMT - *** CASCADED EXTERNAL NOTES 09-Sep-2014 08:02:04 GMT From CASE: 8335069 - Event The IP NOC has further explained that the ongoing planned maintenance activity caused traffic for the Direct Internet Access customers in the United States to black hole. The IP NOC is now rerouting traffic as a temporary resolution to improve service stability until the planned maintenance activity is completed. On Sep 9, 2014, at 3:46 AM, Richard Hicks via Outages <outages@outages.org> wrote:
Has anyone received an RFO for this incident they are willing to share?
Thanks, Rick
On Mon, Sep 8, 2014 at 11:30 PM, Michael Loftis via Outages <outages@outages.org> wrote: Similar issues outbound from Seattle...issue started as serious inbound (mostly EU) issues...inbound seemed OK for a bit but still having outbound issues it looks like so I've removed them from my mix for now.
On Mon, Sep 8, 2014 at 11:18 PM, Bob Hancock via Outages <outages@outages.org> wrote:
Similar results here from Las Vegas to Atlanta.
Bob Hancock Sr Network Engineer Momentum | www.gomomentum.com O: 205.978.6027
On 09/09/2014 01:07 AM, Bill Fehring wrote:
I'm having trouble completing traceroutes, but the packets seem to be getting to their final destinations between San Jose and Frankfurt across Level(3)'s network.
--Bill
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
--
"Genius might be described as a supreme capacity for getting its possessors into trouble of all kinds." -- Samuel Butler _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
On Sep 9, 2014, at 7:24 AM, Joshua McDonald via Outages <outages@outages.org> wrote:
This is what I received from Level3 this morning.
09/09/2014 10:30:16 GMT - *** CASCADED EXTERNAL NOTES 09-Sep-2014 10:29:58 GMT From CASE: 8335069 - Event The planned maintenance activity has been completed for this evening and traffic will now be moved back to its original path. The planned maintenance activity is scheduled to continue on September 10, 2014 from 04:00 GMT to 10:00 GMT. The IP NOC will reroute services during the maintenance window to ensure service stability and no further issues impacts are expected to occur. If additional issues are experienced, please contact the Level 3 Technical Service Center at 1-877-4LEVEL3.
09/09/2014 08:02:05 GMT - *** CASCADED EXTERNAL NOTES 09-Sep-2014 08:02:04 GMT From CASE: 8335069 - Event The IP NOC has further explained that the ongoing planned maintenance activity caused traffic for the Direct Internet Access customers in the United States to black hole. The IP NOC is now rerouting traffic as a temporary resolution to improve service stability until the planned maintenance activity is completed.
Well, that was brilliant, wasn't it? Schedule a big planned maintenance that ends up black holing traffic right before a huge Apple keynote... -Andy
On 09/09/2014 07:52 AM, Andy Ringsmuth via Outages wrote:
On Sep 9, 2014, at 7:24 AM, Joshua McDonald via Outages <outages@outages.org> wrote:
This is what I received from Level3 this morning.
09/09/2014 10:30:16 GMT - *** CASCADED EXTERNAL NOTES 09-Sep-2014 10:29:58 GMT From CASE: 8335069 - Event The planned maintenance activity has been completed for this evening and traffic will now be moved back to its original path. The planned maintenance activity is scheduled to continue on September 10, 2014 from 04:00 GMT to 10:00 GMT. The IP NOC will reroute services during the maintenance window to ensure service stability and no further issues impacts are expected to occur. If additional issues are experienced, please contact the Level 3 Technical Service Center at 1-877-4LEVEL3.
09/09/2014 08:02:05 GMT - *** CASCADED EXTERNAL NOTES 09-Sep-2014 08:02:04 GMT From CASE: 8335069 - Event The IP NOC has further explained that the ongoing planned maintenance activity caused traffic for the Direct Internet Access customers in the United States to black hole. The IP NOC is now rerouting traffic as a temporary resolution to improve service stability until the planned maintenance activity is completed. Well, that was brilliant, wasn't it? Schedule a big planned maintenance that ends up black holing traffic right before a huge Apple keynote...
If an ISP puts off maintenance for every big announcement some other company makes, the number of maintenance windows will be reduced more than what they are. Looks to me like there was a backout or BC plan for this maintenance so this really is a 'nothing to see here, move along' event. --John
-Andy _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Not sure if it's related, but at 10:22am EDT we lost our Level3 BGP session here in White Plains, NY. -----Original Message----- From: Outages [mailto:outages-bounces@outages.org] On Behalf Of me via Outages Sent: Tuesday, September 09, 2014 10:11 AM To: outages@outages.org Subject: Re: [outages] widespread level3 issue amsterdam/western europe? On 09/09/2014 07:52 AM, Andy Ringsmuth via Outages wrote:
On Sep 9, 2014, at 7:24 AM, Joshua McDonald via Outages <outages@outages.org> wrote:
This is what I received from Level3 this morning.
09/09/2014 10:30:16 GMT - *** CASCADED EXTERNAL NOTES 09-Sep-2014 10:29:58 GMT From CASE: 8335069 - Event The planned maintenance activity has been completed for this evening and traffic will now be moved back to its original path. The planned maintenance activity is scheduled to continue on September 10, 2014 from 04:00 GMT to 10:00 GMT. The IP NOC will reroute services during the maintenance window to ensure service stability and no further issues impacts are expected to occur. If additional issues are experienced, please contact the Level 3 Technical Service Center at 1-877-4LEVEL3.
09/09/2014 08:02:05 GMT - *** CASCADED EXTERNAL NOTES 09-Sep-2014 08:02:04 GMT From CASE: 8335069 - Event The IP NOC has further explained that the ongoing planned maintenance activity caused traffic for the Direct Internet Access customers in the United States to black hole. The IP NOC is now rerouting traffic as a temporary resolution to improve service stability until the planned maintenance activity is completed. Well, that was brilliant, wasn't it? Schedule a big planned maintenance that ends up black holing traffic right before a huge Apple keynote...
If an ISP puts off maintenance for every big announcement some other company makes, the number of maintenance windows will be reduced more than what they are. Looks to me like there was a backout or BC plan for this maintenance so this really is a 'nothing to see here, move along' event. --John
-Andy _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
.-- My secret spy satellite informs me that at 2014-09-08 10:31 PM joel jaeggli via Outages wrote:
is anyone else seeing routing implosion on l3 or first symtoms were in western europe.
I had packets bound for l3 ips in ams making lazy circles. started circa 0430 utc
Same here, 100% loss over Level3 between San Jose and LA as of 4:50 UTC. Also seeing issues transatlantic over level3, Amsterdam being one of them. Andree
Similar issues observed in Charlotte NC, had to shutdown that peer for over an hour. Still no response on my initial ticket, which is very uncommon for them. On Mon, 2014-09-08 at 22:31 -0700, joel jaeggli via Outages wrote:
is anyone else seeing routing implosion on l3 or first symtoms were in western europe.
I had packets bound for l3 ips in ams making lazy circles. started circa 0430 utc
joel
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
participants (11)
-
Adam Greene -
Andree Toonk -
Andy Ringsmuth -
Bill Fehring -
Bob Hancock -
joel jaeggli -
Joshua McDonald -
me -
Michael Loftis -
Morgan Miskell -
Richard Hicks