Denver Level3 problems?
Did anybody else see Denver Level3 service take a hit this morning? Our logs indicated nothing, but we saw various blackholing all over their network for ~20 minutes, starting around 9:15am MST. Everybody have a great day - Randal
Yes, we saw a temporary blip on our Level3 at that same time. Our BGP session to them did not drop, but various sites were blackholed while things were reconverging. Ken Matlock Network Analyst Exempla Healthcare (303) 467-4671 matlockk@exempla.org ________________________________ From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On Behalf Of randal k Sent: Tuesday, May 18, 2010 9:52 AM To: outages@outages.org Subject: [outages] Denver Level3 problems? Did anybody else see Denver Level3 service take a hit this morning? Our logs indicated nothing, but we saw various blackholing all over their network for ~20 minutes, starting around 9:15am MST. Everybody have a great day - Randal
We've been having issues staying connected to Google Chat for the last hour or so. And it just so happens we're connected through you. :-) Best Wishes, Jayson A. Baker Peak Internet On Tue, May 18, 2010 at 9:51 AM, randal k <rkohutek@gmail.com> wrote:
Did anybody else see Denver Level3 service take a hit this morning? Our logs indicated nothing, but we saw various blackholing all over their network for ~20 minutes, starting around 9:15am MST.
Everybody have a great day - Randal
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
We had many customer complaints and all our IP SLA tests spiked between 1110 and 1135 est for Level 3 in our Denver market. But we also saw the problems in DAL on our level 3 peer (SLA test as well). Some event happened in the Denver area on level 3, we have opened a ticket with them and they reported to us they have had a lot of customer call in for the same thing. They have not given us an RFO yet... From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On Behalf Of Jayson Baker Sent: Tuesday, May 18, 2010 12:31 To: randal k Cc: outages@outages.org Subject: Re: [outages] Denver Level3 problems? We've been having issues staying connected to Google Chat for the last hour or so. And it just so happens we're connected through you. :-) Best Wishes, Jayson A. Baker Peak Internet On Tue, May 18, 2010 at 9:51 AM, randal k <rkohutek@gmail.com<mailto:rkohutek@gmail.com>> wrote: Did anybody else see Denver Level3 service take a hit this morning? Our logs indicated nothing, but we saw various blackholing all over their network for ~20 minutes, starting around 9:15am MST. Everybody have a great day - Randal _______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages
We also saw issues with customer end points that took level 3 around the same time but in NYC - LA and DFW POPS. Joseph From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On Behalf Of Sean Clark Sent: Tuesday, May 18, 2010 11:53 AM To: Jayson Baker; randal k Cc: outages@outages.org Subject: Re: [outages] Denver Level3 problems? We had many customer complaints and all our IP SLA tests spiked between 1110 and 1135 est for Level 3 in our Denver market. But we also saw the problems in DAL on our level 3 peer (SLA test as well). Some event happened in the Denver area on level 3, we have opened a ticket with them and they reported to us they have had a lot of customer call in for the same thing. They have not given us an RFO yet... From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On Behalf Of Jayson Baker Sent: Tuesday, May 18, 2010 12:31 To: randal k Cc: outages@outages.org Subject: Re: [outages] Denver Level3 problems? We've been having issues staying connected to Google Chat for the last hour or so. And it just so happens we're connected through you. :-) Best Wishes, Jayson A. Baker Peak Internet On Tue, May 18, 2010 at 9:51 AM, randal k <rkohutek@gmail.com<mailto:rkohutek@gmail.com>> wrote: Did anybody else see Denver Level3 service take a hit this morning? Our logs indicated nothing, but we saw various blackholing all over their network for ~20 minutes, starting around 9:15am MST. Everybody have a great day - Randal _______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages
An internal source has informed me there was an issue with a Data Center in the Denver area in which something ³blew up² and this caused a chain effect with anything going through Denver to any other pop in the USA. The problem since has been resolved and was for a very short period. Brad On 5/18/10 2:32 PM, "Joseph Jackson" <jjackson@aninetworks.net> wrote:
We also saw issues with customer end points that took level 3 around the same time but in NYC LA and DFW POPS.
Joseph
From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On Behalf Of Sean Clark Sent: Tuesday, May 18, 2010 11:53 AM To: Jayson Baker; randal k Cc: outages@outages.org Subject: Re: [outages] Denver Level3 problems?
We had many customer complaints and all our IP SLA tests spiked between 1110 and 1135 est for Level 3 in our Denver market. But we also saw the problems in DAL on our level 3 peer (SLA test as well). Some event happened in the Denver area on level 3, we have opened a ticket with them and they reported to us they have had a lot of customer call in for the same thing. They have not given us an RFO yet
From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On Behalf Of Jayson Baker Sent: Tuesday, May 18, 2010 12:31 To: randal k Cc: outages@outages.org Subject: Re: [outages] Denver Level3 problems?
We've been having issues staying connected to Google Chat for the last hour or so.
And it just so happens we're connected through you. :-)
Best Wishes, Jayson A. Baker Peak Internet
On Tue, May 18, 2010 at 9:51 AM, randal k <rkohutek@gmail.com> wrote: Did anybody else see Denver Level3 service take a hit this morning? Our logs indicated nothing, but we saw various blackholing all over their network for ~20 minutes, starting around 9:15am MST.
Everybody have a great day - Randal
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Bradley D. Bopp Director Of Engineering http://www.nationalnet.com http://as22384.peeringdb.net NationalNet is committed to the highest level of Customer Service available in the Web Hosting business. If you have any questions, comments or concerns feel free to contact us at 678-247-7000 ext 1 (or toll-free, 888-4-NATNET).
They have told us that were trying to isolate some dirty circuits from an etherchannel bundle and that went way wrong... In the Denver area... data center/POP .. whatever.. Level 3 rocks... From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On Behalf Of Bradley Bopp Sent: Tuesday, May 18, 2010 15:04 To: Joseph Jackson Cc: outages@outages.org Subject: Re: [outages] Denver Level3 problems? An internal source has informed me there was an issue with a Data Center in the Denver area in which something "blew up" and this caused a chain effect with anything going through Denver to any other pop in the USA. The problem since has been resolved and was for a very short period. Brad On 5/18/10 2:32 PM, "Joseph Jackson" <jjackson@aninetworks.net> wrote: We also saw issues with customer end points that took level 3 around the same time but in NYC - LA and DFW POPS. Joseph From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On Behalf Of Sean Clark Sent: Tuesday, May 18, 2010 11:53 AM To: Jayson Baker; randal k Cc: outages@outages.org Subject: Re: [outages] Denver Level3 problems? We had many customer complaints and all our IP SLA tests spiked between 1110 and 1135 est for Level 3 in our Denver market. But we also saw the problems in DAL on our level 3 peer (SLA test as well). Some event happened in the Denver area on level 3, we have opened a ticket with them and they reported to us they have had a lot of customer call in for the same thing. They have not given us an RFO yet... From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On Behalf Of Jayson Baker Sent: Tuesday, May 18, 2010 12:31 To: randal k Cc: outages@outages.org Subject: Re: [outages] Denver Level3 problems? We've been having issues staying connected to Google Chat for the last hour or so. And it just so happens we're connected through you. :-) Best Wishes, Jayson A. Baker Peak Internet On Tue, May 18, 2010 at 9:51 AM, randal k <rkohutek@gmail.com> wrote: Did anybody else see Denver Level3 service take a hit this morning? Our logs indicated nothing, but we saw various blackholing all over their network for ~20 minutes, starting around 9:15am MST. Everybody have a great day - Randal _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages ________________________________ _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages Bradley D. Bopp Director Of Engineering http://www.nationalnet.com http://as22384.peeringdb.net NationalNet is committed to the highest level of Customer Service available in the Web Hosting business. If you have any questions, comments or concerns feel free to contact us at 678-247-7000 ext 1 (or toll-free, 888-4-NATNET).
L3 NOC called us back with reason for outage yesterday: PEBKAC. - K Bradley Bopp wrote:
An internal source has informed me there was an issue with a Data Center in the Denver area in which something ³blew up² and this caused a chain effect with anything going through Denver to any other pop in the USA.
The problem since has been resolved and was for a very short period.
Brad
On 5/18/10 2:32 PM, "Joseph Jackson" <jjackson@aninetworks.net> wrote:
We also saw issues with customer end points that took level 3 around the same time but in NYC LA and DFW POPS.
Joseph
From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On Behalf Of Sean Clark Sent: Tuesday, May 18, 2010 11:53 AM To: Jayson Baker; randal k Cc: outages@outages.org Subject: Re: [outages] Denver Level3 problems?
We had many customer complaints and all our IP SLA tests spiked between 1110 and 1135 est for Level 3 in our Denver market. But we also saw the problems in DAL on our level 3 peer (SLA test as well). Some event happened in the Denver area on level 3, we have opened a ticket with them and they reported to us they have had a lot of customer call in for the same thing. They have not given us an RFO yetŠ
From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On Behalf Of Jayson Baker Sent: Tuesday, May 18, 2010 12:31 To: randal k Cc: outages@outages.org Subject: Re: [outages] Denver Level3 problems?
We've been having issues staying connected to Google Chat for the last hour or so.
And it just so happens we're connected through you. :-)
Best Wishes, Jayson A. Baker Peak Internet
On Tue, May 18, 2010 at 9:51 AM, randal k <rkohutek@gmail.com> wrote: Did anybody else see Denver Level3 service take a hit this morning? Our logs indicated nothing, but we saw various blackholing all over their network for ~20 minutes, starting around 9:15am MST.
Everybody have a great day - Randal
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Bradley D. Bopp Director Of Engineering http://www.nationalnet.com http://as22384.peeringdb.net
NationalNet is committed to the highest level of Customer Service available in the Web Hosting business. If you have any questions, comments or concerns feel free to contact us at 678-247-7000 ext 1 (or toll-free, 888-4-NATNET).
------------------------------------------------------------------------
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Sorry to dredge up an old issue, but did anyone get anything better than "PEBKAC". We were having some issues with voice that was going to Level3 last Tuesday and trying to figure out if what you all saw was related or not (I use Level3 for transit and voice termination), but since I didn't open a ticket at the time (we thought it was an internal issue), Level3 says they can't research the issue. Any help? -Sean -----Original Message----- From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On Behalf Of Kevin Broderick Sent: Wednesday, May 19, 2010 10:24 AM To: outages@outages.org Subject: Re: [outages] Denver Level3 problems? L3 NOC called us back with reason for outage yesterday: PEBKAC. - K Bradley Bopp wrote:
An internal source has informed me there was an issue with a Data Center in the Denver area in which something ³blew up² and this caused a chain effect with anything going through Denver to any other pop in the USA.
The problem since has been resolved and was for a very short period.
Brad
On 5/18/10 2:32 PM, "Joseph Jackson" <jjackson@aninetworks.net> wrote:
We also saw issues with customer end points that took level 3 around the same time but in NYC LA and DFW POPS.
Joseph
From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On Behalf Of Sean Clark Sent: Tuesday, May 18, 2010 11:53 AM To: Jayson Baker; randal k Cc: outages@outages.org Subject: Re: [outages] Denver Level3 problems?
We had many customer complaints and all our IP SLA tests spiked between 1110 and 1135 est for Level 3 in our Denver market. But we also saw the problems in DAL on our level 3 peer (SLA test as well). Some event happened in the Denver area on level 3, we have opened a ticket with them and they reported to us they have had a lot of customer call in for the same thing. They have not given us an RFO yetŠ
From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On Behalf Of Jayson Baker Sent: Tuesday, May 18, 2010 12:31 To: randal k Cc: outages@outages.org Subject: Re: [outages] Denver Level3 problems?
We've been having issues staying connected to Google Chat for the last hour or so.
And it just so happens we're connected through you. :-)
Best Wishes, Jayson A. Baker Peak Internet
On Tue, May 18, 2010 at 9:51 AM, randal k <rkohutek@gmail.com> wrote: Did anybody else see Denver Level3 service take a hit this morning? Our logs indicated nothing, but we saw various blackholing all over their network for ~20 minutes, starting around 9:15am MST.
Everybody have a great day - Randal
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Bradley D. Bopp Director Of Engineering http://www.nationalnet.com http://as22384.peeringdb.net
NationalNet is committed to the highest level of Customer Service available in the Web Hosting business. If you have any questions, comments or concerns feel free to contact us at 678-247-7000 ext 1 (or toll-free, 888-4-NATNET).
------------------------------------------------------------------------
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
We have a ticket open. MTR between our network (level 3 customer) and another network (level 3 customer) was showing that ae-1-100.ebr2.Denver1.Level3.net was running 50%-90% packet loss for the past 6+ hours. Well, that doesn't really tell the whole story since routers routinely drop ICMP as a matter of due course during periods of high load. But, point to point pings to far end destinations that would normally be 40-60ms were varying regularly (every 60 seconds) to 240-260ms. And, of course the voice calls on that path sounded like crap. What it looks like is a router that is experiencing consistently high CPU load that is chewing on some process every 60 seconds. That explains the MTR packet loss numbers that are probably innocuous. But, while the processor is chewing on that process, the throughput latency goes through the roof. This is a very common thing that we have seen before . . . just not in the middle of the Level3 network. The Level3 tech said that since he could not replicate the problem, he would leave the ticket open until tomorrow. Bryan Sean Crandall wrote:
Sorry to dredge up an old issue, but did anyone get anything better than "PEBKAC".
We were having some issues with voice that was going to Level3 last Tuesday and trying to figure out if what you all saw was related or not (I use Level3 for transit and voice termination), but since I didn't open a ticket at the time (we thought it was an internal issue), Level3 says they can't research the issue.
Any help?
-Sean
-----Original Message----- From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On Behalf Of Kevin Broderick Sent: Wednesday, May 19, 2010 10:24 AM To: outages@outages.org Subject: Re: [outages] Denver Level3 problems?
L3 NOC called us back with reason for outage yesterday: PEBKAC.
- K
Bradley Bopp wrote:
An internal source has informed me there was an issue with a Data Center in the Denver area in which something ³blew up² and this caused a chain effect with anything going through Denver to any other pop in the USA.
The problem since has been resolved and was for a very short period.
Brad
On 5/18/10 2:32 PM, "Joseph Jackson" <jjackson@aninetworks.net> wrote:
We also saw issues with customer end points that took level 3 around the same time but in NYC LA and DFW POPS.
Joseph
From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On Behalf Of Sean Clark Sent: Tuesday, May 18, 2010 11:53 AM To: Jayson Baker; randal k Cc: outages@outages.org Subject: Re: [outages] Denver Level3 problems?
We had many customer complaints and all our IP SLA tests spiked between 1110 and 1135 est for Level 3 in our Denver market. But we also saw the problems in DAL on our level 3 peer (SLA test as well). Some event happened in the Denver area on level 3, we have opened a ticket with them and they reported to us they have had a lot of customer call in for the same thing. They have not given us an RFO yetŠ
From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On Behalf Of Jayson Baker Sent: Tuesday, May 18, 2010 12:31 To: randal k Cc: outages@outages.org Subject: Re: [outages] Denver Level3 problems?
We've been having issues staying connected to Google Chat for the last hour or so.
And it just so happens we're connected through you. :-)
Best Wishes, Jayson A. Baker Peak Internet
On Tue, May 18, 2010 at 9:51 AM, randal k <rkohutek@gmail.com> wrote: Did anybody else see Denver Level3 service take a hit this morning? Our logs indicated nothing, but we saw various blackholing all over their network for ~20 minutes, starting around 9:15am MST.
Everybody have a great day - Randal
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Bradley D. Bopp Director Of Engineering http://www.nationalnet.com http://as22384.peeringdb.net
NationalNet is committed to the highest level of Customer Service available in the Web Hosting business. If you have any questions, comments or concerns feel free to contact us at 678-247-7000 ext 1 (or toll-free, 888-4-NATNET).
------------------------------------------------------------------------
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
We saw a huge hit earlier in Denver (around 9:20am) through Level3, but only partially dropped our traffic. (about 1/3 was still available) Shawn From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On Behalf Of randal k Sent: Tuesday, May 18, 2010 9:52 AM To: outages@outages.org Subject: [outages] Denver Level3 problems? Did anybody else see Denver Level3 service take a hit this morning? Our logs indicated nothing, but we saw various blackholing all over their network for ~20 minutes, starting around 9:15am MST. Everybody have a great day - Randal
participants (10)
-
Bradley Bopp -
Bryan Campbell -
Jayson Baker -
Joseph Jackson -
Kevin Broderick -
Matlock, Kenneth L -
randal k -
Sean Clark -
Sean Crandall -
Shawn Clemens