So is this issue all related to a fiber cut or a DC/Peering point having issues? http://www.thewhir.com/web-hosting-news/liquidweb-among-companies-affected-m... Michael Peterman -----Original Message----- From: Outages [mailto:outages-bounces@outages.org] On Behalf Of outages-request@outages.org Sent: Tuesday, August 12, 2014 11:00 AM To: outages@outages.org Subject: Outages Digest, Vol 75, Issue 9 Send Outages mailing list submissions to outages@outages.org To subscribe or unsubscribe via the World Wide Web, visit https://puck.nether.net/mailman/listinfo/outages or, via email, send a message with subject or body 'help' to outages-request@outages.org You can reach the person managing the list at outages-owner@outages.org When replying, please edit your Subject line so it is more specific than "Re: Contents of Outages digest..." Today's Topics: 1. Windstream (Mike Walter) 2. CoreSite Any2 LA (jd) 3. Re: CoreSite Any2 LA (Paul Stewart) 4. LastPass.com has a DC outage (Chris Egeland) 5. Major outages today, not much info at this time (Hoyle Anderson (AM)) 6. Re: Major outages today, not much info at this time (Teun Vink) ---------------------------------------------------------------------- Message: 1 Date: Tue, 12 Aug 2014 13:19:08 +0000 From: Mike Walter <mwalter@3z.net> To: "'outages@outages.org'" <outages@outages.org> Subject: [outages] Windstream Message-ID: <D7092CF606B8964193F248C92106B7810FCDE655@zionex01.ZION.local> Content-Type: text/plain; charset="us-ascii" Just an FYI, there is a Windstream fiber cut effecting Ohio and Kentucky customers, no ETA to repair based on their phone message. Still on hold waiting to open case to find out more. -Mike
I think it is related to this. http://www.cisco.com/c/en/us/support/docs/switches/catalyst-6500-series-swit... But it is probably several things including 512k route limit on cisco cats. (coffee != sleep) & (!coffee == sleep) Donald.Smith@centurylink.com<mailto:Donald.Smith@centurylink.com> ________________________________ From: Outages [outages-bounces@outages.org] on behalf of Michael Peterman via Outages [outages@outages.org] Sent: Tuesday, August 12, 2014 12:21 PM To: outages@outages.org Subject: [outages] So what is broken So is this issue all related to a fiber cut or a DC/Peering point having issues? http://www.thewhir.com/web-hosting-news/liquidweb-among-companies-affected-m... Michael Peterman -----Original Message----- From: Outages [mailto:outages-bounces@outages.org] On Behalf Of outages-request@outages.org Sent: Tuesday, August 12, 2014 11:00 AM To: outages@outages.org Subject: Outages Digest, Vol 75, Issue 9 Send Outages mailing list submissions to outages@outages.org To subscribe or unsubscribe via the World Wide Web, visit https://puck.nether.net/mailman/listinfo/outages or, via email, send a message with subject or body 'help' to outages-request@outages.org You can reach the person managing the list at outages-owner@outages.org When replying, please edit your Subject line so it is more specific than "Re: Contents of Outages digest..." Today's Topics: 1. Windstream (Mike Walter) 2. CoreSite Any2 LA (jd) 3. Re: CoreSite Any2 LA (Paul Stewart) 4. LastPass.com has a DC outage (Chris Egeland) 5. Major outages today, not much info at this time (Hoyle Anderson (AM)) 6. Re: Major outages today, not much info at this time (Teun Vink) ---------------------------------------------------------------------- Message: 1 Date: Tue, 12 Aug 2014 13:19:08 +0000 From: Mike Walter <mwalter@3z.net> To: "'outages@outages.org'" <outages@outages.org> Subject: [outages] Windstream Message-ID: <D7092CF606B8964193F248C92106B7810FCDE655@zionex01.ZION.local> Content-Type: text/plain; charset="us-ascii" Just an FYI, there is a Windstream fiber cut effecting Ohio and Kentucky customers, no ETA to repair based on their phone message. Still on hold waiting to open case to find out more. -Mike
I would imagine it's related to the global routing table hitting 512k routes. http://www.cidr-report.org/cgi-bin/plot?file=%2fvar%2fdata%2fbgp%2fas2.0%2fb... Regards, SG On 8/12/2014 12:21 PM, Michael Peterman via Outages wrote:
So is this issue all related to a fiber cut or a DC/Peering point having issues?
http://www.thewhir.com/web-hosting-news/liquidweb-among-companies-affected-m...
Michael Peterman
-----Original Message----- From: Outages [mailto:outages-bounces@outages.org] On Behalf Of outages-request@outages.org Sent: Tuesday, August 12, 2014 11:00 AM To: outages@outages.org Subject: Outages Digest, Vol 75, Issue 9
Send Outages mailing list submissions to outages@outages.org
To subscribe or unsubscribe via the World Wide Web, visit https://puck.nether.net/mailman/listinfo/outages or, via email, send a message with subject or body 'help' to outages-request@outages.org
You can reach the person managing the list at outages-owner@outages.org
When replying, please edit your Subject line so it is more specific than "Re: Contents of Outages digest..."
Today's Topics:
1. Windstream (Mike Walter) 2. CoreSite Any2 LA (jd) 3. Re: CoreSite Any2 LA (Paul Stewart) 4. LastPass.com has a DC outage (Chris Egeland) 5. Major outages today, not much info at this time (Hoyle Anderson (AM)) 6. Re: Major outages today, not much info at this time (Teun Vink)
----------------------------------------------------------------------
Message: 1 Date: Tue, 12 Aug 2014 13:19:08 +0000 From: Mike Walter <mwalter@3z.net> To: "'outages@outages.org'" <outages@outages.org> Subject: [outages] Windstream Message-ID: <D7092CF606B8964193F248C92106B7810FCDE655@zionex01.ZION.local> Content-Type: text/plain; charset="us-ascii"
Just an FYI, there is a Windstream fiber cut effecting Ohio and Kentucky customers, no ETA to repair based on their phone message. Still on hold waiting to open case to find out more. -Mike
I'm not so sure. None of my four upstreams is showing that many. Level3 (as an example) is 497,389 right now. This might be an issue for some places but probably isn't the full "this is why everyone is down today" explanation. -Bill
-----Original Message----- From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Spencer Gaw via Outages Sent: Tuesday, August 12, 2014 3:20 PM To: Michael Peterman; outages@outages.org Subject: Re: [outages] So what is broken
I would imagine it's related to the global routing table hitting 512k routes.
http://www.cidr-report.org/cgi- bin/plot?file=%2fvar%2fdata%2fbgp%2fas2.0%2fbgp%2dactive%2etxt&start =1407179655&end=1407870855&width=0%2e9&height=0%2e3&with=step&y label=BGP+entries
Regards,
SG
On 8/12/2014 12:21 PM, Michael Peterman via Outages wrote:
So is this issue all related to a fiber cut or a DC/Peering point having issues?
http://www.thewhir.com/web-hosting-news/liquidweb-among- companies-affe cted-major-outage-across-us-network-providers
Michael Peterman
-----Original Message----- From: Outages [mailto:outages-bounces@outages.org] On Behalf Of outages-request@outages.org Sent: Tuesday, August 12, 2014 11:00 AM To: outages@outages.org Subject: Outages Digest, Vol 75, Issue 9
Send Outages mailing list submissions to outages@outages.org
To subscribe or unsubscribe via the World Wide Web, visit https://puck.nether.net/mailman/listinfo/outages or, via email, send a message with subject or body 'help' to outages-request@outages.org
You can reach the person managing the list at outages-owner@outages.org
When replying, please edit your Subject line so it is more specific than "Re: Contents of Outages digest..."
Today's Topics:
1. Windstream (Mike Walter) 2. CoreSite Any2 LA (jd) 3. Re: CoreSite Any2 LA (Paul Stewart) 4. LastPass.com has a DC outage (Chris Egeland) 5. Major outages today, not much info at this time (Hoyle Anderson (AM)) 6. Re: Major outages today, not much info at this time (Teun Vink)
----------------------------------------------------------------------
Message: 1 Date: Tue, 12 Aug 2014 13:19:08 +0000 From: Mike Walter <mwalter@3z.net> To: "'outages@outages.org'" <outages@outages.org> Subject: [outages] Windstream Message-ID:
<D7092CF606B8964193F248C92106B7810FCDE655@zionex01.ZION.loc al>
Content-Type: text/plain; charset="us-ascii"
Just an FYI, there is a Windstream fiber cut effecting Ohio and Kentucky customers, no ETA to repair based on their phone message. Still on hold waiting to open case to find out more. -Mike
-------------- next part -------------- An HTML attachment was scrubbed... URL:
<https://puck.nether.net/pipermail/outages/attachments/20140812/194c6c
4b/attachment-0001.html>
------------------------------
Message: 2 Date: Tue, 12 Aug 2014 07:19:04 -0700 From: jd <jd@boldnetworks.net> To: "'outages@outages.org'" <outages@outages.org> Subject: [outages] CoreSite Any2 LA Message-ID: <53EA2258.2020601@boldnetworks.net> Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Is anyone having issues with Any2 in LA? I have no evidence but I'm hearing some things, if anyone could debunk.
Thanks JD
------------------------------
Message: 3 Date: Tue, 12 Aug 2014 10:42:39 -0400 From: Paul Stewart <paul@paulstewart.org> To: "'outages@outages.org'" <outages@outages.org> Subject: Re: [outages] CoreSite Any2 LA Message-ID: <D00FA00A.806C1%paul@paulstewart.org> Content-Type: text/plain; charset="UTF-8"
No issues here at this point?. At least that have become evident?
Paul
On 2014-08-12, 10:19 AM, "jd via Outages" <outages@outages.org> wrote:
Is anyone having issues with Any2 in LA? I have no evidence but I'm hearing some things, if anyone could debunk.
Thanks JD _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
------------------------------
Message: 4 Date: Tue, 12 Aug 2014 15:04:58 +0000 From: Chris Egeland <chris.egeland@ferrotherm.com> To: "outages@outages.org List" <outages@outages.org> Subject: [outages] LastPass.com has a DC outage Message-ID: <EA49DAFC5FDC1A40B16DEE2D6ED5469A9FD277AB@mail01> Content-Type: text/plain; charset="us-ascii"
https://twitter.com/LastPass/status/499199566701658112
"Update: One of our data centers went down, we have the service fully running on another data center & are working to minimize impact."
I'm getting "An error occurred while attempting to contact the server. Please check your Internet connection." In my LastPass bar on Firefox.
So, if you're getting errors with LastPass, this is why.
________________________________
This e-mail and any attachments may contain privileged or confidential information of Ferrotherm Corporation. This e-mail is intended solely for the use of the individual or entity to which it is addressed. If you are not the intended recipient of this e-mail, you are hereby notified that any copying, distribution, dissemination or action taken in relation to the contents of this e-mail and any of its attachments is strictly prohibited and may be unlawful. If you have received this e-mail in error, please notify the sender immediately and permanently delete the original e-mail and destroy any copies or printouts of this e-mail as well as any attachments. -------------- next part -------------- An HTML attachment was scrubbed... URL:
<https://puck.nether.net/pipermail/outages/attachments/20140812/837627
0e/attachment-0001.html>
------------------------------
Message: 5 Date: Tue, 12 Aug 2014 15:20:45 +0000 From: "Hoyle Anderson (AM)" <hoyle.anderson@dimensiondata.com> To: "outages@outages.org" <outages@outages.org> Subject: [outages] Major outages today, not much info at this time Message-ID:
<7C6FA1D8224C7944A005E6587BDEA50A03F692@usispsvexdb03.na.didata.lo cal>
Content-Type: text/plain; charset="utf-8"
I know this isn?t much help, but there are major problems with multiple ISPs since around 4-5 AM EST. I really don?t have much detail, but I have sites that are unreachable from some providers. Looks like Comcast, level3, ATT, cogent, etc.
So, it?s probably not just you, but I?m afraid I don?t know who it is. I heard one report of a datacenter outage.
Sources: http://downdetector.com/ https://twitter.com/liquidweb
-------------- next part -------------- An HTML attachment was scrubbed... URL:
<https://puck.nether.net/pipermail/outages/attachments/20140812/fa57b8
50/attachment-0001.html>
------------------------------
Message: 6 Date: Tue, 12 Aug 2014 17:42:05 +0200 From: Teun Vink <teun@teun.tv> To: "Hoyle Anderson (AM)" <hoyle.anderson@dimensiondata.com> Cc: "outages@outages.org" <outages@outages.org> Subject: Re: [outages] Major outages today, not much info at this time Message-ID: <1407858125.27157.21.camel@moridin> Content-Type: text/plain; charset="UTF-8"
On di, 2014-08-12 at 15:20 +0000, Hoyle Anderson (AM) via Outages wrote:
I know this isn?t much help, but there are major problems with multiple ISPs since around 4-5 AM EST. I really don?t have much detail, but I have sites that are unreachable from some providers. Looks like Comcast, level3, ATT, cogent, etc.
So, it?s probably not just you, but I?m afraid I don?t know who it is. I heard one report of a datacenter outage.
Hi,
Some routing tables hit 512K routes today. Some old hardware and software can't handle that and either crash or ignore newly learned routes. So this may cause some disturbances in the force.
HTH, Teun
------------------------------
Subject: Digest Footer
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
------------------------------
End of Outages Digest, Vol 75, Issue 9 **************************************
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
I go with the fiber cut issue. And maybe the other bgp sessions are getting overloaded because thing have to re-route. Michael Peterman President Small Business Technology Solutions Inc. "You worry about running your business; we will worry about the technology" Main Office 7513 7th Avenue Suite 3 Kenosha, WI 53143 Phone: (262) 671-4898 ext. 202 Phone: (608) 554-4858 ext. 202 Cell: (262) 496-3306 www.seeus4it.com Michael@seeus4it.com -----Original Message----- From: Bill Wichers [mailto:billw@waveform.net] Sent: Tuesday, August 12, 2014 2:27 PM To: Spencer Gaw; Michael Peterman Cc: outages@outages.org Subject: RE: [outages] So what is broken I'm not so sure. None of my four upstreams is showing that many. Level3 (as an example) is 497,389 right now. This might be an issue for some places but probably isn't the full "this is why everyone is down today" explanation. -Bill
-----Original Message----- From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Spencer Gaw via Outages Sent: Tuesday, August 12, 2014 3:20 PM To: Michael Peterman; outages@outages.org Subject: Re: [outages] So what is broken
I would imagine it's related to the global routing table hitting 512k routes.
http://www.cidr-report.org/cgi- bin/plot?file=%2fvar%2fdata%2fbgp%2fas2.0%2fbgp%2dactive%2etxt&start =1407179655&end=1407870855&width=0%2e9&height=0%2e3&with=step&y label=BGP+entries
Regards,
SG
On 8/12/2014 12:21 PM, Michael Peterman via Outages wrote:
So is this issue all related to a fiber cut or a DC/Peering point having issues?
http://www.thewhir.com/web-hosting-news/liquidweb-among- companies-affe cted-major-outage-across-us-network-providers
Michael Peterman
-----Original Message----- From: Outages [mailto:outages-bounces@outages.org] On Behalf Of outages-request@outages.org Sent: Tuesday, August 12, 2014 11:00 AM To: outages@outages.org Subject: Outages Digest, Vol 75, Issue 9
Send Outages mailing list submissions to outages@outages.org
To subscribe or unsubscribe via the World Wide Web, visit https://puck.nether.net/mailman/listinfo/outages or, via email, send a message with subject or body 'help' to outages-request@outages.org
You can reach the person managing the list at outages-owner@outages.org
When replying, please edit your Subject line so it is more specific than "Re: Contents of Outages digest..."
Today's Topics:
1. Windstream (Mike Walter) 2. CoreSite Any2 LA (jd) 3. Re: CoreSite Any2 LA (Paul Stewart) 4. LastPass.com has a DC outage (Chris Egeland) 5. Major outages today, not much info at this time (Hoyle Anderson (AM)) 6. Re: Major outages today, not much info at this time (Teun Vink)
-------------------------------------------------------------------- --
Message: 1 Date: Tue, 12 Aug 2014 13:19:08 +0000 From: Mike Walter <mwalter@3z.net> To: "'outages@outages.org'" <outages@outages.org> Subject: [outages] Windstream Message-ID:
<D7092CF606B8964193F248C92106B7810FCDE655@zionex01.ZION.loc al>
Content-Type: text/plain; charset="us-ascii"
Just an FYI, there is a Windstream fiber cut effecting Ohio and Kentucky customers, no ETA to repair based on their phone message. Still on hold waiting to open case to find out more. -Mike
-------------- next part -------------- An HTML attachment was scrubbed... URL:
<https://puck.nether.net/pipermail/outages/attachments/20140812/194c6c
4b/attachment-0001.html>
------------------------------
Message: 2 Date: Tue, 12 Aug 2014 07:19:04 -0700 From: jd <jd@boldnetworks.net> To: "'outages@outages.org'" <outages@outages.org> Subject: [outages] CoreSite Any2 LA Message-ID: <53EA2258.2020601@boldnetworks.net> Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Is anyone having issues with Any2 in LA? I have no evidence but I'm hearing some things, if anyone could debunk.
Thanks JD
------------------------------
Message: 3 Date: Tue, 12 Aug 2014 10:42:39 -0400 From: Paul Stewart <paul@paulstewart.org> To: "'outages@outages.org'" <outages@outages.org> Subject: Re: [outages] CoreSite Any2 LA Message-ID: <D00FA00A.806C1%paul@paulstewart.org> Content-Type: text/plain; charset="UTF-8"
No issues here at this point?. At least that have become evident?
Paul
On 2014-08-12, 10:19 AM, "jd via Outages" <outages@outages.org> wrote:
Is anyone having issues with Any2 in LA? I have no evidence but I'm hearing some things, if anyone could debunk.
Thanks JD _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
------------------------------
Message: 4 Date: Tue, 12 Aug 2014 15:04:58 +0000 From: Chris Egeland <chris.egeland@ferrotherm.com> To: "outages@outages.org List" <outages@outages.org> Subject: [outages] LastPass.com has a DC outage Message-ID: <EA49DAFC5FDC1A40B16DEE2D6ED5469A9FD277AB@mail01> Content-Type: text/plain; charset="us-ascii"
https://twitter.com/LastPass/status/499199566701658112
"Update: One of our data centers went down, we have the service fully running on another data center & are working to minimize impact."
I'm getting "An error occurred while attempting to contact the server. Please check your Internet connection." In my LastPass bar on Firefox.
So, if you're getting errors with LastPass, this is why.
________________________________
This e-mail and any attachments may contain privileged or confidential information of Ferrotherm Corporation. This e-mail is intended solely for the use of the individual or entity to which it is addressed. If you are not the intended recipient of this e-mail, you are hereby notified that any copying, distribution, dissemination or action taken in relation to the contents of this e-mail and any of its attachments is strictly prohibited and may be unlawful. If you have received this e-mail in error, please notify the sender immediately and permanently delete the original e-mail and destroy any copies or printouts of this e-mail as well as any attachments. -------------- next part -------------- An HTML attachment was scrubbed... URL:
<https://puck.nether.net/pipermail/outages/attachments/20140812/837627
0e/attachment-0001.html>
------------------------------
Message: 5 Date: Tue, 12 Aug 2014 15:20:45 +0000 From: "Hoyle Anderson (AM)" <hoyle.anderson@dimensiondata.com> To: "outages@outages.org" <outages@outages.org> Subject: [outages] Major outages today, not much info at this time Message-ID:
<7C6FA1D8224C7944A005E6587BDEA50A03F692@usispsvexdb03.na.didata.lo cal>
Content-Type: text/plain; charset="utf-8"
I know this isn?t much help, but there are major problems with multiple ISPs since around 4-5 AM EST. I really don?t have much detail, but I have sites that are unreachable from some providers. Looks like Comcast, level3, ATT, cogent, etc.
So, it?s probably not just you, but I?m afraid I don?t know who it is. I heard one report of a datacenter outage.
Sources: http://downdetector.com/ https://twitter.com/liquidweb
-------------- next part -------------- An HTML attachment was scrubbed... URL:
<https://puck.nether.net/pipermail/outages/attachments/20140812/fa57b8
50/attachment-0001.html>
------------------------------
Message: 6 Date: Tue, 12 Aug 2014 17:42:05 +0200 From: Teun Vink <teun@teun.tv> To: "Hoyle Anderson (AM)" <hoyle.anderson@dimensiondata.com> Cc: "outages@outages.org" <outages@outages.org> Subject: Re: [outages] Major outages today, not much info at this time Message-ID: <1407858125.27157.21.camel@moridin> Content-Type: text/plain; charset="UTF-8"
On di, 2014-08-12 at 15:20 +0000, Hoyle Anderson (AM) via Outages wrote:
I know this isn?t much help, but there are major problems with multiple ISPs since around 4-5 AM EST. I really don?t have much detail, but I have sites that are unreachable from some providers. Looks like Comcast, level3, ATT, cogent, etc.
So, it?s probably not just you, but I?m afraid I don?t know who it is. I heard one report of a datacenter outage.
Hi,
Some routing tables hit 512K routes today. Some old hardware and software can't handle that and either crash or ignore newly learned routes. So this may cause some disturbances in the force.
HTH, Teun
------------------------------
Subject: Digest Footer
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
------------------------------
End of Outages Digest, Vol 75, Issue 9 **************************************
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
AS 2381 definitely saw Level3 sending > 512K for a brief time this AM, say 02:50 to 03:25. Perhaps those affected haven't figured it out yet. My below graph has timestamps in CDT. https://www.eng.wiscnet.net/stats/cgi-bin/genstatspage.fcgi?width=750&height... fwiw, -Michael On 8/12/2014 2:27 PM, Bill Wichers via Outages wrote:
I'm not so sure. None of my four upstreams is showing that many. Level3 (as an example) is 497,389 right now.
This might be an issue for some places but probably isn't the full "this is why everyone is down today" explanation.
-Bill
-----Original Message----- From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Spencer Gaw via Outages Sent: Tuesday, August 12, 2014 3:20 PM To: Michael Peterman; outages@outages.org Subject: Re: [outages] So what is broken
I would imagine it's related to the global routing table hitting 512k routes.
http://www.cidr-report.org/cgi- bin/plot?file=%2fvar%2fdata%2fbgp%2fas2.0%2fbgp%2dactive%2etxt&start =1407179655&end=1407870855&width=0%2e9&height=0%2e3&with=step&y label=BGP+entries
Regards,
SG
On 8/12/2014 12:21 PM, Michael Peterman via Outages wrote:
So is this issue all related to a fiber cut or a DC/Peering point having issues?
http://www.thewhir.com/web-hosting-news/liquidweb-among- companies-affe cted-major-outage-across-us-network-providers
Michael Peterman
-----Original Message----- From: Outages [mailto:outages-bounces@outages.org] On Behalf Of outages-request@outages.org Sent: Tuesday, August 12, 2014 11:00 AM To: outages@outages.org Subject: Outages Digest, Vol 75, Issue 9
Send Outages mailing list submissions to outages@outages.org
To subscribe or unsubscribe via the World Wide Web, visit https://puck.nether.net/mailman/listinfo/outages or, via email, send a message with subject or body 'help' to outages-request@outages.org
You can reach the person managing the list at outages-owner@outages.org
When replying, please edit your Subject line so it is more specific than "Re: Contents of Outages digest..."
Today's Topics:
1. Windstream (Mike Walter) 2. CoreSite Any2 LA (jd) 3. Re: CoreSite Any2 LA (Paul Stewart) 4. LastPass.com has a DC outage (Chris Egeland) 5. Major outages today, not much info at this time (Hoyle Anderson (AM)) 6. Re: Major outages today, not much info at this time (Teun Vink)
----------------------------------------------------------------------
Message: 1 Date: Tue, 12 Aug 2014 13:19:08 +0000 From: Mike Walter <mwalter@3z.net> To: "'outages@outages.org'" <outages@outages.org> Subject: [outages] Windstream Message-ID:
<D7092CF606B8964193F248C92106B7810FCDE655@zionex01.ZION.loc al>
Content-Type: text/plain; charset="us-ascii"
Just an FYI, there is a Windstream fiber cut effecting Ohio and Kentucky customers, no ETA to repair based on their phone message. Still on hold waiting to open case to find out more. -Mike
-------------- next part -------------- An HTML attachment was scrubbed... URL:
<https://puck.nether.net/pipermail/outages/attachments/20140812/194c6c
4b/attachment-0001.html>
------------------------------
Message: 2 Date: Tue, 12 Aug 2014 07:19:04 -0700 From: jd <jd@boldnetworks.net> To: "'outages@outages.org'" <outages@outages.org> Subject: [outages] CoreSite Any2 LA Message-ID: <53EA2258.2020601@boldnetworks.net> Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Is anyone having issues with Any2 in LA? I have no evidence but I'm hearing some things, if anyone could debunk.
Thanks JD
------------------------------
Message: 3 Date: Tue, 12 Aug 2014 10:42:39 -0400 From: Paul Stewart <paul@paulstewart.org> To: "'outages@outages.org'" <outages@outages.org> Subject: Re: [outages] CoreSite Any2 LA Message-ID: <D00FA00A.806C1%paul@paulstewart.org> Content-Type: text/plain; charset="UTF-8"
No issues here at this point?. At least that have become evident?
Paul
On 2014-08-12, 10:19 AM, "jd via Outages" <outages@outages.org> wrote:
Is anyone having issues with Any2 in LA? I have no evidence but I'm hearing some things, if anyone could debunk.
Thanks JD _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
------------------------------
Message: 4 Date: Tue, 12 Aug 2014 15:04:58 +0000 From: Chris Egeland <chris.egeland@ferrotherm.com> To: "outages@outages.org List" <outages@outages.org> Subject: [outages] LastPass.com has a DC outage Message-ID: <EA49DAFC5FDC1A40B16DEE2D6ED5469A9FD277AB@mail01> Content-Type: text/plain; charset="us-ascii"
https://twitter.com/LastPass/status/499199566701658112
"Update: One of our data centers went down, we have the service fully running on another data center & are working to minimize impact."
I'm getting "An error occurred while attempting to contact the server. Please check your Internet connection." In my LastPass bar on Firefox.
So, if you're getting errors with LastPass, this is why.
________________________________
This e-mail and any attachments may contain privileged or confidential information of Ferrotherm Corporation. This e-mail is intended solely for the use of the individual or entity to which it is addressed. If you are not the intended recipient of this e-mail, you are hereby notified that any copying, distribution, dissemination or action taken in relation to the contents of this e-mail and any of its attachments is strictly prohibited and may be unlawful. If you have received this e-mail in error, please notify the sender immediately and permanently delete the original e-mail and destroy any copies or printouts of this e-mail as well as any attachments. -------------- next part -------------- An HTML attachment was scrubbed... URL:
<https://puck.nether.net/pipermail/outages/attachments/20140812/837627
0e/attachment-0001.html>
------------------------------
Message: 5 Date: Tue, 12 Aug 2014 15:20:45 +0000 From: "Hoyle Anderson (AM)" <hoyle.anderson@dimensiondata.com> To: "outages@outages.org" <outages@outages.org> Subject: [outages] Major outages today, not much info at this time Message-ID:
<7C6FA1D8224C7944A005E6587BDEA50A03F692@usispsvexdb03.na.didata.lo cal>
Content-Type: text/plain; charset="utf-8"
I know this isn?t much help, but there are major problems with multiple ISPs since around 4-5 AM EST. I really don?t have much detail, but I have sites that are unreachable from some providers. Looks like Comcast, level3, ATT, cogent, etc.
So, it?s probably not just you, but I?m afraid I don?t know who it is. I heard one report of a datacenter outage.
Sources: http://downdetector.com/ https://twitter.com/liquidweb
-------------- next part -------------- An HTML attachment was scrubbed... URL:
<https://puck.nether.net/pipermail/outages/attachments/20140812/fa57b8
50/attachment-0001.html>
------------------------------
Message: 6 Date: Tue, 12 Aug 2014 17:42:05 +0200 From: Teun Vink <teun@teun.tv> To: "Hoyle Anderson (AM)" <hoyle.anderson@dimensiondata.com> Cc: "outages@outages.org" <outages@outages.org> Subject: Re: [outages] Major outages today, not much info at this time Message-ID: <1407858125.27157.21.camel@moridin> Content-Type: text/plain; charset="UTF-8"
On di, 2014-08-12 at 15:20 +0000, Hoyle Anderson (AM) via Outages wrote:
I know this isn?t much help, but there are major problems with multiple ISPs since around 4-5 AM EST. I really don?t have much detail, but I have sites that are unreachable from some providers. Looks like Comcast, level3, ATT, cogent, etc.
So, it?s probably not just you, but I?m afraid I don?t know who it is. I heard one report of a datacenter outage.
Hi,
Some routing tables hit 512K routes today. Some old hardware and software can't handle that and either crash or ignore newly learned routes. So this may cause some disturbances in the force.
HTH, Teun
------------------------------
Subject: Digest Footer
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
------------------------------
End of Outages Digest, Vol 75, Issue 9 **************************************
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
CIDR report is showing over 512k around the same time. On Aug 12, 2014 4:09 PM, "Michael Hare via Outages" <outages@outages.org> wrote:
AS 2381 definitely saw Level3 sending > 512K for a brief time this AM, say 02:50 to 03:25. Perhaps those affected haven't figured it out yet. My below graph has timestamps in CDT.
https://www.eng.wiscnet.net/stats/cgi-bin/genstatspage. fcgi?width=750&height=200&rrdfunc=MAX&rrdfile=%2Fvar%2Fflows%2Frrds% 2FjnxBgpM2Prefixes%2Fdevice%3Dr-uwmadison-isp.wiscnet.net_ family%3Dipv4_class%3Dunicast_ip%3D4.53.34.5_ jnxBgpM2PrefixInPrefixesAccepted.rrd&time=CUSTOM&start_time= 08%2F12%2F2014+2am&end_time=08%2F12%2F2014+6am
fwiw, -Michael
On 8/12/2014 2:27 PM, Bill Wichers via Outages wrote:
I'm not so sure. None of my four upstreams is showing that many. Level3 (as an example) is 497,389 right now.
This might be an issue for some places but probably isn't the full "this is why everyone is down today" explanation.
-Bill
-----Original Message-----
From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Spencer Gaw via Outages Sent: Tuesday, August 12, 2014 3:20 PM To: Michael Peterman; outages@outages.org Subject: Re: [outages] So what is broken
I would imagine it's related to the global routing table hitting 512k routes.
http://www.cidr-report.org/cgi- bin/plot?file=%2fvar%2fdata%2fbgp%2fas2.0%2fbgp%2dactive%2etxt&start =1407179655&end=1407870855&width=0%2e9&height=0%2e3&with=step&y label=BGP+entries
Regards,
SG
On 8/12/2014 12:21 PM, Michael Peterman via Outages wrote:
So is this issue all related to a fiber cut or a DC/Peering point having issues?
companies-affe
cted-major-outage-across-us-network-providers
Michael Peterman
-----Original Message----- From: Outages [mailto:outages-bounces@outages.org] On Behalf Of outages-request@outages.org Sent: Tuesday, August 12, 2014 11:00 AM To: outages@outages.org Subject: Outages Digest, Vol 75, Issue 9
Send Outages mailing list submissions to outages@outages.org
To subscribe or unsubscribe via the World Wide Web, visit https://puck.nether.net/mailman/listinfo/outages or, via email, send a message with subject or body 'help' to outages-request@outages.org
You can reach the person managing the list at outages-owner@outages.org
When replying, please edit your Subject line so it is more specific than "Re:
Contents of Outages digest..."
Today's Topics:
1. Windstream (Mike Walter) 2. CoreSite Any2 LA (jd) 3. Re: CoreSite Any2 LA (Paul Stewart) 4. LastPass.com has a DC outage (Chris Egeland) 5. Major outages today, not much info at this time (Hoyle Anderson (AM)) 6. Re: Major outages today, not much info at this time (Teun Vink)
----------------------------------------------------------------------
Message: 1 Date: Tue, 12 Aug 2014 13:19:08 +0000 From: Mike Walter <mwalter@3z.net> To: "'outages@outages.org'" <outages@outages.org> Subject: [outages] Windstream Message-ID:
<D7092CF606B8964193F248C92106B7810FCDE655@zionex01.ZION.loc
al>
Content-Type: text/plain; charset="us-ascii"
Just an FYI, there is a Windstream fiber cut effecting Ohio and Kentucky
customers, no ETA to repair based on their phone message. Still on hold waiting to open case to find out more.
-Mike
-------------- next part -------------- An HTML attachment was scrubbed... URL:
<https://puck.nether.net/pipermail/outages/attachments/20140812/194c6c
4b/attachment-0001.html>
------------------------------
Message: 2 Date: Tue, 12 Aug 2014 07:19:04 -0700 From: jd <jd@boldnetworks.net> To: "'outages@outages.org'" <outages@outages.org> Subject: [outages] CoreSite Any2 LA Message-ID: <53EA2258.2020601@boldnetworks.net> Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Is anyone having issues with Any2 in LA? I have no evidence but I'm hearing
some things, if anyone could debunk.
Thanks JD
------------------------------
Message: 3 Date: Tue, 12 Aug 2014 10:42:39 -0400 From: Paul Stewart <paul@paulstewart.org> To: "'outages@outages.org'" <outages@outages.org> Subject: Re: [outages] CoreSite Any2 LA Message-ID: <D00FA00A.806C1%paul@paulstewart.org> Content-Type: text/plain; charset="UTF-8"
No issues here at this point?. At least that have become evident?
Paul
On 2014-08-12, 10:19 AM, "jd via Outages" <outages@outages.org> wrote:
Is anyone having issues with Any2 in LA? I have no evidence but I'm
hearing some things, if anyone could debunk.
Thanks JD _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
------------------------------
Message: 4 Date: Tue, 12 Aug 2014 15:04:58 +0000 From: Chris Egeland <chris.egeland@ferrotherm.com> To: "outages@outages.org List" <outages@outages.org> Subject: [outages] LastPass.com has a DC outage Message-ID: <EA49DAFC5FDC1A40B16DEE2D6ED5469A9FD277AB@mail01> Content-Type: text/plain; charset="us-ascii"
https://twitter.com/LastPass/status/499199566701658112
"Update: One of our data centers went down, we have the service fully
running on another data center & are working to minimize impact."
I'm getting "An error occurred while attempting to contact the server.
Please check your Internet connection." In my LastPass bar on Firefox.
So, if you're getting errors with LastPass, this is why.
________________________________
This e-mail and any attachments may contain privileged or confidential
information of Ferrotherm Corporation. This e-mail is intended solely for the use of the individual or entity to which it is addressed. If you are not the intended recipient of this e-mail, you are hereby notified that any copying, distribution, dissemination or action taken in relation to the contents of this e-mail and any of its attachments is strictly prohibited and may be unlawful. If you have received this e-mail in error, please notify the sender immediately and permanently delete the original e-mail and destroy any copies or printouts of this e-mail as well as any attachments.
-------------- next part -------------- An HTML attachment was scrubbed... URL:
<https://puck.nether.net/pipermail/outages/attachments/20140812/837627
0e/attachment-0001.html>
------------------------------
Message: 5 Date: Tue, 12 Aug 2014 15:20:45 +0000 From: "Hoyle Anderson (AM)" <hoyle.anderson@dimensiondata.com> To: "outages@outages.org" <outages@outages.org> Subject: [outages] Major outages today, not much info at this time Message-ID:
<7C6FA1D8224C7944A005E6587BDEA50A03F692@usispsvexdb03.na.didata.lo cal>
Content-Type: text/plain; charset="utf-8"
I know this isn?t much help, but there are major problems with multiple
ISPs since around 4-5 AM EST. I really don?t have much detail, but I have sites that are unreachable from some providers. Looks like Comcast, level3, ATT, cogent, etc.
So, it?s probably not just you, but I?m afraid I don?t know who it is. I heard
one report of a datacenter outage.
Sources: http://downdetector.com/ https://twitter.com/liquidweb
-------------- next part -------------- An HTML attachment was scrubbed... URL:
<https://puck.nether.net/pipermail/outages/attachments/20140812/fa57b8
50/attachment-0001.html>
------------------------------
Message: 6 Date: Tue, 12 Aug 2014 17:42:05 +0200 From: Teun Vink <teun@teun.tv> To: "Hoyle Anderson (AM)" <hoyle.anderson@dimensiondata.com> Cc: "outages@outages.org" <outages@outages.org> Subject: Re: [outages] Major outages today, not much info at this time Message-ID: <1407858125.27157.21.camel@moridin> Content-Type: text/plain; charset="UTF-8"
On di, 2014-08-12 at 15:20 +0000, Hoyle Anderson (AM) via Outages wrote:
I know this isn?t much help, but there are major problems with multiple ISPs since around 4-5 AM EST. I really don?t have much detail, but I have sites that are unreachable from some providers. Looks like Comcast, level3, ATT, cogent, etc.
So, it?s probably not just you, but I?m afraid I don?t know who it is. I heard one report of a datacenter outage.
Hi,
Some routing tables hit 512K routes today. Some old hardware and
software can't handle that and either crash or ignore newly learned routes. So this may cause some disturbances in the force.
HTH, Teun
------------------------------
Subject: Digest Footer
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
------------------------------
End of Outages Digest, Vol 75, Issue 9 **************************************
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
When did Windstream come back up after their big outage from the fiber cut? AS7029 has a couple thousand routes. Just a possible correlation I observed. On Aug 12, 2014 4:12 PM, "Sean Hayes" <sean@engage.pro> wrote:
CIDR report is showing over 512k around the same time. On Aug 12, 2014 4:09 PM, "Michael Hare via Outages" <outages@outages.org> wrote:
AS 2381 definitely saw Level3 sending > 512K for a brief time this AM, say 02:50 to 03:25. Perhaps those affected haven't figured it out yet. My below graph has timestamps in CDT.
https://www.eng.wiscnet.net/stats/cgi-bin/genstatspage. fcgi?width=750&height=200&rrdfunc=MAX&rrdfile=%2Fvar%2Fflows%2Frrds% 2FjnxBgpM2Prefixes%2Fdevice%3Dr-uwmadison-isp.wiscnet.net_ family%3Dipv4_class%3Dunicast_ip%3D4.53.34.5_ jnxBgpM2PrefixInPrefixesAccepted.rrd&time=CUSTOM&start_time= 08%2F12%2F2014+2am&end_time=08%2F12%2F2014+6am
fwiw, -Michael
On 8/12/2014 2:27 PM, Bill Wichers via Outages wrote:
I'm not so sure. None of my four upstreams is showing that many. Level3 (as an example) is 497,389 right now.
This might be an issue for some places but probably isn't the full "this is why everyone is down today" explanation.
-Bill
-----Original Message-----
From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Spencer Gaw via Outages Sent: Tuesday, August 12, 2014 3:20 PM To: Michael Peterman; outages@outages.org Subject: Re: [outages] So what is broken
I would imagine it's related to the global routing table hitting 512k routes.
http://www.cidr-report.org/cgi- bin/plot?file=%2fvar%2fdata%2fbgp%2fas2.0%2fbgp%2dactive%2etxt&start =1407179655&end=1407870855&width=0%2e9&height=0%2e3&with=step&y label=BGP+entries
Regards,
SG
On 8/12/2014 12:21 PM, Michael Peterman via Outages wrote:
So is this issue all related to a fiber cut or a DC/Peering point having issues?
companies-affe
cted-major-outage-across-us-network-providers
Michael Peterman
-----Original Message----- From: Outages [mailto:outages-bounces@outages.org] On Behalf Of outages-request@outages.org Sent: Tuesday, August 12, 2014 11:00 AM To: outages@outages.org Subject: Outages Digest, Vol 75, Issue 9
Send Outages mailing list submissions to outages@outages.org
To subscribe or unsubscribe via the World Wide Web, visit https://puck.nether.net/mailman/listinfo/outages or, via email, send a message with subject or body 'help' to outages-request@outages.org
You can reach the person managing the list at outages-owner@outages.org
When replying, please edit your Subject line so it is more specific than "Re:
Contents of Outages digest..."
Today's Topics:
1. Windstream (Mike Walter) 2. CoreSite Any2 LA (jd) 3. Re: CoreSite Any2 LA (Paul Stewart) 4. LastPass.com has a DC outage (Chris Egeland) 5. Major outages today, not much info at this time (Hoyle Anderson (AM)) 6. Re: Major outages today, not much info at this time (Teun Vink)
----------------------------------------------------------------------
Message: 1 Date: Tue, 12 Aug 2014 13:19:08 +0000 From: Mike Walter <mwalter@3z.net> To: "'outages@outages.org'" <outages@outages.org> Subject: [outages] Windstream Message-ID:
<D7092CF606B8964193F248C92106B7810FCDE655@zionex01.ZION.loc
al>
Content-Type: text/plain; charset="us-ascii"
Just an FYI, there is a Windstream fiber cut effecting Ohio and Kentucky
customers, no ETA to repair based on their phone message. Still on hold waiting to open case to find out more.
-Mike
-------------- next part -------------- An HTML attachment was scrubbed... URL:
<https://puck.nether.net/pipermail/outages/attachments/ 20140812/194c6c
4b/attachment-0001.html>
------------------------------
Message: 2 Date: Tue, 12 Aug 2014 07:19:04 -0700 From: jd <jd@boldnetworks.net> To: "'outages@outages.org'" <outages@outages.org> Subject: [outages] CoreSite Any2 LA Message-ID: <53EA2258.2020601@boldnetworks.net> Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Is anyone having issues with Any2 in LA? I have no evidence but I'm hearing
some things, if anyone could debunk.
Thanks JD
------------------------------
Message: 3 Date: Tue, 12 Aug 2014 10:42:39 -0400 From: Paul Stewart <paul@paulstewart.org> To: "'outages@outages.org'" <outages@outages.org> Subject: Re: [outages] CoreSite Any2 LA Message-ID: <D00FA00A.806C1%paul@paulstewart.org> Content-Type: text/plain; charset="UTF-8"
No issues here at this point?. At least that have become evident?
Paul
On 2014-08-12, 10:19 AM, "jd via Outages" <outages@outages.org> wrote:
Is anyone having issues with Any2 in LA? I have no evidence but I'm
hearing some things, if anyone could debunk.
Thanks JD _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
------------------------------
Message: 4 Date: Tue, 12 Aug 2014 15:04:58 +0000 From: Chris Egeland <chris.egeland@ferrotherm.com> To: "outages@outages.org List" <outages@outages.org> Subject: [outages] LastPass.com has a DC outage Message-ID: <EA49DAFC5FDC1A40B16DEE2D6ED5469A9FD277AB@mail01> Content-Type: text/plain; charset="us-ascii"
https://twitter.com/LastPass/status/499199566701658112
"Update: One of our data centers went down, we have the service fully
running on another data center & are working to minimize impact."
I'm getting "An error occurred while attempting to contact the server.
Please check your Internet connection." In my LastPass bar on Firefox.
So, if you're getting errors with LastPass, this is why.
________________________________
This e-mail and any attachments may contain privileged or confidential
information of Ferrotherm Corporation. This e-mail is intended solely for the use of the individual or entity to which it is addressed. If you are not the intended recipient of this e-mail, you are hereby notified that any copying, distribution, dissemination or action taken in relation to the contents of this e-mail and any of its attachments is strictly prohibited and may be unlawful. If you have received this e-mail in error, please notify the sender immediately and permanently delete the original e-mail and destroy any copies or printouts of this e-mail as well as any attachments.
-------------- next part -------------- An HTML attachment was scrubbed... URL:
<https://puck.nether.net/pipermail/outages/attachments/ 20140812/837627
0e/attachment-0001.html>
------------------------------
Message: 5 Date: Tue, 12 Aug 2014 15:20:45 +0000 From: "Hoyle Anderson (AM)" <hoyle.anderson@dimensiondata.com> To: "outages@outages.org" <outages@outages.org> Subject: [outages] Major outages today, not much info at this time Message-ID:
<7C6FA1D8224C7944A005E6587BDEA50A03F692@usispsvexdb03.na.didata.lo cal>
Content-Type: text/plain; charset="utf-8"
I know this isn?t much help, but there are major problems with multiple
ISPs since around 4-5 AM EST. I really don?t have much detail, but I have sites that are unreachable from some providers. Looks like Comcast, level3, ATT, cogent, etc.
So, it?s probably not just you, but I?m afraid I don?t know who it is. I heard
one report of a datacenter outage.
Sources: http://downdetector.com/ https://twitter.com/liquidweb
-------------- next part -------------- An HTML attachment was scrubbed... URL:
20140812/fa57b8
50/attachment-0001.html>
------------------------------
Message: 6 Date: Tue, 12 Aug 2014 17:42:05 +0200 From: Teun Vink <teun@teun.tv> To: "Hoyle Anderson (AM)" <hoyle.anderson@dimensiondata.com> Cc: "outages@outages.org" <outages@outages.org> Subject: Re: [outages] Major outages today, not much info at this time Message-ID: <1407858125.27157.21.camel@moridin> Content-Type: text/plain; charset="UTF-8"
On di, 2014-08-12 at 15:20 +0000, Hoyle Anderson (AM) via Outages wrote:
I know this isn?t much help, but there are major problems with multiple ISPs since around 4-5 AM EST. I really don?t have much detail, but I have sites that are unreachable from some providers. Looks like Comcast, level3, ATT, cogent, etc.
So, it?s probably not just you, but I?m afraid I don?t know who it is. I heard one report of a datacenter outage.
Hi,
Some routing tables hit 512K routes today. Some old hardware and
software can't handle that and either crash or ignore newly learned routes. So this may cause some disturbances in the force.
HTH, Teun
------------------------------
Subject: Digest Footer
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
------------------------------
End of Outages Digest, Vol 75, Issue 9 **************************************
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
On 8/12/14 12:27 PM, Bill Wichers via Outages wrote:
I'm not so sure. None of my four upstreams is showing that many. Level3 (as an example) is 497,389 right now.
This might be an issue for some places but probably isn't the full "this is why everyone is down today" explanation.
The vantage-point of as2.0 or routeviews is a little bit different then that of the average moderately well connected asn. Previous $ employers crossed the 500k fib route point around 2007 I'm sure there were plenty that were quite bit earlier than that, given when million route fibs started shipping. joel
-Bill
-----Original Message----- From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Spencer Gaw via Outages Sent: Tuesday, August 12, 2014 3:20 PM To: Michael Peterman; outages@outages.org Subject: Re: [outages] So what is broken
I would imagine it's related to the global routing table hitting 512k routes.
http://www.cidr-report.org/cgi- bin/plot?file=%2fvar%2fdata%2fbgp%2fas2.0%2fbgp%2dactive%2etxt&start =1407179655&end=1407870855&width=0%2e9&height=0%2e3&with=step&y label=BGP+entries
Regards,
SG
On 8/12/2014 12:21 PM, Michael Peterman via Outages wrote:
So is this issue all related to a fiber cut or a DC/Peering point having issues?
http://www.thewhir.com/web-hosting-news/liquidweb-among- companies-affe cted-major-outage-across-us-network-providers
Michael Peterman
-----Original Message----- From: Outages [mailto:outages-bounces@outages.org] On Behalf Of outages-request@outages.org Sent: Tuesday, August 12, 2014 11:00 AM To: outages@outages.org Subject: Outages Digest, Vol 75, Issue 9
Send Outages mailing list submissions to outages@outages.org
To subscribe or unsubscribe via the World Wide Web, visit https://puck.nether.net/mailman/listinfo/outages or, via email, send a message with subject or body 'help' to outages-request@outages.org
You can reach the person managing the list at outages-owner@outages.org
When replying, please edit your Subject line so it is more specific than "Re: Contents of Outages digest..."
Today's Topics:
1. Windstream (Mike Walter) 2. CoreSite Any2 LA (jd) 3. Re: CoreSite Any2 LA (Paul Stewart) 4. LastPass.com has a DC outage (Chris Egeland) 5. Major outages today, not much info at this time (Hoyle Anderson (AM)) 6. Re: Major outages today, not much info at this time (Teun Vink)
----------------------------------------------------------------------
Message: 1 Date: Tue, 12 Aug 2014 13:19:08 +0000 From: Mike Walter <mwalter@3z.net> To: "'outages@outages.org'" <outages@outages.org> Subject: [outages] Windstream Message-ID:
<D7092CF606B8964193F248C92106B7810FCDE655@zionex01.ZION.loc al>
Content-Type: text/plain; charset="us-ascii"
Just an FYI, there is a Windstream fiber cut effecting Ohio and Kentucky customers, no ETA to repair based on their phone message. Still on hold waiting to open case to find out more. -Mike
-------------- next part -------------- An HTML attachment was scrubbed... URL:
<https://puck.nether.net/pipermail/outages/attachments/20140812/194c6c
4b/attachment-0001.html>
------------------------------
Message: 2 Date: Tue, 12 Aug 2014 07:19:04 -0700 From: jd <jd@boldnetworks.net> To: "'outages@outages.org'" <outages@outages.org> Subject: [outages] CoreSite Any2 LA Message-ID: <53EA2258.2020601@boldnetworks.net> Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Is anyone having issues with Any2 in LA? I have no evidence but I'm hearing some things, if anyone could debunk.
Thanks JD
------------------------------
Message: 3 Date: Tue, 12 Aug 2014 10:42:39 -0400 From: Paul Stewart <paul@paulstewart.org> To: "'outages@outages.org'" <outages@outages.org> Subject: Re: [outages] CoreSite Any2 LA Message-ID: <D00FA00A.806C1%paul@paulstewart.org> Content-Type: text/plain; charset="UTF-8"
No issues here at this point?. At least that have become evident?
Paul
On 2014-08-12, 10:19 AM, "jd via Outages" <outages@outages.org> wrote:
Is anyone having issues with Any2 in LA? I have no evidence but I'm hearing some things, if anyone could debunk.
Thanks JD _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
------------------------------
Message: 4 Date: Tue, 12 Aug 2014 15:04:58 +0000 From: Chris Egeland <chris.egeland@ferrotherm.com> To: "outages@outages.org List" <outages@outages.org> Subject: [outages] LastPass.com has a DC outage Message-ID: <EA49DAFC5FDC1A40B16DEE2D6ED5469A9FD277AB@mail01> Content-Type: text/plain; charset="us-ascii"
https://twitter.com/LastPass/status/499199566701658112
"Update: One of our data centers went down, we have the service fully running on another data center & are working to minimize impact."
I'm getting "An error occurred while attempting to contact the server. Please check your Internet connection." In my LastPass bar on Firefox.
So, if you're getting errors with LastPass, this is why.
________________________________
This e-mail and any attachments may contain privileged or confidential information of Ferrotherm Corporation. This e-mail is intended solely for the use of the individual or entity to which it is addressed. If you are not the intended recipient of this e-mail, you are hereby notified that any copying, distribution, dissemination or action taken in relation to the contents of this e-mail and any of its attachments is strictly prohibited and may be unlawful. If you have received this e-mail in error, please notify the sender immediately and permanently delete the original e-mail and destroy any copies or printouts of this e-mail as well as any attachments. -------------- next part -------------- An HTML attachment was scrubbed... URL:
<https://puck.nether.net/pipermail/outages/attachments/20140812/837627
0e/attachment-0001.html>
------------------------------
Message: 5 Date: Tue, 12 Aug 2014 15:20:45 +0000 From: "Hoyle Anderson (AM)" <hoyle.anderson@dimensiondata.com> To: "outages@outages.org" <outages@outages.org> Subject: [outages] Major outages today, not much info at this time Message-ID:
<7C6FA1D8224C7944A005E6587BDEA50A03F692@usispsvexdb03.na.didata.lo cal>
Content-Type: text/plain; charset="utf-8"
I know this isn?t much help, but there are major problems with multiple ISPs since around 4-5 AM EST. I really don?t have much detail, but I have sites that are unreachable from some providers. Looks like Comcast, level3, ATT, cogent, etc.
So, it?s probably not just you, but I?m afraid I don?t know who it is. I heard one report of a datacenter outage.
Sources: http://downdetector.com/ https://twitter.com/liquidweb
-------------- next part -------------- An HTML attachment was scrubbed... URL:
<https://puck.nether.net/pipermail/outages/attachments/20140812/fa57b8
50/attachment-0001.html>
------------------------------
Message: 6 Date: Tue, 12 Aug 2014 17:42:05 +0200 From: Teun Vink <teun@teun.tv> To: "Hoyle Anderson (AM)" <hoyle.anderson@dimensiondata.com> Cc: "outages@outages.org" <outages@outages.org> Subject: Re: [outages] Major outages today, not much info at this time Message-ID: <1407858125.27157.21.camel@moridin> Content-Type: text/plain; charset="UTF-8"
On di, 2014-08-12 at 15:20 +0000, Hoyle Anderson (AM) via Outages wrote:
I know this isn?t much help, but there are major problems with multiple ISPs since around 4-5 AM EST. I really don?t have much detail, but I have sites that are unreachable from some providers. Looks like Comcast, level3, ATT, cogent, etc.
So, it?s probably not just you, but I?m afraid I don?t know who it is. I heard one report of a datacenter outage.
Hi,
Some routing tables hit 512K routes today. Some old hardware and software can't handle that and either crash or ignore newly learned routes. So this may cause some disturbances in the force.
HTH, Teun
------------------------------
Subject: Digest Footer
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
------------------------------
End of Outages Digest, Vol 75, Issue 9 **************************************
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Cisco gear can't support more than 512K routes... We hit that or close to that at around 4am EDT today. ISPs are scrambling to update memory in their routers. -------- Original message -------- From: Michael Peterman via Outages Date:08/12/2014 1:00 PM (GMT-07:00) To: outages@outages.org Subject: [outages] So what is broken So is this issue all related to a fiber cut or a DC/Peering point having issues? http://www.thewhir.com/web-hosting-news/liquidweb-among-companies-affected-m... Michael Peterman -----Original Message----- From: Outages [mailto:outages-bounces@outages.org] On Behalf Of outages-request@outages.org Sent: Tuesday, August 12, 2014 11:00 AM To: outages@outages.org Subject: Outages Digest, Vol 75, Issue 9 Send Outages mailing list submissions to outages@outages.org To subscribe or unsubscribe via the World Wide Web, visit https://puck.nether.net/mailman/listinfo/outages or, via email, send a message with subject or body 'help' to outages-request@outages.org You can reach the person managing the list at outages-owner@outages.org When replying, please edit your Subject line so it is more specific than "Re: Contents of Outages digest..." Today's Topics: 1. Windstream (Mike Walter) 2. CoreSite Any2 LA (jd) 3. Re: CoreSite Any2 LA (Paul Stewart) 4. LastPass.com has a DC outage (Chris Egeland) 5. Major outages today, not much info at this time (Hoyle Anderson (AM)) 6. Re: Major outages today, not much info at this time (Teun Vink) ---------------------------------------------------------------------- Message: 1 Date: Tue, 12 Aug 2014 13:19:08 +0000 From: Mike Walter <mwalter@3z.net> To: "'outages@outages.org'" <outages@outages.org> Subject: [outages] Windstream Message-ID: <D7092CF606B8964193F248C92106B7810FCDE655@zionex01.ZION.local> Content-Type: text/plain; charset="us-ascii" Just an FYI, there is a Windstream fiber cut effecting Ohio and Kentucky customers, no ETA to repair based on their phone message. Still on hold waiting to open case to find out more. -Mike
Only seeing 497, 554 IPv4 routes and 18,128 IPv6 routes here as of right now through L3. Chris On 08/12/2014 01:44 PM, Geoffrey Mina via Outages wrote:
Cisco gear can't support more than 512K routes... We hit that or close to that at around 4am EDT today.
ISPs are scrambling to update memory in their routers.
-------- Original message -------- From: Michael Peterman via Outages Date:08/12/2014 1:00 PM (GMT-07:00) To: outages@outages.org Subject: [outages] So what is broken
So is this issue all related to a fiber cut or a DC/Peering point having issues?
http://www.thewhir.com/web-hosting-news/liquidweb-among-companies-affected-m...
Michael Peterman
-----Original Message----- From: Outages [mailto:outages-bounces@outages.org] On Behalf Of outages-request@outages.org Sent: Tuesday, August 12, 2014 11:00 AM To: outages@outages.org Subject: Outages Digest, Vol 75, Issue 9
Send Outages mailing list submissions to outages@outages.org
To subscribe or unsubscribe via the World Wide Web, visit https://puck.nether.net/mailman/listinfo/outages or, via email, send a message with subject or body 'help' to outages-request@outages.org
You can reach the person managing the list at outages-owner@outages.org
When replying, please edit your Subject line so it is more specific than "Re: Contents of Outages digest..."
Today's Topics:
1. Windstream (Mike Walter) 2. CoreSite Any2 LA (jd) 3. Re: CoreSite Any2 LA (Paul Stewart) 4. LastPass.com has a DC outage (Chris Egeland) 5. Major outages today, not much info at this time (Hoyle Anderson (AM)) 6. Re: Major outages today, not much info at this time (Teun Vink)
----------------------------------------------------------------------
Message: 1 Date: Tue, 12 Aug 2014 13:19:08 +0000 From: Mike Walter <mwalter@3z.net> To: "'outages@outages.org'" <outages@outages.org> Subject: [outages] Windstream Message-ID: <D7092CF606B8964193F248C92106B7810FCDE655@zionex01.ZION.local> Content-Type: text/plain; charset="us-ascii"
Just an FYI, there is a Windstream fiber cut effecting Ohio and Kentucky customers, no ETA to repair based on their phone message. Still on hold waiting to open case to find out more. -Mike
If it helps anybody: SSH@MLX-SRHWT#show ip route summary IP Routing Table - 507053 entries 19 connected, 4 static, 0 RIP, 96 OSPF, 506934 BGP, 0 ISIS Number of prefixes: /0: 1 /8: 16 /9: 13 /10: 30 /11: 91 /12: 259 /13: 498 /14: 987 /15: 1713 /16: 13069 /17: 7134 /18: 11727 /19: 24996 /20: 35523 /21: 38175 /22: 55029 /23: 48036 /24: 269269 ...snip... Nexthop Table Entry - 23 entries We are AS19754 (http://bgp.he.net/AS19754) and have seen no issues today nor had any reports of issues. jwb -------------- James W. Breeden | CEO | LiveAir Networks | it's the way we think Smithville West Operations Center | 125 Kellar Rd, Suite C2 | Smithville, TX 78957 email/sip: jwb@liveair.net<mailto:jwb@liveair.net> | o: 512.360.4273 x7011 | tf: 877.360.4273 | c: 512.304.0745 | f: 866.396.8990 | www.liveair.net<http://www.liveair.net/> social net: me - twitter.com/meltedfire<http://twitter.com/jwb4273> | facebook.com/meltedfire<http://facebook.com/meltedfire> | linkedin.com/in/meltedfire<http://www.linkedin.com/in/meltedfire> social net: liveair - twitter.com/liveair<http://twitter.com/liveair> | facebook.com/liveair<http://facebook.com/liveair> internet and data services . infosys building blocks . digital phone . managed services . wholesale LiveAir Networks: Proven Better Value. More Reliable Service. Enhanced Local Support. Shop Local.
Be environmentally responsible. Only print this email when necessary.
"Here's to the crazy ones, the misfits, the rebels, the troublemakers, the round pegs in the square holes... the ones who see things differently -- they're not fond of rules... You can quote them, disagree with them, glorify or vilify them, but the only thing you can't do is ignore them because they change things... they push the human race forward, and while some may see them as the crazy ones, we see genius, because the ones who are crazy enough to think that they can change the world, are the ones who do." - Steve Jobs From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Christopher L. Stone via Outages Sent: Tuesday, August 12, 2014 3:38 PM To: outages@outages.org<mailto:outages@outages.org> Subject: Re: [outages] So what is broken Only seeing 497, 554 IPv4 routes and 18,128 IPv6 routes here as of right now through L3. Chris On 08/12/2014 01:44 PM, Geoffrey Mina via Outages wrote: Cisco gear can't support more than 512K routes... We hit that or close to that at around 4am EDT today. ISPs are scrambling to update memory in their routers. -------- Original message -------- From: Michael Peterman via Outages Date:08/12/2014 1:00 PM (GMT-07:00) To: outages@outages.org<mailto:outages@outages.org> Subject: [outages] So what is broken So is this issue all related to a fiber cut or a DC/Peering point having issues? http://www.thewhir.com/web-hosting-news/liquidweb-among-companies-affected-m... Michael Peterman -----Original Message----- From: Outages [mailto:outages-bounces@outages.org] On Behalf Of outages-request@outages.org<mailto:outages-request@outages.org> Sent: Tuesday, August 12, 2014 11:00 AM To: outages@outages.org<mailto:outages@outages.org> Subject: Outages Digest, Vol 75, Issue 9 Send Outages mailing list submissions to outages@outages.org<mailto:outages@outages.org> To subscribe or unsubscribe via the World Wide Web, visit https://puck.nether.net/mailman/listinfo/outages or, via email, send a message with subject or body 'help' to outages-request@outages.org<mailto:outages-request@outages.org> You can reach the person managing the list at outages-owner@outages.org<mailto:outages-owner@outages.org> When replying, please edit your Subject line so it is more specific than "Re: Contents of Outages digest..." Today's Topics: 1. Windstream (Mike Walter) 2. CoreSite Any2 LA (jd) 3. Re: CoreSite Any2 LA (Paul Stewart) 4. LastPass.com has a DC outage (Chris Egeland) 5. Major outages today, not much info at this time (Hoyle Anderson (AM)) 6. Re: Major outages today, not much info at this time (Teun Vink) ---------------------------------------------------------------------- Message: 1 Date: Tue, 12 Aug 2014 13:19:08 +0000 From: Mike Walter <mwalter@3z.net><mailto:mwalter@3z.net> To: "'outages@outages.org'"<mailto:'outages@outages.org'> <outages@outages.org><mailto:outages@outages.org> Subject: [outages] Windstream Message-ID: <D7092CF606B8964193F248C92106B7810FCDE655@zionex01.ZION.local><mailto:D7092CF606B8964193F248C92106B7810FCDE655@zionex01.ZION.local> Content-Type: text/plain; charset="us-ascii" Just an FYI, there is a Windstream fiber cut effecting Ohio and Kentucky customers, no ETA to repair based on their phone message. Still on hold waiting to open case to find out more. -Mike
On 12/08/2014 22:38, Christopher L. Stone via Outages wrote:
Only seeing 497, 554 IPv4 routes and 18,128 IPv6 routes here as of right now through L3.
Yesterday at 11PM (GMT+2), my IPv4/v6 sessions with L3 were not getting any announcements (from router cdg2.gblx.net).
Caron, this might have been because you are still connected to one of the older GC 7609 aggregation switches and it couldn't handle the full RIB table for the Internet routes. Most of those 7609's should have been replaced, you should contact Level 3 and ask when you are getting moved to the new Aggregation switches? Regards, Joe Sanchez (Please excuse the brevity, misspelled words and sentence structure of this email as it was sent via a supposed smart device.)
On Aug 13, 2014, at 1:01 AM, Laurent CARON via Outages <outages@outages.org> wrote:
On 12/08/2014 22:38, Christopher L. Stone via Outages wrote: Only seeing 497, 554 IPv4 routes and 18,128 IPv6 routes here as of right now through L3.
Yesterday at 11PM (GMT+2), my IPv4/v6 sessions with L3 were not getting any announcements (from router cdg2.gblx.net).
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Looks like "full route" routing tables peaked at ~515k routes briefly due to an aggregation issue at Verizon. We saw the increase in several of our full table feeds from Level3 today. More details here: http://www.bgpmon.net/what-caused-todays-internet-hiccup/ Cheers, Andree .-- My secret spy satellite informs me that at 2014-08-12 1:38 PM Christopher L. Stone via Outages wrote:
Only seeing 497, 554 IPv4 routes and 18,128 IPv6 routes here as of right now through L3.
Chris
On 08/12/2014 01:44 PM, Geoffrey Mina via Outages wrote:
Cisco gear can't support more than 512K routes... We hit that or close to that at around 4am EDT today.
ISPs are scrambling to update memory in their routers.
-------- Original message -------- From: Michael Peterman via Outages Date:08/12/2014 1:00 PM (GMT-07:00) To: outages@outages.org Subject: [outages] So what is broken
So is this issue all related to a fiber cut or a DC/Peering point having issues?
http://www.thewhir.com/web-hosting-news/liquidweb-among-companies-affected-m...
Michael Peterman
-----Original Message----- From: Outages [mailto:outages-bounces@outages.org] On Behalf Of outages-request@outages.org Sent: Tuesday, August 12, 2014 11:00 AM To: outages@outages.org Subject: Outages Digest, Vol 75, Issue 9
Send Outages mailing list submissions to outages@outages.org
To subscribe or unsubscribe via the World Wide Web, visit https://puck.nether.net/mailman/listinfo/outages or, via email, send a message with subject or body 'help' to outages-request@outages.org
You can reach the person managing the list at outages-owner@outages.org
When replying, please edit your Subject line so it is more specific than "Re: Contents of Outages digest..."
Today's Topics:
1. Windstream (Mike Walter) 2. CoreSite Any2 LA (jd) 3. Re: CoreSite Any2 LA (Paul Stewart) 4. LastPass.com has a DC outage (Chris Egeland) 5. Major outages today, not much info at this time (Hoyle Anderson (AM)) 6. Re: Major outages today, not much info at this time (Teun Vink)
----------------------------------------------------------------------
Message: 1 Date: Tue, 12 Aug 2014 13:19:08 +0000 From: Mike Walter <mwalter@3z.net> To: "'outages@outages.org'" <outages@outages.org> Subject: [outages] Windstream Message-ID: <D7092CF606B8964193F248C92106B7810FCDE655@zionex01.ZION.local> Content-Type: text/plain; charset="us-ascii"
Just an FYI, there is a Windstream fiber cut effecting Ohio and Kentucky customers, no ETA to repair based on their phone message. Still on hold waiting to open case to find out more. -Mike
That's a tcam limit on the 6500/7600 platform. It's not upgradable, but it is configurable. The gsr platform doesn't have a hard limit like this, its just memory (And is upgradeable), but most with that gear will have already had to upgrade well before now. Sent from my iPhone On Aug 12, 2014, at 4:14 PM, "Geoffrey Mina via Outages" <outages@outages.org<mailto:outages@outages.org>> wrote: Cisco gear can't support more than 512K routes... We hit that or close to that at around 4am EDT today. ISPs are scrambling to update memory in their routers. -------- Original message -------- From: Michael Peterman via Outages Date:08/12/2014 1:00 PM (GMT-07:00) To: outages@outages.org<mailto:outages@outages.org> Subject: [outages] So what is broken So is this issue all related to a fiber cut or a DC/Peering point having issues? http://www.thewhir.com/web-hosting-news/liquidweb-among-companies-affected-m... Michael Peterman -----Original Message----- From: Outages [mailto:outages-bounces@outages.org] On Behalf Of outages-request@outages.org<mailto:outages-request@outages.org> Sent: Tuesday, August 12, 2014 11:00 AM To: outages@outages.org<mailto:outages@outages.org> Subject: Outages Digest, Vol 75, Issue 9 Send Outages mailing list submissions to outages@outages.org<mailto:outages@outages.org> To subscribe or unsubscribe via the World Wide Web, visit https://puck.nether.net/mailman/listinfo/outages or, via email, send a message with subject or body 'help' to outages-request@outages.org<mailto:outages-request@outages.org> You can reach the person managing the list at outages-owner@outages.org<mailto:outages-owner@outages.org> When replying, please edit your Subject line so it is more specific than "Re: Contents of Outages digest..." Today's Topics: 1. Windstream (Mike Walter) 2. CoreSite Any2 LA (jd) 3. Re: CoreSite Any2 LA (Paul Stewart) 4. LastPass.com<http://LastPass.com> has a DC outage (Chris Egeland) 5. Major outages today, not much info at this time (Hoyle Anderson (AM)) 6. Re: Major outages today, not much info at this time (Teun Vink) ---------------------------------------------------------------------- Message: 1 Date: Tue, 12 Aug 2014 13:19:08 +0000 From: Mike Walter <mwalter@3z.net<mailto:mwalter@3z.net>> To: "'outages@outages.org<mailto:outages@outages.org>'" <outages@outages.org<mailto:outages@outages.org>> Subject: [outages] Windstream Message-ID: <D7092CF606B8964193F248C92106B7810FCDE655@zionex01.ZION.local<mailto:D7092CF606B8964193F248C92106B7810FCDE655@zionex01.ZION.local>> Content-Type: text/plain; charset="us-ascii" Just an FYI, there is a Windstream fiber cut effecting Ohio and Kentucky customers, no ETA to repair based on their phone message. Still on hold waiting to open case to find out more. -Mike
We were told by Quality Technology Services in Atlanta that they were "installing additional memory" to correct the issue. Perhaps they meant they were actually reconfiguring/bouncing the routers.... I know they had Cisco engineers on site in GA, CA, and NJ today. Either way, after our network link went totally down and came back up, the issue was resolved. ________________________________ From: Bill Wichers [billw@waveform.net] Sent: Tuesday, August 12, 2014 3:27 PM To: Geoffrey Mina Cc: Michael Peterman; outages@outages.org Subject: Re: [outages] So what is broken That's a tcam limit on the 6500/7600 platform. It's not upgradable, but it is configurable. The gsr platform doesn't have a hard limit like this, its just memory (And is upgradeable), but most with that gear will have already had to upgrade well before now. Sent from my iPhone On Aug 12, 2014, at 4:14 PM, "Geoffrey Mina via Outages" <outages@outages.org<mailto:outages@outages.org>> wrote: Cisco gear can't support more than 512K routes... We hit that or close to that at around 4am EDT today. ISPs are scrambling to update memory in their routers. -------- Original message -------- From: Michael Peterman via Outages Date:08/12/2014 1:00 PM (GMT-07:00) To: outages@outages.org<mailto:outages@outages.org> Subject: [outages] So what is broken So is this issue all related to a fiber cut or a DC/Peering point having issues? http://www.thewhir.com/web-hosting-news/liquidweb-among-companies-affected-m... Michael Peterman -----Original Message----- From: Outages [mailto:outages-bounces@outages.org] On Behalf Of outages-request@outages.org<mailto:outages-request@outages.org> Sent: Tuesday, August 12, 2014 11:00 AM To: outages@outages.org<mailto:outages@outages.org> Subject: Outages Digest, Vol 75, Issue 9 Send Outages mailing list submissions to outages@outages.org<mailto:outages@outages.org> To subscribe or unsubscribe via the World Wide Web, visit https://puck.nether.net/mailman/listinfo/outages or, via email, send a message with subject or body 'help' to outages-request@outages.org<mailto:outages-request@outages.org> You can reach the person managing the list at outages-owner@outages.org<mailto:outages-owner@outages.org> When replying, please edit your Subject line so it is more specific than "Re: Contents of Outages digest..." Today's Topics: 1. Windstream (Mike Walter) 2. CoreSite Any2 LA (jd) 3. Re: CoreSite Any2 LA (Paul Stewart) 4. LastPass.com<http://LastPass.com> has a DC outage (Chris Egeland) 5. Major outages today, not much info at this time (Hoyle Anderson (AM)) 6. Re: Major outages today, not much info at this time (Teun Vink) ---------------------------------------------------------------------- Message: 1 Date: Tue, 12 Aug 2014 13:19:08 +0000 From: Mike Walter <mwalter@3z.net<mailto:mwalter@3z.net>> To: "'outages@outages.org<mailto:outages@outages.org>'" <outages@outages.org<mailto:outages@outages.org>> Subject: [outages] Windstream Message-ID: <D7092CF606B8964193F248C92106B7810FCDE655@zionex01.ZION.local<mailto:D7092CF606B8964193F248C92106B7810FCDE655@zionex01.ZION.local>> Content-Type: text/plain; charset="us-ascii" Just an FYI, there is a Windstream fiber cut effecting Ohio and Kentucky customers, no ETA to repair based on their phone message. Still on hold waiting to open case to find out more. -Mike
On Tue, Aug 12, 2014 at 5:29 PM, Geoffrey Mina via Outages <outages@outages.org> wrote:
We were told by Quality Technology Services in Atlanta that they were "installing additional memory" to correct the issue. Perhaps they meant they were actually reconfiguring/bouncing the routers.... I know they had Cisco engineers on site in GA, CA, and NJ today.
Probably they were recarving the TCAM allocation. Quick summary document for those having the issue: http://www.cisco.com/c/en/us/support/docs/switches/catalyst-6500-series-swit... This increases the size of the TCAM allocated for v4 routes, at the cost of v6 space. This doesn't actually *solve* the issue, but does at least kick the can down the road a bit longer... TCAM based solutions for FIB, while cheap to implement, leads to pain... W
Either way, after our network link went totally down and came back up, the issue was resolved.
________________________________ From: Bill Wichers [billw@waveform.net] Sent: Tuesday, August 12, 2014 3:27 PM To: Geoffrey Mina Cc: Michael Peterman; outages@outages.org Subject: Re: [outages] So what is broken
That's a tcam limit on the 6500/7600 platform. It's not upgradable, but it is configurable.
The gsr platform doesn't have a hard limit like this, its just memory (And is upgradeable), but most with that gear will have already had to upgrade well before now.
Sent from my iPhone
On Aug 12, 2014, at 4:14 PM, "Geoffrey Mina via Outages" <outages@outages.org> wrote:
Cisco gear can't support more than 512K routes... We hit that or close to that at around 4am EDT today.
ISPs are scrambling to update memory in their routers.
-------- Original message -------- From: Michael Peterman via Outages Date:08/12/2014 1:00 PM (GMT-07:00) To: outages@outages.org Subject: [outages] So what is broken
So is this issue all related to a fiber cut or a DC/Peering point having issues?
http://www.thewhir.com/web-hosting-news/liquidweb-among-companies-affected-m...
Michael Peterman
-----Original Message----- From: Outages [mailto:outages-bounces@outages.org] On Behalf Of outages-request@outages.org Sent: Tuesday, August 12, 2014 11:00 AM To: outages@outages.org Subject: Outages Digest, Vol 75, Issue 9
Send Outages mailing list submissions to outages@outages.org
To subscribe or unsubscribe via the World Wide Web, visit https://puck.nether.net/mailman/listinfo/outages or, via email, send a message with subject or body 'help' to outages-request@outages.org
You can reach the person managing the list at outages-owner@outages.org
When replying, please edit your Subject line so it is more specific than "Re: Contents of Outages digest..."
Today's Topics:
1. Windstream (Mike Walter) 2. CoreSite Any2 LA (jd) 3. Re: CoreSite Any2 LA (Paul Stewart) 4. LastPass.com has a DC outage (Chris Egeland) 5. Major outages today, not much info at this time (Hoyle Anderson (AM)) 6. Re: Major outages today, not much info at this time (Teun Vink)
----------------------------------------------------------------------
Message: 1 Date: Tue, 12 Aug 2014 13:19:08 +0000 From: Mike Walter <mwalter@3z.net> To: "'outages@outages.org'" <outages@outages.org> Subject: [outages] Windstream Message-ID: <D7092CF606B8964193F248C92106B7810FCDE655@zionex01.ZION.local> Content-Type: text/plain; charset="us-ascii"
Just an FYI, there is a Windstream fiber cut effecting Ohio and Kentucky customers, no ETA to repair based on their phone message. Still on hold waiting to open case to find out more. -Mike
The sup2t-xl for the 6500 series should not affected by this. The 1M TCAM space is shared and not default configured for 512k. On August 12, 2014 at 6:32:27 PM, Bill Wichers via Outages (outages@outages.org) wrote: That's a tcam limit on the 6500/7600 platform. It's not upgradable, but it is configurable. The gsr platform doesn't have a hard limit like this, its just memory (And is upgradeable), but most with that gear will have already had to upgrade well before now. Sent from my iPhone On Aug 12, 2014, at 4:14 PM, "Geoffrey Mina via Outages" <outages@outages.org> wrote: Cisco gear can't support more than 512K routes... We hit that or close to that at around 4am EDT today. ISPs are scrambling to update memory in their routers. -------- Original message -------- From: Michael Peterman via Outages Date:08/12/2014 1:00 PM (GMT-07:00) To: outages@outages.org Subject: [outages] So what is broken So is this issue all related to a fiber cut or a DC/Peering point having issues? http://www.thewhir.com/web-hosting-news/liquidweb-among-companies-affected-m... Michael Peterman -----Original Message----- From: Outages [mailto:outages-bounces@outages.org] On Behalf Of outages-request@outages.org Sent: Tuesday, August 12, 2014 11:00 AM To: outages@outages.org Subject: Outages Digest, Vol 75, Issue 9 Send Outages mailing list submissions to outages@outages.org To subscribe or unsubscribe via the World Wide Web, visit https://puck.nether.net/mailman/listinfo/outages or, via email, send a message with subject or body 'help' to outages-request@outages.org You can reach the person managing the list at outages-owner@outages.org When replying, please edit your Subject line so it is more specific than "Re: Contents of Outages digest..." Today's Topics: 1. Windstream (Mike Walter) 2. CoreSite Any2 LA (jd) 3. Re: CoreSite Any2 LA (Paul Stewart) 4. LastPass.com has a DC outage (Chris Egeland) 5. Major outages today, not much info at this time (Hoyle Anderson (AM)) 6. Re: Major outages today, not much info at this time (Teun Vink) ---------------------------------------------------------------------- Message: 1 Date: Tue, 12 Aug 2014 13:19:08 +0000 From: Mike Walter <mwalter@3z.net> To: "'outages@outages.org'" <outages@outages.org> Subject: [outages] Windstream Message-ID: <D7092CF606B8964193F248C92106B7810FCDE655@zionex01.ZION.local> Content-Type: text/plain; charset="us-ascii" Just an FYI, there is a Windstream fiber cut effecting Ohio and Kentucky customers, no ETA to repair based on their phone message. Still on hold waiting to open case to find out more. -Mike
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 I noticed that Cisco published this blog entry today: http://blogs.cisco.com/sp/global-internet-routing-table-reaches-512k-milesto... FYI, - - ferg On 8/13/2014 3:21 PM, Adam Kaminski via Outages wrote:
The sup2t-xl for the 6500 series should not affected by this. The 1M TCAM space is shared and not default configured for 512k.
On August 12, 2014 at 6:32:27 PM, Bill Wichers via Outages (outages@outages.org <mailto:outages@outages.org>) wrote:
That's a tcam limit on the 6500/7600 platform. It's not upgradable, but it is configurable.
The gsr platform doesn't have a hard limit like this, its just memory (And is upgradeable), but most with that gear will have already had to upgrade well before now.
Sent from my iPhone
On Aug 12, 2014, at 4:14 PM, "Geoffrey Mina via Outages" <outages@outages.org <mailto:outages@outages.org>> wrote:
Cisco gear can't support more than 512K routes... We hit that or close to that at around 4am EDT today.
ISPs are scrambling to update memory in their routers.
-------- Original message -------- From: Michael Peterman via Outages Date:08/12/2014 1:00 PM (GMT-07:00) To: outages@outages.org <mailto:outages@outages.org> Subject: [outages] So what is broken
So is this issue all related to a fiber cut or a DC/Peering point having issues?
http://www.thewhir.com/web-hosting-news/liquidweb-among-companies-affected-m...
Michael Peterman
- -- Paul Ferguson VP Threat Intelligence, IID PGP Public Key ID: 0x54DC85B2 Key fingerprint: 19EC 2945 FEE8 D6C8 58A1 CE53 2896 AC75 54DC 85B2 -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (MingW32) iF4EAREIAAYFAlPr7ZIACgkQKJasdVTchbJhMAEAhN29FzHxGFZ8uxpehiYttF8w 38jN8uiIwsWZ16tMB7EBALf62Ubg5gfi7gUn3A5uv5TTGVO+Ulr5O80+GJFDjDbk =iT32 -----END PGP SIGNATURE-----
Looks like the internet route table got over 512k routes. Guessing some 7600's did not like that. On Aug 12, 2014 1:57 PM, "Michael Peterman via Outages" <outages@outages.org> wrote:
So is this issue all related to a fiber cut or a DC/Peering point having issues?
http://www.thewhir.com/web-hosting-news/liquidweb-among-companies-affected-m...
Michael Peterman
-----Original Message----- From: Outages [mailto:outages-bounces@outages.org] On Behalf Of outages-request@outages.org Sent: Tuesday, August 12, 2014 11:00 AM To: outages@outages.org Subject: Outages Digest, Vol 75, Issue 9
Send Outages mailing list submissions to outages@outages.org
To subscribe or unsubscribe via the World Wide Web, visit https://puck.nether.net/mailman/listinfo/outages or, via email, send a message with subject or body 'help' to outages-request@outages.org
You can reach the person managing the list at outages-owner@outages.org
When replying, please edit your Subject line so it is more specific than "Re: Contents of Outages digest..."
Today's Topics:
1. Windstream (Mike Walter) 2. CoreSite Any2 LA (jd) 3. Re: CoreSite Any2 LA (Paul Stewart) 4. LastPass.com has a DC outage (Chris Egeland) 5. Major outages today, not much info at this time (Hoyle Anderson (AM)) 6. Re: Major outages today, not much info at this time (Teun Vink)
----------------------------------------------------------------------
Message: 1 Date: Tue, 12 Aug 2014 13:19:08 +0000 From: Mike Walter <mwalter@3z.net> To: "'outages@outages.org'" <outages@outages.org> Subject: [outages] Windstream Message-ID: <D7092CF606B8964193F248C92106B7810FCDE655@zionex01.ZION.local> Content-Type: text/plain; charset="us-ascii"
Just an FYI, there is a Windstream fiber cut effecting Ohio and Kentucky customers, no ETA to repair based on their phone message. Still on hold waiting to open case to find out more. -Mike
participants (17)
-
Adam Kaminski -
Andree Toonk -
Bill Wichers -
Breeden, James W. -
Christopher L. Stone -
Eric Rees -
Geoffrey Mina -
Joe Sanchez -
joel jaeggli -
Laurent CARON -
Michael Hare -
Michael Peterman -
Paul Ferguson -
Sean Hayes -
Smith, Donald -
Spencer Gaw -
Warren Kumari