March 2019 - Outages - Outages Lists

Re: [outages] Equinix Chicago IX IPv4 Outage
by Brock Richards 27 Mar '19

27 Mar '19

Not an Equinix issue..... AS32703 -> AS36280 -> AS6327 -> 208.115.136.0/23 Shaw (AS6327) is advertising me 208.115.136.0/23 on their transit which is overriding Equinix (208.115.136.0/22). This looks to have come from Shaw's upstream AS36280 and then from AS32703. My filters didn't catch it, shame on me. Thanks, ---------------------------------------- Brock Richards Senior Advisor Network & Telecom Nutrien Ltd Saskatoon, SK T: (306) 717-1316 www.nutrien.com<http://www.nutrien.com/> [cid:image001.jpg@01D382D3.48EA47A0] Nutrien wants to peer with you! http://as393891.peeringdb.com<http://as393891.peeringdb.com/> http://as137945.peeringdb.com<http://as137945.peeringdb.com/> Happy Peering! From: Jake Zack [mailto:jake.zack@cira.ca] Sent: Wednesday, March 27, 2019 12:11 PM To: Brock Richards <Brock.Richards(a)nutrien.com> Subject: [EXT] RE: Equinix Chicago IX IPv4 Outage WARNING: This email originated from outside of the organization. Exercise caution when viewing attachments, clicking links, or responding to requests. Our v4 sessions @ Equinix Chicago are still up. -jake From: Outages <outages-bounces(a)outages.org<mailto:outages-bounces@outages.org>> On Behalf Of Brock Richards via Outages Sent: March-27-19 2:09 PM To: outages(a)outages.org<mailto:outages@outages.org> Subject: [outages] Equinix Chicago IX IPv4 Outage Hello, All our IPv4 sessions In Chicago Equinix have gone down but IPv6 remains. Anyone else seeing this? Thanks, ---------------------------------------- Brock Richards Senior Advisor Network & Telecom Nutrien Ltd Saskatoon, SK T: (306) 717-1316 www.nutrien.com<http://www.nutrien.com/> [cid:image001.jpg@01D382D3.48EA47A0] Nutrien wants to peer with you! http://as393891.peeringdb.com<https://urldefense.proofpoint.com/v2/url?u=http-3A__as393891.peeringdb.com_…> http://as137945.peeringdb.com<https://urldefense.proofpoint.com/v2/url?u=http-3A__as137945.peeringdb.com_…> Happy Peering! For more information on Nutrien's email policy or to unsubscribe, click here: https://www.nutrien.com/important-notice Pour plus de renseignements sur la politique de courrier électronique de Nutrien ou pour vous désabonner, cliquez ici: https://www.nutrien.com/avis-important For more information on Nutrien's email policy or to unsubscribe, click here: https://www.nutrien.com/important-notice Pour plus de renseignements sur la politique de courrier électronique de Nutrien ou pour vous désabonner, cliquez ici: https://www.nutrien.com/avis-important

1 0

Equinix Chicago IX IPv4 Outage
by Brock Richards 27 Mar '19

27 Mar '19

Hello, All our IPv4 sessions In Chicago Equinix have gone down but IPv6 remains. Anyone else seeing this? Thanks, ---------------------------------------- Brock Richards Senior Advisor Network & Telecom Nutrien Ltd Saskatoon, SK T: (306) 717-1316 www.nutrien.com<http://www.nutrien.com/> [cid:image001.jpg@01D382D3.48EA47A0] Nutrien wants to peer with you! http://as393891.peeringdb.com<http://as393891.peeringdb.com/> http://as137945.peeringdb.com<http://as137945.peeringdb.com/> Happy Peering! For more information on Nutrien's email policy or to unsubscribe, click here: https://www.nutrien.com/important-notice Pour plus de renseignements sur la politique de courrier ?lectronique de Nutrien ou pour vous d?sabonner, cliquez ici: https://www.nutrien.com/avis-important

3 2

Cloudwatch outage?
by Nathan Sinton 27 Mar '19

27 Mar '19

Cloudwatch appears to be down in us-west-2. No graphs will load and API calls return a 503 error. Is anyone else seeing issues?

3 3

Cox outage?
by Rob Hall 27 Mar '19

27 Mar '19

I'm getting reports that there's a nationwide Cox outage ; anyone else seeing this? Thanks! -- Rob Hall, CCNP R&S, CCP-V, CCA-N I.T. Manager @Work Personnel 1040 Tidewater Court Kingsport, TN 37660 Office: (423)272-7897, x10113 Mobile: (423)312-6003

5 6

Amazon AS16509 Equinix Chicago IX issues?
by Erich Kaiser 25 Mar '19

25 Mar '19

Anyone else see them not sending routes via the IX? Sessions are up but no routes coming from them. Looks to me traffic shifted around 8:30am CST. Erich Kaiser The Fusion Network www.gotfusion.net ASN 19754

2 2

L3/CenturyLink - Cuts in NC and FL
by Alex Lembesis 22 Mar '19

22 Mar '19

Rough week... We have two diverse paths on unprotected services between PA and FL that are currently down. CenturyLink is reporting fiber cuts on both. First Cut.... 03/22/2019 10:45:28 GMT - *** CASCADED EXTERNAL NOTES 22-Mar-2019 10:42:19 GMT From CASE: 16079534 - Event Splice repair has commenced at the second damage site and is ongoing at this time. The field crew is experiencing some delays due to water in the excavation area that must be routinely pumped out of the work area for fiber repairs to be made. An ETR remains unavailable. 03/22/2019 09:32:10 GMT - *** CASCADED EXTERNAL NOTES 22-Mar-2019 09:28:55 GMT From CASE: 16079534 - Event Splicing is complete at the first point of damage. Approximately 1000 ft of replacement cable is currently being pulled into a manhole and laid into proper placement for splice repair to commence at the next damage site. 03/22/2019 08:22:38 GMT - *** CASCADED EXTERNAL NOTES 22-Mar-2019 08:19:27 GMT From CASE: 16079534 - Event Splice repair of the affected fiber is ongoing at this time. An ETR remains unavailable. 03/22/2019 07:18:03 GMT - *** CASCADED EXTERNAL NOTES 22-Mar-2019 07:14:49 GMT From CASE: 16079534 - Event Repair of the damaged fiber span has commenced. An estimated time of restoral (ETR) is currently unavailable. 03/22/2019 06:15:28 GMT - *** CASCADED EXTERNAL NOTES 22-Mar-2019 06:12:18 GMT From CASE: 16079534 - Event The local providers field crew has exposed the affected span. The 98-count fiber was found to be damaged in multiple places. Analysis of the damage is currently underway to determine the best course of repair. 03/22/2019 05:01:02 GMT - *** CASCADED EXTERNAL NOTES 22-Mar-2019 04:57:51 GMT From CASE: 16079534 - Event Excavation efforts to expose the affected span are ongoing at this time. 03/22/2019 04:01:26 GMT - *** CASCADED EXTERNAL NOTES 22-Mar-2019 03:58:05 GMT From CASE: 16079534 - Event The local provider reports that the construction crew continues to excavate the terrain. Personnel have advised that fiber span is deeper than expected which is prolonging the exposure of the fiber cable. 03/22/2019 03:05:49 GMT - *** CASCADED EXTERNAL NOTES 22-Mar-2019 03:02:29 GMT From CASE: 16079534 - Event The local provider reported that construction crews continue to excavate the terrain. The Transport NOC has escalated with the local provider as an effort to obtain more information. 03/22/2019 02:06:28 GMT - *** CASCADED EXTERNAL NOTES 22-Mar-2019 02:03:08 GMT From CASE: 16079534 - Event The Transport NOC has been engaged the local provider for a status update. As such, more information will be provided momentarily. 03/22/2019 01:12:38 GMT - *** CASCADED EXTERNAL NOTES 22-Mar-2019 01:09:11 GMT From CASE: 16079534 - Event The local provider reports that a construction crew is on site and is currently excavating both ends of the fiber damage. Once excavation is completed, two new splice enclosures along with replacement cabling will be installed. 03/22/2019 00:29:42 GMT - *** CASCADED EXTERNAL NOTES 22-Mar-2019 00:26:23 GMT From CASE: 16079534 - Event The local provider reports that the ETA of the splicing crew remains unavailable at this time. 03/21/2019 23:42:24 GMT - *** CASCADED EXTERNAL NOTES 21-Mar-2019 23:39:16 GMT From CASE: 16079534 - Event The Transport NOC reports that the local provider has engaged and dispatched a splicing crew to the fault location to begin the repairs. However, an ETA for splicing personnel to be on-site is currently unavailable. 03/21/2019 22:45:37 GMT - *** CASCADED EXTERNAL NOTES 21-Mar-2019 22:42:25 GMT From CASE: 16079534 - Event The local provider reports that the field crew arrive at the site in Dunn, NC and identified a fiber cut that was caused a construction crew who was performing a directional boring operation. Onsite personnel at this time are currently assessing the fiber damage further to unveil the extent of the repair required. 03/21/2019 21:48:14 GMT - *** CASCADED EXTERNAL NOTES 21-Mar-2019 21:45:08 GMT From CASE: 16079534 - Event The local provider reports that additional fiber testing has further isolated the fault towards Dunn, NC. As such, the local providers field crew has dispatch to Dunn, NC to continue isolation; however, an ETA is currently unavailable. 03/21/2019 20:47:47 GMT - *** CASCADED EXTERNAL NOTES 21-Mar-2019 20:44:30 GMT From CASE: 16079534 - Event The local provider reports that the field crew has arrived at a site located in Benson, NC and additional light level fiber testing is underway. 03/21/2019 19:46:12 GMT - *** CASCADED EXTERNAL NOTES 21-Mar-2019 19:43:03 GMT From CASE: 16079534 - Event The Transport NOC reports that the local provider has isolated a potential fiber cut on their network. The local provider has engaged and dispatched field crews to suspected fault location for onsite isolation; however, an ETA is currently unavailable. 03/21/2019 19:02:30 GMT - *** CASCADED EXTERNAL NOTES 21-Mar-2019 18:47:59 GMT From CASE: 16079534 - Event The Transport NOC was engaged, and initial investigation suggested a fiber cut was impacting unprotected services. As such, Field Operations were engaged, and personnel have completed initial light level fiber testing. The test results isolated a fault approximately 14,000ft from the testing facility towards a local provider's network. At this time, the Transport NOC has engaged the associated local provider to investigate their network for any potential faults. Second Cut... 03/22/2019 14:32:01 GMT - *** CASCADED EXTERNAL NOTES 22-Mar-2019 14:21:09 GMT From CASE: 16085394 - Event Field Operations tested the fiber and it appears that the issue a partial fiber damage 60 miles from the testing site. Field Operations are enroute to the other side of the span to test from that direction and investigate for good spare fibers to migrate services. Field Operations gave an estimated time of arrival to the other site of 17:00 GMT. The next update will be provided when Field Operations arrives on site or more information becomes available. 03/22/2019 13:49:08 GMT - 03/22/2019 13:17:19 GMT - The Transport NOC engaged Field Operations who traveled to the site and is setting up to begin performing light level measurements to determine the location of the fiber issue. 03/22/2019 12:56:16 GMT - On March 22, 2019, at 12:37 GMT, CenturyLink identified a service impact between Bradenton and Tampa, FL. The NOC is engaged and investigating in order to isolate the cause. Please be advised that updates for this event will be relayed at a minimum of hourly unless otherwise noted. The information conveyed hereafter is associated to live troubleshooting effort and as the discovery process evolves through to service resolution, ticket closure, or post incident review, details may evolve. This message is intended solely for the designated recipient(s). It may contain confidential or proprietary information and may be subject to attorney-client privilege or other confidentiality protections. If you are not a designated recipient you may not review, copy or distribute this message. If you receive this in error, please notify the sender by reply e-mail and delete this message. Thank you.

2 1

Calls to China.
by Adam Moss 21 Mar '19

21 Mar '19

Seeing almost every carrier fail calls into China this morning, has anyone seen any official statements from any big carriers that could be referenced, or any news coverage? Feel free to reply offlist. Thanks. ADAM MOSSTechnical Account Manager | Sales Engineer Customer Successwww.flowroute.com

4 3

L3/CenturyLink Memphis/Nashville Tennessee Unreachable
by Benjamin Wesemann 20 Mar '19

20 Mar '19

Had multiple circuits drop connectivity due to unreachability outbound. My ICMP packets get a return of unreachable from L3/Clink when attempting a connection to AWS and Google DNS. Anyone else seeing the same?

1 0

Zayo issues in EU?
by netops＠totaluptime.com 19 Mar '19

19 Mar '19

Is anyone in Europe seeing issues with Zayo AS6461? We do not have transit or peering directly with them, but when traffic goes through their network, we're seeing it blackhole, especially to British Telecom AS2856, but perhaps others. This started just before noon UTC. A trace to an affected destination goes one or two hops into their network before dying. Thanks! Charles H. <http://totaluptime.com/> Network Operations Team Queue Direct: # below, option 3 NA: +1 800.584.1514 EU: +44 (0)330.808.0228 <https://www.facebook.com/totaluptime> <https://www.linkedin.com/company/total-uptime-technologies-llc> <https://twitter.com/totaluptime> <https://plus.google.com/+Totaluptime> DISCLAIMER: Privileged/Confidential Information is contained in this message. This email is intended only for the addressee indicated in this message (or party responsible for delivery of the message to such person). This email message and its information may not be publicly copied, shared, pasted, forwarded or delivered to any other individual, business entity, online forum or community without the written consent of the sender. If you are not the addressee (or party responsible for delivery of the message to such person), you should destroy this message and kindly notify the sender by reply email. Please advise immediately if you or your employer does not consent to email or messages of this kind. Opinions, conclusions and other information in this message that do not relate to the official business of the firm or its subsidiaries or business units shall be understood as neither given nor endorsed by it.

1 0

Facebook and Instagram are down for some users
by Robbie Trencheny 15 Mar '19

15 Mar '19

Seeing broken pages here in Oakland, CA. https://www.theverge.com/2019/3/13/18264092/facebook-instagram-down-partial… -- Robbie Trencheny (@robbie <http://twitter.com/robbie>) 925-884-3728 robbie.io

7 7