There is a Level 3 cut in Houston, TX, of a 168-count and a 96-count cable, that is apparently impacting lots of east-west traffic. MN seems a bit far from Houston, TX, to be impacted, but perhaps? *** CASCADED EXTERNAL NOTES 31-Jan-2017 17:48:35 GMT From CASE: 12152675 - Event The Transport NOC has advised that Field Services have confirmed that both the 168 count cable and the 96 count cable onsite have been cut. Outside plant engineers have pulled 80 feet of slack on the 168 count cable from the existing splice point, which is being used to repair that fiber. At this time, the fiber is being prepared for the ensuing fiber repair. Outside plant engineers were unable to pull any slack on the 96 count cable from the splice point. As such, the splice crew is preparing to splice a temporary cable throw from the break point to the existing splice point. There is no ETTR for this case at this time. 01/31/2017 16:21:26 GMT - *** CASCADED EXTERNAL NOTES 31-Jan-2017 16:19:28 GMT From CASE: 12152675 - Event The Transport NOC has advised that the damaged cable is a 168 count cable that was struck by a backhoe during a department of transportation construction project onsite. The construction project may have also damaged a 96 count cable onsite. Field Services and the Transport NOC are working to confirm the damage to the second cable. The excavation/splicing contractor is still en route; however outside plant engineers are still unable to provide an ETA for the contractors. Outside plant engineers have begun excavating effort onsite with the available construction equipment and have identified an existing splice point nearby. At this time, effort to pull the necessary slack from the existing splice point are underway. 01/31/2017 15:04:17 GMT - 01/31/2017 15:03:31 GMT - The Transport NOC has advised that Field Services have located the fiber damage and determined the break was the result of a construction project. Outside plant engineers are onsite and are waiting for the splicing contractors to arrive. There is no ETA for the splicing contractors at this time. 01/31/2017 14:03:16 GMT - The Transport NOC has advised that Field Services have arrived onsite and are preparing to test the affected fiber to confirm the existence and location of the damaged cable. Fiber testing will commence momentarily. 01/31/2017 13:59:00 GMT - A fiber cut in Houston, TX is impacting unprotected services. The Transport NOC has engaged Field Services to dispatch a field technician to the failure site. There is no ETA for the field technician at this time. Please be advised that updates for this event will be relayed at a minimum of hourly unless otherwise noted. The information conveyed hereafter is associated to live troubleshooting effort and as the discovery process evolves through to service resolution, ticket closure, or post incident review, details may evolve. -- Hugo On Tue 2017-Jan-31 18:07:29 +0000, Derek J. Nicholls via Outages <outages@outages.org> wrote:
We had to turn down our BGP with Level3 due to almost complete loss for ~5 minutes. I also have a layer2 circuit with them which has come back up since but is experiencing a lot of loss. We’re out of Eagan, MN
Derek Nicholls Sr. Network Administrator | Tempworks Software & Payroll Funding Office: 651-452-0366 3140 Neil Armstrong Blvd Suite 205 | Eagan | Minnesota | 55121 dnicholls@tempworks.com<mailto:dnicholls@tempworks.com> | www.tempworks.com<http://www.tempworks.com/>
From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Ca By via Outages Sent: Tuesday, January 31, 2017 12:01 PM To: outages@outages.org Subject: [outages] Level3 Looking Glassdown
Level3 folks, please fix.
wget -T 5 lg.level3.net<https://linkprotect.cudasvc.com/url?a=http://lg.level3.net&c=E,1,65ZeY0snVET9qaPqRdvh2Sa3RT7MZE61MghnAkIys49KsaIlZSkH-GtXO3UbHtv2Tz-iL-gSz4v2aleWIgoksfVGKFUGXKhWXDqG_i1xqYPn43LsI-D1ZA,,&typo=1> converted 'http://lg.level3.net<https://linkprotect.cudasvc.com/url?a=http://lg.level3.net&c=E,1,4pEiSraAcLmXw830KB1JR7WeRwgOlgQFIBIPE8Gm84BKrZIFHpG7QPLdTZ4G_AdEqSy2NiglSzfexPkGh0EWnfS7KZiGVjS4ciGLmXz64GQRdupa7kAV&typo=1>' (ANSI_X3.4-1968) -> 'http://lg.level3.net<https://linkprotect.cudasvc.com/url?a=http://lg.level3.net&c=E,1,zSJO5Pjj73AErWit4kheKsWg3pyJroiNZICzn480acGF2WsydN64mQ54LYkLwJ0wqWsoX2n11UU0gBoyCU5Hpcw3MWxQGnIOxrjnbU2CTxOt4HAebWsL86dX&typo=1>' (UTF-8) --2017-01-31 12:58:57-- http://lg.level3.net/<https://linkprotect.cudasvc.com/url?a=http://lg.level3.net/&c=E,1,LE6Ov7vn2fjHkJN2Q1Iu3UvrqWFeoCcL2-OHR34YathQl0arHzdDrK7AhWKZTjSqvd6YRtz11eU7J3EFdosafQqRtHMt-y_stnSujqGPhpRkcQ,,&typo=1> Resolving lg.level3.net<https://linkprotect.cudasvc.com/url?a=http://lg.level3.net&c=E,1,RQ3mPbIz98skzt2ykO8BCSVmUwynT2-OPtvgxIcq-Rakc53Vp83Yj7VYZ-kBD_TVF1NcVuIeFPlTW5NwmZKVhnjpH2j6QPNvhkDpD_Km0du0fdF3DE8iBjLFXA,,&typo=1> (lg.level3.net<https://linkprotect.cudasvc.com/url?a=http://lg.level3.net&c=E,1,63yd2QpqHgbEbSui2UG9EXJP7kDPyUjxo5FxuBFeCdxxIGKRmeNjzDw3xCrurqHne6N9nb9tSBJTijm74oLNTb4inp3D_Pcf-inup6CxE2-J10dODroe&typo=1>)... 209.245.18.181 Connecting to lg.level3.net<https://linkprotect.cudasvc.com/url?a=http://lg.level3.net&c=E,1,CvU5bdnbR5iloFh9gq6ytXW2dwqgB7rZcr5lkZemNddocPBuOA176nIIq7aVo1eUxUiOnZyEdzIIMJMl6LRZxlliVpqBTbAXVvS0TiyCpuBQbuw,&typo=1> (lg.level3.net<https://linkprotect.cudasvc.com/url?a=http://lg.level3.net&c=E,1,1mCfvG2agTr2J86kkpqv1SWVR8weolRUs0iD4SUpZGsLlS9a3kgRi2_LnCv8MTYjOayj9rqxlNP_Tnuf8riIc6tMWn1rCGoFz-UtyAxYgJsVgty4&typo=1>)|209.245.18.181|:80... failed: Connection timed out. Retrying.
traceroute lg.level3.net<https://linkprotect.cudasvc.com/url?a=http://lg.level3.net&c=E,1,ygQIZyLx3YeBLoyeSml5g1LLLnbzrdFkSYw1IPJ7MYAw3xnbCPJu5o4_mq3j72d-xteZistsl4CX9oQRs8o9vIPefDFo1znO962KiJGxb2RDTg,,&typo=1> traceroute to lg.level3.net<https://linkprotect.cudasvc.com/url?a=http://lg.level3.net&c=E,1,lvNtrJjWmsdmwMjC26klIHSgkEgQNn8c684jZ787A4WUsvJlWB113O2VAzhXPjOu2QzyXkphGPlCGd-YZ4Y--7T4SIOWBfXbwJdMsESd3A,,&typo=1> (209.245.18.181), 30 hops max, 60 byte packets 1 * * * 2 xe-0-1-0-17.r04.atlnga05.us.bb.gin.ntt.net<https://linkprotect.cudasvc.com/url?a=http://xe-0-1-0-17.r04.atlnga05.us.bb.gin.ntt.net&c=E,1,u2dYqQiQcFjAQ0JYGBEzE3-MtU-PYnHvK5jbtKaLII9Qy56WQwaGJtvIloTA00Mxu_2AjwoYUfDOhq2Re_q5dK7YmwjyLnvF9IjBwOA8vwLUxYySrsk,&typo=1> (129.250.203.221) 1.284 ms 1.277 ms ae7-300.cr2-atl1.ip4.gtt.net<https://linkprotect.cudasvc.com/url?a=http://ae7-300.cr2-atl1.ip4.gtt.net&c=E,1,YpIlJ_1gNF4En7uHt16oCzOdtuETGxfx_YY4isLvhloZU2LC3KWTD-y4F-1Il_SsQ3RTsqvk78ffUOMVwBwS0ZsqzJ-O85JRPR6jmLHd7_5_e_1-&typo=1> (173.205.55.97) 0.241 ms 3 ae-4.r03.atlnga05.us.bb.gin.ntt.net<https://linkprotect.cudasvc.com/url?a=http://ae-4.r03.atlnga05.us.bb.gin.ntt.net&c=E,1,CKY1IS6w12E6GEvwL3JejiYzxBuA55OSlIEb0tnDaxSVeg9mhZxU-wf6L__WuKmKxhWBn47D-SyiZaXImzXyXrVrgC1Ed4-R_LZsDBtt0nRUS5E,&typo=1> (129.250.5.204) 0.427 ms ae4.edge2.Atlanta4.Level3.net<https://linkprotect.cudasvc.com/url?a=http://ae4.edge2.Atlanta4.Level3.net&c=E,1,fDraKWbUYry75w12FGbrDKixsWxRFYqrTAw-kbx2nERVfPcPf2q1ebrUxU4kqZGjYK6UUVHzv_sm8UZJnR08PPi6ct1hcUxdsInWhB-kmRPJaupxn3WxKwEpVA,,&typo=1> (4.68.63.157) 0.222 ms ae-4.r03.atlnga05.us.bb.gin.ntt.net<https://linkprotect.cudasvc.com/url?a=http://ae-4.r03.atlnga05.us.bb.gin.ntt.net&c=E,1,hAWB87_8a3WaKyBRueB6UrEfpNZbjJvYuEIDB7mf5zhJoAWy0SuQlwi3lCdN5JHR7Py3TFbA7JXK7e_a5Mt1BWr_R7qfv10DJjGRskcJHA6ZqPbUddFp&typo=1> (129.250.5.204) 0.464 ms 4 * * ae-1-51.edge6.Denver1.Level3.net<https://linkprotect.cudasvc.com/url?a=http://ae-1-51.edge6.Denver1.Level3.net&c=E,1,yElQnfHCp3eYR020r23JE_ZND0SzQWPyWG00Prgu5Oc80Eb68nmqO9M5rvwuHqPBOLpWkE3a5Tc-PMm_9mpIBGJWt8GheXUmIcIcuw0JmSaTKZfvOCUq&typo=1> (4.69.147.74) 34.727 ms 5 ae-1-51.edge6.Denver1.Level3.net<https://linkprotect.cudasvc.com/url?a=http://ae-1-51.edge6.Denver1.Level3.net&c=E,1,ERQdEewwLju-Mx1WPVGTYsFWy43HWzvPXw1xyPDWHhMcDsp2Oblj7CSqw-DD0F-KxgZc8GxH_bY753XaximlM97NU_Jy7_W_p0ZHeT5yNQ,,&typo=1> (4.69.147.74) 34.655 ms 34.639 ms 34.641 ms 6 * * * 7 * * * 8 * * * 9 * * * 10 * * *
THIS MESSAGE IS ONLY INTENDED FOR THE USE OF THE INTENDED RECIPIENT(S) AND MAY CONTAIN INFORMATION THAT IS PRIVILEGED, PROPRIETARY AND/OR CONFIDENTIAL. If you are not the intended recipient, you are hereby notified that any review, re-transmission, dissemination, distribution, copying, conversion to hard copy or other use of this communication is strictly prohibited. If you are not the intended recipient and have received this message in error, please notify me by return e-mail and delete this message from your system. TempWorks Software, Inc.
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages