November 2015 - Outages - Outages Lists

Rogers fiber cut between Vancouver and Seattle?
by Hugo Slabbert 02 Dec '15

02 Dec '15

Hearing of a Rogers fiber cut that's affected some of our transport between Burnaby (BC, Canada) and Seattle. I don't have much in terms of details, though. Anyone else affected in the know of details or ETR? -- Hugo

2 7

Level3 LA - Packet loss 25-50%
by Kyle M 28 Nov '15

28 Nov '15

Alerting Fired from google compute node, seeing major packet loss on Level3 have ticket in with Level3 support. Cross checked from Seattle and Dallas Seattle Traceroute My traceroute [v0.85] localhost.localdomain (0.0.0.0) Fri Nov 27 19:56:35 2015 Keys: Help Display mode Restart statistics Order of fields quit Packets Pings Host Loss% Snt Last Avg Best Wrst StDev 1. 10.8.0.1 0.0% 41 135.0 189.8 88.5 266.3 54.7 2. v2010.er01.sea.as15003.net 0.0% 41 147.3 224.8 123.9 1050. 139.1 3. v3508.er01.sea.ubiquity.io 0.0% 41 160.7 233.7 131.2 972.5 127.3 4. 38.88.0.25 0.0% 41 237.4 246.5 99.5 986.6 140.3 5. te0-0-1-1.agr11.sea03.atlas.coge 0.0% 41 247.5 270.3 154.3 908.4 131.7 6. 154.24.20.181 0.0% 40 258.9 296.0 170.1 829.9 133.3 7. te0-2-0-3.ccr22.sea01.atlas.coge 0.0% 40 180.2 286.6 180.2 751.3 123.2 8. be2084.ccr21.sea02.atlas.cogentc 0.0% 40 192.6 276.2 190.0 673.9 99.1 9. level3.sea02.atlas.cogentco.com 0.0% 40 207.6 248.2 185.0 595.2 77.6 10. ae-4-90.edge3.LosAngeles1.Level3 20.0% 40 222.8 238.3 191.9 516.9 60.2 11. ae-4-90.edge3.LosAngeles1.Level3 38.5% 40 249.3 229.4 171.9 438.2 52.6 12. COMPANY.edge3.LosAngeles1.Le 22.5% 40 203.0 232.4 166.9 366.1 45.6 13. IP 28.2% 40 289.7 237.5 129.5 297.7 40.7 -- Kyle M <lostcuaz(a)gmail.com>

2 1

Verizon and Telia daytime issue for several weeks now
by Norman Brown 28 Nov '15

28 Nov '15

Does anyone know what's going on between Verizon and TeliaSonera? There appears to be a daytime only (10am to 6pm daily) issue impacting the Asbhurn, VA peering points between Verizon AS701 and TeliaSonera AS1299. I've not been able to get confirmation of the issue from either party. Latency spikes and throughput drops like a rock off and on throughout the entire day. This has been going on for approximately three weeks now. |------------------------------------------------------------------------------------------| | WinMTR statistics | | Host - % | Sent | Recv | Best | Avrg | Wrst | Last | |------------------------------------------------|------|------|------|------|------|------| |lo0-100.PITBPA-VFTTP-303.verizon-gni.net - 0 | 6 | 6 | 3 | 7 | 26 | 4 | | T0-5-0-6.PITBPA-LCR-21.verizon-gni.net - 0 | 6 | 6 | 3 | 5 | 7 | 5 | | No response from host - 100 | 1 | 0 | 0 | 0 | 0 | 0 | | No response from host - 100 | 1 | 0 | 0 | 0 | 0 | 0 | | 0.ae1.GW11.IAD8.ALTER.NET - 0 | 6 | 6 | 14 | 15 | 17 | 14 | | teliasonera-gw.customer.alter.net - 0 | 6 | 6 | 81 | 87 | 94 | 81 | | ash-bb4-link.telia.net - 0 | 6 | 6 | 84 | 90 | 96 | 84 | | nyk-bb2-link.telia.net - 0 | 6 | 6 | 88 | 97 | 123 | 123 | | nyk-b3-link.telia.net - 0 | 6 | 6 | 84 | 91 | 99 | 84 | |digitalocean-ic-306498-nyk-b3.c.telia.net - 0 | 6 | 6 | 86 | 92 | 99 | 86 | | 162.243.188.222 - 0 | 6 | 6 | 83 | 91 | 99 | 83 | | 104.131.177.000 - 0 | 6 | 6 | 85 | 91 | 98 | 85 | |________________________________________________|______|______|______|______|______|______| WinMTR v0.92 GPL V2 by Appnor MSP - Fully Managed Hosting & Cloud Provider My traceroute [v0.85] fafnir (0.0.0.0) Fri Nov 27 12:53:10 2015 Packets Pings Host Loss% Snt Last Avg Best Wrst StDev 1. 104.131.0.253 0.0% 8 1.5 1.6 0.3 4.0 1.5 2. 162.243.188.241 0.0% 8 0.3 0.6 0.2 2.5 0.5 3. nyk-b3-link.telia.net 0.0% 8 1.0 2.0 0.9 9.1 2.8 4. nyk-bb1-link.telia.net 0.0% 7 1.0 3.8 1.0 20.4 7.3 5. nyk-b2-link.telia.net 0.0% 7 1.4 1.4 1.4 1.5 0.0 6. xe-3-3-3.GW10.EWR6.ALTER.NET 0.0% 7 1.6 1.7 1.4 2.4 0.0 7. ??? 8. xe-7-1-2-0.PITBPA-VFTTP-303.veri 0.0% 7 99.7 84.9 74.7 99.7 9.0 9. pool-173-75-56-000.pitbpa.fios.ve 0.0% 7 81.4 87.5 73.8 112.2 15.4

4 4

Bluehost (and affiliates) down
by Frank Bulk 25 Nov '15

25 Nov '15

Looks like Bluehost and some of its affiliates are having issues right now. Bluehost is part of Endurance International Group (EIG), and has sister companies, HostMonster, FastDomain and iPage. https://en.wikipedia.org/wiki/Bluehost Our last delivered email to a Bluehost hosted site was 9:25 am (Central). http://downdetector.com/status/bluehost https://twitter.com/bluehostsupport/ https://currentlydown.com/hostmonster.com https://twitter.com/search?f=tweets&vertical=default&q=hostmonster&src=typd Unfortunately their status sites are down, too: https://www.bluehost.com/cgi/serverstatus/ https://www.hostmonster.com/cgi/serverstatus/ Some others are complaining about inability to hit their hosted sites and others about DNS not working. Frank

3 4

Level 3 UK
by Ryan Burke 25 Nov '15

25 Nov '15

Anybody seeing issues with Level 3 at the moment? 5 8 ms 8 ms 8 ms 10.200.0.17 6 212 ms 17 ms 10 ms te-4-3.car5.london1.level3.net [217.163.45.205] 7 * 11 ms * ae-124-3510.edge5.london1.level3.net [4.69.166.3 7] 8 * 26 ms * b.resolvers.level3.net [4.2.2.2] 9 * * * Request timed out. 10 8 ms 8 ms 8 ms b.resolvers.level3.net [4.2.2.2] Ryan Burke

3 3

Slack Outage
by Anthony Hook 23 Nov '15

23 Nov '15

Multiple sources on Twitter are confirming widespread Slack outages. Their status.slack.com appears intermittently down as well. -AH

2 1

Re: [outages] packet loss to us-east-1
by Gunnar Guðvarðarson 23 Nov '15

23 Nov '15

Confirmed from Iceland ~ Gunnar On Nov 23, 2015 19:55, Grant Ridder via Outages <outages(a)outages.org> wrote: It is related for some since Slack is in us-east-1 On Mon, Nov 23, 2015 at 11:52 AM, Chris Swingler <chris(a)chrisswingler.com<mailto:chris@chrisswingler.com>> wrote: I'm not sure these are related (I'm seeing zero packet loss to Slack from Chicago on Cogent) but Slack's definitely broken. Seemed to have been a cascading failure, a colleague of mine complained over IRC he got booted from his, then I got booted from one Slack team, then another. On Nov 23, 2015, at 1:49 PM, Justin Oeder via Outages <outages(a)outages.org<mailto:outages@outages.org>> wrote: Slack, which is hosted with Amazon is currently down. ________________________________ From: "Grant Ridder via Outages" <outages(a)outages.org<mailto:outages@outages.org>> To: "Mike Lyon" <mike.lyon(a)gmail.com<mailto:mike.lyon@gmail.com>> Cc: outages(a)outages.org<mailto:outages@outages.org>, "Tomoc Field" <tomoc.field(a)sonic.com<mailto:tomoc.field@sonic.com>> Sent: Monday, November 23, 2015 2:43:57 PM Subject: Re: [outages] packet loss to us-east-1 Seeing this issue again 4. AS46375 50.ae4.gw.pao1.sonic.net<http://50.ae4.gw.pao1.sonic.net/> 0.0% 10 15.5 23.2 2.9 89.3 28.6 5. AS8121 199.127.107.189 0.0% 10 64.2 26.6 3.7 85.7 28.2 6. AS8121 xe0-0-0-7.core2.sv8.layer42.net<http://xe0-0-0-7.core2.sv8.layer42.net/> 0.0% 10 66.6 24.8 3.5 102.1 33.4 7. AS16509 72.21.221.176 0.0% 10 64.2 26.9 3.0 99.1 32.4 8. AS??? 54.240.243.50 0.0% 10 147.4 115.9 88.2 176.9 27.9 9. AS??? 54.240.243.61 90.0% 10 78.4 78.4 78.4 78.4 0.0 10. AS??? 54.239.42.38 10.0% 10 130.4 106.9 71.8 167.6 31.8 11. AS16509 54.239.108.120 70.0% 10 178.9 125.2 90.0 178.9 47.2 12. AS16509 54.239.111.235 20.0% 10 143.1 99.0 70.8 158.1 34.7 13. AS16509 54.239.111.230 70.0% 10 89.5 93.9 88.7 103.5 8.3 -Grant On Fri, Nov 20, 2015 at 5:27 PM, Mike Lyon <mike.lyon(a)gmail.com<mailto:mike.lyon@gmail.com>> wrote: Hmmm, It shows as if it is getting there but look at that jump from 15ms to 85ms on hops 7 and 8: traceroute amazon.com<http://amazon.com/> traceroute: Warning: amazon.com<http://amazon.com/> has multiple addresses; using 176.32.103.205 traceroute to amazon.com<http://amazon.com/> (176.32.103.205), 64 hops max, 52 byte packets 1 192.168.20.1 (192.168.20.1) 0.708 ms 0.395 ms 0.328 ms 2 162.250.208.141 (162.250.208.141) 0.671 ms 0.622 ms 0.588 ms 3 10.25.0.1 (10.25.0.1) 1.985 ms 4.165 ms 21.626 ms 4 10.20.0.1 (10.20.0.1) 7.852 ms 7.626 ms 5.214 ms 5 10.0.0.1 (10.0.0.1) 8.041 ms 17.839 ms 34.089 ms 6 69.36.226.193 (69.36.226.193) 5.477 ms 14.228 ms 20.203 ms 7 xe0-2-0-4.core4.scl.layer42.net<http://xe0-2-0-4.core4.scl.layer42.net/> (65.50.198.237) 15.153 ms 13.475 ms 8.706 ms 8 205.251.229.145 (205.251.229.145) 85.231 ms 54.240.242.108 (54.240.242.108) 96.157 ms 205.251.229.145 (205.251.229.145) 175.633 ms 9 54.240.242.141 (54.240.242.141) 100.007 ms 234.602 ms 54.240.242.121 (54.240.242.121) 94.263 ms 10 54.239.109.102 (54.239.109.102) 110.966 ms 54.239.109.136 (54.239.109.136) 123.322 ms 54.239.110.50 (54.239.110.50) 105.029 ms 11 54.239.109.143 (54.239.109.143) 84.239 ms 54.239.110.23 (54.239.110.23) 91.540 ms 54.239.110.57 (54.239.110.57) 86.620 ms 12 54.239.109.143 (54.239.109.143) 91.382 ms 205.251.245.168 (205.251.245.168) 87.930 ms 54.239.109.143 (54.239.109.143) 76.879 ms 13 * 205.251.245.168 (205.251.245.168) 90.780 ms 96.386 ms On Fri, Nov 20, 2015 at 5:24 PM, Grant Ridder <shortdudey123(a)gmail.com<mailto:shortdudey123@gmail.com>> wrote: Looks like they solved it by routing around layer42 and go via cogent. That in itself is a bad idea :/ 4. AS46375 50.ae4.gw.pao1.sonic.net<http://50.ae4.gw.pao1.sonic.net/> 1.2% 81 3.5 12.0 2.6 91.0 16.2 5. AS174 te0-0-0-15.ccr21.sjc04.atlas.cogentco.com<http://te0-0-0-15.ccr21.sjc04.atlas.cogentco.com/> 6.2% 81 6.6 10.5 3.2 79.9 11.7 6. AS174 be2016.ccr22.sfo01.atlas.cogentco.com<http://be2016.ccr22.sfo01.atlas.cogentco.com/> 3.7% 81 15.3 14.6 4.2 98.9 16.3 7. AS174 be2133.ccr22.mci01.atlas.cogentco.com<http://be2133.ccr22.mci01.atlas.cogentco.com/> 4.9% 81 47.7 50.4 41.8 113.6 13.3 8. AS174 be2157.ccr42.ord01.atlas.cogentco.com<http://be2157.ccr42.ord01.atlas.cogentco.com/> 70.0% 81 54.1 67.2 53.9 140.2 21.3 9. AS174 be2217.ccr41.ord03.atlas.cogentco.com<http://be2217.ccr41.ord03.atlas.cogentco.com/> 33.3% 81 57.3 63.6 54.3 158.4 17.0 10. AS174 ntt.ord03.atlas.cogentco.com<http://ntt.ord03.atlas.cogentco.com/> 66.2% 81 56.9 72.0 56.5 126.8 20.6 11. AS2914 ae-3.amazon.chcgil09.us.bb.gin.ntt.net<http://ae-3.amazon.chcgil09.us.bb.gin.ntt.net/> 34.6% 81 61.5 61.8 55.9 89.7 8.1 -Grant On Fri, Nov 20, 2015 at 5:14 PM, Mike Lyon <mike.lyon(a)gmail.com<mailto:mike.lyon@gmail.com>> wrote: Oh thank god! I was seeing it too but wasn't sure if was Layer42 or Amazon that was having the issue... Thanks, Mike On Fri, Nov 20, 2015 at 5:09 PM, Grant Ridder via Outages <outages(a)outages.org<mailto:outages@outages.org>> wrote: Saw that, thanks! -Grant On Fri, Nov 20, 2015 at 4:49 PM, Tomoc Field <tomoc.field(a)sonic.com<mailto:tomoc.field@sonic.com>> wrote: Yes. 70%+ packet loss from 7065 to all Amazon services as far as I can tell. This was experienced via both layer42 and XO, but Amazon appears to have resolved their issues at this time. -Tomoc On Fri, Nov 20, 2015 at 04:34:42PM -0800, Grant Ridder via Outages wrote: > Anyone else seeing packet log into us-east-1? > > 54.240.243.121 (AS16509) is Amazon > > MTR report excerpt > 4. AS46375 50.ae4.gw.pao1.sonic.net<http://50.ae4.gw.pao1.sonic.net/> 0.0% 10 2.5 > 8.3 2.5 37.5 10.7 > 5. AS8121 199.127.107.189 0.0% 10 3.8 > 7.9 3.4 34.9 9.5 > 6. AS8121 xe0-0-0-7.core2.sv8.layer42.net<http://xe0-0-0-7.core2.sv8.layer42.net/> 0.0% 10 3.9 > 6.0 3.4 17.6 4.6 > 7. AS16509 72.21.221.176 0.0% 10 4.6 > 5.2 3.0 15.5 3.6 > 8. AS??? 54.240.243.110 0.0% 10 98.6 > 96.4 86.9 104.4 6.2 > 9. AS??? 54.240.243.121 90.0% 10 85.7 > 85.7 85.7 85.7 0.0 > 10. AS16509 205.251.229.104 20.0% 10 87.5 > 85.2 82.5 87.8 2.0 > 11. AS??? 54.239.42.29 70.0% 10 85.0 > 84.3 82.8 85.2 1.0 > 12. AS16509 54.239.109.238 30.0% 10 87.4 > 96.8 87.4 105.2 6.5 > AS16509 54.239.110.16 > > > -Grant > _______________________________________________ > Outages mailing list > Outages(a)outages.org<mailto:Outages@outages.org> > https://puck.nether.net/mailman/listinfo/outages _______________________________________________ Outages mailing list Outages(a)outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages -- Mike Lyon 408-621-4826 mike.lyon(a)gmail.com<mailto:mike.lyon@gmail.com> http://www.linkedin.com/in/mlyon -- Mike Lyon 408-621-4826 mike.lyon(a)gmail.com<mailto:mike.lyon@gmail.com> http://www.linkedin.com/in/mlyon _______________________________________________ Outages mailing list Outages(a)outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages _______________________________________________ Outages mailing list Outages(a)outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages

1 0

Re: [outages] packet loss to us-east-1
by Grant Ridder 23 Nov '15

23 Nov '15

Saw that, thanks! -Grant On Fri, Nov 20, 2015 at 4:49 PM, Tomoc Field <tomoc.field(a)sonic.com> wrote: > Yes. 70%+ packet loss from 7065 to all Amazon services as far as I can > tell. This was experienced via both layer42 and XO, but Amazon appears to > have resolved their issues at this time. > > -Tomoc > > On Fri, Nov 20, 2015 at 04:34:42PM -0800, Grant Ridder via Outages wrote: > > Anyone else seeing packet log into us-east-1? > > > > 54.240.243.121 (AS16509) is Amazon > > > > MTR report excerpt > > 4. AS46375 50.ae4.gw.pao1.sonic.net 0.0% 10 2.5 > > 8.3 2.5 37.5 10.7 > > 5. AS8121 199.127.107.189 0.0% 10 3.8 > > 7.9 3.4 34.9 9.5 > > 6. AS8121 xe0-0-0-7.core2.sv8.layer42.net 0.0% 10 3.9 > > 6.0 3.4 17.6 4.6 > > 7. AS16509 72.21.221.176 0.0% 10 4.6 > > 5.2 3.0 15.5 3.6 > > 8. AS??? 54.240.243.110 0.0% 10 98.6 > > 96.4 86.9 104.4 6.2 > > 9. AS??? 54.240.243.121 90.0% 10 85.7 > > 85.7 85.7 85.7 0.0 > > 10. AS16509 205.251.229.104 20.0% 10 87.5 > > 85.2 82.5 87.8 2.0 > > 11. AS??? 54.239.42.29 70.0% 10 85.0 > > 84.3 82.8 85.2 1.0 > > 12. AS16509 54.239.109.238 30.0% 10 87.4 > > 96.8 87.4 105.2 6.5 > > AS16509 54.239.110.16 > > > > > > -Grant > > > _______________________________________________ > > Outages mailing list > > Outages(a)outages.org > > https://puck.nether.net/mailman/listinfo/outages > >

4 7

packet loss to us-east-1
by Grant Ridder 20 Nov '15

20 Nov '15

Anyone else seeing packet log into us-east-1? 54.240.243.121 (AS16509) is Amazon MTR report excerpt 4. AS46375 50.ae4.gw.pao1.sonic.net 0.0% 10 2.5 8.3 2.5 37.5 10.7 5. AS8121 199.127.107.189 0.0% 10 3.8 7.9 3.4 34.9 9.5 6. AS8121 xe0-0-0-7.core2.sv8.layer42.net 0.0% 10 3.9 6.0 3.4 17.6 4.6 7. AS16509 72.21.221.176 0.0% 10 4.6 5.2 3.0 15.5 3.6 8. AS??? 54.240.243.110 0.0% 10 98.6 96.4 86.9 104.4 6.2 9. AS??? 54.240.243.121 90.0% 10 85.7 85.7 85.7 85.7 0.0 10. AS16509 205.251.229.104 20.0% 10 87.5 85.2 82.5 87.8 2.0 11. AS??? 54.239.42.29 70.0% 10 85.0 84.3 82.8 85.2 1.0 12. AS16509 54.239.109.238 30.0% 10 87.4 96.8 87.4 105.2 6.5 AS16509 54.239.110.16 -Grant

1 0

XO / Level 3 Outage | Chicago - St Louis | 11/19/2015
by Thom Cleary 20 Nov '15

20 Nov '15

Here's the latest: *** CASCADED EXTERNAL NOTES 20-Nov-2015 00:36:21 GMT From CASE: 10003567 - Event Field Services continue preparation efforts at this time. 11/20/2015 00:40:36 GMT - *** CASCADED EXTERNAL NOTES 20-Nov-2015 00:36:21 GMT From CASE: 10003567 - Event Field Services continue preparation efforts at this time. 11/19/2015 23:47:36 GMT - *** CASCADED EXTERNAL NOTES 19-Nov-2015 23:43:49 GMT From CASE: 10003567 - Event Restoration efforts are still ongoing, an estimated restoral time has not been provided. 11/19/2015 23:05:21 GMT - *** CASCADED EXTERNAL NOTES 19-Nov-2015 23:00:53 GMT From CASE: 10003567 - Event Field Services have excavated three hand holes to reach the 108-count fiber cable for repair. Efforts are ongoing at the site with no estimated restoral time currently available. 11/19/2015 22:29:51 GMT - *** CASCADED EXTERNAL NOTES 19-Nov-2015 22:25:57 GMT From CASE: 10003567 - Event Field Services have accessed one hand hole and are working on accessing a second. Evaluation of the cable damage is underway. 11/19/2015 21:51:05 GMT - *** CASCADED EXTERNAL NOTES 19-Nov-2015 21:47:47 GMT From CASE: 10003567 - Event Field Services are now excavating to the hand hole with a backhoe. Once access is available, evaluation of the fault will continue. 11/19/2015 20:54:23 GMT - *** CASCADED EXTERNAL NOTES 19-Nov-2015 20:50:55 GMT From CASE: 10003567 - Event Field Services are manually digging to access the hand hole closest to the suspected damage point. Once accessed, Field Services will be able to better evaluate the issue. 11/19/2015 20:12:15 GMT - *** CASCADED EXTERNAL NOTES 19-Nov-2015 20:08:11 GMT From CASE: 10003567 - Event The plow work at the site has made isolation of the damage point difficult to pinpoint. Field Services are working to access hand holes in order to continue investigation and restoration efforts. 11/19/2015 19:08:37 GMT - *** CASCADED EXTERNAL NOTES 19-Nov-2015 19:04:13 GMT From CASE: 10003567 - Event Field Services are onsite in the construction area working to locate the exact damage point. Replacement cable is being dispatched to the site to have available for repair and should arrive by 20:00 GMT.

6 11