Confirmed from Iceland ~ Gunnar On Nov 23, 2015 19:55, Grant Ridder via Outages <outages@outages.org> wrote: It is related for some since Slack is in us-east-1 On Mon, Nov 23, 2015 at 11:52 AM, Chris Swingler <chris@chrisswingler.com<mailto:chris@chrisswingler.com>> wrote: I'm not sure these are related (I'm seeing zero packet loss to Slack from Chicago on Cogent) but Slack's definitely broken. Seemed to have been a cascading failure, a colleague of mine complained over IRC he got booted from his, then I got booted from one Slack team, then another. On Nov 23, 2015, at 1:49 PM, Justin Oeder via Outages <outages@outages.org<mailto:outages@outages.org>> wrote: Slack, which is hosted with Amazon is currently down. ________________________________ From: "Grant Ridder via Outages" <outages@outages.org<mailto:outages@outages.org>> To: "Mike Lyon" <mike.lyon@gmail.com<mailto:mike.lyon@gmail.com>> Cc: outages@outages.org<mailto:outages@outages.org>, "Tomoc Field" <tomoc.field@sonic.com<mailto:tomoc.field@sonic.com>> Sent: Monday, November 23, 2015 2:43:57 PM Subject: Re: [outages] packet loss to us-east-1 Seeing this issue again 4. AS46375 50.ae4.gw.pao1.sonic.net<http://50.ae4.gw.pao1.sonic.net/> 0.0% 10 15.5 23.2 2.9 89.3 28.6 5. AS8121 199.127.107.189 0.0% 10 64.2 26.6 3.7 85.7 28.2 6. AS8121 xe0-0-0-7.core2.sv8.layer42.net<http://xe0-0-0-7.core2.sv8.layer42.net/> 0.0% 10 66.6 24.8 3.5 102.1 33.4 7. AS16509 72.21.221.176 0.0% 10 64.2 26.9 3.0 99.1 32.4 8. AS??? 54.240.243.50 0.0% 10 147.4 115.9 88.2 176.9 27.9 9. AS??? 54.240.243.61 90.0% 10 78.4 78.4 78.4 78.4 0.0 10. AS??? 54.239.42.38 10.0% 10 130.4 106.9 71.8 167.6 31.8 11. AS16509 54.239.108.120 70.0% 10 178.9 125.2 90.0 178.9 47.2 12. AS16509 54.239.111.235 20.0% 10 143.1 99.0 70.8 158.1 34.7 13. AS16509 54.239.111.230 70.0% 10 89.5 93.9 88.7 103.5 8.3 -Grant On Fri, Nov 20, 2015 at 5:27 PM, Mike Lyon <mike.lyon@gmail.com<mailto:mike.lyon@gmail.com>> wrote: Hmmm, It shows as if it is getting there but look at that jump from 15ms to 85ms on hops 7 and 8: traceroute amazon.com<http://amazon.com/> traceroute: Warning: amazon.com<http://amazon.com/> has multiple addresses; using 176.32.103.205 traceroute to amazon.com<http://amazon.com/> (176.32.103.205), 64 hops max, 52 byte packets 1 192.168.20.1 (192.168.20.1) 0.708 ms 0.395 ms 0.328 ms 2 162.250.208.141 (162.250.208.141) 0.671 ms 0.622 ms 0.588 ms 3 10.25.0.1 (10.25.0.1) 1.985 ms 4.165 ms 21.626 ms 4 10.20.0.1 (10.20.0.1) 7.852 ms 7.626 ms 5.214 ms 5 10.0.0.1 (10.0.0.1) 8.041 ms 17.839 ms 34.089 ms 6 69.36.226.193 (69.36.226.193) 5.477 ms 14.228 ms 20.203 ms 7 xe0-2-0-4.core4.scl.layer42.net<http://xe0-2-0-4.core4.scl.layer42.net/> (65.50.198.237) 15.153 ms 13.475 ms 8.706 ms 8 205.251.229.145 (205.251.229.145) 85.231 ms 54.240.242.108 (54.240.242.108) 96.157 ms 205.251.229.145 (205.251.229.145) 175.633 ms 9 54.240.242.141 (54.240.242.141) 100.007 ms 234.602 ms 54.240.242.121 (54.240.242.121) 94.263 ms 10 54.239.109.102 (54.239.109.102) 110.966 ms 54.239.109.136 (54.239.109.136) 123.322 ms 54.239.110.50 (54.239.110.50) 105.029 ms 11 54.239.109.143 (54.239.109.143) 84.239 ms 54.239.110.23 (54.239.110.23) 91.540 ms 54.239.110.57 (54.239.110.57) 86.620 ms 12 54.239.109.143 (54.239.109.143) 91.382 ms 205.251.245.168 (205.251.245.168) 87.930 ms 54.239.109.143 (54.239.109.143) 76.879 ms 13 * 205.251.245.168 (205.251.245.168) 90.780 ms 96.386 ms On Fri, Nov 20, 2015 at 5:24 PM, Grant Ridder <shortdudey123@gmail.com<mailto:shortdudey123@gmail.com>> wrote: Looks like they solved it by routing around layer42 and go via cogent. That in itself is a bad idea :/ 4. AS46375 50.ae4.gw.pao1.sonic.net<http://50.ae4.gw.pao1.sonic.net/> 1.2% 81 3.5 12.0 2.6 91.0 16.2 5. AS174 te0-0-0-15.ccr21.sjc04.atlas.cogentco.com<http://te0-0-0-15.ccr21.sjc04.atlas.cogentco.com/> 6.2% 81 6.6 10.5 3.2 79.9 11.7 6. AS174 be2016.ccr22.sfo01.atlas.cogentco.com<http://be2016.ccr22.sfo01.atlas.cogentco.com/> 3.7% 81 15.3 14.6 4.2 98.9 16.3 7. AS174 be2133.ccr22.mci01.atlas.cogentco.com<http://be2133.ccr22.mci01.atlas.cogentco.com/> 4.9% 81 47.7 50.4 41.8 113.6 13.3 8. AS174 be2157.ccr42.ord01.atlas.cogentco.com<http://be2157.ccr42.ord01.atlas.cogentco.com/> 70.0% 81 54.1 67.2 53.9 140.2 21.3 9. AS174 be2217.ccr41.ord03.atlas.cogentco.com<http://be2217.ccr41.ord03.atlas.cogentco.com/> 33.3% 81 57.3 63.6 54.3 158.4 17.0 10. AS174 ntt.ord03.atlas.cogentco.com<http://ntt.ord03.atlas.cogentco.com/> 66.2% 81 56.9 72.0 56.5 126.8 20.6 11. AS2914 ae-3.amazon.chcgil09.us.bb.gin.ntt.net<http://ae-3.amazon.chcgil09.us.bb.gin.ntt.net/> 34.6% 81 61.5 61.8 55.9 89.7 8.1 -Grant On Fri, Nov 20, 2015 at 5:14 PM, Mike Lyon <mike.lyon@gmail.com<mailto:mike.lyon@gmail.com>> wrote: Oh thank god! I was seeing it too but wasn't sure if was Layer42 or Amazon that was having the issue... Thanks, Mike On Fri, Nov 20, 2015 at 5:09 PM, Grant Ridder via Outages <outages@outages.org<mailto:outages@outages.org>> wrote: Saw that, thanks! -Grant On Fri, Nov 20, 2015 at 4:49 PM, Tomoc Field <tomoc.field@sonic.com<mailto:tomoc.field@sonic.com>> wrote: Yes. 70%+ packet loss from 7065 to all Amazon services as far as I can tell. This was experienced via both layer42 and XO, but Amazon appears to have resolved their issues at this time. -Tomoc On Fri, Nov 20, 2015 at 04:34:42PM -0800, Grant Ridder via Outages wrote:

Anyone else seeing packet log into us-east-1?

54.240.243.121 (AS16509) is Amazon

MTR report excerpt 4. AS46375 50.ae4.gw.pao1.sonic.net<http://50.ae4.gw.pao1.sonic.net/> 0.0% 10 2.5 8.3 2.5 37.5 10.7 5. AS8121 199.127.107.189 0.0% 10 3.8 7.9 3.4 34.9 9.5 6. AS8121 xe0-0-0-7.core2.sv8.layer42.net<http://xe0-0-0-7.core2.sv8.layer42.net/> 0.0% 10 3.9 6.0 3.4 17.6 4.6 7. AS16509 72.21.221.176 0.0% 10 4.6 5.2 3.0 15.5 3.6 8. AS??? 54.240.243.110 0.0% 10 98.6 96.4 86.9 104.4 6.2 9. AS??? 54.240.243.121 90.0% 10 85.7 85.7 85.7 85.7 0.0 10. AS16509 205.251.229.104 20.0% 10 87.5 85.2 82.5 87.8 2.0 11. AS??? 54.239.42.29 70.0% 10 85.0 84.3 82.8 85.2 1.0 12. AS16509 54.239.109.238 30.0% 10 87.4 96.8 87.4 105.2 6.5 AS16509 54.239.110.16

-Grant

_______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages

_______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages -- Mike Lyon 408-621-4826 mike.lyon@gmail.com<mailto:mike.lyon@gmail.com> http://www.linkedin.com/in/mlyon -- Mike Lyon 408-621-4826 mike.lyon@gmail.com<mailto:mike.lyon@gmail.com> http://www.linkedin.com/in/mlyon _______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages _______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages