Re: [outages] packet loss to us-east-1
Saw that, thanks! -Grant On Fri, Nov 20, 2015 at 4:49 PM, Tomoc Field <tomoc.field@sonic.com> wrote:
Yes. 70%+ packet loss from 7065 to all Amazon services as far as I can tell. This was experienced via both layer42 and XO, but Amazon appears to have resolved their issues at this time.
-Tomoc
On Fri, Nov 20, 2015 at 04:34:42PM -0800, Grant Ridder via Outages wrote:
Anyone else seeing packet log into us-east-1?
54.240.243.121 (AS16509) is Amazon
MTR report excerpt 4. AS46375 50.ae4.gw.pao1.sonic.net 0.0% 10 2.5 8.3 2.5 37.5 10.7 5. AS8121 199.127.107.189 0.0% 10 3.8 7.9 3.4 34.9 9.5 6. AS8121 xe0-0-0-7.core2.sv8.layer42.net 0.0% 10 3.9 6.0 3.4 17.6 4.6 7. AS16509 72.21.221.176 0.0% 10 4.6 5.2 3.0 15.5 3.6 8. AS??? 54.240.243.110 0.0% 10 98.6 96.4 86.9 104.4 6.2 9. AS??? 54.240.243.121 90.0% 10 85.7 85.7 85.7 85.7 0.0 10. AS16509 205.251.229.104 20.0% 10 87.5 85.2 82.5 87.8 2.0 11. AS??? 54.239.42.29 70.0% 10 85.0 84.3 82.8 85.2 1.0 12. AS16509 54.239.109.238 30.0% 10 87.4 96.8 87.4 105.2 6.5 AS16509 54.239.110.16
-Grant
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Oh thank god! I was seeing it too but wasn't sure if was Layer42 or Amazon that was having the issue... Thanks, Mike On Fri, Nov 20, 2015 at 5:09 PM, Grant Ridder via Outages < outages@outages.org> wrote:
Saw that, thanks!
-Grant
On Fri, Nov 20, 2015 at 4:49 PM, Tomoc Field <tomoc.field@sonic.com> wrote:
Yes. 70%+ packet loss from 7065 to all Amazon services as far as I can tell. This was experienced via both layer42 and XO, but Amazon appears to have resolved their issues at this time.
-Tomoc
On Fri, Nov 20, 2015 at 04:34:42PM -0800, Grant Ridder via Outages wrote:
Anyone else seeing packet log into us-east-1?
54.240.243.121 (AS16509) is Amazon
MTR report excerpt 4. AS46375 50.ae4.gw.pao1.sonic.net 0.0% 10 2.5 8.3 2.5 37.5 10.7 5. AS8121 199.127.107.189 0.0% 10 3.8 7.9 3.4 34.9 9.5 6. AS8121 xe0-0-0-7.core2.sv8.layer42.net 0.0% 10 3.9 6.0 3.4 17.6 4.6 7. AS16509 72.21.221.176 0.0% 10 4.6 5.2 3.0 15.5 3.6 8. AS??? 54.240.243.110 0.0% 10 98.6 96.4 86.9 104.4 6.2 9. AS??? 54.240.243.121 90.0% 10 85.7 85.7 85.7 85.7 0.0 10. AS16509 205.251.229.104 20.0% 10 87.5 85.2 82.5 87.8 2.0 11. AS??? 54.239.42.29 70.0% 10 85.0 84.3 82.8 85.2 1.0 12. AS16509 54.239.109.238 30.0% 10 87.4 96.8 87.4 105.2 6.5 AS16509 54.239.110.16
-Grant
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
-- Mike Lyon 408-621-4826 mike.lyon@gmail.com http://www.linkedin.com/in/mlyon
Looks like they solved it by routing around layer42 and go via cogent. That in itself is a bad idea :/ 4. AS46375 50.ae4.gw.pao1.sonic.net 1.2% 81 3.5 12.0 2.6 91.0 16.2 5. AS174 te0-0-0-15.ccr21.sjc04.atlas.cogentco.com 6.2% 81 6.6 10.5 3.2 79.9 11.7 6. AS174 be2016.ccr22.sfo01.atlas.cogentco.com 3.7% 81 15.3 14.6 4.2 98.9 16.3 7. AS174 be2133.ccr22.mci01.atlas.cogentco.com 4.9% 81 47.7 50.4 41.8 113.6 13.3 8. AS174 be2157.ccr42.ord01.atlas.cogentco.com 70.0% 81 54.1 67.2 53.9 140.2 21.3 9. AS174 be2217.ccr41.ord03.atlas.cogentco.com 33.3% 81 57.3 63.6 54.3 158.4 17.0 10. AS174 ntt.ord03.atlas.cogentco.com 66.2% 81 56.9 72.0 56.5 126.8 20.6 11. AS2914 ae-3.amazon.chcgil09.us.bb.gin.ntt.net 34.6% 81 61.5 61.8 55.9 89.7 8.1 -Grant On Fri, Nov 20, 2015 at 5:14 PM, Mike Lyon <mike.lyon@gmail.com> wrote:
Oh thank god! I was seeing it too but wasn't sure if was Layer42 or Amazon that was having the issue...
Thanks, Mike
On Fri, Nov 20, 2015 at 5:09 PM, Grant Ridder via Outages < outages@outages.org> wrote:
Saw that, thanks!
-Grant
On Fri, Nov 20, 2015 at 4:49 PM, Tomoc Field <tomoc.field@sonic.com> wrote:
Yes. 70%+ packet loss from 7065 to all Amazon services as far as I can tell. This was experienced via both layer42 and XO, but Amazon appears to have resolved their issues at this time.
-Tomoc
On Fri, Nov 20, 2015 at 04:34:42PM -0800, Grant Ridder via Outages wrote:
Anyone else seeing packet log into us-east-1?
54.240.243.121 (AS16509) is Amazon
MTR report excerpt 4. AS46375 50.ae4.gw.pao1.sonic.net 0.0% 10 2.5 8.3 2.5 37.5 10.7 5. AS8121 199.127.107.189 0.0% 10 3.8 7.9 3.4 34.9 9.5 6. AS8121 xe0-0-0-7.core2.sv8.layer42.net 0.0% 10 3.9 6.0 3.4 17.6 4.6 7. AS16509 72.21.221.176 0.0% 10 4.6 5.2 3.0 15.5 3.6 8. AS??? 54.240.243.110 0.0% 10 98.6 96.4 86.9 104.4 6.2 9. AS??? 54.240.243.121 90.0% 10 85.7 85.7 85.7 85.7 0.0 10. AS16509 205.251.229.104 20.0% 10 87.5 85.2 82.5 87.8 2.0 11. AS??? 54.239.42.29 70.0% 10 85.0 84.3 82.8 85.2 1.0 12. AS16509 54.239.109.238 30.0% 10 87.4 96.8 87.4 105.2 6.5 AS16509 54.239.110.16
-Grant
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
-- Mike Lyon 408-621-4826 mike.lyon@gmail.com
Hmmm, It shows as if it is getting there but look at that jump from 15ms to 85ms on hops 7 and 8: traceroute amazon.com traceroute: Warning: amazon.com has multiple addresses; using 176.32.103.205 traceroute to amazon.com (176.32.103.205), 64 hops max, 52 byte packets 1 192.168.20.1 (192.168.20.1) 0.708 ms 0.395 ms 0.328 ms 2 162.250.208.141 (162.250.208.141) 0.671 ms 0.622 ms 0.588 ms 3 10.25.0.1 (10.25.0.1) 1.985 ms 4.165 ms 21.626 ms 4 10.20.0.1 (10.20.0.1) 7.852 ms 7.626 ms 5.214 ms 5 10.0.0.1 (10.0.0.1) 8.041 ms 17.839 ms 34.089 ms 6 69.36.226.193 (69.36.226.193) 5.477 ms 14.228 ms 20.203 ms 7 xe0-2-0-4.core4.scl.layer42.net (65.50.198.237) 15.153 ms 13.475 ms 8.706 ms 8 205.251.229.145 (205.251.229.145) 85.231 ms 54.240.242.108 (54.240.242.108) 96.157 ms 205.251.229.145 (205.251.229.145) 175.633 ms 9 54.240.242.141 (54.240.242.141) 100.007 ms 234.602 ms 54.240.242.121 (54.240.242.121) 94.263 ms 10 54.239.109.102 (54.239.109.102) 110.966 ms 54.239.109.136 (54.239.109.136) 123.322 ms 54.239.110.50 (54.239.110.50) 105.029 ms 11 54.239.109.143 (54.239.109.143) 84.239 ms 54.239.110.23 (54.239.110.23) 91.540 ms 54.239.110.57 (54.239.110.57) 86.620 ms 12 54.239.109.143 (54.239.109.143) 91.382 ms 205.251.245.168 (205.251.245.168) 87.930 ms 54.239.109.143 (54.239.109.143) 76.879 ms 13 * 205.251.245.168 (205.251.245.168) 90.780 ms 96.386 ms On Fri, Nov 20, 2015 at 5:24 PM, Grant Ridder <shortdudey123@gmail.com> wrote:
Looks like they solved it by routing around layer42 and go via cogent. That in itself is a bad idea :/
4. AS46375 50.ae4.gw.pao1.sonic.net 1.2% 81 3.5 12.0 2.6 91.0 16.2 5. AS174 te0-0-0-15.ccr21.sjc04.atlas.cogentco.com 6.2% 81 6.6 10.5 3.2 79.9 11.7 6. AS174 be2016.ccr22.sfo01.atlas.cogentco.com 3.7% 81 15.3 14.6 4.2 98.9 16.3 7. AS174 be2133.ccr22.mci01.atlas.cogentco.com 4.9% 81 47.7 50.4 41.8 113.6 13.3 8. AS174 be2157.ccr42.ord01.atlas.cogentco.com 70.0% 81 54.1 67.2 53.9 140.2 21.3 9. AS174 be2217.ccr41.ord03.atlas.cogentco.com 33.3% 81 57.3 63.6 54.3 158.4 17.0 10. AS174 ntt.ord03.atlas.cogentco.com 66.2% 81 56.9 72.0 56.5 126.8 20.6 11. AS2914 ae-3.amazon.chcgil09.us.bb.gin.ntt.net 34.6% 81 61.5 61.8 55.9 89.7 8.1
-Grant
On Fri, Nov 20, 2015 at 5:14 PM, Mike Lyon <mike.lyon@gmail.com> wrote:
Oh thank god! I was seeing it too but wasn't sure if was Layer42 or Amazon that was having the issue...
Thanks, Mike
On Fri, Nov 20, 2015 at 5:09 PM, Grant Ridder via Outages < outages@outages.org> wrote:
Saw that, thanks!
-Grant
On Fri, Nov 20, 2015 at 4:49 PM, Tomoc Field <tomoc.field@sonic.com> wrote:
Yes. 70%+ packet loss from 7065 to all Amazon services as far as I can tell. This was experienced via both layer42 and XO, but Amazon appears to have resolved their issues at this time.
-Tomoc
On Fri, Nov 20, 2015 at 04:34:42PM -0800, Grant Ridder via Outages wrote:
Anyone else seeing packet log into us-east-1?
54.240.243.121 (AS16509) is Amazon
MTR report excerpt 4. AS46375 50.ae4.gw.pao1.sonic.net 0.0% 10 2.5 8.3 2.5 37.5 10.7 5. AS8121 199.127.107.189 0.0% 10 3.8 7.9 3.4 34.9 9.5 6. AS8121 xe0-0-0-7.core2.sv8.layer42.net 0.0% 10 3.9 6.0 3.4 17.6 4.6 7. AS16509 72.21.221.176 0.0% 10 4.6 5.2 3.0 15.5 3.6 8. AS??? 54.240.243.110 0.0% 10 98.6 96.4 86.9 104.4 6.2 9. AS??? 54.240.243.121 90.0% 10 85.7 85.7 85.7 85.7 0.0 10. AS16509 205.251.229.104 20.0% 10 87.5 85.2 82.5 87.8 2.0 11. AS??? 54.239.42.29 70.0% 10 85.0 84.3 82.8 85.2 1.0 12. AS16509 54.239.109.238 30.0% 10 87.4 96.8 87.4 105.2 6.5 AS16509 54.239.110.16
-Grant
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
-- Mike Lyon 408-621-4826 mike.lyon@gmail.com
-- Mike Lyon 408-621-4826 mike.lyon@gmail.com http://www.linkedin.com/in/mlyon
Seeing this issue again 4. AS46375 50.ae4.gw.pao1.sonic.net 0.0% 10 15.5 23.2 2.9 89.3 28.6 5. AS8121 199.127.107.189 0.0% 10 64.2 26.6 3.7 85.7 28.2 6. AS8121 xe0-0-0-7.core2.sv8.layer42.net 0.0% 10 66.6 24.8 3.5 102.1 33.4 7. AS16509 72.21.221.176 0.0% 10 64.2 26.9 3.0 99.1 32.4 8. AS??? 54.240.243.50 0.0% 10 147.4 115.9 88.2 176.9 27.9 9. AS??? 54.240.243.61 90.0% 10 78.4 78.4 78.4 78.4 0.0 10. AS??? 54.239.42.38 10.0% 10 130.4 106.9 71.8 167.6 31.8 11. AS16509 54.239.108.120 70.0% 10 178.9 125.2 90.0 178.9 47.2 12. AS16509 54.239.111.235 20.0% 10 143.1 99.0 70.8 158.1 34.7 13. AS16509 54.239.111.230 70.0% 10 89.5 93.9 88.7 103.5 8.3 -Grant On Fri, Nov 20, 2015 at 5:27 PM, Mike Lyon <mike.lyon@gmail.com> wrote:
Hmmm,
It shows as if it is getting there but look at that jump from 15ms to 85ms on hops 7 and 8:
traceroute amazon.com
traceroute: Warning: amazon.com has multiple addresses; using 176.32.103.205
traceroute to amazon.com (176.32.103.205), 64 hops max, 52 byte packets
1 192.168.20.1 (192.168.20.1) 0.708 ms 0.395 ms 0.328 ms
2 162.250.208.141 (162.250.208.141) 0.671 ms 0.622 ms 0.588 ms
3 10.25.0.1 (10.25.0.1) 1.985 ms 4.165 ms 21.626 ms
4 10.20.0.1 (10.20.0.1) 7.852 ms 7.626 ms 5.214 ms
5 10.0.0.1 (10.0.0.1) 8.041 ms 17.839 ms 34.089 ms
6 69.36.226.193 (69.36.226.193) 5.477 ms 14.228 ms 20.203 ms
7 xe0-2-0-4.core4.scl.layer42.net (65.50.198.237) 15.153 ms 13.475 ms 8.706 ms
8 205.251.229.145 (205.251.229.145) 85.231 ms
54.240.242.108 (54.240.242.108) 96.157 ms
205.251.229.145 (205.251.229.145) 175.633 ms
9 54.240.242.141 (54.240.242.141) 100.007 ms 234.602 ms
54.240.242.121 (54.240.242.121) 94.263 ms
10 54.239.109.102 (54.239.109.102) 110.966 ms
54.239.109.136 (54.239.109.136) 123.322 ms
54.239.110.50 (54.239.110.50) 105.029 ms
11 54.239.109.143 (54.239.109.143) 84.239 ms
54.239.110.23 (54.239.110.23) 91.540 ms
54.239.110.57 (54.239.110.57) 86.620 ms
12 54.239.109.143 (54.239.109.143) 91.382 ms
205.251.245.168 (205.251.245.168) 87.930 ms
54.239.109.143 (54.239.109.143) 76.879 ms
13 * 205.251.245.168 (205.251.245.168) 90.780 ms 96.386 ms
On Fri, Nov 20, 2015 at 5:24 PM, Grant Ridder <shortdudey123@gmail.com> wrote:
Looks like they solved it by routing around layer42 and go via cogent. That in itself is a bad idea :/
4. AS46375 50.ae4.gw.pao1.sonic.net 1.2% 81 3.5 12.0 2.6 91.0 16.2 5. AS174 te0-0-0-15.ccr21.sjc04.atlas.cogentco.com 6.2% 81 6.6 10.5 3.2 79.9 11.7 6. AS174 be2016.ccr22.sfo01.atlas.cogentco.com 3.7% 81 15.3 14.6 4.2 98.9 16.3 7. AS174 be2133.ccr22.mci01.atlas.cogentco.com 4.9% 81 47.7 50.4 41.8 113.6 13.3 8. AS174 be2157.ccr42.ord01.atlas.cogentco.com 70.0% 81 54.1 67.2 53.9 140.2 21.3 9. AS174 be2217.ccr41.ord03.atlas.cogentco.com 33.3% 81 57.3 63.6 54.3 158.4 17.0 10. AS174 ntt.ord03.atlas.cogentco.com 66.2% 81 56.9 72.0 56.5 126.8 20.6 11. AS2914 ae-3.amazon.chcgil09.us.bb.gin.ntt.net 34.6% 81 61.5 61.8 55.9 89.7 8.1
-Grant
On Fri, Nov 20, 2015 at 5:14 PM, Mike Lyon <mike.lyon@gmail.com> wrote:
Oh thank god! I was seeing it too but wasn't sure if was Layer42 or Amazon that was having the issue...
Thanks, Mike
On Fri, Nov 20, 2015 at 5:09 PM, Grant Ridder via Outages < outages@outages.org> wrote:
Saw that, thanks!
-Grant
On Fri, Nov 20, 2015 at 4:49 PM, Tomoc Field <tomoc.field@sonic.com> wrote:
Yes. 70%+ packet loss from 7065 to all Amazon services as far as I can tell. This was experienced via both layer42 and XO, but Amazon appears to have resolved their issues at this time.
-Tomoc
On Fri, Nov 20, 2015 at 04:34:42PM -0800, Grant Ridder via Outages wrote:
Anyone else seeing packet log into us-east-1?
54.240.243.121 (AS16509) is Amazon
MTR report excerpt 4. AS46375 50.ae4.gw.pao1.sonic.net 0.0% 10 2.5 8.3 2.5 37.5 10.7 5. AS8121 199.127.107.189 0.0% 10 3.8 7.9 3.4 34.9 9.5 6. AS8121 xe0-0-0-7.core2.sv8.layer42.net 0.0% 10 3.9 6.0 3.4 17.6 4.6 7. AS16509 72.21.221.176 0.0% 10 4.6 5.2 3.0 15.5 3.6 8. AS??? 54.240.243.110 0.0% 10 98.6 96.4 86.9 104.4 6.2 9. AS??? 54.240.243.121 90.0% 10 85.7 85.7 85.7 85.7 0.0 10. AS16509 205.251.229.104 20.0% 10 87.5 85.2 82.5 87.8 2.0 11. AS??? 54.239.42.29 70.0% 10 85.0 84.3 82.8 85.2 1.0 12. AS16509 54.239.109.238 30.0% 10 87.4 96.8 87.4 105.2 6.5 AS16509 54.239.110.16
-Grant
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
-- Mike Lyon 408-621-4826 mike.lyon@gmail.com
-- Mike Lyon 408-621-4826 mike.lyon@gmail.com
Slack, which is hosted with Amazon is currently down. From: "Grant Ridder via Outages" <outages@outages.org> To: "Mike Lyon" <mike.lyon@gmail.com> Cc: outages@outages.org, "Tomoc Field" <tomoc.field@sonic.com> Sent: Monday, November 23, 2015 2:43:57 PM Subject: Re: [outages] packet loss to us-east-1 Seeing this issue again 4. AS46375 50.ae4.gw.pao1.sonic.net 0.0% 10 15.5 23.2 2.9 89.3 28.6 5. AS8121 199.127.107.189 0.0% 10 64.2 26.6 3.7 85.7 28.2 6. AS8121 xe0-0-0-7.core2.sv8.layer42.net 0.0% 10 66.6 24.8 3.5 102.1 33.4 7. AS16509 72.21.221.176 0.0% 10 64.2 26.9 3.0 99.1 32.4 8. AS??? 54.240.243.50 0.0% 10 147.4 115.9 88.2 176.9 27.9 9. AS??? 54.240.243.61 90.0% 10 78.4 78.4 78.4 78.4 0.0 10. AS??? 54.239.42.38 10.0% 10 130.4 106.9 71.8 167.6 31.8 11. AS16509 54.239.108.120 70.0% 10 178.9 125.2 90.0 178.9 47.2 12. AS16509 54.239.111.235 20.0% 10 143.1 99.0 70.8 158.1 34.7 13. AS16509 54.239.111.230 70.0% 10 89.5 93.9 88.7 103.5 8.3 -Grant On Fri, Nov 20, 2015 at 5:27 PM, Mike Lyon < mike.lyon@gmail.com > wrote: Hmmm, It shows as if it is getting there but look at that jump from 15ms to 85ms on hops 7 and 8: traceroute amazon.com traceroute: Warning: amazon.com has multiple addresses; using 176.32.103.205 traceroute to amazon.com (176.32.103.205), 64 hops max, 52 byte packets 1 192.168.20.1 (192.168.20.1) 0.708 ms 0.395 ms 0.328 ms 2 162.250.208.141 (162.250.208.141) 0.671 ms 0.622 ms 0.588 ms 3 10.25.0.1 (10.25.0.1) 1.985 ms 4.165 ms 21.626 ms 4 10.20.0.1 (10.20.0.1) 7.852 ms 7.626 ms 5.214 ms 5 10.0.0.1 (10.0.0.1) 8.041 ms 17.839 ms 34.089 ms 6 69.36.226.193 (69.36.226.193) 5.477 ms 14.228 ms 20.203 ms 7 xe0-2-0-4.core4.scl.layer42.net (65.50.198.237) 15.153 ms 13.475 ms 8.706 ms 8 205.251.229.145 (205.251.229.145) 85.231 ms 54.240.242.108 (54.240.242.108) 96.157 ms 205.251.229.145 (205.251.229.145) 175.633 ms 9 54.240.242.141 (54.240.242.141) 100.007 ms 234.602 ms 54.240.242.121 (54.240.242.121) 94.263 ms 10 54.239.109.102 (54.239.109.102) 110.966 ms 54.239.109.136 (54.239.109.136) 123.322 ms 54.239.110.50 (54.239.110.50) 105.029 ms 11 54.239.109.143 (54.239.109.143) 84.239 ms 54.239.110.23 (54.239.110.23) 91.540 ms 54.239.110.57 (54.239.110.57) 86.620 ms 12 54.239.109.143 (54.239.109.143) 91.382 ms 205.251.245.168 (205.251.245.168) 87.930 ms 54.239.109.143 (54.239.109.143) 76.879 ms 13 * 205.251.245.168 (205.251.245.168) 90.780 ms 96.386 ms On Fri, Nov 20, 2015 at 5:24 PM, Grant Ridder < shortdudey123@gmail.com > wrote: BQ_BEGIN Looks like they solved it by routing around layer42 and go via cogent. That in itself is a bad idea :/ 4. AS46375 50.ae4.gw.pao1.sonic.net 1.2% 81 3.5 12.0 2.6 91.0 16.2 5. AS174 te0-0-0-15.ccr21.sjc04.atlas.cogentco.com 6.2% 81 6.6 10.5 3.2 79.9 11.7 6. AS174 be2016.ccr22.sfo01.atlas.cogentco.com 3.7% 81 15.3 14.6 4.2 98.9 16.3 7. AS174 be2133.ccr22.mci01.atlas.cogentco.com 4.9% 81 47.7 50.4 41.8 113.6 13.3 8. AS174 be2157.ccr42.ord01.atlas.cogentco.com 70.0% 81 54.1 67.2 53.9 140.2 21.3 9. AS174 be2217.ccr41.ord03.atlas.cogentco.com 33.3% 81 57.3 63.6 54.3 158.4 17.0 10. AS174 ntt.ord03.atlas.cogentco.com 66.2% 81 56.9 72.0 56.5 126.8 20.6 11. AS2914 ae-3.amazon.chcgil09.us.bb.gin.ntt.net 34.6% 81 61.5 61.8 55.9 89.7 8.1 -Grant On Fri, Nov 20, 2015 at 5:14 PM, Mike Lyon < mike.lyon@gmail.com > wrote: BQ_BEGIN Oh thank god! I was seeing it too but wasn't sure if was Layer42 or Amazon that was having the issue... Thanks, Mike On Fri, Nov 20, 2015 at 5:09 PM, Grant Ridder via Outages < outages@outages.org > wrote: BQ_BEGIN Saw that, thanks! -Grant On Fri, Nov 20, 2015 at 4:49 PM, Tomoc Field < tomoc.field@sonic.com > wrote: BQ_BEGIN Yes. 70%+ packet loss from 7065 to all Amazon services as far as I can tell. This was experienced via both layer42 and XO, but Amazon appears to have resolved their issues at this time. -Tomoc On Fri, Nov 20, 2015 at 04:34:42PM -0800, Grant Ridder via Outages wrote:
Anyone else seeing packet log into us-east-1?
54.240.243.121 (AS16509) is Amazon
MTR report excerpt 4. AS46375 50.ae4.gw.pao1.sonic.net 0.0% 10 2.5 8.3 2.5 37.5 10.7 5. AS8121 199.127.107.189 0.0% 10 3.8 7.9 3.4 34.9 9.5 6. AS8121 xe0-0-0-7.core2.sv8.layer42.net 0.0% 10 3.9 6.0 3.4 17.6 4.6 7. AS16509 72.21.221.176 0.0% 10 4.6 5.2 3.0 15.5 3.6 8. AS??? 54.240.243.110 0.0% 10 98.6 96.4 86.9 104.4 6.2 9. AS??? 54.240.243.121 90.0% 10 85.7 85.7 85.7 85.7 0.0 10. AS16509 205.251.229.104 20.0% 10 87.5 85.2 82.5 87.8 2.0 11. AS??? 54.239.42.29 70.0% 10 85.0 84.3 82.8 85.2 1.0 12. AS16509 54.239.109.238 30.0% 10 87.4 96.8 87.4 105.2 6.5 AS16509 54.239.110.16
-Grant
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages BQ_END -- Mike Lyon 408-621-4826 mike.lyon@gmail.com http://www.linkedin.com/in/mlyon BQ_END BQ_END -- Mike Lyon 408-621-4826 mike.lyon@gmail.com http://www.linkedin.com/in/mlyon BQ_END _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
I'm not sure these are related (I'm seeing zero packet loss to Slack from Chicago on Cogent) but Slack's definitely broken. Seemed to have been a cascading failure, a colleague of mine complained over IRC he got booted from his, then I got booted from one Slack team, then another.
On Nov 23, 2015, at 1:49 PM, Justin Oeder via Outages <outages@outages.org> wrote:
Slack, which is hosted with Amazon is currently down.
From: "Grant Ridder via Outages" <outages@outages.org> To: "Mike Lyon" <mike.lyon@gmail.com> Cc: outages@outages.org, "Tomoc Field" <tomoc.field@sonic.com> Sent: Monday, November 23, 2015 2:43:57 PM Subject: Re: [outages] packet loss to us-east-1
Seeing this issue again 4. AS46375 50.ae4.gw.pao1.sonic.net <http://50.ae4.gw.pao1.sonic.net/> 0.0% 10 15.5 23.2 2.9 89.3 28.6 5. AS8121 199.127.107.189 0.0% 10 64.2 26.6 3.7 85.7 28.2 6. AS8121 xe0-0-0-7.core2.sv8.layer42.net <http://xe0-0-0-7.core2.sv8.layer42.net/> 0.0% 10 66.6 24.8 3.5 102.1 33.4 7. AS16509 72.21.221.176 0.0% 10 64.2 26.9 3.0 99.1 32.4 8. AS??? 54.240.243.50 0.0% 10 147.4 115.9 88.2 176.9 27.9 9. AS??? 54.240.243.61 90.0% 10 78.4 78.4 78.4 78.4 0.0 10. AS??? 54.239.42.38 10.0% 10 130.4 106.9 71.8 167.6 31.8 11. AS16509 54.239.108.120 70.0% 10 178.9 125.2 90.0 178.9 47.2 12. AS16509 54.239.111.235 20.0% 10 143.1 99.0 70.8 158.1 34.7 13. AS16509 54.239.111.230 70.0% 10 89.5 93.9 88.7 103.5 8.3
-Grant
On Fri, Nov 20, 2015 at 5:27 PM, Mike Lyon <mike.lyon@gmail.com <mailto:mike.lyon@gmail.com>> wrote: Hmmm, It shows as if it is getting there but look at that jump from 15ms to 85ms on hops 7 and 8:
traceroute amazon.com <http://amazon.com/> traceroute: Warning: amazon.com <http://amazon.com/> has multiple addresses; using 176.32.103.205
traceroute to amazon.com <http://amazon.com/> (176.32.103.205), 64 hops max, 52 byte packets
1 192.168.20.1 (192.168.20.1) 0.708 ms 0.395 ms 0.328 ms
2 162.250.208.141 (162.250.208.141) 0.671 ms 0.622 ms 0.588 ms
3 10.25.0.1 (10.25.0.1) 1.985 ms 4.165 ms 21.626 ms
4 10.20.0.1 (10.20.0.1) 7.852 ms 7.626 ms 5.214 ms
5 10.0.0.1 (10.0.0.1) 8.041 ms 17.839 ms 34.089 ms
6 69.36.226.193 (69.36.226.193) 5.477 ms 14.228 ms 20.203 ms
7 xe0-2-0-4.core4.scl.layer42.net <http://xe0-2-0-4.core4.scl.layer42.net/> (65.50.198.237) 15.153 ms 13.475 ms 8.706 ms
8 205.251.229.145 (205.251.229.145) 85.231 ms
54.240.242.108 (54.240.242.108) 96.157 ms
205.251.229.145 (205.251.229.145) 175.633 ms
9 54.240.242.141 (54.240.242.141) 100.007 ms 234.602 ms
54.240.242.121 (54.240.242.121) 94.263 ms
10 54.239.109.102 (54.239.109.102) 110.966 ms
54.239.109.136 (54.239.109.136) 123.322 ms
54.239.110.50 (54.239.110.50) 105.029 ms
11 54.239.109.143 (54.239.109.143) 84.239 ms
54.239.110.23 (54.239.110.23) 91.540 ms
54.239.110.57 (54.239.110.57) 86.620 ms
12 54.239.109.143 (54.239.109.143) 91.382 ms
205.251.245.168 (205.251.245.168) 87.930 ms
54.239.109.143 (54.239.109.143) 76.879 ms
13 * 205.251.245.168 (205.251.245.168) 90.780 ms 96.386 ms
On Fri, Nov 20, 2015 at 5:24 PM, Grant Ridder <shortdudey123@gmail.com <mailto:shortdudey123@gmail.com>> wrote: Looks like they solved it by routing around layer42 and go via cogent. That in itself is a bad idea :/ 4. AS46375 50.ae4.gw.pao1.sonic.net <http://50.ae4.gw.pao1.sonic.net/> 1.2% 81 3.5 12.0 2.6 91.0 16.2 5. AS174 te0-0-0-15.ccr21.sjc04.atlas.cogentco.com <http://te0-0-0-15.ccr21.sjc04.atlas.cogentco.com/> 6.2% 81 6.6 10.5 3.2 79.9 11.7 6. AS174 be2016.ccr22.sfo01.atlas.cogentco.com <http://be2016.ccr22.sfo01.atlas.cogentco.com/> 3.7% 81 15.3 14.6 4.2 98.9 16.3 7. AS174 be2133.ccr22.mci01.atlas.cogentco.com <http://be2133.ccr22.mci01.atlas.cogentco.com/> 4.9% 81 47.7 50.4 41.8 113.6 13.3 8. AS174 be2157.ccr42.ord01.atlas.cogentco.com <http://be2157.ccr42.ord01.atlas.cogentco.com/> 70.0% 81 54.1 67.2 53.9 140.2 21.3 9. AS174 be2217.ccr41.ord03.atlas.cogentco.com <http://be2217.ccr41.ord03.atlas.cogentco.com/> 33.3% 81 57.3 63.6 54.3 158.4 17.0 10. AS174 ntt.ord03.atlas.cogentco.com <http://ntt.ord03.atlas.cogentco.com/> 66.2% 81 56.9 72.0 56.5 126.8 20.6 11. AS2914 ae-3.amazon.chcgil09.us.bb.gin.ntt.net <http://ae-3.amazon.chcgil09.us.bb.gin.ntt.net/> 34.6% 81 61.5 61.8 55.9 89.7 8.1
-Grant
On Fri, Nov 20, 2015 at 5:14 PM, Mike Lyon <mike.lyon@gmail.com <mailto:mike.lyon@gmail.com>> wrote: Oh thank god! I was seeing it too but wasn't sure if was Layer42 or Amazon that was having the issue... Thanks, Mike
On Fri, Nov 20, 2015 at 5:09 PM, Grant Ridder via Outages <outages@outages.org <mailto:outages@outages.org>> wrote: Saw that, thanks! -Grant
On Fri, Nov 20, 2015 at 4:49 PM, Tomoc Field <tomoc.field@sonic.com <mailto:tomoc.field@sonic.com>> wrote: Yes. 70%+ packet loss from 7065 to all Amazon services as far as I can tell. This was experienced via both layer42 and XO, but Amazon appears to have resolved their issues at this time.
-Tomoc
On Fri, Nov 20, 2015 at 04:34:42PM -0800, Grant Ridder via Outages wrote:
Anyone else seeing packet log into us-east-1?
54.240.243.121 (AS16509) is Amazon
MTR report excerpt 4. AS46375 50.ae4.gw.pao1.sonic.net <http://50.ae4.gw.pao1.sonic.net/> 0.0% 10 2.5 8.3 2.5 37.5 10.7 <tel:8.3%20%20%202.5%20%2037.5%20%2010.7> 5. AS8121 199.127.107.189 0.0% 10 3.8 7.9 3.4 34.9 9.5 6. AS8121 xe0-0-0-7.core2.sv8.layer42.net <http://xe0-0-0-7.core2.sv8.layer42.net/> 0.0% 10 3.9 6.0 3.4 17.6 4.6 7. AS16509 72.21.221.176 0.0% 10 4.6 5.2 3.0 15.5 3.6 8. AS??? 54.240.243.110 0.0% 10 98.6 96.4 86.9 104.4 6.2 9. AS??? 54.240.243.121 90.0% 10 85.7 85.7 85.7 85.7 0.0 10. AS16509 205.251.229.104 20.0% 10 87.5 85.2 82.5 87.8 2.0 11. AS??? 54.239.42.29 70.0% 10 85.0 84.3 82.8 85.2 1.0 12. AS16509 54.239.109.238 30.0% 10 87.4 96.8 87.4 105.2 6.5 AS16509 54.239.110.16
-Grant
_______________________________________________ Outages mailing list Outages@outages.org <mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages <https://puck.nether.net/mailman/listinfo/outages>
_______________________________________________ Outages mailing list Outages@outages.org <mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages <https://puck.nether.net/mailman/listinfo/outages>
-- Mike Lyon 408-621-4826 <tel:408-621-4826> mike.lyon@gmail.com <mailto:mike.lyon@gmail.com> http://www.linkedin.com/in/mlyon <http://www.linkedin.com/in/mlyon>
-- Mike Lyon 408-621-4826 <tel:408-621-4826> mike.lyon@gmail.com <mailto:mike.lyon@gmail.com> http://www.linkedin.com/in/mlyon <http://www.linkedin.com/in/mlyon>
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
It is related for some since Slack is in us-east-1 On Mon, Nov 23, 2015 at 11:52 AM, Chris Swingler <chris@chrisswingler.com> wrote:
I'm not sure these are related (I'm seeing zero packet loss to Slack from Chicago on Cogent) but Slack's definitely broken. Seemed to have been a cascading failure, a colleague of mine complained over IRC he got booted from his, then I got booted from one Slack team, then another.
On Nov 23, 2015, at 1:49 PM, Justin Oeder via Outages <outages@outages.org> wrote:
Slack, which is hosted with Amazon is currently down.
------------------------------ *From: *"Grant Ridder via Outages" <outages@outages.org> *To: *"Mike Lyon" <mike.lyon@gmail.com> *Cc: *outages@outages.org, "Tomoc Field" <tomoc.field@sonic.com> *Sent: *Monday, November 23, 2015 2:43:57 PM *Subject: *Re: [outages] packet loss to us-east-1
Seeing this issue again 4. AS46375 50.ae4.gw.pao1.sonic.net 0.0% 10 15.5 23.2 2.9 89.3 28.6 5. AS8121 199.127.107.189 0.0% 10 64.2 26.6 3.7 85.7 28.2 6. AS8121 xe0-0-0-7.core2.sv8.layer42.net 0.0% 10 66.6 24.8 3.5 102.1 33.4 7. AS16509 72.21.221.176 0.0% 10 64.2 26.9 3.0 99.1 32.4 8. AS??? 54.240.243.50 0.0% 10 147.4 115.9 88.2 176.9 27.9 9. AS??? 54.240.243.61 90.0% 10 78.4 78.4 78.4 78.4 0.0 10. AS??? 54.239.42.38 10.0% 10 130.4 106.9 71.8 167.6 31.8 11. AS16509 54.239.108.120 70.0% 10 178.9 125.2 90.0 178.9 47.2 12. AS16509 54.239.111.235 20.0% 10 143.1 99.0 70.8 158.1 34.7 13. AS16509 54.239.111.230 70.0% 10 89.5 93.9 88.7 103.5 8.3
-Grant
On Fri, Nov 20, 2015 at 5:27 PM, Mike Lyon <mike.lyon@gmail.com> wrote:
Hmmm, It shows as if it is getting there but look at that jump from 15ms to 85ms on hops 7 and 8:
traceroute amazon.com
traceroute: Warning: amazon.com has multiple addresses; using 176.32.103.205
traceroute to amazon.com (176.32.103.205), 64 hops max, 52 byte packets
1 192.168.20.1 (192.168.20.1) 0.708 ms 0.395 ms 0.328 ms
2 162.250.208.141 (162.250.208.141) 0.671 ms 0.622 ms 0.588 ms
3 10.25.0.1 (10.25.0.1) 1.985 ms 4.165 ms 21.626 ms
4 10.20.0.1 (10.20.0.1) 7.852 ms 7.626 ms 5.214 ms
5 10.0.0.1 (10.0.0.1) 8.041 ms 17.839 ms 34.089 ms
6 69.36.226.193 (69.36.226.193) 5.477 ms 14.228 ms 20.203 ms
7 xe0-2-0-4.core4.scl.layer42.net (65.50.198.237) 15.153 ms 13.475 ms 8.706 ms
8 205.251.229.145 (205.251.229.145) 85.231 ms
54.240.242.108 (54.240.242.108) 96.157 ms
205.251.229.145 (205.251.229.145) 175.633 ms
9 54.240.242.141 (54.240.242.141) 100.007 ms 234.602 ms
54.240.242.121 (54.240.242.121) 94.263 ms
10 54.239.109.102 (54.239.109.102) 110.966 ms
54.239.109.136 (54.239.109.136) 123.322 ms
54.239.110.50 (54.239.110.50) 105.029 ms
11 54.239.109.143 (54.239.109.143) 84.239 ms
54.239.110.23 (54.239.110.23) 91.540 ms
54.239.110.57 (54.239.110.57) 86.620 ms
12 54.239.109.143 (54.239.109.143) 91.382 ms
205.251.245.168 (205.251.245.168) 87.930 ms
54.239.109.143 (54.239.109.143) 76.879 ms
13 * 205.251.245.168 (205.251.245.168) 90.780 ms 96.386 ms
On Fri, Nov 20, 2015 at 5:24 PM, Grant Ridder <shortdudey123@gmail.com> wrote:
Looks like they solved it by routing around layer42 and go via cogent. That in itself is a bad idea :/ 4. AS46375 50.ae4.gw.pao1.sonic.net 1.2% 81 3.5 12.0 2.6 91.0 16.2 5. AS174 te0-0-0-15.ccr21.sjc04.atlas.cogentco.com 6.2% 81 6.6 10.5 3.2 79.9 11.7 6. AS174 be2016.ccr22.sfo01.atlas.cogentco.com 3.7% 81 15.3 14.6 4.2 98.9 16.3 7. AS174 be2133.ccr22.mci01.atlas.cogentco.com 4.9% 81 47.7 50.4 41.8 113.6 13.3 8. AS174 be2157.ccr42.ord01.atlas.cogentco.com 70.0% 81 54.1 67.2 53.9 140.2 21.3 9. AS174 be2217.ccr41.ord03.atlas.cogentco.com 33.3% 81 57.3 63.6 54.3 158.4 17.0 10. AS174 ntt.ord03.atlas.cogentco.com 66.2% 81 56.9 72.0 56.5 126.8 20.6 11. AS2914 ae-3.amazon.chcgil09.us.bb.gin.ntt.net 34.6% 81 61.5 61.8 55.9 89.7 8.1
-Grant
On Fri, Nov 20, 2015 at 5:14 PM, Mike Lyon <mike.lyon@gmail.com> wrote:
Oh thank god! I was seeing it too but wasn't sure if was Layer42 or Amazon that was having the issue... Thanks, Mike
On Fri, Nov 20, 2015 at 5:09 PM, Grant Ridder via Outages < outages@outages.org> wrote:
Saw that, thanks! -Grant
On Fri, Nov 20, 2015 at 4:49 PM, Tomoc Field <tomoc.field@sonic.com> wrote:
Yes. 70%+ packet loss from 7065 to all Amazon services as far as I can tell. This was experienced via both layer42 and XO, but Amazon appears to have resolved their issues at this time.
-Tomoc
On Fri, Nov 20, 2015 at 04:34:42PM -0800, Grant Ridder via Outages wrote: > Anyone else seeing packet log into us-east-1? > > 54.240.243.121 (AS16509) is Amazon > > MTR report excerpt > 4. AS46375 50.ae4.gw.pao1.sonic.net 0.0% 10 2.5 > 8.3 2.5 37.5 10.7 > 5. AS8121 199.127.107.189 0.0% 10 3.8 > 7.9 3.4 34.9 9.5 > 6. AS8121 xe0-0-0-7.core2.sv8.layer42.net 0.0% 10 3.9 > 6.0 3.4 17.6 4.6 > 7. AS16509 72.21.221.176 0.0% 10 4.6 > 5.2 3.0 15.5 3.6 > 8. AS??? 54.240.243.110 0.0% 10 98.6 > 96.4 86.9 104.4 6.2 > 9. AS??? 54.240.243.121 90.0% 10 85.7 > 85.7 85.7 85.7 0.0 > 10. AS16509 205.251.229.104 20.0% 10 87.5 > 85.2 82.5 87.8 2.0 > 11. AS??? 54.239.42.29 70.0% 10 85.0 > 84.3 82.8 85.2 1.0 > 12. AS16509 54.239.109.238 30.0% 10 87.4 > 96.8 87.4 105.2 6.5 > AS16509 54.239.110.16 > > > -Grant
> _______________________________________________ > Outages mailing list > Outages@outages.org > https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
-- Mike Lyon 408-621-4826 mike.lyon@gmail.com
-- Mike Lyon 408-621-4826 mike.lyon@gmail.com
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
participants (4)
-
Chris Swingler -
Grant Ridder -
Justin Oeder -
Mike Lyon