Possible outage affecting Amazon.com?
Is anyone else seeing traffic routing all over the place trying to get to Amazon.com? Chris
No problems in East King County, WA. traceroute to www.amazon.com (205.251.242.54), 64 hops max, 52 byte packets 1 * gw (192.168.1.1) 1.424 ms 0.733 ms 2 10.60.128.1 (10.60.128.1) 9.041 ms 8.490 ms 11.409 ms 3 24-113-43-225.wavecable.com (24.113.43.225) 8.609 ms 8.934 ms 11.994 ms 4 24-113-43-30.wavecable.com (24.113.43.30) 11.154 ms 10.728 ms 14.028 ms 5 24-113-43-34.wavecable.com (24.113.43.34) 14.343 ms 11.580 ms 10.602 ms 6 24-113-43-93.wavecable.com (24.113.43.93) 14.553 ms 15.583 ms 16.086 ms 7 agg2-sea-b-p200.bb.spectrumnet.us (216.243.24.93) 13.873 ms 16.139 ms 13.539 ms 8 xe-0-5-0-3.r04.sttlwa01.us.bb.gin.ntt.net (198.104.202.17) 14.380 ms 13.713 ms 15.128 ms 9 ae-6.r20.sttlwa01.us.bb.gin.ntt.net (129.250.5.42) 15.613 ms 28.168 ms 14.479 ms 10 ae-5.r21.snjsca04.us.bb.gin.ntt.net (129.250.3.39) 39.677 ms ae-0.r20.sttlwa01.us.bb.gin.ntt.net (129.250.2.53) 13.502 ms 42.304 ms 11 ae-5.r21.dllstx09.us.bb.gin.ntt.net (129.250.4.24) 73.945 ms ae-5.r21.snjsca04.us.bb.gin.ntt.net (129.250.3.39) 30.644 ms 32.001 ms 12 ae-2.r08.dllstx09.us.bb.gin.ntt.net (129.250.3.81) 80.939 ms 73.379 ms ae-5.r21.dllstx09.us.bb.gin.ntt.net (129.250.4.24) 74.897 ms 13 ae-2.amazon.dllstx09.us.bb.gin.ntt.net (129.250.201.158) 63.314 ms ae-1.amazon.dllstx09.us.bb.gin.ntt.net (129.250.201.154) 65.644 ms 63.799 ms 14 54.240.229.106 (54.240.229.106) 99.587 ms ae-1.amazon.dllstx09.us.bb.gin.ntt.net (129.250.201.154) 68.553 ms 54.240.229.108 (54.240.229.108) 96.919 ms 15 54.240.229.14 (54.240.229.14) 109.246 ms 54.240.229.18 (54.240.229.18) 92.771 ms 54.240.229.104 (54.240.229.104) 97.733 ms 16 54.240.229.2 (54.240.229.2) 92.322 ms 54.240.229.16 (54.240.229.16) 94.506 ms 54.240.229.8 (54.240.229.8) 114.901 ms 17 54.240.229.174 (54.240.229.174) 91.388 ms * 54.240.229.170 (54.240.229.170) 99.532 ms 18 205.251.245.33 (205.251.245.33) 90.515 ms 92.953 ms 54.240.229.129 (54.240.229.129) 111.709 ms 19 205.251.245.33 (205.251.245.33) 95.427 ms 205.251.245.41 (205.251.245.41) 93.439 ms 205.251.245.121 (205.251.245.121) 109.442 ms 20 * * * --Dj On Feb 21, 2014, at 3:36 PM, chris@htswireless.com wrote:
Is anyone else seeing traffic routing all over the place trying to get to Amazon.com?
Chris
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
I happen to be in Marrakech at the moment and am able to access it. ______________________________________________________________ Clayton Dukes ______________________________________________________________ On Sat, Feb 22, 2014 at 12:36 AM, <chris@htswireless.com> wrote:
Is anyone else seeing traffic routing all over the place trying to get to Amazon.com?
Chris
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
No problems from comcast in Mountain View ca On Fri, Feb 21, 2014 at 3:47 PM, Clayton Dukes <cdukes@gmail.com> wrote:
I happen to be in Marrakech at the moment and am able to access it.
______________________________________________________________
Clayton Dukes ______________________________________________________________
On Sat, Feb 22, 2014 at 12:36 AM, <chris@htswireless.com> wrote:
Is anyone else seeing traffic routing all over the place trying to get to Amazon.com?
Chris
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Looks like (more) Level3 problems... --Dj On Feb 21, 2014, at 3:57 PM, Grant Ridder <shortdudey123@gmail.com> wrote:
No problems from comcast in Mountain View ca
On Fri, Feb 21, 2014 at 3:47 PM, Clayton Dukes <cdukes@gmail.com> wrote: I happen to be in Marrakech at the moment and am able to access it.
______________________________________________________________
Clayton Dukes ______________________________________________________________
On Sat, Feb 22, 2014 at 12:36 AM, <chris@htswireless.com> wrote: Is anyone else seeing traffic routing all over the place trying to get to Amazon.com?
Chris
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Am in Colorado on CenturyLink DSL. When I try and trace to www.amazon.com, it fails: Start: Fri Feb 21 17:07:37 2014 HOST: orion Loss% Snt Last Avg Best Wrst StDev 1.|-- 0.0% 10 0.4 0.4 0.3 0.5 0.0 2.|-- hlrn-dsl-gw07.hlrn.qwest. 0.0% 10 35.4 38.8 34.9 70.7 11.2 3.|-- hlrn-agw1.inet.qwest.net 0.0% 10 35.2 35.6 34.7 40.6 1.7 4.|-- dca2-edge-01.inet.qwest.n 50.0% 10 85.0 85.2 84.9 85.6 0.0 5.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0 But the web site comes up fine in my browser. Checked IPs and tried with that and still the same - site comes up fine but trace and pings fail at CTL: 17:08:49-cls@orion:~/scripts/ctr$ host www.amazon.com www.amazon.com has address 72.21.215.232 17:09:40-cls@orion:~/scripts/ctr$ mtr 72.21.215.232 17:09:49-cls@orion:~/scripts/ctr$ mtr --report --report-cycles=10 72.21.215.232 Start: Fri Feb 21 17:10:02 2014 HOST: orion Loss% Snt Last Avg Best Wrst StDev 1.|-- 0.0% 10 0.5 0.5 0.4 1.3 0.0 2.|-- hlrn-dsl-gw07.hlrn.qwest. 0.0% 10 35.4 41.0 35.1 89.4 17.0 3.|-- hlrn-agw1.inet.qwest.net 0.0% 10 35.1 35.0 34.9 35.4 0.0 4.|-- dca2-edge-01.inet.qwest.n 0.0% 10 85.1 85.2 84.9 85.5 0.0 5.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0 17:10:17-cls@orion:~/scripts/ctr$ ping 72.21.215.232 PING 72.21.215.232 (72.21.215.232) 56(84) bytes of data. ^C --- 72.21.215.232 ping statistics --- 1 packets transmitted, 0 received, 100% packet loss, time 0ms Same going out our data center in Denver on Cogent: HOST: pegasus Loss% Snt Last Avg Best Wrst StDev 1. core 0.0% 3 0.6 0.8 0.6 1.1 0.3 2. fa0-2.na01.b009854-0.den01.a 0.0% 3 0.9 1.8 0.9 2.9 1.0 3. vl3509.mag02.den01.atlas.cog 0.0% 3 1.9 1.3 0.9 1.9 0.5 4. te0-7-0-5.ccr21.den01.atlas. 0.0% 3 1.1 1.1 1.1 1.1 0.0 5. be2129.mpd21.mci01.atlas.cog 0.0% 3 13.1 13.0 12.8 13.1 0.1 6. be2064.ccr21.dfw01.atlas.cog 0.0% 3 22.8 23.0 22.8 23.3 0.3 7. be2031.ccr21.dfw03.atlas.cog 0.0% 3 23.5 23.9 23.5 24.2 0.4 8. ntt.dfw03.atlas.cogentco.com 0.0% 3 23.7 23.9 23.7 24.5 0.4 9. ae-1.r08.dllstx09.us.bb.gin. 0.0% 3 25.2 24.2 23.7 25.2 0.8 10. ??? 100.0 3 0.0 0.0 0.0 0.0 0.0 Chris On 02/21/2014 05:00 PM, Dj Padzensky wrote:
Looks like (more) Level3 problems...
--Dj
On Feb 21, 2014, at 3:57 PM, Grant Ridder <shortdudey123@gmail.com <mailto:shortdudey123@gmail.com>> wrote:
No problems from comcast in Mountain View ca
On Fri, Feb 21, 2014 at 3:47 PM, Clayton Dukes <cdukes@gmail.com <mailto:cdukes@gmail.com>> wrote:
I happen to be in Marrakech at the moment and am able to access it.
______________________________________________________________
Clayton Dukes ______________________________________________________________
On Sat, Feb 22, 2014 at 12:36 AM, <chris@htswireless.com <mailto:chris@htswireless.com>> wrote:
Is anyone else seeing traffic routing all over the place trying to get to Amazon.com <http://Amazon.com>?
Chris
_______________________________________________ Outages mailing list Outages@outages.org <mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org <mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org <mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
What you're seeing is commonplace too. I would say both ICMP TTL exceeded (used for route/path discovery) and ECHO/ECHO REPLY look to be filtered at border routers, probably border routers that Amazon owns/maintains. While mtr by default uses ICMP exclusively, ICMP TTL exceeded is still used for path discovery, hence people thinking "there must be something wrong". mtr does have a -T flag that uses TCP SYN instead of ICMP ECHO for endpoint reachability, i.e. mtr -T -P 80 www.amazon.com, but all it does for me on FreeBSD is bail out with a hard-to-read error ("bind(): No error: 0") and doesn't reset tty params on failure, resulting in a wonderful mess of a terminal (requires "stty icanon isig iexten echo icrnl ixon brkint onlcr" be run before things begin behaving again). But traceroute -P tcp -p 80 www.amazon.com doesn't improve things either, going back to my theory that TTL exceeded is probably filtered. Testing end-to-end like you did (in a browser, telnet port 80, etc.) is one of the only things one can do in this situation. <cry> I love how network admins still to this day filter all this, making troubleshooting virtually impossible. The mentality I find hilarious; "if we permit it, someone could ping -f the router's IP that responds to TTL exceeded and cause high CPU" -- and if that doesn't work, guess what the packet kids do? Send 200gbit/sec of traffic at your uplink, effectively accomplishing the same result of knocking you offline. And no I will not respond to commentaries about this section of my mail. </cry> -- | Jeremy Chadwick jdc@koitsu.org | | UNIX Systems Administrator http://jdc.koitsu.org/ | | Making life hard for others since 1977. PGP 4BD6C0CB | On Fri, Feb 21, 2014 at 05:13:45PM -0700, Chris Stone wrote:
Am in Colorado on CenturyLink DSL. When I try and trace to www.amazon.com, it fails:
Start: Fri Feb 21 17:07:37 2014 HOST: orion Loss% Snt Last Avg Best Wrst StDev 1.|-- 0.0% 10 0.4 0.4 0.3 0.5 0.0 2.|-- hlrn-dsl-gw07.hlrn.qwest. 0.0% 10 35.4 38.8 34.9 70.7 11.2 3.|-- hlrn-agw1.inet.qwest.net 0.0% 10 35.2 35.6 34.7 40.6 1.7 4.|-- dca2-edge-01.inet.qwest.n 50.0% 10 85.0 85.2 84.9 85.6 0.0 5.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0
But the web site comes up fine in my browser. Checked IPs and tried with that and still the same - site comes up fine but trace and pings fail at CTL:
17:08:49-cls@orion:~/scripts/ctr$ host www.amazon.com www.amazon.com has address 72.21.215.232 17:09:40-cls@orion:~/scripts/ctr$ mtr 72.21.215.232 17:09:49-cls@orion:~/scripts/ctr$ mtr --report --report-cycles=10 72.21.215.232 Start: Fri Feb 21 17:10:02 2014 HOST: orion Loss% Snt Last Avg Best Wrst StDev 1.|-- 0.0% 10 0.5 0.5 0.4 1.3 0.0 2.|-- hlrn-dsl-gw07.hlrn.qwest. 0.0% 10 35.4 41.0 35.1 89.4 17.0 3.|-- hlrn-agw1.inet.qwest.net 0.0% 10 35.1 35.0 34.9 35.4 0.0 4.|-- dca2-edge-01.inet.qwest.n 0.0% 10 85.1 85.2 84.9 85.5 0.0 5.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0 17:10:17-cls@orion:~/scripts/ctr$ ping 72.21.215.232 PING 72.21.215.232 (72.21.215.232) 56(84) bytes of data. ^C --- 72.21.215.232 ping statistics --- 1 packets transmitted, 0 received, 100% packet loss, time 0ms
Same going out our data center in Denver on Cogent:
HOST: pegasus Loss% Snt Last Avg Best Wrst StDev 1. core 0.0% 3 0.6 0.8 0.6 1.1 0.3 2. fa0-2.na01.b009854-0.den01.a 0.0% 3 0.9 1.8 0.9 2.9 1.0 3. vl3509.mag02.den01.atlas.cog 0.0% 3 1.9 1.3 0.9 1.9 0.5 4. te0-7-0-5.ccr21.den01.atlas. 0.0% 3 1.1 1.1 1.1 1.1 0.0 5. be2129.mpd21.mci01.atlas.cog 0.0% 3 13.1 13.0 12.8 13.1 0.1 6. be2064.ccr21.dfw01.atlas.cog 0.0% 3 22.8 23.0 22.8 23.3 0.3 7. be2031.ccr21.dfw03.atlas.cog 0.0% 3 23.5 23.9 23.5 24.2 0.4 8. ntt.dfw03.atlas.cogentco.com 0.0% 3 23.7 23.9 23.7 24.5 0.4 9. ae-1.r08.dllstx09.us.bb.gin. 0.0% 3 25.2 24.2 23.7 25.2 0.8 10. ??? 100.0 3 0.0 0.0 0.0 0.0 0.0
Chris
On 02/21/2014 05:00 PM, Dj Padzensky wrote:
Looks like (more) Level3 problems...
--Dj
On Feb 21, 2014, at 3:57 PM, Grant Ridder <shortdudey123@gmail.com <mailto:shortdudey123@gmail.com>> wrote:
No problems from comcast in Mountain View ca
On Fri, Feb 21, 2014 at 3:47 PM, Clayton Dukes <cdukes@gmail.com <mailto:cdukes@gmail.com>> wrote:
I happen to be in Marrakech at the moment and am able to access it.
______________________________________________________________
Clayton Dukes ______________________________________________________________
On Sat, Feb 22, 2014 at 12:36 AM, <chris@htswireless.com <mailto:chris@htswireless.com>> wrote:
Is anyone else seeing traffic routing all over the place trying to get to Amazon.com <http://Amazon.com>?
Chris
_______________________________________________ Outages mailing list Outages@outages.org <mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org <mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org <mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
The main problem we were seeing is that the main page would only partially load, and you could not browse to other pages. Although it does seem to be working much better now. Chris -----Original Message----- From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Jeremy Chadwick Sent: Friday, February 21, 2014 6:47 PM To: Chris Stone Cc: outages@outages.org Subject: Re: [outages] Possible outage affecting Amazon.com? What you're seeing is commonplace too. I would say both ICMP TTL exceeded (used for route/path discovery) and ECHO/ECHO REPLY look to be filtered at border routers, probably border routers that Amazon owns/maintains. While mtr by default uses ICMP exclusively, ICMP TTL exceeded is still used for path discovery, hence people thinking "there must be something wrong". mtr does have a -T flag that uses TCP SYN instead of ICMP ECHO for endpoint reachability, i.e. mtr -T -P 80 www.amazon.com, but all it does for me on FreeBSD is bail out with a hard-to-read error ("bind(): No error: 0") and doesn't reset tty params on failure, resulting in a wonderful mess of a terminal (requires "stty icanon isig iexten echo icrnl ixon brkint onlcr" be run before things begin behaving again). But traceroute -P tcp -p 80 www.amazon.com doesn't improve things either, going back to my theory that TTL exceeded is probably filtered. Testing end-to-end like you did (in a browser, telnet port 80, etc.) is one of the only things one can do in this situation. <cry> I love how network admins still to this day filter all this, making troubleshooting virtually impossible. The mentality I find hilarious; "if we permit it, someone could ping -f the router's IP that responds to TTL exceeded and cause high CPU" -- and if that doesn't work, guess what the packet kids do? Send 200gbit/sec of traffic at your uplink, effectively accomplishing the same result of knocking you offline. And no I will not respond to commentaries about this section of my mail. </cry> -- | Jeremy Chadwick jdc@koitsu.org | | UNIX Systems Administrator http://jdc.koitsu.org/ | | Making life hard for others since 1977. PGP 4BD6C0CB | On Fri, Feb 21, 2014 at 05:13:45PM -0700, Chris Stone wrote:
Am in Colorado on CenturyLink DSL. When I try and trace to www.amazon.com, it fails:
Start: Fri Feb 21 17:07:37 2014 HOST: orion Loss% Snt Last Avg Best Wrst StDev 1.|-- 0.0% 10 0.4 0.4 0.3 0.5 0.0 2.|-- hlrn-dsl-gw07.hlrn.qwest. 0.0% 10 35.4 38.8 34.9 70.7 11.2 3.|-- hlrn-agw1.inet.qwest.net 0.0% 10 35.2 35.6 34.7 40.6 1.7 4.|-- dca2-edge-01.inet.qwest.n 50.0% 10 85.0 85.2 84.9 85.6 0.0 5.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0
But the web site comes up fine in my browser. Checked IPs and tried with that and still the same - site comes up fine but trace and pings fail at CTL:
17:08:49-cls@orion:~/scripts/ctr$ host www.amazon.com www.amazon.com has address 72.21.215.232 17:09:40-cls@orion:~/scripts/ctr$ mtr 72.21.215.232 17:09:49-cls@orion:~/scripts/ctr$ mtr --report --report-cycles=10 72.21.215.232 Start: Fri Feb 21 17:10:02 2014 HOST: orion Loss% Snt Last Avg Best Wrst StDev 1.|-- 0.0% 10 0.5 0.5 0.4 1.3 0.0 2.|-- hlrn-dsl-gw07.hlrn.qwest. 0.0% 10 35.4 41.0 35.1 89.4 17.0 3.|-- hlrn-agw1.inet.qwest.net 0.0% 10 35.1 35.0 34.9 35.4 0.0 4.|-- dca2-edge-01.inet.qwest.n 0.0% 10 85.1 85.2 84.9 85.5 0.0 5.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0 17:10:17-cls@orion:~/scripts/ctr$ ping 72.21.215.232 PING 72.21.215.232 (72.21.215.232) 56(84) bytes of data. ^C --- 72.21.215.232 ping statistics --- 1 packets transmitted, 0 received, 100% packet loss, time 0ms
Same going out our data center in Denver on Cogent:
HOST: pegasus Loss% Snt Last Avg Best Wrst StDev 1. core 0.0% 3 0.6 0.8 0.6 1.1 0.3 2. fa0-2.na01.b009854-0.den01.a 0.0% 3 0.9 1.8 0.9 2.9 1.0 3. vl3509.mag02.den01.atlas.cog 0.0% 3 1.9 1.3 0.9 1.9 0.5 4. te0-7-0-5.ccr21.den01.atlas. 0.0% 3 1.1 1.1 1.1 1.1 0.0 5. be2129.mpd21.mci01.atlas.cog 0.0% 3 13.1 13.0 12.8 13.1 0.1 6. be2064.ccr21.dfw01.atlas.cog 0.0% 3 22.8 23.0 22.8 23.3 0.3 7. be2031.ccr21.dfw03.atlas.cog 0.0% 3 23.5 23.9 23.5 24.2 0.4 8. ntt.dfw03.atlas.cogentco.com 0.0% 3 23.7 23.9 23.7 24.5 0.4 9. ae-1.r08.dllstx09.us.bb.gin. 0.0% 3 25.2 24.2 23.7 25.2 0.8 10. ??? 100.0 3 0.0 0.0 0.0 0.0 0.0
Chris
On 02/21/2014 05:00 PM, Dj Padzensky wrote:
Looks like (more) Level3 problems...
--Dj
On Feb 21, 2014, at 3:57 PM, Grant Ridder <shortdudey123@gmail.com <mailto:shortdudey123@gmail.com>> wrote:
No problems from comcast in Mountain View ca
On Fri, Feb 21, 2014 at 3:47 PM, Clayton Dukes <cdukes@gmail.com <mailto:cdukes@gmail.com>> wrote:
I happen to be in Marrakech at the moment and am able to access it.
______________________________________________________________
Clayton Dukes ______________________________________________________________
On Sat, Feb 22, 2014 at 12:36 AM, <chris@htswireless.com <mailto:chris@htswireless.com>> wrote:
Is anyone else seeing traffic routing all over the place trying to get to Amazon.com <http://Amazon.com>?
Chris
_______________________________________________ Outages mailing list Outages@outages.org <mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org <mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org <mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
participants (7)
-
Chris Hudson -
Chris Stone -
chris＠htswireless.com
-
Clayton Dukes -
Dj Padzensky -
Grant Ridder -
Jeremy Chadwick