Anyone else see the same? root@nagios:/# wget -4 www.amazon.com --2016-04-22 12:19:00-- http://www.amazon.com/ Resolving www.amazon.com... 54.239.26.128 Connecting to www.amazon.com|54.239.26.128|:80... failed: Connection timed out. Retrying. --2016-04-22 12:19:22-- (try: 2) http://www.amazon.com/ Connecting to www.amazon.com|54.239.26.128|:80... failed: Connection timed out. Retrying. --2016-04-22 12:19:45-- (try: 3) http://www.amazon.com/ Connecting to www.amazon.com|54.239.26.128|:80... ^C root@nagios:/# root@nagios:/etc/cron.weekly# tcptraceroute www.amazon.com Selected device eth0.3, address 96.31.0.5, port 41361 for outgoing packets Tracing the path to www.amazon.com (54.239.26.128) on TCP port 80 (www), 30 hops max 1 router-core-inside.mtcnet.net (96.31.0.254) 0.258 ms 0.366 ms 0.356 ms 2 sxct.sxcy.mtcnet.net (167.142.156.197) 0.274 ms 0.173 ms 0.150 ms 3 premier.sxcy-mlx.fbnt.netins.net (173.215.60.5) 2.131 ms 1.453 ms 1.448 ms 4 ins-db4-te-0-6-0-4.219.desm.netins.net (167.142.60.157) 7.797 ms 7.594 ms 7.592 ms 5 ins-dc1-po200.desm.netins.net (167.142.67.193) 9.304 ms 9.177 ms 9.219 ms 6 ins-kc2-po20.kmrr.netins.net (167.142.67.30) 9.407 ms 10.021 ms 9.228 ms 7 v504.core1.oma1.he.net (184.105.18.169) 39.532 ms 32.855 ms 41.492 ms 8 10ge11-15.core2.chi1.he.net (184.105.81.229) 20.037 ms 20.076 ms 21.987 ms 9 100ge5-2.core1.chi1.he.net (184.105.81.98) 20.034 ms 20.106 ms 19.997 ms 10 100ge5-2.core1.nyc4.he.net (184.105.223.162) 40.097 ms 47.921 ms 38.699 ms 11 100ge11-1.core1.par2.he.net (184.105.81.78) 121.681 ms 113.899 ms * 12 10ge3-2.core1.zrh1.he.net (184.105.222.50) 140.341 ms 131.058 ms 139.553 ms 13 * * * 14 * * * 15 * * * 16 * * * 17 * * * 18 * * * ^C root@nagios:/etc/cron.weekly# -----Original Message----- From: nagios@nagios.mtcnet.net [mailto:nagios@nagios.mtcnet.net] Sent: Friday, April 22, 2016 12:18 PM Subject: PROBLEM: Host amazon is DOWN! ***** Nagios ***** Host: This is a PROBLEM notice that Amazon is DOWN Date/Time: Fri Apr 22 12:18:06 CDT 2016 CRITICAL - Socket timeout after 10 seconds
Beginning at 10:12AM PST I had many (12 or so) customers on Comcast and Qwest go down. Our primary path to them was via Hurricane Elec. Some are coming back up now. -----Original Message----- From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Frank Bulk via Outages Sent: Friday, April 22, 2016 10:21 AM To: outages@outages.org Subject: [outages] HTTP access to www.amazon.com is down for us Anyone else see the same? root@nagios:/# wget -4 www.amazon.com --2016-04-22 12:19:00-- http://www.amazon.com/ Resolving www.amazon.com... 54.239.26.128 Connecting to www.amazon.com|54.239.26.128|:80... failed: Connection timed out. Retrying. --2016-04-22 12:19:22-- (try: 2) http://www.amazon.com/ Connecting to www.amazon.com|54.239.26.128|:80... failed: Connection timed out. Retrying. --2016-04-22 12:19:45-- (try: 3) http://www.amazon.com/ Connecting to www.amazon.com|54.239.26.128|:80... ^C root@nagios:/# root@nagios:/etc/cron.weekly# tcptraceroute www.amazon.com Selected device eth0.3, address 96.31.0.5, port 41361 for outgoing packets Tracing the path to www.amazon.com (54.239.26.128) on TCP port 80 (www), 30 hops max 1 router-core-inside.mtcnet.net (96.31.0.254) 0.258 ms 0.366 ms 0.356 ms 2 sxct.sxcy.mtcnet.net (167.142.156.197) 0.274 ms 0.173 ms 0.150 ms 3 premier.sxcy-mlx.fbnt.netins.net (173.215.60.5) 2.131 ms 1.453 ms 1.448 ms 4 ins-db4-te-0-6-0-4.219.desm.netins.net (167.142.60.157) 7.797 ms 7.594 ms 7.592 ms 5 ins-dc1-po200.desm.netins.net (167.142.67.193) 9.304 ms 9.177 ms 9.219 ms 6 ins-kc2-po20.kmrr.netins.net (167.142.67.30) 9.407 ms 10.021 ms 9.228 ms 7 v504.core1.oma1.he.net (184.105.18.169) 39.532 ms 32.855 ms 41.492 ms 8 10ge11-15.core2.chi1.he.net (184.105.81.229) 20.037 ms 20.076 ms 21.987 ms 9 100ge5-2.core1.chi1.he.net (184.105.81.98) 20.034 ms 20.106 ms 19.997 ms 10 100ge5-2.core1.nyc4.he.net (184.105.223.162) 40.097 ms 47.921 ms 38.699 ms 11 100ge11-1.core1.par2.he.net (184.105.81.78) 121.681 ms 113.899 ms * 12 10ge3-2.core1.zrh1.he.net (184.105.222.50) 140.341 ms 131.058 ms 139.553 ms 13 * * * 14 * * * 15 * * * 16 * * * 17 * * * 18 * * * ^C root@nagios:/etc/cron.weekly# -----Original Message----- From: nagios@nagios.mtcnet.net [mailto:nagios@nagios.mtcnet.net] Sent: Friday, April 22, 2016 12:18 PM Subject: PROBLEM: Host amazon is DOWN! ***** Nagios ***** Host: This is a PROBLEM notice that Amazon is DOWN Date/Time: Fri Apr 22 12:18:06 CDT 2016 CRITICAL - Socket timeout after 10 seconds _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages Attention: Information contained in this message and or attachments is intended only for the recipient(s) named above and may contain confidential and or privileged material that is protected under State or Federal law. If you are not the intended recipient, any disclosure, copying, distribution or action taken on it is prohibited. If you believe you have received this email in error, please contact the sender with a copy to compliance@ochin.org, delete this email and destroy all copies.
Working fine from Des Moines Mediacom via AT&T. lft -d 443 www.amazon.com Tracing ...............*.********.**T TTL LFT trace to 54.239.26.128:443/tcp 1 x.x.x.x 0.3ms 2 10.148.x.x 7.2ms 3 172.30.73.1 10.4ms 4 172.30.1.145 9.0ms 5 12.249.52.13 14.8ms 6 cr81.desia.ip.att.net (12.122.153.50) 14.8ms ** [neglected] no reply packets received from TTL 7 8 gar8.cgcil.ip.att.net (12.122.99.5) 16.6ms 9 206.121.76.82 14.0ms 10 52.95.62.70 37.8ms 11 52.95.62.79 37.2ms ** [neglected] no reply packets received from TTL 12 13 54.240.228.230 40.4ms 14 54.240.229.185 39.9ms 15 54.240.228.161 39.3ms 16 54.239.110.44 50.4ms 17 54.239.111.79 39.3ms 18 205.251.244.193 38.7ms ** [neglected] no reply packets received from TTLs 19 through 27 28 [target open] 54.239.26.128:443 38.0ms $ date Fri Apr 22 12:29:13 CDT 2016 On Fri, Apr 22, 2016 at 12:22 PM, Tyler Applebaum via Outages < outages@outages.org> wrote:
Beginning at 10:12AM PST I had many (12 or so) customers on Comcast and Qwest go down. Our primary path to them was via Hurricane Elec. Some are coming back up now.
-----Original Message----- From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Frank Bulk via Outages Sent: Friday, April 22, 2016 10:21 AM To: outages@outages.org Subject: [outages] HTTP access to www.amazon.com is down for us
Anyone else see the same?
root@nagios:/# wget -4 www.amazon.com --2016-04-22 12:19:00-- http://www.amazon.com/ Resolving www.amazon.com... 54.239.26.128 Connecting to www.amazon.com|54.239.26.128|:80... failed: Connection timed out. Retrying.
--2016-04-22 12:19:22-- (try: 2) http://www.amazon.com/ Connecting to www.amazon.com|54.239.26.128|:80... failed: Connection timed out. Retrying.
--2016-04-22 12:19:45-- (try: 3) http://www.amazon.com/ Connecting to www.amazon.com|54.239.26.128|:80... ^C root@nagios:/#
root@nagios:/etc/cron.weekly# tcptraceroute www.amazon.com Selected device eth0.3, address 96.31.0.5, port 41361 for outgoing packets Tracing the path to www.amazon.com (54.239.26.128) on TCP port 80 (www), 30 hops max 1 router-core-inside.mtcnet.net (96.31.0.254) 0.258 ms 0.366 ms 0.356 ms 2 sxct.sxcy.mtcnet.net (167.142.156.197) 0.274 ms 0.173 ms 0.150 ms 3 premier.sxcy-mlx.fbnt.netins.net (173.215.60.5) 2.131 ms 1.453 ms 1.448 ms 4 ins-db4-te-0-6-0-4.219.desm.netins.net (167.142.60.157) 7.797 ms 7.594 ms 7.592 ms 5 ins-dc1-po200.desm.netins.net (167.142.67.193) 9.304 ms 9.177 ms 9.219 ms 6 ins-kc2-po20.kmrr.netins.net (167.142.67.30) 9.407 ms 10.021 ms 9.228 ms 7 v504.core1.oma1.he.net (184.105.18.169) 39.532 ms 32.855 ms 41.492 ms 8 10ge11-15.core2.chi1.he.net (184.105.81.229) 20.037 ms 20.076 ms 21.987 ms 9 100ge5-2.core1.chi1.he.net (184.105.81.98) 20.034 ms 20.106 ms 19.997 ms 10 100ge5-2.core1.nyc4.he.net (184.105.223.162) 40.097 ms 47.921 ms 38.699 ms 11 100ge11-1.core1.par2.he.net (184.105.81.78) 121.681 ms 113.899 ms * 12 10ge3-2.core1.zrh1.he.net (184.105.222.50) 140.341 ms 131.058 ms 139.553 ms 13 * * * 14 * * * 15 * * * 16 * * * 17 * * * 18 * * * ^C root@nagios:/etc/cron.weekly#
-----Original Message----- From: nagios@nagios.mtcnet.net [mailto:nagios@nagios.mtcnet.net] Sent: Friday, April 22, 2016 12:18 PM Subject: PROBLEM: Host amazon is DOWN!
***** Nagios *****
Host: This is a PROBLEM notice that Amazon is DOWN
Date/Time: Fri Apr 22 12:18:06 CDT 2016
CRITICAL - Socket timeout after 10 seconds
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages Attention: Information contained in this message and or attachments is intended only for the recipient(s) named above and may contain confidential and or privileged material that is protected under State or Federal law. If you are not the intended recipient, any disclosure, copying, distribution or action taken on it is prohibited. If you believe you have received this email in error, please contact the sender with a copy to compliance@ochin.org, delete this email and destroy all copies. _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
How seeing a new IP address, and so now HTTP loads give us a 503: root@nagios:/# wget -4 www.amazon.com --2016-04-22 12:21:03-- http://www.amazon.com/ Resolving www.amazon.com... 54.239.25.192 Connecting to www.amazon.com|54.239.25.192|:80... connected. HTTP request sent, awaiting response... 503 Service Unavailable 2016-04-22 12:21:03 ERROR 503: Service Unavailable. root@nagios:/# This IP address now goes via ZAYO rather than HE, but at least completes. Frank -----Original Message----- From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Frank Bulk via Outages Sent: Friday, April 22, 2016 12:21 PM To: outages@outages.org Subject: [outages] HTTP access to www.amazon.com is down for us Anyone else see the same? root@nagios:/# wget -4 www.amazon.com --2016-04-22 12:19:00-- http://www.amazon.com/ Resolving www.amazon.com... 54.239.26.128 Connecting to www.amazon.com|54.239.26.128|:80... failed: Connection timed out. Retrying. --2016-04-22 12:19:22-- (try: 2) http://www.amazon.com/ Connecting to www.amazon.com|54.239.26.128|:80... failed: Connection timed out. Retrying. --2016-04-22 12:19:45-- (try: 3) http://www.amazon.com/ Connecting to www.amazon.com|54.239.26.128|:80... ^C root@nagios:/# root@nagios:/etc/cron.weekly# tcptraceroute www.amazon.com Selected device eth0.3, address 96.31.0.5, port 41361 for outgoing packets Tracing the path to www.amazon.com (54.239.26.128) on TCP port 80 (www), 30 hops max 1 router-core-inside.mtcnet.net (96.31.0.254) 0.258 ms 0.366 ms 0.356 ms 2 sxct.sxcy.mtcnet.net (167.142.156.197) 0.274 ms 0.173 ms 0.150 ms 3 premier.sxcy-mlx.fbnt.netins.net (173.215.60.5) 2.131 ms 1.453 ms 1.448 ms 4 ins-db4-te-0-6-0-4.219.desm.netins.net (167.142.60.157) 7.797 ms 7.594 ms 7.592 ms 5 ins-dc1-po200.desm.netins.net (167.142.67.193) 9.304 ms 9.177 ms 9.219 ms 6 ins-kc2-po20.kmrr.netins.net (167.142.67.30) 9.407 ms 10.021 ms 9.228 ms 7 v504.core1.oma1.he.net (184.105.18.169) 39.532 ms 32.855 ms 41.492 ms 8 10ge11-15.core2.chi1.he.net (184.105.81.229) 20.037 ms 20.076 ms 21.987 ms 9 100ge5-2.core1.chi1.he.net (184.105.81.98) 20.034 ms 20.106 ms 19.997 ms 10 100ge5-2.core1.nyc4.he.net (184.105.223.162) 40.097 ms 47.921 ms 38.699 ms 11 100ge11-1.core1.par2.he.net (184.105.81.78) 121.681 ms 113.899 ms * 12 10ge3-2.core1.zrh1.he.net (184.105.222.50) 140.341 ms 131.058 ms 139.553 ms 13 * * * 14 * * * 15 * * * 16 * * * 17 * * * 18 * * * ^C root@nagios:/etc/cron.weekly# -----Original Message----- From: nagios@nagios.mtcnet.net [mailto:nagios@nagios.mtcnet.net] Sent: Friday, April 22, 2016 12:18 PM Subject: PROBLEM: Host amazon is DOWN! ***** Nagios ***** Host: This is a PROBLEM notice that Amazon is DOWN Date/Time: Fri Apr 22 12:18:06 CDT 2016 CRITICAL - Socket timeout after 10 seconds _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
We experienced some connectivity issues as well. It looks like HE propagated a more specific route, black holing traffic. Before outage: 62.23.0.0/16 *[BGP/170] 2d 06:45:04, MED 200, localpref 100 AS path: 3356 8220 I, During outage: 62.23.128.0/17 *[BGP/170] 00:08:29, MED 250, localpref 100 AS path: 6939 200759 65021 I Things have normalized now. Ilya ----- Original Message ----- From: "Frank Bulk via Outages" <outages@outages.org> To: outages@outages.org Sent: Friday, April 22, 2016 1:22:39 PM Subject: Re: [outages] HTTP access to www.amazon.com is down for us How seeing a new IP address, and so now HTTP loads give us a 503: root@nagios:/# wget -4 www.amazon.com --2016-04-22 12:21:03-- http://www.amazon.com/ Resolving www.amazon.com... 54.239.25.192 Connecting to www.amazon.com|54.239.25.192|:80... connected. HTTP request sent, awaiting response... 503 Service Unavailable 2016-04-22 12:21:03 ERROR 503: Service Unavailable. root@nagios:/# This IP address now goes via ZAYO rather than HE, but at least completes. Frank -----Original Message----- From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Frank Bulk via Outages Sent: Friday, April 22, 2016 12:21 PM To: outages@outages.org Subject: [outages] HTTP access to www.amazon.com is down for us Anyone else see the same? root@nagios:/# wget -4 www.amazon.com --2016-04-22 12:19:00-- http://www.amazon.com/ Resolving www.amazon.com... 54.239.26.128 Connecting to www.amazon.com|54.239.26.128|:80... failed: Connection timed out. Retrying. --2016-04-22 12:19:22-- (try: 2) http://www.amazon.com/ Connecting to www.amazon.com|54.239.26.128|:80... failed: Connection timed out. Retrying. --2016-04-22 12:19:45-- (try: 3) http://www.amazon.com/ Connecting to www.amazon.com|54.239.26.128|:80... ^C root@nagios:/# root@nagios:/etc/cron.weekly# tcptraceroute www.amazon.com Selected device eth0.3, address 96.31.0.5, port 41361 for outgoing packets Tracing the path to www.amazon.com (54.239.26.128) on TCP port 80 (www), 30 hops max 1 router-core-inside.mtcnet.net (96.31.0.254) 0.258 ms 0.366 ms 0.356 ms 2 sxct.sxcy.mtcnet.net (167.142.156.197) 0.274 ms 0.173 ms 0.150 ms 3 premier.sxcy-mlx.fbnt.netins.net (173.215.60.5) 2.131 ms 1.453 ms 1.448 ms 4 ins-db4-te-0-6-0-4.219.desm.netins.net (167.142.60.157) 7.797 ms 7.594 ms 7.592 ms 5 ins-dc1-po200.desm.netins.net (167.142.67.193) 9.304 ms 9.177 ms 9.219 ms 6 ins-kc2-po20.kmrr.netins.net (167.142.67.30) 9.407 ms 10.021 ms 9.228 ms 7 v504.core1.oma1.he.net (184.105.18.169) 39.532 ms 32.855 ms 41.492 ms 8 10ge11-15.core2.chi1.he.net (184.105.81.229) 20.037 ms 20.076 ms 21.987 ms 9 100ge5-2.core1.chi1.he.net (184.105.81.98) 20.034 ms 20.106 ms 19.997 ms 10 100ge5-2.core1.nyc4.he.net (184.105.223.162) 40.097 ms 47.921 ms 38.699 ms 11 100ge11-1.core1.par2.he.net (184.105.81.78) 121.681 ms 113.899 ms * 12 10ge3-2.core1.zrh1.he.net (184.105.222.50) 140.341 ms 131.058 ms 139.553 ms 13 * * * 14 * * * 15 * * * 16 * * * 17 * * * 18 * * * ^C root@nagios:/etc/cron.weekly# -----Original Message----- From: nagios@nagios.mtcnet.net [mailto:nagios@nagios.mtcnet.net] Sent: Friday, April 22, 2016 12:18 PM Subject: PROBLEM: Host amazon is DOWN! ***** Nagios ***** Host: This is a PROBLEM notice that Amazon is DOWN Date/Time: Fri Apr 22 12:18:06 CDT 2016 CRITICAL - Socket timeout after 10 seconds _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
I saw some weirdness from my home connection here in the Czech Republic (my ISP is AS39761, really small). Routes that were supposed to be local were routed via HE.NET Zurich POP for some reason. Also saw some mentions of "init7.net" in the traceroute (which I of course forgot to save). Everything seems alright now. So it was probably a global issue. Best Regards, Filip Hruska On 04/22/2016 07:22 PM, Frank Bulk via Outages wrote:
How seeing a new IP address, and so now HTTP loads give us a 503:
root@nagios:/# wget -4 www.amazon.com --2016-04-22 12:21:03-- http://www.amazon.com/ Resolving www.amazon.com... 54.239.25.192 Connecting to www.amazon.com|54.239.25.192|:80... connected. HTTP request sent, awaiting response... 503 Service Unavailable 2016-04-22 12:21:03 ERROR 503: Service Unavailable.
root@nagios:/#
This IP address now goes via ZAYO rather than HE, but at least completes.
Frank
-----Original Message----- From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Frank Bulk via Outages Sent: Friday, April 22, 2016 12:21 PM To: outages@outages.org Subject: [outages] HTTP access to www.amazon.com is down for us
Anyone else see the same?
root@nagios:/# wget -4 www.amazon.com --2016-04-22 12:19:00-- http://www.amazon.com/ Resolving www.amazon.com... 54.239.26.128 Connecting to www.amazon.com|54.239.26.128|:80... failed: Connection timed out. Retrying.
--2016-04-22 12:19:22-- (try: 2) http://www.amazon.com/ Connecting to www.amazon.com|54.239.26.128|:80... failed: Connection timed out. Retrying.
--2016-04-22 12:19:45-- (try: 3) http://www.amazon.com/ Connecting to www.amazon.com|54.239.26.128|:80... ^C root@nagios:/#
root@nagios:/etc/cron.weekly# tcptraceroute www.amazon.com Selected device eth0.3, address 96.31.0.5, port 41361 for outgoing packets Tracing the path to www.amazon.com (54.239.26.128) on TCP port 80 (www), 30 hops max 1 router-core-inside.mtcnet.net (96.31.0.254) 0.258 ms 0.366 ms 0.356 ms 2 sxct.sxcy.mtcnet.net (167.142.156.197) 0.274 ms 0.173 ms 0.150 ms 3 premier.sxcy-mlx.fbnt.netins.net (173.215.60.5) 2.131 ms 1.453 ms 1.448 ms 4 ins-db4-te-0-6-0-4.219.desm.netins.net (167.142.60.157) 7.797 ms 7.594 ms 7.592 ms 5 ins-dc1-po200.desm.netins.net (167.142.67.193) 9.304 ms 9.177 ms 9.219 ms 6 ins-kc2-po20.kmrr.netins.net (167.142.67.30) 9.407 ms 10.021 ms 9.228 ms 7 v504.core1.oma1.he.net (184.105.18.169) 39.532 ms 32.855 ms 41.492 ms 8 10ge11-15.core2.chi1.he.net (184.105.81.229) 20.037 ms 20.076 ms 21.987 ms 9 100ge5-2.core1.chi1.he.net (184.105.81.98) 20.034 ms 20.106 ms 19.997 ms 10 100ge5-2.core1.nyc4.he.net (184.105.223.162) 40.097 ms 47.921 ms 38.699 ms 11 100ge11-1.core1.par2.he.net (184.105.81.78) 121.681 ms 113.899 ms * 12 10ge3-2.core1.zrh1.he.net (184.105.222.50) 140.341 ms 131.058 ms 139.553 ms 13 * * * 14 * * * 15 * * * 16 * * * 17 * * * 18 * * * ^C root@nagios:/etc/cron.weekly#
-----Original Message----- From: nagios@nagios.mtcnet.net [mailto:nagios@nagios.mtcnet.net] Sent: Friday, April 22, 2016 12:18 PM Subject: PROBLEM: Host amazon is DOWN!
***** Nagios *****
Host: This is a PROBLEM notice that Amazon is DOWN
Date/Time: Fri Apr 22 12:18:06 CDT 2016
CRITICAL - Socket timeout after 10 seconds
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Working just fine for me C:\Users\jluthman>tracert -d www.amazon.com Tracing route to www.amazon.com [54.239.17.7] over a maximum of 30 hops: 1 1 ms 1 ms 1 ms 10.10.10.1 2 2 ms 1 ms 1 ms 192.149.254.1 3 5 ms 5 ms 5 ms 45.52.8.205 4 20 ms 20 ms 20 ms 74.40.5.58 5 20 ms 19 ms 20 ms 74.42.149.189 6 20 ms 19 ms 20 ms 74.40.4.138 7 20 ms 19 ms 20 ms 206.223.119.98 8 37 ms 37 ms 36 ms 52.95.62.84 9 33 ms 30 ms 30 ms 52.95.62.97 10 30 ms 30 ms 34 ms 54.239.42.63 11 30 ms 30 ms 33 ms 54.239.42.69 12 46 ms 47 ms 48 ms 54.239.109.200 13 31 ms 30 ms 30 ms 54.239.109.217 14 30 ms 30 ms 30 ms 205.251.245.121 15 * * * Request timed out. 16 * * * Request timed out. 17 * ^C C:\Users\jluthman>host www.amazon.com www.amazon.com has address 54.239.17.7 Josh Luthman Office: 937-552-2340 Direct: 937-552-2343 1100 Wayne St Suite 1337 Troy, OH 45373 On Fri, Apr 22, 2016 at 1:34 PM, Filip Hruska via Outages < outages@outages.org> wrote:
I saw some weirdness from my home connection here in the Czech Republic (my ISP is AS39761, really small). Routes that were supposed to be local were routed via HE.NET Zurich POP for some reason. Also saw some mentions of "init7.net" in the traceroute (which I of course forgot to save).
Everything seems alright now.
So it was probably a global issue.
Best Regards, Filip Hruska
On 04/22/2016 07:22 PM, Frank Bulk via Outages wrote:
How seeing a new IP address, and so now HTTP loads give us a 503:
root@nagios:/# wget -4 www.amazon.com --2016-04-22 12:21:03-- http://www.amazon.com/ Resolving www.amazon.com... 54.239.25.192 Connecting to www.amazon.com|54.239.25.192|:80... connected. HTTP request sent, awaiting response... 503 Service Unavailable 2016-04-22 12:21:03 ERROR 503: Service Unavailable.
root@nagios:/#
This IP address now goes via ZAYO rather than HE, but at least completes.
Frank
-----Original Message----- From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Frank Bulk via Outages Sent: Friday, April 22, 2016 12:21 PM To: outages@outages.org Subject: [outages] HTTP access to www.amazon.com is down for us
Anyone else see the same?
root@nagios:/# wget -4 www.amazon.com --2016-04-22 12:19:00-- http://www.amazon.com/ Resolving www.amazon.com... 54.239.26.128 Connecting to www.amazon.com|54.239.26.128|:80... failed: Connection timed out. Retrying.
--2016-04-22 12:19:22-- (try: 2) http://www.amazon.com/ Connecting to www.amazon.com|54.239.26.128|:80... failed: Connection timed out. Retrying.
--2016-04-22 12:19:45-- (try: 3) http://www.amazon.com/ Connecting to www.amazon.com|54.239.26.128|:80... ^C root@nagios:/#
root@nagios:/etc/cron.weekly# tcptraceroute www.amazon.com Selected device eth0.3, address 96.31.0.5, port 41361 for outgoing packets Tracing the path to www.amazon.com (54.239.26.128) on TCP port 80 (www), 30 hops max 1 router-core-inside.mtcnet.net (96.31.0.254) 0.258 ms 0.366 ms 0.356 ms 2 sxct.sxcy.mtcnet.net (167.142.156.197) 0.274 ms 0.173 ms 0.150 ms 3 premier.sxcy-mlx.fbnt.netins.net (173.215.60.5) 2.131 ms 1.453 ms 1.448 ms 4 ins-db4-te-0-6-0-4.219.desm.netins.net (167.142.60.157) 7.797 ms 7.594 ms 7.592 ms 5 ins-dc1-po200.desm.netins.net (167.142.67.193) 9.304 ms 9.177 ms 9.219 ms 6 ins-kc2-po20.kmrr.netins.net (167.142.67.30) 9.407 ms 10.021 ms 9.228 ms 7 v504.core1.oma1.he.net (184.105.18.169) 39.532 ms 32.855 ms 41.492 ms 8 10ge11-15.core2.chi1.he.net (184.105.81.229) 20.037 ms 20.076 ms 21.987 ms 9 100ge5-2.core1.chi1.he.net (184.105.81.98) 20.034 ms 20.106 ms 19.997 ms 10 100ge5-2.core1.nyc4.he.net (184.105.223.162) 40.097 ms 47.921 ms 38.699 ms 11 100ge11-1.core1.par2.he.net (184.105.81.78) 121.681 ms 113.899 ms * 12 10ge3-2.core1.zrh1.he.net (184.105.222.50) 140.341 ms 131.058 ms 139.553 ms 13 * * * 14 * * * 15 * * * 16 * * * 17 * * * 18 * * * ^C root@nagios:/etc/cron.weekly#
-----Original Message----- From: nagios@nagios.mtcnet.net [mailto:nagios@nagios.mtcnet.net] Sent: Friday, April 22, 2016 12:18 PM Subject: PROBLEM: Host amazon is DOWN!
***** Nagios *****
Host: This is a PROBLEM notice that Amazon is DOWN
Date/Time: Fri Apr 22 12:18:06 CDT 2016
CRITICAL - Socket timeout after 10 seconds
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
I saw traffic to Zurich HE as well. AWS announced their Zurich office today. I’m gonna guess these two things are not a coincidence. https://aws.amazon.com/switzerland
On Apr 22, 2016, at 10:34 AM, Filip Hruska via Outages <outages@outages.org> wrote:
I saw some weirdness from my home connection here in the Czech Republic (my ISP is AS39761, really small). Routes that were supposed to be local were routed via HE.NET Zurich POP for some reason. Also saw some mentions of "init7.net" in the traceroute (which I of course forgot to save).
Everything seems alright now.
So it was probably a global issue.
Best Regards, Filip Hruska
On 04/22/2016 07:22 PM, Frank Bulk via Outages wrote:
How seeing a new IP address, and so now HTTP loads give us a 503:
root@nagios:/# wget -4 www.amazon.com --2016-04-22 12:21:03-- http://www.amazon.com/ Resolving www.amazon.com... 54.239.25.192 Connecting to www.amazon.com|54.239.25.192|:80... connected. HTTP request sent, awaiting response... 503 Service Unavailable 2016-04-22 12:21:03 ERROR 503: Service Unavailable.
root@nagios:/#
This IP address now goes via ZAYO rather than HE, but at least completes.
Frank
-----Original Message----- From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Frank Bulk via Outages Sent: Friday, April 22, 2016 12:21 PM To: outages@outages.org Subject: [outages] HTTP access to www.amazon.com is down for us
Anyone else see the same?
root@nagios:/# wget -4 www.amazon.com --2016-04-22 12:19:00-- http://www.amazon.com/ Resolving www.amazon.com... 54.239.26.128 Connecting to www.amazon.com|54.239.26.128|:80... failed: Connection timed out. Retrying.
--2016-04-22 12:19:22-- (try: 2) http://www.amazon.com/ Connecting to www.amazon.com|54.239.26.128|:80... failed: Connection timed out. Retrying.
--2016-04-22 12:19:45-- (try: 3) http://www.amazon.com/ Connecting to www.amazon.com|54.239.26.128|:80... ^C root@nagios:/#
root@nagios:/etc/cron.weekly# tcptraceroute www.amazon.com Selected device eth0.3, address 96.31.0.5, port 41361 for outgoing packets Tracing the path to www.amazon.com (54.239.26.128) on TCP port 80 (www), 30 hops max 1 router-core-inside.mtcnet.net (96.31.0.254) 0.258 ms 0.366 ms 0.356 ms 2 sxct.sxcy.mtcnet.net (167.142.156.197) 0.274 ms 0.173 ms 0.150 ms 3 premier.sxcy-mlx.fbnt.netins.net (173.215.60.5) 2.131 ms 1.453 ms 1.448 ms 4 ins-db4-te-0-6-0-4.219.desm.netins.net (167.142.60.157) 7.797 ms 7.594 ms 7.592 ms 5 ins-dc1-po200.desm.netins.net (167.142.67.193) 9.304 ms 9.177 ms 9.219 ms 6 ins-kc2-po20.kmrr.netins.net (167.142.67.30) 9.407 ms 10.021 ms 9.228 ms 7 v504.core1.oma1.he.net (184.105.18.169) 39.532 ms 32.855 ms 41.492 ms 8 10ge11-15.core2.chi1.he.net (184.105.81.229) 20.037 ms 20.076 ms 21.987 ms 9 100ge5-2.core1.chi1.he.net (184.105.81.98) 20.034 ms 20.106 ms 19.997 ms 10 100ge5-2.core1.nyc4.he.net (184.105.223.162) 40.097 ms 47.921 ms 38.699 ms 11 100ge11-1.core1.par2.he.net (184.105.81.78) 121.681 ms 113.899 ms * 12 10ge3-2.core1.zrh1.he.net (184.105.222.50) 140.341 ms 131.058 ms 139.553 ms 13 * * * 14 * * * 15 * * * 16 * * * 17 * * * 18 * * * ^C root@nagios:/etc/cron.weekly#
-----Original Message----- From: nagios@nagios.mtcnet.net [mailto:nagios@nagios.mtcnet.net] Sent: Friday, April 22, 2016 12:18 PM Subject: PROBLEM: Host amazon is DOWN!
***** Nagios *****
Host: This is a PROBLEM notice that Amazon is DOWN
Date/Time: Fri Apr 22 12:18:06 CDT 2016
CRITICAL - Socket timeout after 10 seconds
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
We also have HE.Net as a BGP provider, and we saw traffic to Google.com and Facebook temporarily get routed to zrh1.he.net. Traffic to AOL.com worked fine, so that route was unaffected. It is back up now, but was out for a good 5-10 minutes here in Indiana, US. Eric Rogers PDS Connect www.pdsconnect.me (317) 831-3000 x200 -----Original Message----- From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Frank Bulk via Outages Sent: Friday, April 22, 2016 1:21 PM To: outages@outages.org Subject: [outages] HTTP access to www.amazon.com is down for us Anyone else see the same? root@nagios:/# wget -4 www.amazon.com --2016-04-22 12:19:00-- http://www.amazon.com/ Resolving www.amazon.com... 54.239.26.128 Connecting to www.amazon.com|54.239.26.128|:80... failed: Connection timed out. Retrying. --2016-04-22 12:19:22-- (try: 2) http://www.amazon.com/ Connecting to www.amazon.com|54.239.26.128|:80... failed: Connection timed out. Retrying. --2016-04-22 12:19:45-- (try: 3) http://www.amazon.com/ Connecting to www.amazon.com|54.239.26.128|:80... ^C root@nagios:/# root@nagios:/etc/cron.weekly# tcptraceroute www.amazon.com Selected device eth0.3, address 96.31.0.5, port 41361 for outgoing packets Tracing the path to www.amazon.com (54.239.26.128) on TCP port 80 (www), 30 hops max 1 router-core-inside.mtcnet.net (96.31.0.254) 0.258 ms 0.366 ms 0.356 ms 2 sxct.sxcy.mtcnet.net (167.142.156.197) 0.274 ms 0.173 ms 0.150 ms 3 premier.sxcy-mlx.fbnt.netins.net (173.215.60.5) 2.131 ms 1.453 ms 1.448 ms 4 ins-db4-te-0-6-0-4.219.desm.netins.net (167.142.60.157) 7.797 ms 7.594 ms 7.592 ms 5 ins-dc1-po200.desm.netins.net (167.142.67.193) 9.304 ms 9.177 ms 9.219 ms 6 ins-kc2-po20.kmrr.netins.net (167.142.67.30) 9.407 ms 10.021 ms 9.228 ms 7 v504.core1.oma1.he.net (184.105.18.169) 39.532 ms 32.855 ms 41.492 ms 8 10ge11-15.core2.chi1.he.net (184.105.81.229) 20.037 ms 20.076 ms 21.987 ms 9 100ge5-2.core1.chi1.he.net (184.105.81.98) 20.034 ms 20.106 ms 19.997 ms 10 100ge5-2.core1.nyc4.he.net (184.105.223.162) 40.097 ms 47.921 ms 38.699 ms 11 100ge11-1.core1.par2.he.net (184.105.81.78) 121.681 ms 113.899 ms * 12 10ge3-2.core1.zrh1.he.net (184.105.222.50) 140.341 ms 131.058 ms 139.553 ms 13 * * * 14 * * * 15 * * * 16 * * * 17 * * * 18 * * * ^C root@nagios:/etc/cron.weekly# -----Original Message----- From: nagios@nagios.mtcnet.net [mailto:nagios@nagios.mtcnet.net] Sent: Friday, April 22, 2016 12:18 PM Subject: PROBLEM: Host amazon is DOWN! ***** Nagios ***** Host: This is a PROBLEM notice that Amazon is DOWN Date/Time: Fri Apr 22 12:18:06 CDT 2016 CRITICAL - Socket timeout after 10 seconds _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Not sure if it's relevant, but we received word from HE that they experienced a route leak from AS200759. It affected about 30% of our traffic for 10 minutes about an hour ago. *Alex Forster* Network Engineer, Linode On Fri, Apr 22, 2016 at 2:08 PM, Eric Rogers via Outages < outages@outages.org> wrote:
We also have HE.Net as a BGP provider, and we saw traffic to Google.com and Facebook temporarily get routed to zrh1.he.net. Traffic to AOL.com worked fine, so that route was unaffected. It is back up now, but was out for a good 5-10 minutes here in Indiana, US.
Eric Rogers PDS Connect www.pdsconnect.me (317) 831-3000 x200
-----Original Message----- From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Frank Bulk via Outages Sent: Friday, April 22, 2016 1:21 PM To: outages@outages.org Subject: [outages] HTTP access to www.amazon.com is down for us
Anyone else see the same?
root@nagios:/# wget -4 www.amazon.com --2016-04-22 12:19:00-- http://www.amazon.com/ Resolving www.amazon.com... 54.239.26.128 Connecting to www.amazon.com|54.239.26.128|:80... failed: Connection timed out. Retrying.
--2016-04-22 12:19:22-- (try: 2) http://www.amazon.com/ Connecting to www.amazon.com|54.239.26.128|:80... failed: Connection timed out. Retrying.
--2016-04-22 12:19:45-- (try: 3) http://www.amazon.com/ Connecting to www.amazon.com|54.239.26.128|:80... ^C root@nagios:/#
root@nagios:/etc/cron.weekly# tcptraceroute www.amazon.com Selected device eth0.3, address 96.31.0.5, port 41361 for outgoing packets Tracing the path to www.amazon.com (54.239.26.128) on TCP port 80 (www), 30 hops max 1 router-core-inside.mtcnet.net (96.31.0.254) 0.258 ms 0.366 ms 0.356 ms 2 sxct.sxcy.mtcnet.net (167.142.156.197) 0.274 ms 0.173 ms 0.150 ms 3 premier.sxcy-mlx.fbnt.netins.net (173.215.60.5) 2.131 ms 1.453 ms 1.448 ms 4 ins-db4-te-0-6-0-4.219.desm.netins.net (167.142.60.157) 7.797 ms 7.594 ms 7.592 ms 5 ins-dc1-po200.desm.netins.net (167.142.67.193) 9.304 ms 9.177 ms 9.219 ms 6 ins-kc2-po20.kmrr.netins.net (167.142.67.30) 9.407 ms 10.021 ms 9.228 ms 7 v504.core1.oma1.he.net (184.105.18.169) 39.532 ms 32.855 ms 41.492 ms 8 10ge11-15.core2.chi1.he.net (184.105.81.229) 20.037 ms 20.076 ms 21.987 ms 9 100ge5-2.core1.chi1.he.net (184.105.81.98) 20.034 ms 20.106 ms 19.997 ms 10 100ge5-2.core1.nyc4.he.net (184.105.223.162) 40.097 ms 47.921 ms 38.699 ms 11 100ge11-1.core1.par2.he.net (184.105.81.78) 121.681 ms 113.899 ms * 12 10ge3-2.core1.zrh1.he.net (184.105.222.50) 140.341 ms 131.058 ms 139.553 ms 13 * * * 14 * * * 15 * * * 16 * * * 17 * * * 18 * * * ^C root@nagios:/etc/cron.weekly#
-----Original Message----- From: nagios@nagios.mtcnet.net [mailto:nagios@nagios.mtcnet.net] Sent: Friday, April 22, 2016 12:18 PM Subject: PROBLEM: Host amazon is DOWN!
***** Nagios *****
Host: This is a PROBLEM notice that Amazon is DOWN
Date/Time: Fri Apr 22 12:18:06 CDT 2016
CRITICAL - Socket timeout after 10 seconds
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
participants (9)
-
Alex Forster -
Andrew Smith -
Eric Rogers -
Filip Hruska -
Frank Bulk -
Ilya Sokolov -
John Kinsella -
Josh Luthman -
Tyler Applebaum