April 2016 - Outages - Outages Lists

Fw: new message
by Gary Staff 30 Apr '16

30 Apr '16

Hello! You have a new message, please read <http://omginfographics.com/behave.php?1g> Gary Staff

1 0

Level 3 and/or Nitel Outage in Miami, FL Area
by Joe Klein 29 Apr '16

29 Apr '16

Received an email from our carrier that they're experiencing an outage in the Miami, FL market - Hello Nitel customer, We are currently experiencing a major outage in Level 3's network affecting multiple customers in the Miami market. Nitel engineers are working with the carrier to restore your service as soon as possible. Our circuit at the airport went down at 1725 site local time and is still down (2146).

1 0

menards.com down
by Andy Ringsmuth 29 Apr '16

29 Apr '16

Anyone else seeing this? downforeveryoneorjustme.com also reports it down. I get a strange error message, “Error Code 16. This request was blocked by security rules.” For me, their iOS app kinda works, leading me to believe something is indeed mucked up on their end. ---- Andy Ringsmuth 5609 Harding Drive Lincoln, NE 68521-5831 (402) 304-0083 andy(a)andyring.com

3 3

Verizon issues?
by Robert Glover 27 Apr '16

27 Apr '16

Is anyone seeing weirdness to/from Verizon? Keynote sees a problem: http://internetpulse.keynote.com/

1 0

HE.net Chicago router link issue?
by Peter Beckman 26 Apr '16

26 Apr '16

A few minutes ago I was notified that one of my servers in Denver, CO was down (ip: 72.18.145.210). Anyone seeing a similar loss on he.net in Chicago? nog2.angryox.com (0.0.0.0) Tue Apr 26 15:54:54 2016 Packets Pings Host Loss% Snt Last Avg Best Wrst StDev 1. xl0.tic.nova.org 0.0% 36 0.1 0.1 0.1 1.0 0.0 2. wsip-70-168-111-17.dc.dc.cox.net 2.8% 36 1.8 13.3 1.1 183.2 40.1 3. ip68-100-3-109.dc.dc.cox.net 0.0% 36 1.1 3.0 1.1 25.6 4.9 4. ashbbprj01-ae5.0.rd.as.cox.net 0.0% 36 1.7 3.6 1.7 20.7 4.3 5. 68.105.30.114 2.8% 36 7.8 15.1 1.8 120.4 21.0 6. 10ge3-6.core2.chi1.he.net 2.8% 36 33.2 35.2 28.1 51.4 6.9 7. ?? But from Verizon it is up: max.angryox.com (0.0.0.0) Tue Apr 26 15:56:08 2016 Packets Pings Host Loss% Snt Last Avg Best Wrst StDev 1. tomato.angryox.com 0.0% 158 0.3 0.4 0.3 2.4 0.2 2. lo0-100.WASHDC-VFTTP-312.verizon-gni 0.0% 158 3.7 7.0 3.0 153.9 13.1 3. T1-3-0-4.WASHDC-LCR-21.verizon-gni.n 0.0% 158 7.9 8.1 3.1 156.9 12.0 4. ??? 5. ??? 6. 0.ae3.GW11.IAD8.ALTER.NET 0.0% 157 6.5 8.5 5.4 118.2 12.1 7. teliasonera-gw.customer.alter.net 0.0% 157 7.2 9.0 5.5 164.1 15.7 8. ash-bb3-link.telia.net 0.0% 157 8.0 8.9 5.5 148.9 14.4 9. sjo-b21-link.telia.net 0.0% 157 71.1 72.6 68.8 150.8 7.2 10. den-b1-link.telia.net 0.0% 157 99.3 99.6 98.0 122.4 2.4 11. handynet-ic-145041-den-b1.c.telia.ne 0.0% 157 130.4 100.1 98.0 130.4 3.3 12. ae2-14.core1.denver2.handynetworks.c 0.0% 157 97.1 97.1 95.5 113.8 1.8 13. ae0-10.dist2.denver2.handynetworks.c 0.0% 157 99.5 103.9 95.6 828.6 58.2 14. vlan954.sw2-g4.denver2.handynetworks 0.0% 157 98.5 104.7 98.1 761.2 52.8 15. vlan11.sw1-h3.softsyshosting.com 0.0% 157 99.8 103.8 97.1 430.8 33.8 16. 72.18.145.210 0.0% 157 97.3 97.1 95.5 128.1 2.7 --------------------------------------------------------------------------- Peter Beckman Internet Guy beckman(a)angryox.com http://www.angryox.com/ ---------------------------------------------------------------------------

6 8

AWS issues
by John Kinsella 22 Apr '16

22 Apr '16

Currently able to connect to multiple services hosted by AWS (Slack, iTunes, and my own EC2 instances). http://status.aws.amazon.com/ claims situation normal, so far, although I can’t get into my AWS dashboard. Might be wider than AWS, but I seem to have connectivity out to other major services... Packets Pings Host Loss% Snt Last Avg Best Wrst StDev 1. 10.0.1.1 0.0% 29 1.0 10.7 0.9 123.2 26.2 2. 10.0.1.49 0.0% 29 1.7 12.1 1.5 101.1 26.7 3. 173.247.205.38 0.0% 29 3.7 24.3 3.5 120.4 36.8 4. x.196.247.173.web-pass.com 0.0% 29 3.3 20.3 3.2 127.4 36.9 5. x.196.247.173.web-pass.com 0.0% 29 3.9 21.0 3.6 106.7 31.3 6. v505.core1.sfo1.he.net 0.0% 29 3.9 19.7 3.8 123.5 30.3 7. 10ge11-2.core1.sjc2.he.net 0.0% 29 14.8 27.7 4.9 165.2 39.3 8. 100ge1-2.core1.nyc4.he.net 6.9% 29 74.4 112.9 66.5 285.1 71.3 9. 100ge11-1.core1.par2.he.net 21.4% 29 296.4 171.1 137.0 296.4 57.3 10. 10ge3-2.core1.zrh1.he.net 0.0% 28 175.8 173.3 150.4 280.4 34.0 11. ??? 12. ??? 13. ??? 14. ??? 15. te2-2.er01.zrh01.ip-max.net 92.6% 28 391.1 318.5 246.0 391.1 102.6 16. xe0-0-3.cr02.gva253.ip-max.net 96.3% 28 331.7 331.7 331.7 331.7 0.0 17. xe0-0-1.cr02.gva252.ip-max.net 96.2% 27 321.6 321.6 321.6 321.6 0.0 18. ???

8 9

HTTP access to www.amazon.com is down for us
by Frank Bulk 22 Apr '16

22 Apr '16

Anyone else see the same? root@nagios:/# wget -4 www.amazon.com --2016-04-22 12:19:00-- http://www.amazon.com/ Resolving www.amazon.com... 54.239.26.128 Connecting to www.amazon.com|54.239.26.128|:80... failed: Connection timed out. Retrying. --2016-04-22 12:19:22-- (try: 2) http://www.amazon.com/ Connecting to www.amazon.com|54.239.26.128|:80... failed: Connection timed out. Retrying. --2016-04-22 12:19:45-- (try: 3) http://www.amazon.com/ Connecting to www.amazon.com|54.239.26.128|:80... ^C root@nagios:/# root@nagios:/etc/cron.weekly# tcptraceroute www.amazon.com Selected device eth0.3, address 96.31.0.5, port 41361 for outgoing packets Tracing the path to www.amazon.com (54.239.26.128) on TCP port 80 (www), 30 hops max 1 router-core-inside.mtcnet.net (96.31.0.254) 0.258 ms 0.366 ms 0.356 ms 2 sxct.sxcy.mtcnet.net (167.142.156.197) 0.274 ms 0.173 ms 0.150 ms 3 premier.sxcy-mlx.fbnt.netins.net (173.215.60.5) 2.131 ms 1.453 ms 1.448 ms 4 ins-db4-te-0-6-0-4.219.desm.netins.net (167.142.60.157) 7.797 ms 7.594 ms 7.592 ms 5 ins-dc1-po200.desm.netins.net (167.142.67.193) 9.304 ms 9.177 ms 9.219 ms 6 ins-kc2-po20.kmrr.netins.net (167.142.67.30) 9.407 ms 10.021 ms 9.228 ms 7 v504.core1.oma1.he.net (184.105.18.169) 39.532 ms 32.855 ms 41.492 ms 8 10ge11-15.core2.chi1.he.net (184.105.81.229) 20.037 ms 20.076 ms 21.987 ms 9 100ge5-2.core1.chi1.he.net (184.105.81.98) 20.034 ms 20.106 ms 19.997 ms 10 100ge5-2.core1.nyc4.he.net (184.105.223.162) 40.097 ms 47.921 ms 38.699 ms 11 100ge11-1.core1.par2.he.net (184.105.81.78) 121.681 ms 113.899 ms * 12 10ge3-2.core1.zrh1.he.net (184.105.222.50) 140.341 ms 131.058 ms 139.553 ms 13 * * * 14 * * * 15 * * * 16 * * * 17 * * * 18 * * * ^C root@nagios:/etc/cron.weekly# -----Original Message----- From: nagios(a)nagios.mtcnet.net [mailto:nagios@nagios.mtcnet.net] Sent: Friday, April 22, 2016 12:18 PM Subject: PROBLEM: Host amazon is DOWN! ***** Nagios ***** Host: This is a PROBLEM notice that Amazon is DOWN Date/Time: Fri Apr 22 12:18:06 CDT 2016 CRITICAL - Socket timeout after 10 seconds

9 9

Level 3 issues?
by Jeff Agle 22 Apr '16

22 Apr '16

Anyone else seeing peering issues / loss in Level3 this morning? The below is to/from Eastern Canada. Keep getting route changes in the path as well. Hop Sent PL% Min Max Avg Host Name/IP 1 1535 0 0.58 22.72 0.87 192.168.224.1 [192.168.224.1] 2 1535 0 0.83 284.45 19.79 142.166.210.153 [142.166.210.153] 3 765 0 1.31 202.85 17.73 142.166.211.113 [142.166.211.113] 4 775 0 18.50 289.86 34.47 207.231.227.53 [207.231.227.53] 5 771 0 16.26 278.12 34.58 ae52.edge1.toronto2.level3.net [4.31.208.9] 6 1532 90 84.42 289.61 94.59 ae-1-60.ear1.sanjose1.level3.net [4.69.152.23] <--------- 7 1532 90 84.40 304.22 93.41 ae-1-60.ear1.sanjose1.level3.net [4.69.152.23] <-------- 8 1535 0 84.62 416.69 100.82 one-communi.ear1.sanjose1.level3.net [4.15.122.98] 9 1535 0 83.31 410.75 98.50 user-uinjs2k.biz.mindspring.com [165.121.240.84] 10 1535 0 84.63 406.80 97.94 66.32.80.116 [66.32.80.116] 11 1535 0 84.63 367.45 101.48 199.180.64.8 [199.180.64.8] Hop Sent PL% Min Max Avg Host Name / IP 1 299 0 1 3.99 1.75 108.59.240.61 [108.59.240.61] 2 299 0 10.44 31.52 13.29 165.121.238.180 [165.121.238.180] 3 299 0 10.32 29.26 12.99 209.178.150.68 [209.178.150.68] 4 297 11 10 2437.3 74.79 5-1-34.ear2.newyork1.level3.net [4.35.90.65] <---------- 5 299 0 10.46 14.35 10.99 4.71.230.102 [4.71.230.102] 6 299 0 26 30.17 26.82 be5.cr01.stjh.nb.aliant.net [207.231.227.113] 7 299 0 25.76 40.32 30.25 loop0.1kw.ba06.stjh.nb.aliant.net [142.166.182.14] ~jeff

2 1

us-east-1 network partition?
by Grant Ridder 21 Apr '16

21 Apr '16

Looks like 1 zone in us-east-1 got network partitioned for ~5 min Start: 21:01:10,698 UTC Recover: 21:06:38,946 UTC VPC / Classic both affected -Grant

2 3

Verizon Issues?
by chris 21 Apr '16

21 Apr '16

Anyone aware of any known outages/issues for FIOS? Have a handful of customers in northern NJ with business FIOS with 50%+ packet loss on all verizon hops. We have checked these customers and theres no congestion on their networks. We're aware of the Verizon strike in progress and wondering if maybe some repair or maintenance is not being done because of the strike? If anyone from VZ has ability to dig deeper or if anyone else is seeing similar issues would be good to hear. thanks chris

7 6