September 2014 - Outages

wiki.outages.org restored
by virendra rode 29 Sep '14

29 Sep '14

Hello All - Just a quick note, wiki.outages.org has been restored and has a new home and dual stack (v4/v6). Hats off to Bill Woodcock and his team at PCH in getting this up and running regards, /virendra

1 0

Packets lost from 38.104.251.246
by Samer Ziadeh 26 Sep '14

26 Sep '14

Does anyone know anything about the IP 38.104.251.246 ? After looking at google, it seems like it's for psi-net.com, and it's one of the routers on the way to my server. Running a ping on that IP, majority of the packets are dropped, and it's causing my site to go down from time to time. ^_^ [~]$ ping 38.104.251.246 PING 38.104.251.246 (38.104.251.246): 56 data bytes Request timeout for icmp_seq 0 Request timeout for icmp_seq 1 Request timeout for icmp_seq 2 64 bytes from 38.104.251.246: icmp_seq=3 ttl=56 time=5.468 ms Request timeout for icmp_seq 4 Request timeout for icmp_seq 5 Request timeout for icmp_seq 6 Request timeout for icmp_seq 7 Request timeout for icmp_seq 8 64 bytes from 38.104.251.246: icmp_seq=9 ttl=56 time=19.022 ms Request timeout for icmp_seq 10 Request timeout for icmp_seq 11 Request timeout for icmp_seq 12 Request timeout for icmp_seq 13 Request timeout for icmp_seq 14 Request timeout for icmp_seq 15 Request timeout for icmp_seq 16 Request timeout for icmp_seq 17 64 bytes from 38.104.251.246: icmp_seq=18 ttl=56 time=3.957 ms Request timeout for icmp_seq 19 Request timeout for icmp_seq 20 64 bytes from 38.104.251.246: icmp_seq=21 ttl=56 time=6.522 ms --- 38.104.251.246 ping statistics --- 23 packets transmitted, 4 packets received, 82.6% packet loss round-trip min/avg/max/stddev = 3.957/8.742/19.022/6.005 ms -- - Samer

7 7

pacwest down?
by jd 26 Sep '14

26 Sep '14

is anyone else experiencing a pacwest outage again? thanks

4 6

Hangouts Chrome Extension
by Anthony Hook 25 Sep '14

25 Sep '14

Google Hangouts Chrome Extension appears to be down. Mobile works, and integrated from Gmail is up. Nothing yet at http://www.google.com/appsstatus#hl=en&v=status I am experiencing this, and am getting reports from users around the state. Thanks, -- - Anthony Hook

8 9

Issues to Digital Ocean?
by Dovid B 24 Sep '14

24 Sep '14

Is anyone else having a problem getting to Digital Ocean? >From HE: (0.0.0.0) Wed Sep 24 12:22:38 2014 Keys: Help Display mode Restart statistics Order of fields quit Packets Pings Host Loss% Snt Last Avg Best Wrst StDev 1. ge2-20.core1.nyc5.he.net 0.0% 5 0.3 0.9 0.2 3.5 1.5 2. 10ge6-1.core1.nyc6.he.net 0.0% 5 0.3 4.7 0.3 10.5 4.3 3. nyc2edge1.digitalocean.com 0.0% 5 0.7 0.6 0.6 0.7 0.0 4. ??? >From DedNow: My traceroute [v0.81] (0.0.0.0) Wed Sep 24 12:25:23 2014 Resolver: Received error response 5. (refused by name server)elds quit Packets Pings Host Loss% Snt Last Avg Best Wrst StDev 1. 50.60.143.33 0.0% 7 5.7 2.6 1.0 6.2 2.3 2. 65.98.0.152 0.0% 7 0.2 2.9 0.2 18.9 7.0 3. xe-7-0-1.ar2.nyc3.us.as4436.gtt.net 16.7% 7 4.8 3.8 2.1 4.8 1.3 4. as2914.ae9.ar2.nyc3.us.as4436.gtt.net 0.0% 7 0.8 0.8 0.8 0.9 0.1 5. xe-0-3-0-21.r05.nycmny01.us.ce.gin.ntt.net 0.0% 7 0.8 1.3 0.7 3.9 1.2 6. ??? Via NTT it's working fine.

5 6

HTTPv6 access to www.facebook.com
by Frank Bulk 23 Sep '14

23 Sep '14

Our monitoring system has seen HTTPv6 access to www.facebook.com timeout (happens at 10 seconds) over a dozen times this afternoon, starting at 2:07 pm Central. Anyone else seeing something similar over IPv4 or v6? Frank root@nagios:~# tcptraceroute6 www.facebook.com traceroute to edge-star6-shv-01-ash5.facebook.com (2a03:2880:2050:1f08:face:b00c:0:1) from 2607:fe28:0:1000::5, port 80, from port 57678, 30 hops max, 60 bytes packets 1 router-core.mtcnet.net (2607:fe28:0:1000::1) 0.285 ms 0.229 ms 0.202 ms 2 sxct.sxcy.mtcnet.net (2607:fe28:11:1002::197) 0.193 ms 0.130 ms 0.132 ms 3 v6-premier.sxcy-mlx.fbnt.netins.net (2001:5f8:7f0a:2::1) 1.634 ms 1.584 ms 1.610 ms 4 v6-ins-db4-te-0-6-0-4-219.desm.netins.net (2001:5f8:1:1::1) 8.184 ms 7.962 ms 7.683 ms 5 v6-ins-dc1-te-7-4.desm.netins.net (2001:5f8::26:2) 7.783 ms 7.785 ms 7.778 ms 6 v6-ins-kc2-po20.kmrr.netins.net (2001:5f8::8:2) 9.023 ms 9.017 ms 9.001 ms 7 v6-ins-kc1-po21.kmrr.netins.net (2001:5f8::14:1) 9.105 ms 9.012 ms 9.033 ms 8 2001:438:fffe::e99 (2001:438:fffe::e99) 46.100 ms 14.780 ms 14.797 ms 9 ae5.cr1.ord2.us.above.net (2001:438:ffff::407d:1e59) 23.269 ms 48.526 ms 23.277 ms 10 ae4.er1.ord7.us.above.net (2001:438:ffff::407d:1c32) 23.340 ms 23.337 ms 23.326 ms 11 2600:805:2::1 (2600:805:2::1) 22.760 ms 22.777 ms 22.772 ms 12 2600:803::32 (2600:803::32) 46.623 ms 46.494 ms 46.549 ms 13 2600:803:97f::3a (2600:803:97f::3a) 39.731 ms 39.706 ms 39.713 ms 14 be2.bb02.iad3.tfbnw.net (2620:0:1cff:dead:beef::8a) 40.324 ms 40.253 ms 40.274 ms 15 ae52.dr03.ash3.tfbnw.net (2620:0:1cff:dead:beef::67f) 46.054 ms 40.027 ms 45.969 ms 16 po1022.csw01a.ash5.tfbnw.net (2620:0:1cff:dead:beee::40f) 48.281 ms 40.471 ms 40.251 ms 17 * * * 18 * * * 19 * * * 20 * * * 21 edge-star6-shv-01-ash5.facebook.com (2a03:2880:2050:1f08:face:b00c:0:1) 39.779 ms [open] 39.641 ms 39.676 ms root@nagios:~#

2 2

In the news: Asia-America Gateway cable broken between HK and Vietnam
by Bill Woodcock 19 Sep '14

19 Sep '14

http://thenextweb.com/asia/2014/09/17/internet-speeds-across-southeast-asia… -Bill

2 1

TimeWarner Congestion in Seattle?
by Radde, Darrin 19 Sep '14

19 Sep '14

Noticing loss through TimeWarner in Seattle to Internet destinations, any known congestion or other issues with TimeWarner in the area? To Google (8.8.8.8) Success rate is 99 percent (9939/10000), round-trip min/avg/max = 4/7/36 ms To 66.192.253.122 (Time Warner internal hop, second in a traceroute to 8.8.8.8) from my router connected to TimeWarner Success rate is 99 percent (9963/10000), round-trip min/avg/max = 1/1/76 ms To 4.2.2.1 Success rate is 99 percent (9990/10000), round-trip min/avg/max = 16/19/44 ms To 64.129.234.202 (Time Warner internal hop, second hop in a traceroute to 4.2.2.1) from my router connected to TimeWarner Success rate is 99 percent (9981/10000), round-trip min/avg/max = 16/21/108 ms Darrin ---------------------------------------------------------------------- This message and any attachments are intended only for the use of the addressee and may contain information that is privileged and confidential. If the reader of the message is not the intended recipient or an authorized representative of the intended recipient, you are hereby notified that any dissemination of this communication is strictly prohibited. If you have received this communication in error, notify the sender immediately by return email and delete the message and any attachments from your system.

1 0

ipv6.cnn.com down
by Frank Bulk 19 Sep '14

19 Sep '14

Access to ipv6.cnn.com went down at 10:45 am. Level3's looking glass shows that 2620:100:e000::/48 is still being advertised, but traceouroutes don't get into TimeWarner's network. root@nagios:# tcptraceroute6 ipv6.cnn.com traceroute to ipv6.cnn.com (2620:100:e000::8001) from 2607:fe28:0:1000::5, port 80, from port 43374, 30 hops max, 60 bytes packets 1 router-core.mtcnet.net (2607:fe28:0:1000::1) 0.248 ms 0.217 ms 0.201 ms 2 sxct.spnc.mtcnet.net (2607:fe28:11:1002::194) 0.198 ms 0.130 ms 0.127 ms 3 v6-premier.movl-mlx.fbnt.netins.net (2001:5f8:7f0a:1::1) 4.921 ms 4.912 ms 4.909 ms 4 v6-ins-kb1-te-12-2-3031.kmrr.netins.net (2001:5f8:2:2::1) 10.723 ms 10.657 ms 10.577 ms 5 v6-ins-kc1-et-9-3.kmrr.netins.net (2001:5f8::28:1) 10.127 ms 10.028 ms 10.042 ms 6 2001:438:fffe::e99 (2001:438:fffe::e99) 15.788 ms 15.803 ms 15.866 ms 7 ae5.cr1.ord2.us.above.net (2001:438:ffff::407d:1e59) 31.760 ms 32.954 ms 24.212 ms 8 ae4.er1.ord7.us.above.net (2001:438:ffff::407d:1c32) 24.282 ms 23.754 ms 23.685 ms 9 2001:438:ffff::407d:c0a (2001:438:ffff::407d:c0a) 23.461 ms 84.608 ms 23.410 ms 10 vl-52.edge2.Chicago2.Level3.net (2001:1900:37:2::4) 23.534 ms 23.491 ms 23.533 ms 11 2001:1900:4:1::be (2001:1900:4:1::be) 37.618 ms 52.090 ms 37.627 ms 12 vl-4040.car1.Atlanta2.Level3.net (2001:1900:4:1::b1) 83.993 ms 53.707 ms 53.662 ms 13 * * * 14 * * * 15 * * * 16 * * * 17 * * * 18 * * * 19 * * * 20 * * * 21 * * * 22 * * * 23 * * * 24 * * * 25 * * * 26 * * * 27 * * * 28 * * * 29 * * * 30 * * * root@nagios:# This is the record from our daily archive (~2 am today): traceroute to ipv6.cnn.com (2620:100:e000::8001) from 2607:fe28:0:1000::5, port 80, from port 60821, 30 hops max, 60 bytes packets 1 router-core.mtcnet.net (2607:fe28:0:1000::1) 0.258 ms 0.203 ms 0.213 ms 2 sxct.spnc.mtcnet.net (2607:fe28:11:1002::194) 0.206 ms 0.124 ms 0.194 ms 3 v6-premier.movl-mlx.fbnt.netins.net (2001:5f8:7f0a:1::1) 4.943 ms 4.948 ms 4.942 ms 4 v6-ins-kb1-te-12-2-3031.kmrr.netins.net (2001:5f8:2:2::1) 10.633 ms 10.735 ms 10.822 ms 5 v6-ins-kc1-et-9-3.kmrr.netins.net (2001:5f8::28:1) 10.041 ms 10.032 ms 11.672 ms 6 2001:438:fffe::e99 (2001:438:fffe::e99) 15.798 ms 15.814 ms 15.784 ms 7 ae5.cr1.ord2.us.above.net (2001:438:ffff::407d:1e59) 24.273 ms 24.270 ms 24.640 ms 8 ae4.er1.ord7.us.above.net (2001:438:ffff::407d:1c32) 23.940 ms 23.895 ms 23.913 ms 9 2001:438:ffff::407d:c0a (2001:438:ffff::407d:c0a) 54.277 ms 23.479 ms 23.482 ms 10 vl-51.edge2.Chicago2.Level3.net (2001:1900:37:1::4) 28.617 ms 23.589 ms 23.586 ms 11 2001:1900:4:1::be (2001:1900:4:1::be) 37.561 ms 37.553 ms 37.538 ms 12 vl-4040.car1.Atlanta2.Level3.net (2001:1900:4:1::b1) 54.006 ms 53.890 ms 53.989 ms 13 vl-52.ear1.Atlanta2.Level3.net (2001:1900:1c:2::13) 933.251 ms 871.418 ms 528.468 ms 14 TIME-WARNER.ear1.Atlanta2.Level3.net (2001:1900:2100::442) 76.361 ms 76.329 ms 76.337 ms 15 2620:100:e000:ffff::96 (2620:100:e000:ffff::96) 76.655 ms 76.631 ms 76.660 ms 16 2620:100:e000:ffff::111 (2620:100:e000:ffff::111) 165.671 ms 77.386 ms 76.685 ms 17 2620:100:e000::d (2620:100:e000::d) 77.111 ms 76.686 ms 76.808 ms 18 2620:100:e000::19 (2620:100:e000::19) 76.638 ms 76.602 ms 76.590 ms 19 2620:100:e000::8001 (2620:100:e000::8001) 76.341 ms [open] 76.341 ms 76.313 ms root@nagios:/home/fbulk#

1 2

Possible XO/ATT peering issue in Chicago
by Bill Wichers 18 Sep '14

18 Sep '14

Is anyone else seeing excessive packet loss between ATT and XO in Chicago? We started seeing this this morning, and the problem looks like it might actually be within XO's network in Chicago but I'm not 100% certain. I have no info from either carrier as of yet. Hop #6 is 216.156.0.185 Hop #9 is 12.122.133.34 Hop #10 in the lower mtr output is 12.122.2.206 Packets Pings Host Loss% Last Avg Best Wrst StDev 5. ae1d0.mcr1.southfield-mi.us.xo.net 0.0% 8.0 10.8 6.6 48.9 7.0 6. vb1730.rar3.chicago-il.us.xo.net 0.0% 9.8 10.0 6.6 14.7 1.4 7. 207.88.14.194.ptr.us.xo.net 46.4% 7.9 8.4 6.5 44.9 3.7 8. 206.111.2.90.ptr.us.xo.net 12.9% 39.5 41.8 38.4 60.6 1.9 9. cr1.cgcil.ip.att.net 15.5% 49.7 49.6 46.3 52.8 1.5 10. 12.122.99.26 17.8% 49.9 48.4 45.0 51.5 1.4 And 5. ae1d0.mcr1.southfield-mi.us.xo.net 0.0% 7.7 9.6 6.5 80.1 7.3 6. vb1730.rar3.chicago-il.us.xo.net 0.0% 10.4 9.0 6.6 11.6 1.3 7. 207.88.14.194.ptr.us.xo.net 44.4% 6.6 7.5 6.5 32.8 2.9 8. 206.111.2.90.ptr.us.xo.net 12.1% 41.5 41.4 38.1 64.2 2.1 9. cr1.cgcil.ip.att.net 14.2% 65.4 66.2 63.8 76.0 1.5 10. cr1.cl2oh.ip.att.net 15.2% 67.5 67.0 64.0 123.6 4.7 11. cr2.nsvtn.ip.att.net 15.2% 69.3 68.6 64.3 163.8 9.3 -Bill

4 4