October 2011 - Outages - Outages Lists

Calls to Integra PRIs from Qwest/Cox/Verizon Wireless fail
by Carlos Alvarez 31 Oct '11

31 Oct '11

"All circuits are busy" message, affecting multiple PRIs in the Phoenix area. -- Carlos Alvarez TelEvolve 602-889-3003

1 0

TWTC IPv6
by Ryan Rawdon 28 Oct '11

28 Oct '11

Starting at 2:50AM (Eastern), we are unable to reach 2001:4870::/32 from a number of locations (all locations that we tested from). ashburn-rtr2#sh bgp ipv6 unicast 2001:4870::/32 % Network not in table ashburn-rtr2#

4 7

Qwest Re-Convergence
by Jason J. W. Williams 26 Oct '11

26 Oct '11

Has anyone noticed issues reaching Qwest destinations around 4:05a-4:07a Mountain Time today? We received alerts that two of our sites that happened to have Qwest connectivity in common were unavailable during that period. Both sites being monitored were in different facilities but both in Boise. -J Traceroute #1 1 ip-67-205-112-66.static.privatedns.com (67.205.112.66) 0.842 ms 0.908 ms 0.936 ms 2 te8-4.v0707.cl-core04.mtl.iweb.com (67.205.127.121) 1.047 ms 1.068 ms 1.089 ms 3 if-9-1.icore1.MTT-Montreal.as6453.net (206.82.135.29) 7.737 ms 7.813 ms 7.837 ms 4 if-10-0-0-19.mcore3.MTT-Montreal.as6453.net (216.6.114.13) 0.907 ms 0.935 ms 0.955 ms 5 if-9-0-0-0.tcore2.NYY-NewYork.as6453.net (216.6.99.9) 7.871 ms 7.904 ms 7.926 ms 6 if-11-2.tcore1.NYY-NewYork.as6453.net (216.6.99.2) 7.815 ms 8.013 ms 8.032 ms 7 jfk-brdr-04.inet.qwest.net (63.146.27.41) 7.942 ms 7.827 ms 7.790 ms Traceroute #2 1 207.97.207.194 (207.97.207.194) 0.549 ms 0.578 ms 0.620 ms 2 core1-5.iad1.rackspace.com (69.20.2.18) 0.303 ms 0.354 ms 0.400 ms 3 vlan901.edge3.iad1.rackspace.com (69.20.1.3) 196.221 ms 196.284 ms 196.328 ms 4 dcx2-edge-01.inet.qwest.net (67.133.246.157) 0.368 ms 0.355 ms 0.349 ms ------------------- Jason J. W. Williams COO/CTO, DigiTar williamsjj(a)digitar.com V: 208.343.8520 M: 208.863.0727 F: 208.322.8522 IM: williamsjj(a)digitar.com

4 4

Re: [outages] [Outages-discussion] UK wide DSL broadband outages
by Jay Ashworth 21 Oct '11

21 Oct '11

----- Original Message ----- > From: "Paul Lettington" <paul(a)plett.co.uk> > On Fri, Oct 21 2011 08:10:02 -0500, Frank Bulk wrote: > > Why did a 15-minute outage cause subtended customers to be out for > > much longer than that? > > Because BT's RADIUS servers which look at the realm of the login and > route the session to the correct ISP are unable to cope with the load of all the > ADSL users in the country continually reconnecting during an outage like > this. > > So when BT can't connect the end user to the ISP they accept the end > user's session themselves with a RFC1918 IP address for exactly an hour and > not connected to the internet, to stop them trying to log in. Sounds to me like they need to change that to "15 minutes + 1 minute * rand(10) Cheers, -- jra -- Jay R. Ashworth Baylink jra(a)baylink.com Designer The Things I Think RFC 2100 Ashworth & Associates http://baylink.pitas.com 2000 Land Rover DII St Petersburg FL USA http://photo.imageinc.us +1 727 647 1274

1 0

UK wide DSL broadband outages
by Martin Hepworth 21 Oct '11

21 Oct '11

Looks like BT are having some major issues with the DSL lines this morning. Seeing several ISP's reporting this.. http://portal.plus.net/supportpages.html?a=2&support_action=messages&ispser… http://status.zensupport.co.uk/active/5/2389 etc -- Martin Hepworth Oxford, UK

2 1

Re: [outages] Intermittent ICMP and DNS for 4.2.2.[123]
by Booher, Mike 19 Oct '11

19 Oct '11

About 3pm we lost some major links and are working a ticket with Level3. They are indicating It is a likely failed module between Houston and Denton, TX. No ETR at this time. Not sure it Is associated with the intermittent ICP/DNS issues... ~ Mike

1 0

Intermittent ICMP and DNS for 4.2.2.[123]
by Dan White 19 Oct '11

19 Oct '11

I've been noticing packet loss while pinging 4.2.2.1, 4.2.2.2, and 4.2.2.3 in the last few minutes, which are usually my go to IP addresses for testing external server connectivity. DNS also seems to be intermittent. -- Dan White

7 11

HTTP was timing out with high latency to v6 version of www.charter.com and ipv6.cnn.com
by Frank Bulk 19 Oct '11

19 Oct '11

Started at 10:45 am Central, cleared up by 11:17 am Central. ============== Before ============== nagios:/var/log# tcptraceroute6 www.charter.com traceroute to www.charter.com (2607:f428:3:1:80:80:80:1) from 2607:fe28:0:1000::5, port 80, from port 47336, 30 hops max, 60 bytes packets 1 2607:fe28:0:1003::2 (2607:fe28:0:1003::2) 1.070 ms 1.490 ms 1.392 ms 2 router-core.mtcnet.net (2607:fe28:0:1000::1) 1.944 ms 1.768 ms 1.661 ms 3 sxct.sxcy.mtcnet.net (2607:fe28:11:1002::197) 2.090 ms 1.906 ms 1.350 ms 4 v6-siouxcenter.sxcy.137.netins.net (2001:5f8:7f06::5) 3.759 ms 2.979 ms 2.658 ms 5 v6-ins-db1-te-13-2-219.desm.netins.net (2001:5f8:1:1::1) 7.358 ms 6.969 ms 6.715 ms 6 v6-ins-dc1-et-8-2.desm.netins.net (2001:5f8::1:1) 9.249 ms 8.823 ms 8.477 ms 7 2001:428:3801:210:0:1:0:1 (2001:428:3801:210:0:1:0:1) 21.343 ms 20.996 ms 20.693 ms 8 2001:428::205:171:202:195 (2001:428::205:171:202:195) 33.249 ms 32.945 ms 32.716 ms 9 * * * 10 * * * 11 2607:f428:3:1:80:80:80:1 (2607:f428:3:1:80:80:80:1) 3184.745 ms [open] 3184.899 ms 183.671 ms nagios:/var/log# ============== After ============== nagios:/var/log# tcptraceroute6 www.charter.com traceroute to www.charter.com (2607:f428:3:1:80:80:80:1) from 2607:fe28:0:1000::5, port 80, from port 37668, 30 hops max, 60 bytes packets 1 2607:fe28:0:1003::2 (2607:fe28:0:1003::2) 1.734 ms 1.230 ms 1.828 ms 2 router-core.mtcnet.net (2607:fe28:0:1000::1) 1.610 ms 3.112 ms 6.192 ms 3 sxct.sxcy.mtcnet.net (2607:fe28:11:1002::197) 6.111 ms 6.498 ms 1.429 ms 4 v6-siouxcenter.sxcy.137.netins.net (2001:5f8:7f06::5) 8.275 ms 3.175 ms 3.589 ms 5 v6-ins-db1-te-13-2-219.desm.netins.net (2001:5f8:1:1::1) 93.316 ms 8.445 ms 6.854 ms 6 v6-ins-dc1-et-8-2.desm.netins.net (2001:5f8::1:1) 10.056 ms 8.804 ms 11.640 ms 7 2001:428:3801:210:0:1:0:1 (2001:428:3801:210:0:1:0:1) 21.810 ms 24.138 ms 23.784 ms 8 2001:428::205:171:202:195 (2001:428::205:171:202:195) 36.477 ms 42.724 ms 32.726 ms 9 * * * 10 * * * 11 2607:f428:3:1:80:80:80:1 (2607:f428:3:1:80:80:80:1) 69.142 ms [open] 68.620 ms 68.139 ms nagios:/var/log# ============== Before ============== nagios:/var/log# tcptraceroute6 ipv6.cnn.com traceroute to ipv6.cnn.com (2620:100:e000::8001) from 2607:fe28:0:1000::5, port 80, from port 52863, 30 hops max, 60 bytes packets 1 2607:fe28:0:1003::2 (2607:fe28:0:1003::2) 1.090 ms 1.209 ms 1.889 ms 2 router-core.mtcnet.net (2607:fe28:0:1000::1) 1.608 ms 2.262 ms 1.480 ms 3 sxct.sxcy.mtcnet.net (2607:fe28:11:1002::197) 2.583 ms 2.059 ms 1.232 ms 4 v6-siouxcenter.sxcy.137.netins.net (2001:5f8:7f06::5) 3.871 ms 3.064 ms 2.676 ms 5 v6-ins-db1-te-13-2-219.desm.netins.net (2001:5f8:1:1::1) 7.450 ms 7.005 ms 6.739 ms 6 v6-ins-dc1-et-8-2.desm.netins.net (2001:5f8::1:1) 9.206 ms 8.867 ms 8.480 ms 7 2001:428:3801:210:0:1:0:1 (2001:428:3801:210:0:1:0:1) 21.453 ms 20.980 ms 20.707 ms 8 2001:428:0:1:205:171:202:209 (2001:428:0:1:205:171:202:209) 45.218 ms 44.844 ms 44.537 ms 9 ae5.edge2.dallas3.level3.net (2001:1900:4:3::1c1) 45.079 ms 44.733 ms 44.361 ms 10 * vl-80.car2.Dallas1.Level3.net (2001:1900:17:3::4) 45.890 ms * 11 vl-4061.car1.Atlanta2.Level3.net (2001:1900:4:1::96) 188.828 ms 188.544 ms 188.134 ms 12 vl-52.car1.Atlanta1.Level3.net (2001:1900:1c:2::5) 188.158 ms 187.907 ms 187.752 ms 13 TIME-WARNER.car1.Atlanta1.Level3.net (2001:1900:2100::496) 189.189 ms * 188.380 ms 14 2620:100:e000:ffff::21 (2620:100:e000:ffff::21) 76.773 ms 76.348 ms 75.971 ms 15 * 2620:100:e000:ffff::12 (2620:100:e000:ffff::12) 190.214 ms * 16 * 2620:100:e000:ffff::25 (2620:100:e000:ffff::25) 189.459 ms 189.274 ms 17 * * * 18 * * 2620:100:e000::8001 (2620:100:e000::8001) 189.209 ms [open] 19 2620:100:e000::8001 (2620:100:e000::8001) 189.731 ms [open] 189.373 ms 189.482 ms nagios:/var/log# ============== After ============== nagios:/var/log# tcptraceroute6 ipv6.cnn.com traceroute to ipv6.cnn.com (2620:100:e000::8001) from 2607:fe28:0:1000::5, port 80, from port 36895, 30 hops max, 60 bytes packets 1 2607:fe28:0:1003::2 (2607:fe28:0:1003::2) 1.120 ms 1.622 ms 1.307 ms 2 router-core.mtcnet.net (2607:fe28:0:1000::1) 1.926 ms 1.698 ms 1.771 ms 3 sxct.sxcy.mtcnet.net (2607:fe28:11:1002::197) 1.997 ms 2.013 ms 1.736 ms 4 v6-siouxcenter.sxcy.137.netins.net (2001:5f8:7f06::5) 3.830 ms 3.122 ms 2.693 ms 5 v6-ins-db1-te-13-2-219.desm.netins.net (2001:5f8:1:1::1) 7.786 ms 7.108 ms 6.820 ms 6 v6-ins-dc1-et-8-2.desm.netins.net (2001:5f8::1:1) 9.158 ms 8.758 ms 8.498 ms 7 2001:428:3801:210:0:1:0:1 (2001:428:3801:210:0:1:0:1) 21.350 ms 21.007 ms 38.574 ms 8 2001:428:0:1:205:171:202:209 (2001:428:0:1:205:171:202:209) 47.081 ms 44.443 ms 44.111 ms 9 ae5.edge2.dallas3.level3.net (2001:1900:4:3::1c1) 44.579 ms 92.648 ms 43.891 ms 10 vl-60.car2.Dallas1.Level3.net (2001:1900:17:1::4) 157.562 ms 142.664 ms 225.796 ms 11 vl-4061.car1.Atlanta2.Level3.net (2001:1900:4:1::96) 146.571 ms 89.600 ms 127.063 ms 12 vl-52.car1.Atlanta1.Level3.net (2001:1900:1c:2::5) 73.790 ms 73.028 ms 72.797 ms 13 TIME-WARNER.car1.Atlanta1.Level3.net (2001:1900:2100::496) 118.018 ms 73.072 ms 72.854 ms 14 2620:100:e000:ffff::21 (2620:100:e000:ffff::21) 75.514 ms 74.957 ms 74.781 ms 15 2620:100:e000:ffff::12 (2620:100:e000:ffff::12) 74.907 ms 74.363 ms 73.949 ms 16 2620:100:e000:ffff::25 (2620:100:e000:ffff::25) 75.807 ms 74.881 ms 74.565 ms 17 2620:100:e000:ffff::32 (2620:100:e000:ffff::32) 76.823 ms 74.865 ms 74.626 ms 18 2620:100:e000::8001 (2620:100:e000::8001) 74.190 ms [open] 73.974 ms 73.231 ms nagios:/var/log#

1 0

META: Reminder: post the trace
by Jay Ashworth 19 Oct '11

19 Oct '11

If you're reporting an outage on the list, and you conducted a traceroute that failed, remember to post it at the end of your message; it's usually more helpful than not. I personally recommend 'mtr -r -c 10' (accompanied with 'mtr -nr -c 10', since sometimes you want all the IPs, and that's mtr's one blind spot), but any trace is better than none, whether you're reporting a problem, a meetoo, or a WFM. Cheers, -- jra -- Jay R. Ashworth Baylink jra(a)baylink.com Designer The Things I Think RFC 2100 Ashworth & Associates http://baylink.pitas.com 2000 Land Rover DII St Petersburg FL USA http://photo.imageinc.us +1 727 647 1274

1 0

AT&T & Google Issues
by Louis Arsenault 18 Oct '11

18 Oct '11

We are noticing a packet loss issue on our AT&T circuit to Google service. I have an open ticket with AT&T but they insist that it is a google issue. We are getting 50% packet loss to both www.google.com and 8.8.8.8. These are both taking the same route jumping from AT&T to Google. Trace route & pings are good up to 12.94.87.18. AT&T says that is google's router even though it is there IP. Anyone know how to let google know there is an issue with one of there AT&T pipes? -Louis

3 3