July 2011 - Outages - Outages Lists

Several IPv6 sites down?
by Frank Bulk 01 Aug '11

01 Aug '11

Just a few minutes ago three of the 20+ IPv6 sites I monitor became inaccessible: ipv6.cnn.com (IPv6-only) HOST: nagios Loss% Snt Last Avg Best Wrst StDev 1. 2607:fe28:0:1003::2 0.0% 10 1.4 1.3 1.1 2.4 0.4 2. router-core.mtcnet.net 0.0% 10 1.0 1.1 0.9 1.8 0.3 3. sxct.sxcy.mtcnet.net 0.0% 10 0.9 1.0 0.9 1.1 0.1 4. v6-siouxcenter.sxcy.137.neti 0.0% 10 2.7 3.5 2.7 4.6 0.7 5. v6-ins-db1-et-11-8-204.desm. 0.0% 10 13.1 11.0 8.3 16.6 3.6 6. v6-ins-dc1-et-8-2.desm.netin 0.0% 10 8.5 8.6 8.5 8.9 0.1 7. 2001:428:3801:210:0:1:0:1 0.0% 10 20.7 27.5 20.7 56.8 14.2 8. ??? 100.0 10 0.0 0.0 0.0 0.0 0.0 9. vl-60.car2.Dallas1.Level3.ne 0.0% 10 116.7 49.0 41.1 116.7 23.8 10. ??? 100.0 10 0.0 0.0 0.0 0.0 0.0 www.charter.com (IPv4 is fine, just not IPv6) HOST: nagios Loss% Snt Last Avg Best Wrst StDev 1. 2607:fe28:0:1003::2 0.0% 10 1.1 1.1 1.1 1.2 0.0 2. router-core.mtcnet.net 0.0% 10 1.0 1.0 0.9 1.4 0.1 3. sxct.sxcy.mtcnet.net 0.0% 10 0.9 0.9 0.9 1.0 0.0 4. v6-siouxcenter.sxcy.137.neti 0.0% 10 3.4 3.8 2.8 4.7 0.6 5. v6-ins-db1-et-11-8-204.desm. 0.0% 10 8.3 8.3 8.3 8.5 0.1 6. v6-ins-dc1-et-8-2.desm.netin 0.0% 10 8.5 24.0 8.5 161.5 48.3 7. 2001:428:3801:210:0:1:0:1 0.0% 10 20.8 24.8 20.7 45.5 8.8 8. ??? 100.0 10 0.0 0.0 0.0 0.0 0.0 ipv6.weather.yahoo.com (there's three AAAA records, just 2a00:1288:f006:1fe::1000 is not working) HOST: nagios Loss% Snt Last Avg Best Wrst StDev 1. 2607:fe28:0:1003::2 0.0% 10 1.2 6.5 1.1 54.3 16.8 2. router-core.mtcnet.net 0.0% 10 1.1 1.1 0.9 1.7 0.2 3. sxct.sxcy.mtcnet.net 0.0% 10 0.9 0.9 0.9 1.1 0.1 4. v6-siouxcenter.sxcy.137.neti 0.0% 10 2.7 3.9 2.7 6.3 1.1 5. v6-ins-db1-et-11-8-204.desm. 0.0% 10 13.6 9.4 8.2 13.9 2.3 6. v6-ins-dc1-et-8-2.desm.netin 0.0% 10 8.6 12.6 8.5 40.8 10.2 7. 2001:428:3801:210:0:1:0:1 0.0% 10 20.6 30.0 20.6 112.9 29.1 8. ??? 100.0 10 0.0 0.0 0.0 0.0 0.0 9. vl-90.car1.Dallas1.Level3.ne 0.0% 10 41.5 41.3 41.1 41.5 0.1 10. vl-4042.car1.NewYork1.Level3 0.0% 10 58.0 58.6 57.7 63.8 1.9 11. vl-4086.edge3.London1.Level3 0.0% 10 127.2 126.9 126.5 127.3 0.3 12. vl-52.car3.London1.Level3.ne 0.0% 10 127.0 130.4 126.1 166.7 12.7 13. YAHOO-INC.car3.London1.Level 0.0% 10 126.3 126.3 126.0 127.1 0.3 14. ??? 100.0 10 0.0 0.0 0.0 0.0 0.0 Can anyone else confirm? It's like our /32 stopped propagating to some networks. Frank

3 9

Fwd: Re: Internet Blip in San Diego at 1pm?
by virendra rode 31 Jul '11

31 Jul '11

FYI -/ regards, /virendra -------- Original Message -------- Subject: Re: Internet Blip in San Diego at 1pm? Date: Sun, 31 Jul 2011 14:05:15 -0700 From: Joe Renwick <joe(a)gonetforward.com> To: virendra.rode(a)gmail.com CC: nanog(a)nanog.org Level3 had an issue between LA and Dallas at 1pm. Apparently large amounts of traffic from San Diego head through Dallas so it appeared at a whole Internet drop. Joe On Sun, Jul 31, 2011 at 1:54 PM, virendra rode <virendra.rode(a)gmail.com>wrote: > On 07/31/2011 01:35 PM, Joe Renwick wrote: > >> Several of my customers in San Diego noticed large drops in traffic at 1pm >> today. Note it was not a total loss in connectivity. Anyone else notice >> this? >> >> ------------------- > Several of our east coat/overseas customers called in about reachability > issues towards our SD pop. I believe this was LEVEL3 peering (?) related. > > Hopefully transparency will follow. > > regards, > /virendra > >

3 2

81.26.208.0/20 not in BGP table?
by Bill Wichers 28 Jul '11

28 Jul '11

Can anyone confirm/deny that this subnet is unreachable? I'm not showing it in my BGP tables from XO or Level(3), and Routeviews also seems to be showing no entries (one entry only, very unusual). I have one report from a customer that it's reachable from Comcast though so I'm looking for some other opinions. -Bill

8 8

Five9 Hosted PBX
by Bryan Riley 27 Jul '11

27 Jul '11

Five9 hosted PBX vendor is apparently down hard. Been about 2 hours anybody else experienced this? [cid:image001.jpg@01CC4C75.7848E6F0] Bryan Riley Operations Manager 757.802.4991 (office) 757.351.4115 (fax) 757.805.4955 (main) www.oncalltelecom.com<http://www.oncalltelecom.com>

1 0

ADMIN:Outages List FAQ/Monthly Post.
by virendra rode 27 Jul '11

27 Jul '11

Please take 5 minutes to read it before you post [again]. http://wiki.outages.org/index.php/Code_of_Conduct#OUTAGES_List_FAQ Your adherence to the guidelines is appreciated. regards, /virendra

1 0

Telecity London power problems
by Martin Hepworth 27 Jul '11

27 Jul '11

http://www.theregister.co.uk/2011/07/27/telecity_power_problems_again/ Noticed a 1 minute blip in certain networks this am but can't see any issues myself at the moment -- Martin Hepworth Oxford, UK

1 0

Facebook Pages
by andrew.wallace 26 Jul '11

26 Jul '11

I'm having a problem reaching any Facebook Pages. Personal profiles are fine. Andrew

10 12

GBLX slash XO issues?
by J. Oquendo 26 Jul '11

26 Jul '11

Anyone seeing hiccups on while traversing po2-20G.ar2.dca3 or po3-20G-ar1.mia2 I'm getting bucket loads of TTL exceeded errors: >From 67.16.142.209 icmp_seq=1177 Time to live exceeded >From 67.16.142.209 icmp_seq=1181 Time to live exceeded >From 67.16.142.209 icmp_seq=1184 Time to live exceeded 64 bytes from 8.20.18x.xxx: icmp_seq=1187 ttl=118 time=94.5 ms 64 bytes from 8.20.18x.xxx: icmp_seq=1188 ttl=118 time=91.5 ms 64 bytes from 8.20.18x.xxx: icmp_seq=1189 ttl=118 time=92.0 ms 64 bytes from 8.20.18x.xxx: icmp_seq=1190 ttl=118 time=91.0 ms ... 1259 packets transmitted, 1098 received, +34 errors, 12% packet loss, time 1259563ms rtt min/avg/max/mdev = 8.192/156.007/8703.504/684.907 ms, pipe 10 Whenever I cross a peering between the two (XO + Global) I am seeing nothing but errors. -- =+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+ J. Oquendo SGFA, SGFE, C|EH, CNDA, CHFI, OSCP, CPT, RWSP, GREM "It takes 20 years to build a reputation and five minutes to ruin it. If you think about that, you'll do things differently." - Warren Buffett 42B0 5A53 6505 6638 44BB 3943 2BF7 D83F 210A 95AF http://pgp.mit.edu:11371/pks/lookup?op=get&search=0x2BF7D83F210A95AF

2 2

AT&T Email - SMS Gateway Outage?
by Drew Linsalata 25 Jul '11

25 Jul '11

It appears that AT&T is having an email to SMS gateway outage this morning. No messages sent to xxxxxxxxxx(a)txt.att.net are being delivered to our handsets. Can anyone else confirm this? We're in the NYC metro area.

9 8

Level3 Denver Outage?
by Shawn Clemens 21 Jul '11

21 Jul '11

Did anyone just experience a 6-8 minute outage on their Level3 Service in the Denver market? This was from 13:54 to 14:02 (20:54 to 21:02 UTC) according to my traffic monitoring. I called customer care, and got someone that could barely speak English, who just told me "No problem". Shawn Clemens MHO Networks

1 0