October 2013 - Outages - Outages Lists

Windstream Internet Michigan
by Brian Chaffins 30 Oct '13

30 Oct '13

Just lost multiple T1 Internet circuits to Windstream for Internet. Their Call Center is just saying they are busy and my wait time is longer them 20 mins. Brian

1 0

SIP problems in So Cal
by Jay Hennigan 29 Oct '13

29 Oct '13

Seeing 503 service unavailable and/or 20-second PDD on Bandwidth.com and Intelepeer. Anyone else? -- -- Jay Hennigan - CCIE #7880 - Network Engineering - jay(a)impulse.net Impulse Internet Service - http://www.impulse.net/ Your local telephone and internet company - 805 884-6323 - WB6RDV

5 4

AT&T UVerse down
by Edward Vielmetti 29 Oct '13

29 Oct '13

Multiple reports from Twitter that AT&T's UVerse is down in multiple cities. I am away from home working so I can't confirm (I have UVerse at home). The Chicago Tribune has a "breaking news" story: http://www.chicagotribune.com/business/breaking/chi-att-uverse-service-outa… By Wailin WongTribune reporter 11:05 a.m. CDT, October 29, 2013 AT&T is working to fix a U-Verse service outage in the Chicago area after a “software upgrade issue,” the company said Tuesday. “A limited number of U-verse customers may be experiencing issues with U-verse services due to a software upgrade issue,” AT&T said in a statement. “Technicians have resolved the issue, and service has been restored for most impacted customers. We expect service to be fully restored for all customers by early afternoon. We apologize for this inconvenience.” What's being reported via Twitter: 1. *A. Frankensteinbaugh* ‏@adamsteinbaugh<https://twitter.com/adamsteinbaugh> 1m <https://twitter.com/adamsteinbaugh/status/395242190072320000> Cool. AT&T's *Uverse* is *down*. And apparently it's *down* everywhere. So the ratio of AOL users is higher than normal. Good luck, Twitter. Expand <https://twitter.com/adamsteinbaugh/status/395242190072320000> 2. *Philo Horrorhagen* ‏@philohagen <https://twitter.com/philohagen>11m<https://twitter.com/philohagen/status/395239723205029889> @*Uverse* <https://twitter.com/Uverse> the internet is still *down* 10 hours later and none of my numerous phone calls and hours on hold have resulted in talking to anyone. ** View conversation<https://twitter.com/philohagen/status/395239723205029889> 3. *Nic Birdsall* ‏@nbirdsall <https://twitter.com/nbirdsall>37m<https://twitter.com/nbirdsall/status/395233099962785792> *Uverse* is still *down* in LA. Coming up on 12 hours now. And I probably still won't get the speed I pay for. Expand <https://twitter.com/nbirdsall/status/395233099962785792> 4. *CUB Illinois* ‏@cubillinois <https://twitter.com/cubillinois>40m<https://twitter.com/cubillinois/status/395232521044385792> #ATT <https://twitter.com/search?q=%23ATT&src=hash> #*UVerse*<https://twitter.com/search?q=%23UVerse&src=hash>phone and Internet service *down* for some #Chicago<https://twitter.com/search?q=%23Chicago&src=hash>area customers http://ow.ly/qi4o9 <http://t.co/KHX8bAmOhp> ** View summary<https://twitter.com/cubillinois/status/395232521044385792> 5. *Bonnie Kourvelas* ‏@producerbonnie<https://twitter.com/producerbonnie> 46m <https://twitter.com/producerbonnie/status/395230935550922752> #*uverse* <https://twitter.com/search?q=%23uverse&src=hash> is *down* in #memphis <https://twitter.com/search?q=%23memphis&src=hash> too. A nationwide outage caused by "rebooting servers"? How about putting a bulletin on your websites, folks? Expand <https://twitter.com/producerbonnie/status/395230935550922752> thanks Ed -- Edward Vielmetti +1 734 330 2465 edward.vielmetti(a)gmail.com

1 0

Outages with google?
by Maynard 28 Oct '13

28 Oct '13

Anyone experiencing outages with google? Gmail and google are not responding in my end v/r, Maynard Azurin ☎ office: (301)-981-9516 ☎ cell: (301)-318-7870

2 1

Re: [outages] DNS timeouts for *some* zones from Sprint 4G/LTE
by Jay Ashworth 28 Oct '13

28 Oct '13

We do what we can. But you'll note I said a) that it was happening to many other domains, and b) that my machines in that domain resolved fine from other connectivity. Cheers, - jra Stephane Bortzmeyer <bortzmeyer(a)nic.fr> wrote: >On Sat, Oct 26, 2013 at 12:27:25AM -0400, > Jay Ashworth <jra(a)baylink.com> wrote > a message of 31 lines which said: > >> I'm seeing, tonight, the inability to resolve DNS lookups for *some* >domains, >> from my Sprint 4G phone. FB and Twitter are fine, and Currents can >get to >> Android Central, frex, but panther.baylink.com, where my weather page >resides, >> won't resolve, > >No info but I note baylink.com has only two name servers and, much >worse, they have the same IP address. So, any network glitch and the >domain is unresolvable. > >% check-soa -i baylink.com >ns7.baylink.com. > 66.221.0.238: OK: 2013073001 (124 ms) >ns8.baylink.com. > 66.221.0.238: OK: 2013073001 (125 ms) -- Sent from my Android phone with K-9 Mail. Please excuse my brevity.

1 0

Daily packet loss/routing issue (Layer42/Verizon/Deutsche Telekom/Comcast)
by Jeremy Chadwick 27 Oct '13

27 Oct '13

Here's another one of these incredibly annoying situations that baffles me (and of course figuring out who's responsible is not something I have the power to do). Been watching this one happen every day, usually between the hours of 1700 and 2000 PDT (UTC-0700). Packet loss and latency lasts anywhere between 30 to 75 minutes. From today around 1915: src IP: 204.8.213.80 dst IP: 76.102.14.35 Host Loss% Snt Last Avg Best Wrst StDev 1. --- 0.0% 161 21.5 1.4 0.3 59.3 6.0 2. --- 0.0% 160 0.4 1.2 0.3 41.4 4.6 3. --- 0.0% 160 0.3 2.2 0.2 44.5 6.5 4. --- 0.0% 160 1.5 3.3 1.2 47.2 6.7 5. xe0-1-0-1.core1.sv8.layer42.net 0.0% 160 2.2 2.1 1.9 2.4 0.1 6. 193.159.165.73 0.0% 160 4.9 4.1 1.9 8.1 1.3 7. 80.156.163.154 24.5% 160 1000. 933.0 6.3 1738. 624.0 8. pos-1-12-0-0-cr01.sanjose.ca.ibone.comcast 27.7% 160 987.5 932.7 7.2 1706. 647.7 9. he-0-6-0-0-ar01.sfsutro.ca.sfba.comcast.ne 26.4% 160 981.4 917.1 7.2 1719. 642.9 10. te-0-4-0-10-ur05.santaclara.ca.sfba.comcas 30.8% 160 980.0 895.1 9.0 1710. 636.9 11. te-6-0-acr03.santaclara.ca.sfba.comcast.ne 17.0% 160 967.3 614.0 6.9 1224. 518.1 12. c-76-102-14-35.hsd1.ca.comcast.net 18.9% 160 970.2 609.1 16.2 1229. 517.8 src IP: 76.102.14.35 dst IP: 204.8.213.80 Host Loss% Snt Rcv Last Avg Best Wrst 1. gw.home.lan 0.0% 563 563 0.3 0.2 0.2 1.9 2. 76.102.12.1 0.0% 563 563 8.4 8.8 7.9 22.8 3. te-0-2-0-5-ur06.santaclara.ca.sfba.comcast. 0.0% 563 563 8.7 8.9 8.1 23.1 4. te-1-1-0-9-ar01.oakland.ca.sfba.comcast.net 0.0% 563 563 13.5 12.4 9.7 30.2 5. be-90-ar01.sfsutro.ca.sfba.comcast.net 0.0% 563 563 12.5 13.7 11.1 41.5 6. he-3-8-0-0-cr01.sanjose.ca.ibone.comcast.ne 0.0% 563 563 13.2 15.0 12.5 25.0 7. be-14-pe02.11greatoaks.ca.ibone.comcast.net 0.0% 562 562 16.5 16.5 15.6 43.1 8. 23-30-206-94-static.hfc.comcastbusiness.net 0.0% 562 562 16.4 22.5 15.6 40.2 9. 0.xe-2-0-8.XL3.SJC7.ALTER.NET 0.7% 562 558 20.3 29.6 15.3 124.7 10. 0.so-4-0-0.XL1.SJC1.ALTER.NET 1.1% 562 556 26.6 22.9 16.8 174.2 0.so-7-0-0.XL1.SJC1.ALTER.NET 11. POS1-0.XR1.SJC1.ALTER.NET 23.0% 562 432 196.6 150.0 17.8 459.6 12. 193.ATM7-0.GW4.SJC1.ALTER.NET 24.6% 562 424 1133. 669.3 17.9 1835. 13. ??? What interests me: hops #6 and #7 shown in the first mtr: 193.159.165.73 = AS3320 = Deutsche Telekom AG 80.156.163.154 = AS3320 = Deutsche Telekom AG Using Layer42's own looking glass long after the issue was over: lg-west>show ip bgp 76.102.14.35 BGP routing table entry for 76.96.0.0/11, version 4070745 Paths: (1 available, best #1, table default) Not advertised to any peer Refresh Epoch 1 8121 3320 7922 69.36.239.8 from 69.36.239.8 (69.36.239.8) Origin IGP, metric 301, localpref 100, valid, external, best rx pathid: 0, tx pathid: 0x0 lg-west>traceroute 76.102.14.35 Type escape sequence to abort. Tracing the route to c-76-102-14-35.hsd1.ca.comcast.net (76.102.14.35) VRF info: (vrf in name/id, vrf out name/id) 1 69.36.229.177 [AS 8121] 1 msec 1 msec 1 msec 2 xe0-0-0-4.core1.sv8.layer42.net (65.50.198.161) [AS 8121] 3 msec 2 msec 2 msec 3 193.159.165.73 [AS 3320] 4 msec 2 msec 3 msec 4 80.156.163.154 [AS 3320] 4 msec 5 msec 3 msec 5 pos-1-12-0-0-cr01.sanjose.ca.ibone.comcast.net (68.86.87.141) [AS 7922] 7 msec 6 msec 4 msec 6 he-0-5-0-0-ar01.sfsutro.ca.sfba.comcast.net (68.86.91.46) [AS 7922] 6 msec 5 msec 8 msec 7 te-0-4-0-6-ur05.santaclara.ca.sfba.comcast.net (69.139.198.169) [AS 7922] 5 msec 6 msec 7 msec 8 te-6-0-acr03.santaclara.ca.sfba.comcast.net (68.86.249.66) [AS 7922] 5 msec 6 msec 5 msec 9 c-76-102-14-35.hsd1.ca.comcast.net (76.102.14.35) [AS 7922] 14 msec 18 msec 14 msec Two final things to note: 1. When the packet loss/latency issue is not occurring (e.g. presently), the paths are still the same. If packets were actually going from Sunnyvale (Layer42) to Germany I'd expect a lot higher than 4ms-7ms latency consistently, 2. The source/destination of 204.8.213.80 is a box at my place of work; border of network intentionally filters. We do peer with Layer42 directly, but I have not pursued this with them yet (as said initially, it's hard for me to accurately point fingers). -- | Jeremy Chadwick jdc(a)koitsu.org | | UNIX Systems Administrator http://jdc.koitsu.org/ | | Making life hard for others since 1977. PGP 4BD6C0CB |

3 2

111 8th power issues last night
by John Kinsella 26 Oct '13

26 Oct '13

A little late notice... Looks like there was a power outage in BT's NYC 111 8th DC around 10PM EST last night. Story to-date is procedures weren't followed during an IR scan maintenance window.

1 0

DNS timeouts for *some* zones from Sprint 4G/LTE
by Jay Ashworth 26 Oct '13

26 Oct '13

I'm seeing, tonight, the inability to resolve DNS lookups for *some* domains, from my Sprint 4G phone. FB and Twitter are fine, and Currents can get to Android Central, frex, but panther.baylink.com, where my weather page resides, won't resolve, as won't tapemonster.com, www. and en.wikipedia.org, and a few others. There seems no particular rhyme or reason to it. Perhaps those zone servers are more than 600mi away? :-) (I have logged into one of my Linux boxen, *through* the 4G connection, and did dig+trace from there; panther.baylink.com resolves just fine that way. And, for the record, the 4G link lets me ping panther by IP just fine; this appears to be solely DNS, and not all of it.) Started around 1800EDTish, I think. Anyone else seeing anything like this? Cheers, -- jra -- Jay R. Ashworth Baylink jra(a)baylink.com Designer The Things I Think RFC 2100 Ashworth & Associates http://baylink.pitas.com 2000 Land Rover DII St Petersburg FL USA #natog +1 727 647 1274

1 0

Issue with Time Warner Cable and akamai in Chicago
by Josh Luthman 25 Oct '13

25 Oct '13

Started about 10 minutes ago. Confirmed with tech support. Josh Luthman Office: 937-552-2340 Direct: 937-552-2343 1100 Wayne St Suite 1337 Troy, OH 45373

2 3

Google servers down for Android
by Jered Morgan 24 Oct '13

24 Oct '13

http://forums.androidcentral.com/google-nexus-4/326749-google-servers-down.… Found this after my own device started displaying the message... Anyone have any info?

9 12