July 2014 - Outages - Outages Lists

wiki.outage.org outage?
by WalkaboutTigger . 04 Aug '14

04 Aug '14

Is the wiki site supposed to be down? (Can't contact the database server: Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (111) (localhost)) Apologies in advance for sending this to the entire group - I do not have the admins' email address. -- ------------------------------------------------------------------- On the Internet, you can be anything you want. It's strange so many people choose to be stupid.

2 1

More on AT&T
by Aaron D. Osgood 01 Aug '14

01 Aug '14

Apparently this is reasonably widespread: http://downdetector.com/status/att Aaron D. Osgood Streamline Solutions L.L.C 274 E. Eau Gallie Blvd. #336 Indian Harbour Beach, FL 32937 TEL: 207-518-8455 MOBILE: 207-831-5829 GTalk: aaron.osgood <mailto:AOsgood@Streamline-Solutions.net> AOsgood(a)Streamline-Solutions.net <http://www.streamline-solutions.net/> http://www.streamline-solutions.net Introducing Efficiency to Business since 1986.

10 9

AT&T SMS outage?
by Aaron D. Osgood 31 Jul '14

31 Jul '14

Is anyone else experiencing an AT&T outage? I have clients stating they have not received any sms/text messages since 5pm last night all with aT&T phones. We have an AT&T SMS modem that seems to be working, at least to other carriers. Verizon phone is receiving texts, MetroPCS phones are receiving texts, but AT&T phones are not These symptoms are being seen in messages originating from the Tampa, FL area Aaron D. Osgood Streamline Solutions L.L.C 274 E. Eau Gallie Blvd. #336 Indian Harbour Beach, FL 32937 TEL: 207-518-8455 MOBILE: 207-831-5829 GTalk: aaron.osgood <mailto:AOsgood@Streamline-Solutions.net> AOsgood(a)Streamline-Solutions.net <http://www.streamline-solutions.net/> http://www.streamline-solutions.net Introducing Efficiency to Business since 1986.

1 0

Re: [outages] Level3 10G DWDM Ckt Detroit to Chicago
by Clayton Zekelman 30 Jul '14

30 Jul '14

Oddly enough, the ckt came up a few hours ago - I'm wondering if they re-routed us. No notes in the ticket (we're getting the updates now... they had a bad email address). At 04:53 PM 30/07/2014, Luke Rockwell wrote: >7/30/14 8:27:19 PM GMT Repair work is ongoing at >the site. Updates will be provided as they are received. >7/30/14 7:32:21 PM GMTField Services continue >excavation efforts and have estimated splicing >will begin in approximately one hour. >7/30/14 7:10:12 PM GMT Repair efforts are >ongoing at the site. No estimated restoral time is currently available. >7/30/14 6:18:25 PM GMTField Services report >excavation efforts are in progress to access the >damaged cable. The splice crew is onsite to >continue with repair following excavation. >7/30/14 5:58:09 PM GMT Repair crews continue en >route to the site to proceed with restoration efforts. >7/30/14 5:18:26 PM GMTA 108-count fiber cut has >been confirmed to have been caused by >construction near the railway. All necessary >repair crews are en route with no ETA available at this time. >7/30/14 4:22:00 PM GMT Field Services report >that there is construction in the area that has >likely caused the fiber cut. An excavation crew >has been engaged to uncover the suspected >damaged portion of the span. No ETA has been provided for the excavation crew. >7/30/14 4:03:40 PM GMT Field Services continue >investigation in conjunction with railway >workers to further isolate the suspected fiber cut. >7/30/14 3:41:00 PM GMTField Services report that >the fault location is in close proximity to a >railway. As such, Field Services is coordinating >with railway workers to ensure safe access to >continue investigation. Splice crews have been >dispatched to the site and are expected to arrive shortly. >7/30/14 2:58:44 PM GMT Field Services arrived in >Southfield, MI and detected a fault >approximately 52 miles from the site. Field >Services is proceeding en route to the fault location. >7/30/14 2:11:36 PM GMTThe Transport NOC detected >a loss of signal in Dexter, MI is impacting >unprotected services. A fiber cut is suspected >and Field Services have been engaged for >investigation. Updates will be provided as they are received. > >..................................... >Luke Rockwell >Systems Administrator >Information Technology > >..................................... > >Cal Alumni Association | UC Berkeley >1 Alumni House, Berkeley, CA 94720 >TÂ 510.900.8196 >FÂ 510.642.6252 > >..................................... > >Commitment, Support, Passion > >_____________________________________ ><http://alumni.berkeley.edu/>alumni.berkeley.edu > >_____________________________________ ><https://www.facebook.com/CalAlumni>FacebookÂ |Â ><http://www.linkedin.com/groups?gid=70245>LinkedIn > > >On Wed, Jul 30, 2014 at 9:41 AM, Bill Wichers ><<mailto:billw@waveform.net>billw(a)waveform.net> wrote: > >This is affecting XO in Detroit too, I can see >them on protect path due to higher than normal latency Detroit<->Chicago. > >Â > >Â -Bill > >Â > >From: Outages >[mailto:outages-bounces@outages.org] On Behalf Of Luke Rockwell via Outages >Sent: Wednesday, July 30, 2014 12:14 PM >To: Clayton Zekelman >Cc: <mailto:Outages@outages.org>Outages(a)outages.org >Subject: Re: [outages] Level3 10G DWDM Ckt Detroit to Chicago > >Â > >Latest update from Level 3 trouble board > >Â > >7/30/14 4:03:40 PM GMT > >Field Services continue investigation in >conjunction with railway workers to further isolate the suspected fiber cut. > >7/30/14 3:41:00 PM GMT > >Field Services report that the fault location is >in close proximity to a railway. As such, Field >Services is coordinating with railway workers to >ensure safe access to continue investigation. >Splice crews have been dispatched to the site >and are expected to arrive shortly. > >7/30/14 2:58:44 PM GMT > >Field Services arrived in Southfield, MI and >detected a fault approximately 52 miles from the >site. Field Services is proceeding en route to the fault location. > >7/30/14 2:11:36 PM GMT > >The Transport NOC detected a loss of signal in >Dexter, MI is impacting unprotected services. A >fiber cut is suspected and Field Services have >been engaged for investigation. Updates will be provided as they are received. > > >..................................... > >Luke Rockwell >Systems Administrator >Information Technology > >..................................... > >Cal Alumni Association | UC Berkeley >1 Alumni House, Berkeley, CA 94720 >TÂ 510.900.8196 >FÂ 510.642.6252 > >..................................... > >Commitment, Support, Passion > >_____________________________________ > ><http://alumni.berkeley.edu/>alumni.berkeley.edu > >_____________________________________ > ><https://www.facebook.com/CalAlumni>FacebookÂ |Â ><http://www.linkedin.com/groups?gid=70245>LinkedIn > >Â > >On Wed, Jul 30, 2014 at 8:47 AM, Clayton >Zekelman via Outages <<mailto:outages@outages.org>outages(a)outages.org> wrote: > > > >Well, the same 10 Gig DWDM is down again - >10:53am. Â Level3 reports a fibre cut this time >- near the same location as the card failure on the 19th... > > > > > >At 11:38 AM 19/07/2014, Clayton Zekelman via Outages wrote: > >Looks like our circuit came back up a few minutes ago. > >We took a couple of brief hits on 2 1Gig DIAs >out of Southfield, MI though - not sure if that was related. > > >At 06:48 AM 19/07/2014, Clayton Zekelman wrote: > >Level3 10G DWDM Ckt from Detroit to Chicago >currently out of service (down at 6:15am EDT). > >Possibly related to some sort of transport >failure in Dexter, MI that happened around the same time. > > >--- > >Â > >--- > >Clayton Zekelman >Managed Network Systems Inc. (MNSi) >3363 Tecumseh Rd. E >Windsor, Ontario >N8W 1H4 > >tel. 519-985-8410 >fax. 519-985-8409 Â Â Â Â >_______________________________________________ >Outages mailing list ><mailto:Outages@outages.org>Outages(a)outages.org >https://puck.nether.net/mailman/listinfo/outages > >Â > --- Clayton Zekelman Managed Network Systems Inc. (MNSi) 3363 Tecumseh Rd. E Windsor, Ontario N8W 1H4 tel. 519-985-8410 fax. 519-985-8409

1 0

Re: [outages] Level3 10G DWDM Ckt Detroit to Chicago
by Clayton Zekelman 30 Jul '14

30 Jul '14

Looks like our circuit came back up a few minutes ago. We took a couple of brief hits on 2 1Gig DIAs out of Southfield, MI though - not sure if that was related. At 06:48 AM 19/07/2014, Clayton Zekelman wrote: >Level3 10G DWDM Ckt from Detroit to Chicago currently out of service >(down at 6:15am EDT). > >Possibly related to some sort of transport failure in Dexter, MI >that happened around the same time. > > >--- > >Clayton Zekelman >Managed Network Systems Inc. (MNSi) >3363 Tecumseh Rd. E >Windsor, Ontario >N8W 1H4 > >tel. 519-985-8410 >fax. 519-985-8409 --- Clayton Zekelman Managed Network Systems Inc. (MNSi) 3363 Tecumseh Rd. E Windsor, Ontario N8W 1H4 tel. 519-985-8410 fax. 519-985-8409

3 4

Anyone else seeing www.ebay.com down?
by Frank Bulk 28 Jul '14

28 Jul '14

Anyone else seeing www.ebay.com down? Here's a wget showing a hit to 66.135.210.181 and also a traceroute to the same. root@nagios:/home/fbulk# wget -4 www.ebay.com --2014-07-28 10:12:09-- http://www.ebay.com/ Resolving www.ebay.com... 66.135.210.181, 66.211.181.161, 66.211.181.181, ... Connecting to www.ebay.com|66.135.210.181|:80... connected. HTTP request sent, awaiting response... 500 Internal Server Error 2014-07-28 10:13:09 ERROR 500: Internal Server Error. root@nagios:/home/fbulk# tcptraceroute www.ebay.com Selected device eth0.3, address 96.31.0.5, port 40467 for outgoing packets Tracing the path to www.ebay.com (66.135.210.181) on TCP port 80 (www), 30 hops max 1 router-core-inside.mtcnet.net (96.31.0.254) 0.265 ms 0.198 ms 0.198 ms 2 sxct.sxcy.mtcnet.net (167.142.156.197) 0.192 ms 0.138 ms 0.124 ms 3 premier.sxcy-mlx.fbnt.netins.net (173.215.60.5) 1.649 ms 1.607 ms 1.555 ms 4 38.104.184.26 7.006 ms 6.794 ms 6.604 ms 5 te0-0-1-1.rcr11.dsm01.atlas.cogentco.com (38.104.184.25) 5.877 ms 5.849 ms 5.790 ms 6 te0-2-0-0.ccr41.ord01.atlas.cogentco.com (154.54.46.237) 13.036 ms 13.190 ms 13.134 ms 7 be2098.ccr41.atl01.atlas.cogentco.com (154.54.29.98) 30.992 ms 30.916 ms 30.948 ms 8 be2050.ccr21.atl02.atlas.cogentco.com (154.54.0.166) 31.088 ms 31.071 ms 31.552 ms 9 qwest.atl02.atlas.cogentco.com (154.54.13.30) 30.773 ms 30.765 ms 30.690 ms 10 dvr-edge-13.inet.qwest.net (67.14.24.93) 46.902 ms 47.658 ms 47.390 ms 11 67-148-54-10.dia.static.qwest.net (67.148.54.10) 47.381 ms 48.089 ms 47.661 ms 12 * * * 13 * * * 14 * * * 15 * * * 16 www.ebay.com (66.135.210.181) [open] 76.083 ms 72.986 ms 72.259 ms root@nagios:/home/fbulk# Frank

9 9

Telstra announces closure of 2G network
by Network IPdog 24 Jul '14

24 Jul '14

http://www.itnews.com.au/News/390220,telstra-announces-closure-of-2g-network .aspx This is going to be hard for the companies that use GSM diallers for alarms and things like that Ruff, Ruff...! Network IPdog Ephesians 4:32 & Cheers!!! A password is like a... toothbrush ;^) Choose a good one, change it regularly and don't share it. Quote of the Day: "I have not failed. I've just found 10,000 ways that won't work." ~ Thomas Edison

1 0

multiple XO circuits down in SF area
by Ray Wong 23 Jul '14

23 Jul '14

anyone else with 10Gbps DWDM and Transit links down around SF? Wondering how widespread it is, while I wait for progress on the ticket...

3 2

VoIP Innovations
by Plexicomm Admin 22 Jul '14

22 Jul '14

SIP not completing calls, anyone know why?

3 2

Megapath DNS down
by Robert Glover 20 Jul '14

20 Jul '14

Looks like Megapath DNS server are non-responsive: 64.105.172.26 64.105.172.27 This same issue occured back in January.. We have a lot of end-users down.. Any from Megapath around today? We have contacted Megapath through the normal channels, no response (ya ya I know it's Sunday). Sincerely, Bobby Glover Director of Information Services SVI Incorporated

1 0