September 2010 - Outages

Facebook Engineering on today's outage
by Jay R. Ashworth 25 Sep '10

25 Sep '10

http://www.facebook.com/notes/facebook-engineering/more-details-on-todays-o… Apparently, our surmise about Akamai notwithstanding, the problem was actually internal to their app-specific caching facilities, which went into Sorcerer's Apprentice mode, and they had to kill them all and let ghod sort them out. More if I get it; hope that posting's public. Cheers, -- jra -- Jay R. Ashworth Baylink jra(a)baylink.com Designer The Things I Think RFC 2100 Ashworth & Associates http://baylink.pitas.com '87 e24 St Petersburg FL USA http://photo.imageinc.us +1 727 647 1274 Start a man a fire, and he'll be warm all night. Set a man on fire, and he'll be warm for the rest of his life.

3 2

peeringdb.com down?
by Jonathan Lassoff 24 Sep '10

24 Sep '10

Peeringdb.com seems to be down for me. I'm resolving www.peeringdb.com and peeringdb.com to 205.234.211.82, and connections to that IP on TCP/80 immediately close, and TCP/443 never sees a ServerHello message in response to a ClientHello. Oh well, I can evaluate this peer tomorrow instead... :) Cheers, jof

5 5

Re: [outages] Facebook
by jon 24 Sep '10

24 Sep '10

Yeah all is well. Thanks for everything. Thread closed. On Sep 23, 2010 11:22 PM, "L.T. Easterly" <corqpub(a)gmail.com> wrote: > Up in Central FL (Brighthouse) > > On Thu, Sep 23, 2010 at 3:21 PM, jon <relicwr(a)gmail.com> wrote: > >> Its been around 30 minutes. Is this the same result from less than 20 hours >> ago? >> >> _______________________________________________ >> Outages mailing list >> Outages(a)outages.org >> https://puck.nether.net/mailman/listinfo/outages >> >>

1 0

Facebook
by jon 24 Sep '10

24 Sep '10

Its been around 30 minutes. Is this the same result from less than 20 hours ago?

17 22

SUBSCRIBE
by Kevin.Smith＠dca.state.fl.us 23 Sep '10

23 Sep '10

4 3

Re: [outages] Qwest WA iQ outage
by Christopher Hunt 23 Sep '10

23 Sep '10

It looks like I spoke too soon. Qwest is still experiencing a "transport issue" in western WA. -- Christopher Hunt ReachONE Internet, Inc. (360)456-5640 www.reachone.com

2 1

Re: [outages] Facebook
by Scott Weeks 23 Sep '10

23 Sep '10

--- josh(a)imaginenetworksllc.com wrote: From: Josh Luthman <josh(a)imaginenetworksllc.com> Yes, it is down. Let's get back to work! ---------------------------------------------- Ewwww, that's dripping with manager-itis. Quick, wipe it off! ;-) scott ---------------------- ----------------------- ----------------------

1 0

Facebook
by William S. Duncanson 23 Sep '10

23 Sep '10

Facebook appears to be having some issues. Unreachable from multiple providers in this neck of the woods, unreachable from downforeveryoneorjustme.com, but reachable via AT&T mobile. I'm sure customers are going to start bitching soon if they can't get to their Farmville. -- William S. Duncanson caesar(a)starkreality.com

1 0

Re: [outages] Qwest WA iQ outage
by Christopher Hunt 23 Sep '10

23 Sep '10

Service seems to have been restored. Qwest Master ticket #NB141070 . -- Christopher Hunt ReachONE Internet, Inc. (360)456-5640 www.reachone.com

1 0

Qwest WA iQ outage
by Christopher Hunt 23 Sep '10

23 Sep '10

I've just learned of an area outage of iQ services in the western Washington state area. One of our T1s is down and their IVR acknowledges the outage. Awaiting further info... -- Christopher Hunt ReachONE Internet, Inc. (360)456-5640 www.reachone.com

1 0

Cogeco mail issues?
by Ross Halliday 22 Sep '10

22 Sep '10

Howdy, Since about 6:50 AM EDT (GMT -04:00) all of our outbound mail hosts have been getting a mix between timeouts and the following back from mx.cogeco.ca (216.221.81.26): 421 #4.4.5 Too many connections to this listener. I tried from a few IPs in subnets that have no previous reputation (that I know of, anyway) and am getting the same thing. Is anyone else experiencing this? Thanks --- Ross Halliday Network Operations WTC Communications Office: 613-547-6939 x203 Helpdesk: 866-547-6939 option 2 http://www.wtccommunications.ca

3 2

XO snafu
by Christopher Hunt 21 Sep '10

21 Sep '10

This morning we're experiencing timeouts to and from a lot of sites. I heard through the grapevine that XO instituted some new route limits on their peers. Their site at http://businesscenter.xo.com is down, their phone number 800-424-0583 is fast busy and their Premier National Accounts 866-295-9696 reports "we are currently experiencing technical difficulties and cannot take your call" then DISCONNECT. UPDATE One of my other vendors says that they have resolved it and it seems to have improved. According to them, It was indeed a cap on the number of routes accepted by XO from their peers. Cheers, Chris

8 9

AT&T iPhone and iPad customers hit by data outage (Houston and parks of Texas)
by Frank Bulk 18 Sep '10

18 Sep '10

http://www.victoriaadvocate.com/weblogs/digital-babble/2010/sep/17/att-iphon e-and-ipad-customers-hit-by-data-outage/ Thought this might be interesting. Frank

1 0

test to outages
by Joseph Jackson 17 Sep '10

17 Sep '10

Not seeing my emails hit the list. Please reply. Sent 11:44am CST

3 2

Major XO issues?
by Bill Wichers 16 Sep '10

16 Sep '10

Has anyone been able to find out what is going on with XO? We're showing spotty connectivity all over their network, and I got disconnected from their PNA support number with a "technical difficulties" message. I'm hoping maybe someone on here knows something? -Bill

15 16

[admin] Delayed messages (fixed).
by virendra rode 16 Sep '10

16 Sep '10

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hello All, First of all, I thank everyone for being patient during this morning drill when messages were being delayed as this may have caused further duplication of messages. The delayed messages issue has been addressed (credit, Jared), it was caused due to anti-spam greylist filtering for outages.org which was temporarily holding email(s) from senders it did not recognize. As puck went through the process of approving mail and releasing them, this caused delays. This was causing disruption to the near-instantaneous nature of email people have come to expect. Like always, please feel free to contact us if you see any further issues. grateful thanks, /virendra -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkySbOsACgkQpbZvCIJx1bc85QCePHXekZhGaA7JiwaZx5RTEZpa kzsAn2ZQEZ0xjRJfy0PGJOBx0z7DwB0m =S0Zz -----END PGP SIGNATURE-----

1 0

XO Outages Today - Explanation.
by James La Vacca 16 Sep '10

16 Sep '10

XO Emergency Maintenance Notification - 902165336 Below are details regarding Emergency maintenance 902165336. If you have any questions please contact the SNA Repair Team at 800.422.5303 or snarepair(a)xo.com . Maintenance ID: 902165336 Location: XO Collocation site: Washington DC Service Impact: Routing of traffic across multiple XO markets Event Window: Network Alert Event Details: Attention XO Strategic Customer, XO is currently experiencing a large scale network issue impacting the routing of traffic across multiple markets. XO Engineers have identified the cause and are working to restore now as soon as possible. As a result, we are experiencing abnormally high call volumes into our Care center impacting your ability to contact us via your standard toll free number. We will continue to keep you updated with information via email. Thank you, XO Strategic and National Accounts Affected Circuits: Routing of traffic across multiple XO markets XO Network Outage

1 0

Re: [outages] XO Routing instability?
by Mark Mayfield 16 Sep '10

16 Sep '10

Called XO. Got to support queue, got message: "Thank you for calling XO. We are currently experiencing technical difficulties, and are unable to take your call." Then it hung up on me. Fantastic. Mark -----Original Message----- From: Mark Mayfield Sent: Thursday, September 16, 2010 9:59 AM To: 'outages(a)outages.org' Subject: XO Routing instability? I have gig service to XO in Minneapolis. I'm having problems with some websites, and no problems with others. www.internetpulse.net shows major latency from XO to several other providers. Anyone have any additional detail? Master ticket? Estimated repair time? Thanks, Mark Mayfield City of Roseville Network Systems Engineer 2660 Civic Center Drive Roseville, MN 55113 651-792-7098 Office Confidentiality Statement: The documents accompanying this transmission contain confidential information that is legally privileged. This information is intended only for the use of the individuals or entities listed above. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution, or action taken in reliance on the contents of these documents is strictly prohibited. If you have received this information in error, please notify the sender immediately and arrange for the return or destruction of these documents.

4 3

XO Routing instability?
by Mark Mayfield 16 Sep '10

16 Sep '10

I have gig service to XO in Minneapolis. I'm having problems with some websites, and no problems with others. www.internetpulse.net shows major latency from XO to several other providers. Anyone have any additional detail? Master ticket? Estimated repair time? Thanks, Mark Mayfield City of Roseville Network Systems Engineer 2660 Civic Center Drive Roseville, MN 55113 651-792-7098 Office Confidentiality Statement: The documents accompanying this transmission contain confidential information that is legally privileged. This information is intended only for the use of the individuals or entities listed above. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution, or action taken in reliance on the contents of these documents is strictly prohibited. If you have received this information in error, please notify the sender immediately and arrange for the return or destruction of these documents.

3 2

XO....
by John Von Essen 16 Sep '10

16 Sep '10

www.xo.com appears to be down. Several of my XO circuits in Philly are down or dropping alot of packets. -John

3 2

Fiber cut somewhere in SF Bay Area?
by Mike Lyon 16 Sep '10

16 Sep '10

Howdy folks, Anyone aware of a large fiber cut in Norcal affecting ATT, XO and I am sure many others? Thanks, Mike

2 1

XO Issues?
by Tim Huffman 16 Sep '10

16 Sep '10

Anyone having slowness through XO? Just looked at internetpulse.net, and they're pretty much red across the board. We are having serious packet loss through their network. Tim Huffman Director of Engineering BOB - Business Only Broadband, LLC O (630) 590-6012 C (630) 340-1925 tim(a)bobbroadband.com<mailto:tim@bobbroadband.com> www.bobbroadband.com<http://www.bobbroadband.com/> [cid:image001.jpg@01CB5587.8D9047A0]

5 5

NYC issues?
by Wallace Keith 16 Sep '10

16 Sep '10

Did anyone just experience a loss of connectivity thru NYC? 11:50-11:55 EDT Seemed to be dying at Level 3. Could not reach multiple hosts in Texas and NH. -Keith

1 0

XO on the east coast
by Sean Clark 16 Sep '10

16 Sep '10

I just had to shut all my BGP peers down with XO in atl, bos, den, hou, dal, lax, mia... Anyone having the same issues?

1 0

L3 -> XO issues?
by Darrell Hyde 16 Sep '10

16 Sep '10

Seeing all kinds of instability with XO on the west coast (SFO and Irvine, CA specifically). Interface /30s are intermittently unreachable and appear to be intermittently in and out of BGP based on L3's LG. Internetpulse.net looks pretty bad for XO as well. Anyone have any details?

1 0