Has anyone been able to find out what is going on with XO? We're showing spotty connectivity all over their network, and I got disconnected from their PNA support number with a "technical difficulties" message. I'm hoping maybe someone on here knows something? -Bill
Website is down/not pingable. The phone lines for XO all say its a DNS resolving issue. A few of my DS-1 circuits are down, and a few have serious packet loss. But some are up?! -John On Sep 16, 2010, at 11:26 AM, Bill Wichers wrote:
Has anyone been able to find out what is going on with XO? We’re showing spotty connectivity all over their network, and I got disconnected from their PNA support number with a “technical difficulties” message. I’m hoping maybe someone on here knows something?
-Bill _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Wish I did.. I am also hearing it's a widespread issue affecting ATT as well. -Mike On Thu, Sep 16, 2010 at 8:26 AM, Bill Wichers <billw@waveform.net> wrote:
Has anyone been able to find out what is going on with XO? We’re showing spotty connectivity all over their network, and I got disconnected from their PNA support number with a “technical difficulties” message. I’m hoping maybe someone on here knows something?
-Bill
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
XO supposedly leaked a bunch of routes (2nd hand info), had at least one internal routing loop between NYC & DC (1st hand info). They had massive reachability problems across the USA (1st hand info). They supposedly fixed the problem(s) starting at 12:14 EDT today, and reachability should return to normal (info from XO). But we are still seeing at least some locations having issues at 12:38 EDT. -- TTFN, patrick On Sep 16, 2010, at 11:58 AM, Mike Lyon wrote:
Wish I did.. I am also hearing it's a widespread issue affecting ATT as well.
-Mike
On Thu, Sep 16, 2010 at 8:26 AM, Bill Wichers <billw@waveform.net> wrote: Has anyone been able to find out what is going on with XO? We’re showing spotty connectivity all over their network, and I got disconnected from their PNA support number with a “technical difficulties” message. I’m hoping maybe someone on here knows something?
-Bill
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Both my XO and ATT peering are fine, I did have odd routing issues about an hour ago though. I am in the SF Bay area. Zaid On 9/16/10 8:58 AM, "Mike Lyon" <mike.lyon@gmail.com> wrote:
Wish I did.. I am also hearing it's a widespread issue affecting ATT as well.
-Mike
On Thu, Sep 16, 2010 at 8:26 AM, Bill Wichers <billw@waveform.net> wrote:
Has anyone been able to find out what is going on with XO? We¹re showing spotty connectivity all over their network, and I got disconnected from their PNA support number with a ³technical difficulties² message. I¹m hoping maybe someone on here knows something?
-Bill
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
I am currently seeing a lot of issues with (currently) unknown carriers and our VoIP calls. We use Bandwidth.com for outbound calls, and I know that XO is one of their upstream providers. We are seeing dropped calls, garbled audio. I have a ticket opened with them, but I don't think they have identified the issue yet. From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On Behalf Of Bill Wichers Sent: Thursday, September 16, 2010 11:27 AM To: outages@outages.org Subject: [outages] Major XO issues? Has anyone been able to find out what is going on with XO? We're showing spotty connectivity all over their network, and I got disconnected from their PNA support number with a "technical difficulties" message. I'm hoping maybe someone on here knows something? -Bill No virus found in this incoming message. Checked by AVG - www.avg.com Version: 8.5.445 / Virus Database: 271.1.1/3135 - Release Date: 09/16/10 06:34:00
Our datacenter in Salt Lake City forwarded us this message from XO: "I wanted to give notice in case you were having issues connecting to various websites today. There is a problem with a peering router which allows XO's network to connect to outside networks. Impact appears to be pretty broad due to the volume of routes that egress/ingress that peer. Our Engineering is working to correct. We don't have an estimated time for resolution just yet, but this our top priority issue right now and all levels of our repair organization are engaged." Philip Holmes From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On Behalf Of Geoff Mina Sent: Thursday, September 16, 2010 9:59 AM To: 'Bill Wichers'; outages@outages.org Subject: Re: [outages] Major XO issues? I am currently seeing a lot of issues with (currently) unknown carriers and our VoIP calls. We use Bandwidth.com for outbound calls, and I know that XO is one of their upstream providers. We are seeing dropped calls, garbled audio. I have a ticket opened with them, but I don't think they have identified the issue yet. From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On Behalf Of Bill Wichers Sent: Thursday, September 16, 2010 11:27 AM To: outages@outages.org Subject: [outages] Major XO issues? Has anyone been able to find out what is going on with XO? We're showing spotty connectivity all over their network, and I got disconnected from their PNA support number with a "technical difficulties" message. I'm hoping maybe someone on here knows something? -Bill No virus found in this incoming message. Checked by AVG - www.avg.com Version: 8.5.445 / Virus Database: 271.1.1/3135 - Release Date: 09/16/10 06:34:00
We are experiencing intermittent issues with their voip orig and voip term products. ************************** This email composed on my iPhone 4 whilst driving at 90 MPH with my knees. Please excuse typo's and brevity due to accidents, cops and other mishaps. Cheers, Michael Self PhoneFusion 954.343.5500 (home/office/cell/fax/SMS) On Sep 16, 2010, at 11:57, "Bill Wichers" <billw@waveform.net<mailto:billw@waveform.net>> wrote: Has anyone been able to find out what is going on with XO? We’re showing spotty connectivity all over their network, and I got disconnected from their PNA support number with a “technical difficulties” message. I’m hoping maybe someone on here knows something? -Bill _______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages
I'd also be curious if anyone has any information. Our data was completely down for almost an hour (returned at 1055 CDT) . We're in the Dallas area (Plano). Things are working now (though it seems as at least a few XO peers we do business with are still routing through other upstreams). No issues with Voice, other than what you mentioned, being disconnected from the PNA number. -Ed On Thu, Sep 16, 2010 at 10:26 AM, Bill Wichers <billw@waveform.net> wrote:
Has anyone been able to find out what is going on with XO? We’re showing spotty connectivity all over their network, and I got disconnected from their PNA support number with a “technical difficulties” message. I’m hoping maybe someone on here knows something?
-Bill
I've been told that there were "peering issues" and that there has not been an official "all clear" issued internally yet. The thing that most concerned be about this issue is that when we figure out XO was having issues, we dropped our peering session with them but they appear to have continued to announce our routes -- at least in some areas. We haven't brought that up with them yet but will soon. -Bill -----Original Message----- From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On Behalf Of Ed W Sent: Thursday, September 16, 2010 12:10 PM To: outages@outages.org Subject: Re: [outages] Major XO issues? I'd also be curious if anyone has any information. Our data was completely down for almost an hour (returned at 1055 CDT) . We're in the Dallas area (Plano). Things are working now (though it seems as at least a few XO peers we do business with are still routing through other upstreams). No issues with Voice, other than what you mentioned, being disconnected from the PNA number. -Ed On Thu, Sep 16, 2010 at 10:26 AM, Bill Wichers <billw@waveform.net> wrote:
Has anyone been able to find out what is going on with XO? We're showing spotty connectivity all over their network, and I got disconnected from their PNA support number with a "technical difficulties" message. I'm hoping maybe someone on here knows something?
-Bill
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Seeing this here as well, of course XO in their usual fashion has no outage notices posted on their site. We saw the issues starting at 10:56 AM EST, and although they are better now we still have at least 1 customer in NY complaining. -Scott From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On Behalf Of Bill Wichers Sent: Thursday, September 16, 2010 11:27 AM To: outages@outages.org Subject: [outages] Major XO issues? Has anyone been able to find out what is going on with XO? We're showing spotty connectivity all over their network, and I got disconnected from their PNA support number with a "technical difficulties" message. I'm hoping maybe someone on here knows something? -Bill
Dead to the world via XO in Herndon, VA. Coincidentally (or not?) a few miles from Ashburn. Jason On Thu, Sep 16, 2010 at 12:11 PM, Scott Berkman <scott@sberkman.net> wrote:
Seeing this here as well, of course XO in their usual fashion has no outage notices posted on their site. We saw the issues starting at 10:56 AM EST, and although they are better now we still have at least 1 customer in NY complaining.
-Scott
*From:* outages-bounces@outages.org [mailto:outages-bounces@outages.org] *On Behalf Of *Bill Wichers *Sent:* Thursday, September 16, 2010 11:27 AM *To:* outages@outages.org *Subject:* [outages] Major XO issues?
Has anyone been able to find out what is going on with XO? We’re showing spotty connectivity all over their network, and I got disconnected from their PNA support number with a “technical difficulties” message. I’m hoping maybe someone on here knows something?
-Bill
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
I am on the West Coast and noticed a huge drop in our traffic this morning. Starting to pick backup. From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On Behalf Of Scott Berkman Sent: Thursday, September 16, 2010 9:11 AM To: 'Bill Wichers'; outages@outages.org Subject: Re: [outages] Major XO issues? Seeing this here as well, of course XO in their usual fashion has no outage notices posted on their site. We saw the issues starting at 10:56 AM EST, and although they are better now we still have at least 1 customer in NY complaining. -Scott From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On Behalf Of Bill Wichers Sent: Thursday, September 16, 2010 11:27 AM To: outages@outages.org Subject: [outages] Major XO issues? Has anyone been able to find out what is going on with XO? We're showing spotty connectivity all over their network, and I got disconnected from their PNA support number with a "technical difficulties" message. I'm hoping maybe someone on here knows something? -Bill
On 9/16/2010 12:38, Jacob wrote:
I am on the West Coast and noticed a huge drop in our traffic this morning. Starting to pick backup.
I saw an abnormal increase in traffic since I do a lot of secondary MX and DNS type services. Traffic is starting to go back to normal now. ~Seth
Repost from NANOG for everyone's benefit: "XO Engineers are telling us that they are aware of packet loss across their network and are looking into it. We're experiencing slow/degraded connectivity out of St. Louis and Nashville but Atlanta and Dallas are problem free." William Collier-Byrd will@collier-byrd.net On Thu, Sep 16, 2010 at 11:26 AM, Bill Wichers <billw@waveform.net> wrote:
Has anyone been able to find out what is going on with XO? We’re showing spotty connectivity all over their network, and I got disconnected from their PNA support number with a “technical difficulties” message. I’m hoping maybe someone on here knows something?
-Bill
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 09/16/2010 08:26 AM, Bill Wichers wrote:
Has anyone been able to find out what is going on with XO? We're showing spotty connectivity all over their network, and I got disconnected from their PNA support number with a "technical difficulties" message. I'm hoping maybe someone on here knows something?
-Bill
Apparently the person I spoke to within XO said they were having headless chicken syndrome and that they were in the midst of bringing up sessions and that was the extend of the information I received. Hopefully we'll see some sort of lesson learned from this. regards, /virendra -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkySSEUACgkQpbZvCIJx1bcV6wCbBBx9IvdsNfwX1tberS7VAdIm 1zMAoJgEPksIGOKkRFN2ENSgIt7AElnc =Hy8x -----END PGP SIGNATURE-----
participants (15)
-
Bill Wichers -
Ed W -
Geoff Mina -
Jacob -
Jason L. Sparks -
John Von Essen -
Michael R. Self -
Mike Lyon -
Patrick W. Gilmore -
Philip Holmes -
Scott Berkman -
Seth Mattinen -
virendra rode -
William Byrd -
Zaid Ali