This morning we're experiencing timeouts to and from a lot of sites. I heard through the grapevine that XO instituted some new route limits on their peers. Their site at http://businesscenter.xo.com is down, their phone number 800-424-0583 is fast busy and their Premier National Accounts 866-295-9696 reports "we are currently experiencing technical difficulties and cannot take your call" then DISCONNECT. UPDATE One of my other vendors says that they have resolved it and it seems to have improved. According to them, It was indeed a cap on the number of routes accepted by XO from their peers. Cheers, Chris
----- Original Message ----- From: "Christopher Hunt" <dharmachris@gmail.com>
This morning we're experiencing timeouts to and from a lot of sites. I heard through the grapevine that XO instituted some new route limits on their peers. Their site at http://businesscenter.xo.com is down, their phone number 800-424-0583 is fast busy and their Premier National Accounts 866-295-9696 reports "we are currently experiencing technical difficulties and cannot take your call" then DISCONNECT.
UPDATE One of my other vendors says that they have resolved it and it seems to have improved. According to them, It was indeed a cap on the number of routes accepted by XO from their peers.
Other reports here in the last hour suggest it was the opposite: XO suddenly started *sending* a vastly increased number of presumably bogus routes to their peers, many of whom tripped filter rules, and either shut the sessions down, or limited them and picked the wrong routes to drop. I expect it will be tomorrow before we assemble a coherent picture of what the actual outage was. Anyone check Renesys' blog yet? Cheers, -- jra
Is there any chance that you guys can please try and look at the existing threads on the list before posting a question that has already been posted? If you have information to add that will help people you can reply to one of the existing threads, but having 15 different threads on the same topic is really annoying. Thanks, -Scott -----Original Message----- From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On Behalf Of Christopher Hunt Sent: Thursday, September 16, 2010 11:41 AM To: outages@outages.org Subject: [outages] XO snafu This morning we're experiencing timeouts to and from a lot of sites. I heard through the grapevine that XO instituted some new route limits on their peers. Their site at http://businesscenter.xo.com is down, their phone number 800-424-0583 is fast busy and their Premier National Accounts 866-295-9696 reports "we are currently experiencing technical difficulties and cannot take your call" then DISCONNECT. UPDATE One of my other vendors says that they have resolved it and it seems to have improved. According to them, It was indeed a cap on the number of routes accepted by XO from their peers. Cheers, Chris _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
I agree and apologize. Apparently the list serve is lagging these messages and I didn't receive most threads until just recently. -Chris On 9/16/2010 10:48 AM, Scott Berkman wrote:
Is there any chance that you guys can please try and look at the existing threads on the list before posting a question that has already been posted? If you have information to add that will help people you can reply to one of the existing threads, but having 15 different threads on the same topic is really annoying.
Thanks,
-Scott
-----Original Message----- From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On Behalf Of Christopher Hunt Sent: Thursday, September 16, 2010 11:41 AM To: outages@outages.org Subject: [outages] XO snafu
This morning we're experiencing timeouts to and from a lot of sites. I heard through the grapevine that XO instituted some new route limits on their peers. Their site at http://businesscenter.xo.com is down, their phone number 800-424-0583 is fast busy and their Premier National Accounts 866-295-9696 reports "we are currently experiencing technical difficulties and cannot take your call" then DISCONNECT.
UPDATE One of my other vendors says that they have resolved it and it seems to have improved. According to them, It was indeed a cap on the number of routes accepted by XO from their peers.
Cheers, Chris _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Interestingly-- some messages are coming in with 1 to 2 min lag.. (I checked some headers) and I see some that are 30 min delayed.... Jim -----Original Message----- From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On Behalf Of Christopher Hunt Sent: Thursday, September 16, 2010 1:52 PM To: Scott Berkman Cc: outages@outages.org; 'Christopher Hunt' Subject: Re: [outages] XO snafu I agree and apologize. Apparently the list serve is lagging these messages and I didn't receive most threads until just recently. -Chris On 9/16/2010 10:48 AM, Scott Berkman wrote:
Is there any chance that you guys can please try and look at the existing threads on the list before posting a question that has already been posted? If you have information to add that will help people you can reply to one of the existing threads, but having 15 different threads on the same topic is really annoying.
Thanks,
-Scott
-----Original Message----- From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On Behalf Of Christopher Hunt Sent: Thursday, September 16, 2010 11:41 AM To: outages@outages.org Subject: [outages] XO snafu
This morning we're experiencing timeouts to and from a lot of sites. I heard through the grapevine that XO instituted some new route limits on their peers. Their site at http://businesscenter.xo.com is down, their phone number 800-424-0583 is fast busy and their Premier National Accounts 866-295-9696 reports "we are currently experiencing technical difficulties and cannot take your call" then DISCONNECT.
UPDATE One of my other vendors says that they have resolved it and it seems to have improved. According to them, It was indeed a cap on the number of routes accepted by XO from their peers.
Cheers, Chris _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Anyone else notice a recurrence of XO issues today at about 11:58 GMT-4? Traces during blip: 1 r1 (216.187.0.1) 0.720 ms 1.282 ms 1.542 ms 2 98.124.133.241 (98.124.133.241) 2.327 ms 15.476 ms 2.370 ms 3 te-1-1.bmf1.nyc1.gt-t.net (208.81.84.5) 2.830 ms 2.489 ms 2.135 ms 4 64.209.110.37 (64.209.110.37) 2.869 ms 2.464 ms 3.097 ms 5 * * * 6 * * * 7 * * * 1 r1 (216.187.0.1) 1.179 ms 0.708 ms 0.715 ms 2 98.124.133.241 (98.124.133.241) 2.131 ms 13.152 ms 2.367 ms 3 te-1-1.bmf1.nyc1.gt-t.net (208.81.84.5) 2.820 ms 2.482 ms 2.100 ms 4 64.209.110.37 (64.209.110.37) 3.013 ms 2.462 ms 3.001 ms 5 * * * 6 * * * 7 * * * 1 r1 (216.187.0.1) 1.714 ms 1.240 ms 1.073 ms 2 98.124.133.241 (98.124.133.241) 1.951 ms 13.566 ms 2.469 ms 3 te-1-1.bmf1.nyc1.gt-t.net (208.81.84.5) 2.802 ms 2.983 ms 2.066 ms 4 64.209.110.37 (64.209.110.37) 3.366 ms 2.215 ms 2.738 ms 5 * * * 6 * * * 7 * * * Traces after blip: 1 r1 (216.187.0.1) 0.890 ms 0.949 ms 1.079 ms 2 98.124.133.241 (98.124.133.241) 1.450 ms 2.248 ms 2.811 ms 3 te-1-1.bmf1.nyc1.gt-t.net (208.81.84.5) 3.487 ms 2.725 ms 2.698 ms 4 64.209.110.37 (64.209.110.37) 2.240 ms 2.322 ms 2.507 ms 5 te2-1-0d0.cir1.nyc-ny.us.xo.net (206.111.13.53) 2.231 ms 3.890 ms 67.921 ms 6 207.88.15.22.ptr.us.xo.net (207.88.15.22) 2.400 ms 2.232 ms 2.903 ms 7 te5-0-0.cvr3.nyc-ny.us.xo.net (207.88.86.194) 3.656 ms 9.213 ms 3.198 ms 8 207.88.182.10.ptr.us.xo.net (207.88.182.10) 3.564 ms 2.559 ms 2.739 ms 1 r1 (216.187.0.1) 1.009 ms 1.061 ms 0.967 ms 2 98.124.133.241 (98.124.133.241) 2.260 ms 1.843 ms 1.736 ms 3 te-1-1.bmf1.nyc1.gt-t.net (208.81.84.5) 2.528 ms 3.096 ms 2.825 ms 4 64.209.110.37 (64.209.110.37) 2.544 ms 2.062 ms 2.708 ms 5 te2-1-0d0.cir1.nyc-ny.us.xo.net (206.111.13.53) 2.836 ms 69.511 ms 2.188 ms 6 207.88.15.22.ptr.us.xo.net (207.88.15.22) 2.375 ms 2.955 ms 3.108 ms 7 te5-0-0.cvr3.nyc-ny.us.xo.net (207.88.86.194) 2.458 ms 8.728 ms 3.686 ms 8 207.88.182.6.ptr.us.xo.net (207.88.182.6) 4.207 ms 4.449 ms 3.012 ms 1 r1 (216.187.0.1) 0.545 ms 0.966 ms 1.565 ms 2 98.124.133.241 (98.124.133.241) 2.129 ms 2.374 ms 2.036 ms 3 te-1-1.bmf1.nyc1.gt-t.net (208.81.84.5) 3.513 ms 2.732 ms 2.872 ms 4 64.209.110.37 (64.209.110.37) 2.323 ms 2.447 ms 2.425 ms 5 te2-1-0d0.cir1.nyc-ny.us.xo.net (206.111.13.53) 3.481 ms 68.279 ms 2.201 ms 6 207.88.15.22.ptr.us.xo.net (207.88.15.22) 2.476 ms 2.940 ms 3.108 ms 7 te5-0-0.cvr3.nyc-ny.us.xo.net (207.88.86.194) 2.304 ms 8.964 ms 3.195 ms 8 207.88.182.10.ptr.us.xo.net (207.88.182.10) 3.523 ms 2.656 ms 2.714 ms On 9/16/2010 2:19 PM, Jim McBurnett wrote:
Interestingly-- some messages are coming in with 1 to 2 min lag.. (I checked some headers) and I see some that are 30 min delayed....
Jim
-----Original Message----- From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On Behalf Of Christopher Hunt Sent: Thursday, September 16, 2010 1:52 PM To: Scott Berkman Cc: outages@outages.org; 'Christopher Hunt' Subject: Re: [outages] XO snafu
I agree and apologize. Apparently the list serve is lagging these messages and I didn't receive most threads until just recently.
-Chris On 9/16/2010 10:48 AM, Scott Berkman wrote:
Is there any chance that you guys can please try and look at the existing threads on the list before posting a question that has already been posted? If you have information to add that will help people you can reply to one of the existing threads, but having 15 different threads on the same topic is really annoying.
Thanks,
-Scott
-----Original Message----- From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On Behalf Of Christopher Hunt Sent: Thursday, September 16, 2010 11:41 AM To: outages@outages.org Subject: [outages] XO snafu
This morning we're experiencing timeouts to and from a lot of sites. I heard through the grapevine that XO instituted some new route limits on their peers. Their site at http://businesscenter.xo.com is down, their phone number 800-424-0583 is fast busy and their Premier National Accounts 866-295-9696 reports "we are currently experiencing technical difficulties and cannot take your call" then DISCONNECT.
UPDATE One of my other vendors says that they have resolved it and it seems to have improved. According to them, It was indeed a cap on the number of routes accepted by XO from their peers.
Cheers, Chris _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
It appears the outages mailing list is running upto 30 min. behind from the time people post. People were most likely thinking they were the first ones to notice and start a thread. I don't know if this was caused by the volume of messages regarding the XO issue or if outages.org possibly was affected by xo. I have seen in the past the outages list run behind. Dylan Ebner, Network Engineer Consulting Radiologists, Ltd. 1221 Nicollet Mall, Minneapolis, MN 55403 ph. 612.573.2236 fax. 612.573.2250 dylan.ebner@crlmed.com www.consultingradiologists.com -----Original Message----- From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On Behalf Of Scott Berkman Sent: Thursday, September 16, 2010 12:49 PM To: 'Christopher Hunt'; outages@outages.org Subject: Re: [outages] XO snafu Is there any chance that you guys can please try and look at the existing threads on the list before posting a question that has already been posted? If you have information to add that will help people you can reply to one of the existing threads, but having 15 different threads on the same topic is really annoying. Thanks, -Scott -----Original Message----- From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On Behalf Of Christopher Hunt Sent: Thursday, September 16, 2010 11:41 AM To: outages@outages.org Subject: [outages] XO snafu This morning we're experiencing timeouts to and from a lot of sites. I heard through the grapevine that XO instituted some new route limits on their peers. Their site at http://businesscenter.xo.com is down, their phone number 800-424-0583 is fast busy and their Premier National Accounts 866-295-9696 reports "we are currently experiencing technical difficulties and cannot take your call" then DISCONNECT. UPDATE One of my other vendors says that they have resolved it and it seems to have improved. According to them, It was indeed a cap on the number of routes accepted by XO from their peers. Cheers, Chris _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
I think perhaps it is people's mail servers that are behind, I've been getting these in seemingly real time all along. Anyway, please excuse my comments if anyone that started a new thread really did not know there were already multiple. -Scott -----Original Message----- From: Dylan Ebner [mailto:dylan.ebner@crlmed.com] Sent: Thursday, September 16, 2010 2:05 PM To: Scott Berkman; 'Christopher Hunt'; outages@outages.org Subject: RE: [outages] XO snafu It appears the outages mailing list is running upto 30 min. behind from the time people post. People were most likely thinking they were the first ones to notice and start a thread. I don't know if this was caused by the volume of messages regarding the XO issue or if outages.org possibly was affected by xo. I have seen in the past the outages list run behind. Dylan Ebner, Network Engineer Consulting Radiologists, Ltd. 1221 Nicollet Mall, Minneapolis, MN 55403 ph. 612.573.2236 fax. 612.573.2250 dylan.ebner@crlmed.com www.consultingradiologists.com -----Original Message----- From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On Behalf Of Scott Berkman Sent: Thursday, September 16, 2010 12:49 PM To: 'Christopher Hunt'; outages@outages.org Subject: Re: [outages] XO snafu Is there any chance that you guys can please try and look at the existing threads on the list before posting a question that has already been posted? If you have information to add that will help people you can reply to one of the existing threads, but having 15 different threads on the same topic is really annoying. Thanks, -Scott -----Original Message----- From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On Behalf Of Christopher Hunt Sent: Thursday, September 16, 2010 11:41 AM To: outages@outages.org Subject: [outages] XO snafu This morning we're experiencing timeouts to and from a lot of sites. I heard through the grapevine that XO instituted some new route limits on their peers. Their site at http://businesscenter.xo.com is down, their phone number 800-424-0583 is fast busy and their Premier National Accounts 866-295-9696 reports "we are currently experiencing technical difficulties and cannot take your call" then DISCONNECT. UPDATE One of my other vendors says that they have resolved it and it seems to have improved. According to them, It was indeed a cap on the number of routes accepted by XO from their peers. Cheers, Chris _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Some people don't post regularly and were being held by milter-greylist. I've now exempted @outages.org from greylist. Please direct hate mail at me. - Jared On Sep 16, 2010, at 2:09 PM, Scott Berkman wrote:
I think perhaps it is people's mail servers that are behind, I've been getting these in seemingly real time all along. Anyway, please excuse my comments if anyone that started a new thread really did not know there were already multiple.
-Scott
-----Original Message----- From: Dylan Ebner [mailto:dylan.ebner@crlmed.com] Sent: Thursday, September 16, 2010 2:05 PM To: Scott Berkman; 'Christopher Hunt'; outages@outages.org Subject: RE: [outages] XO snafu
It appears the outages mailing list is running upto 30 min. behind from the time people post. People were most likely thinking they were the first ones to notice and start a thread. I don't know if this was caused by the volume of messages regarding the XO issue or if outages.org possibly was affected by xo. I have seen in the past the outages list run behind.
Dylan Ebner, Network Engineer Consulting Radiologists, Ltd. 1221 Nicollet Mall, Minneapolis, MN 55403 ph. 612.573.2236 fax. 612.573.2250 dylan.ebner@crlmed.com www.consultingradiologists.com
-----Original Message----- From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On Behalf Of Scott Berkman Sent: Thursday, September 16, 2010 12:49 PM To: 'Christopher Hunt'; outages@outages.org Subject: Re: [outages] XO snafu
Is there any chance that you guys can please try and look at the existing threads on the list before posting a question that has already been posted? If you have information to add that will help people you can reply to one of the existing threads, but having 15 different threads on the same topic is really annoying.
Thanks,
-Scott
-----Original Message----- From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On Behalf Of Christopher Hunt Sent: Thursday, September 16, 2010 11:41 AM To: outages@outages.org Subject: [outages] XO snafu
This morning we're experiencing timeouts to and from a lot of sites. I heard through the grapevine that XO instituted some new route limits on their peers. Their site at http://businesscenter.xo.com is down, their phone number 800-424-0583 is fast busy and their Premier National Accounts 866-295-9696 reports "we are currently experiencing technical difficulties and cannot take your call" then DISCONNECT.
UPDATE One of my other vendors says that they have resolved it and it seems to have improved. According to them, It was indeed a cap on the number of routes accepted by XO from their peers.
Cheers, Chris _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi, On 09/16/2010 11:04 AM, Dylan Ebner wrote:
It appears the outages mailing list is running upto 30 min. behind from the time people post. People were most likely thinking they were the first ones to notice and start a thread. I don't know if this was caused by the volume of messages regarding the XO issue or if outages.org possibly was affected by xo. I have seen in the past the outages list run behind.
Dylan Ebner, Network Engineer Consulting Radiologists, Ltd. 1221 Nicollet Mall, Minneapolis, MN 55403 ph. 612.573.2236 fax. 612.573.2250 dylan.ebner@crlmed.com www.consultingradiologists.com
You are correct in pointing this delay and Jared just addressed this issue similar to what others have reported in the past. Apparently the messages slowness was due to anti-spam greylist filtering and this should be fixed starting now. Then again please let us know if you notice it again. Sorry about that. regards, /virendra
-----Original Message----- From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On Behalf Of Scott Berkman Sent: Thursday, September 16, 2010 12:49 PM To: 'Christopher Hunt'; outages@outages.org Subject: Re: [outages] XO snafu
Is there any chance that you guys can please try and look at the existing threads on the list before posting a question that has already been posted? If you have information to add that will help people you can reply to one of the existing threads, but having 15 different threads on the same topic is really annoying.
Thanks,
-Scott
-----Original Message----- From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On Behalf Of Christopher Hunt Sent: Thursday, September 16, 2010 11:41 AM To: outages@outages.org Subject: [outages] XO snafu
This morning we're experiencing timeouts to and from a lot of sites. I heard through the grapevine that XO instituted some new route limits on their peers. Their site at http://businesscenter.xo.com is down, their phone number 800-424-0583 is fast busy and their Premier National Accounts 866-295-9696 reports "we are currently experiencing technical difficulties and cannot take your call" then DISCONNECT.
UPDATE One of my other vendors says that they have resolved it and it seems to have improved. According to them, It was indeed a cap on the number of routes accepted by XO from their peers.
Cheers, Chris _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkySX3IACgkQpbZvCIJx1beSlQCePuqs7Y5mm9hc/In0VPKh4uhy isYAnA+KL/2Pi3YYcswifRSzRKWqrP+Z =qtmw -----END PGP SIGNATURE-----
participants (8)
-
Adam Greene -
Christopher Hunt -
Dylan Ebner -
Jared Mauch -
Jay R. Ashworth -
Jim McBurnett -
Scott Berkman -
virendra rode