XO in DC/Herndon: Can only ping even numbered IPs?
Greetings list, We have a few customers in the Fairfax County/Washington DC area with XO-provided T1 circuits that can only access hosts with an EVEN numbered last octet in their IP address. For example I personally host a server in Equinix Ashburn (with Carpathia) on a /28 and the XO T1's can only ping the EVEN numbered IPs, for example: 209.222.152.120 - can ping/http/etc 209.222.152.121 - no access 209.222.152.122 - can ping/http/etc 209.222.152.123 - no access The is doubly-confirmed via traceroute as only even numbered routers' responses make it back. Getting very slow/doubtful responses from XO support so I'm just reaching out to this list to see if anyone else is affected. I assume this is some sort of load balancing/spreading algorithm. Certainly never saw this particular outage before during my 15 years in the industry. Thanks all. - Cary Wiedemann --Misc Networking Engineering guy -- Sent from my mobile device
Hi Cary, On Mon, 13 Jun 2011, Cary Wiedemann wrote:
Greetings list,
We have a few customers in the Fairfax County/Washington DC area with XO-provided T1 circuits that can only access hosts with an EVEN numbered last octet in their IP address.
For example I personally host a server in Equinix Ashburn (with Carpathia) on a /28 and the XO T1's can only ping the EVEN numbered IPs, for example: 209.222.152.120 - can ping/http/etc 209.222.152.121 - no access 209.222.152.122 - can ping/http/etc 209.222.152.123 - no access
The is doubly-confirmed via traceroute as only even numbered routers' responses make it back.
XO recently implemented automatic abuse-related filtering at their edge routers. You may want to check to make sure these individual IPs aren't triggering one of the automatic filtering rules that null routes the /32s. Hope this helps, -- William R. Lorenz
William sir, I wish it were that simple. Even google.com and cnn.com's odd-numbered round-robin DNS IPs are unreachable, along with the odd-numbered IPs of any intermediary routers after our second hop. Jonathan Lassoff probably hit the nail on the head with a failed link-aggregation scheme; of course XO hasn't provided any explanation yet. Nobody else in the Fairfax, VA region with an XO-provided T1's is experiencing this one? Thanks everyone for your responses. - Cary Wiedemann --Frantically setting up proxies on even-numbered IPs On 6/13/11, William R. Lorenz <wrl@express.org> wrote:
Hi Cary,
On Mon, 13 Jun 2011, Cary Wiedemann wrote:
Greetings list,
We have a few customers in the Fairfax County/Washington DC area with XO-provided T1 circuits that can only access hosts with an EVEN numbered last octet in their IP address.
For example I personally host a server in Equinix Ashburn (with Carpathia) on a /28 and the XO T1's can only ping the EVEN numbered IPs, for example: 209.222.152.120 - can ping/http/etc 209.222.152.121 - no access 209.222.152.122 - can ping/http/etc 209.222.152.123 - no access
The is doubly-confirmed via traceroute as only even numbered routers' responses make it back.
XO recently implemented automatic abuse-related filtering at their edge routers. You may want to check to make sure these individual IPs aren't triggering one of the automatic filtering rules that null routes the /32s.
Hope this helps,
-- William R. Lorenz
-- Sent from my mobile device
On Mon, Jun 13, 2011 at 11:08 AM, Cary Wiedemann <carywiedemann@gmail.com> wrote:
Greetings list,
We have a few customers in the Fairfax County/Washington DC area with XO-provided T1 circuits that can only access hosts with an EVEN numbered last octet in their IP address.
For example I personally host a server in Equinix Ashburn (with Carpathia) on a /28 and the XO T1's can only ping the EVEN numbered IPs, for example: 209.222.152.120 - can ping/http/etc 209.222.152.121 - no access 209.222.152.122 - can ping/http/etc 209.222.152.123 - no access
The is doubly-confirmed via traceroute as only even numbered routers' responses make it back.
This speaks very strongly of a link-aggregation scheme with an undetected link failure. Many implementations can hash based on the destination IP address to load balance traffic across multiple links. Some NSPs don't properly implement a link aggregation negotiation protocol like LACP or PAGP and just leave things in an enabled state. This leads to failures like this. Level3 seems to have an outage like this every 6 - 9 months, I've found.
participants (3)
-
Cary Wiedemann -
Jonathan Lassoff -
William R. Lorenz