We're showing intermittent (well -- now no) connectivity to XO from Level 3 in NYC (and from the looks of the L3 looking glass other areas of the US.) Anyone else seeing this? NYC to XO Web: traceroute to www.xo.com (205.158.160.76), 30 hops max, 60 byte packets 4 ge-9-23.car1.newyork1.level3.net (4.26.32.89) 4.240 ms 4.210 ms 4.177 ms 5 ae-2-70.edge2.newyork1.level3.net (4.69.155.80) 2.200 ms 2.155 ms 1.895 ms 6 206.111.13.65.ptr.us.xo.net (206.111.13.65) 1.963 ms 1.667 ms 1.624 ms 7 * * * 8 * * * NYC to XO (NYC): traceroute to 207.86.XXX.x (207.86.XXX.x), 30 hops max, 60 byte packets 4 ge-9-23.car1.newyork1.level3.net (4.26.32.89) 2.702 ms 2.678 ms 2.667 ms 5 ae-4-90.edge2.newyork1.level3.net (4.69.155.208) 1.607 ms 1.558 ms 1.538 ms 6 206.111.13.65.ptr.us.xo.net (206.111.13.65) 1.644 ms 1.575 ms 1.549 ms 7 * * * 8 * * * *Show Level 3 (Dallas, TX) Traceroute to 205.158.160.76* 1 ae-1-60.edge2.Dallas3.Level3.net (4.69.145.12) 0 msec ae-2-70.edge2.Dallas3.Level3.net (4.69.145.76) 0 msec ae-4-90.edge2.Dallas3.Level3.net (4.69.145.204) 4 msec 2 XO-Level3.edge2.Dallas3.Level3.net (4.68.127.150) 0 msec 0 msec 12 msec 3 * * * 4 * * * -J
We are seeing the same thing starting around 11:35 EDT this morning. ------------------ Aubrey Wells Director | Network Services VocalCloud 678.248.2637 support@vocalcloud.com www.vocalcloud.com On Wed, Sep 26, 2012 at 12:12 PM, Jayson Navitsky <jnavitsky@gmail.com> wrote:
We're showing intermittent (well -- now no) connectivity to XO from Level 3 in NYC (and from the looks of the L3 looking glass other areas of the US.) Anyone else seeing this?
NYC to XO Web: traceroute to www.xo.com (205.158.160.76), 30 hops max, 60 byte packets 4 ge-9-23.car1.newyork1.level3.net (4.26.32.89) 4.240 ms 4.210 ms 4.177 ms 5 ae-2-70.edge2.newyork1.level3.net (4.69.155.80) 2.200 ms 2.155 ms 1.895 ms 6 206.111.13.65.ptr.us.xo.net (206.111.13.65) 1.963 ms 1.667 ms 1.624 ms 7 * * * 8 * * *
NYC to XO (NYC): traceroute to 207.86.XXX.x (207.86.XXX.x), 30 hops max, 60 byte packets 4 ge-9-23.car1.newyork1.level3.net (4.26.32.89) 2.702 ms 2.678 ms 2.667 ms 5 ae-4-90.edge2.newyork1.level3.net (4.69.155.208) 1.607 ms 1.558 ms 1.538 ms 6 206.111.13.65.ptr.us.xo.net (206.111.13.65) 1.644 ms 1.575 ms 1.549 ms 7 * * * 8 * * *
Show Level 3 (Dallas, TX) Traceroute to 205.158.160.76
1 ae-1-60.edge2.Dallas3.Level3.net (4.69.145.12) 0 msec ae-2-70.edge2.Dallas3.Level3.net (4.69.145.76) 0 msec ae-4-90.edge2.Dallas3.Level3.net (4.69.145.204) 4 msec 2 XO-Level3.edge2.Dallas3.Level3.net (4.68.127.150) 0 msec 0 msec 12 msec 3 * * * 4 * * *
-J
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
We are seeing the same thing at level3 in DFW into XO. ge-7-23.car1.dallas1.level3.net 0.0% 1.0 6.9 1.0 53.9 16.5 ae-4-90.edge2.dallas3.level3.net 10.0% 1.2 16.3 1.2 49.1 17.9 xo-level3.edge2.dallas3.level3.net 0.0% 2.1 1.7 1.0 2.2 0.6 ??? 100.0 0.0 0.0 0.0 0.0 0.0 From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On Behalf Of Jayson Navitsky Sent: Wednesday, September 26, 2012 11:13 AM To: outages@outages.org Subject: [outages] Level 3 -> XO We're showing intermittent (well -- now no) connectivity to XO from Level 3 in NYC (and from the looks of the L3 looking glass other areas of the US.) Anyone else seeing this? NYC to XO Web: traceroute to www.xo.com<http://www.xo.com> (205.158.160.76), 30 hops max, 60 byte packets 4 ge-9-23.car1.newyork1.level3.net<http://ge-9-23.car1.newyork1.level3.net> (4.26.32.89) 4.240 ms 4.210 ms 4.177 ms 5 ae-2-70.edge2.newyork1.level3.net<http://ae-2-70.edge2.newyork1.level3.net> (4.69.155.80) 2.200 ms 2.155 ms 1.895 ms 6 206.111.13.65.ptr.us.xo.net<http://206.111.13.65.ptr.us.xo.net> (206.111.13.65) 1.963 ms 1.667 ms 1.624 ms 7 * * * 8 * * * NYC to XO (NYC): traceroute to 207.86.XXX.x (207.86.XXX.x), 30 hops max, 60 byte packets 4 ge-9-23.car1.newyork1.level3.net<http://ge-9-23.car1.newyork1.level3.net> (4.26.32.89) 2.702 ms 2.678 ms 2.667 ms 5 ae-4-90.edge2.newyork1.level3.net<http://ae-4-90.edge2.newyork1.level3.net> (4.69.155.208) 1.607 ms 1.558 ms 1.538 ms 6 206.111.13.65.ptr.us.xo.net<http://206.111.13.65.ptr.us.xo.net> (206.111.13.65) 1.644 ms 1.575 ms 1.549 ms 7 * * * 8 * * * Show Level 3 (Dallas, TX) Traceroute to 205.158.160.76 1 ae-1-60.edge2.Dallas3.Level3.net<http://ae-1-60.edge2.Dallas3.Level3.net> (4.69.145.12) 0 msec ae-2-70.edge2.Dallas3.Level3.net<http://ae-2-70.edge2.Dallas3.Level3.net> (4.69.145.76) 0 msec ae-4-90.edge2.Dallas3.Level3.net<http://ae-4-90.edge2.Dallas3.Level3.net> (4.69.145.204) 4 msec 2 XO-Level3.edge2.Dallas3.Level3.net<http://XO-Level3.edge2.Dallas3.Level3.net> (4.68.127.150) 0 msec 0 msec 12 msec 3 * * * 4 * * * -J
No issues seen here in Denver via IPv4 (Cogent) or IPv6 (L3). Also looks fine via CenturyLink in Denver on both IPv4 and IPv6. Chris On 09/26/2012 10:12 AM, Jayson Navitsky wrote:
We're showing intermittent (well -- now no) connectivity to XO from Level 3 in NYC (and from the looks of the L3 looking glass other areas of the US.) Anyone else seeing this?
NYC to XO Web: traceroute to www.xo.com <http://www.xo.com> (205.158.160.76), 30 hops max, 60 byte packets 4 ge-9-23.car1.newyork1.level3.net <http://ge-9-23.car1.newyork1.level3.net> (4.26.32.89) 4.240 ms 4.210 ms 4.177 ms 5 ae-2-70.edge2.newyork1.level3.net <http://ae-2-70.edge2.newyork1.level3.net> (4.69.155.80) 2.200 ms 2.155 ms 1.895 ms 6 206.111.13.65.ptr.us.xo.net <http://206.111.13.65.ptr.us.xo.net> (206.111.13.65) 1.963 ms 1.667 ms 1.624 ms 7 * * * 8 * * *
NYC to XO (NYC): traceroute to 207.86.XXX.x (207.86.XXX.x), 30 hops max, 60 byte packets 4 ge-9-23.car1.newyork1.level3.net <http://ge-9-23.car1.newyork1.level3.net> (4.26.32.89) 2.702 ms 2.678 ms 2.667 ms 5 ae-4-90.edge2.newyork1.level3.net <http://ae-4-90.edge2.newyork1.level3.net> (4.69.155.208) 1.607 ms 1.558 ms 1.538 ms 6 206.111.13.65.ptr.us.xo.net <http://206.111.13.65.ptr.us.xo.net> (206.111.13.65) 1.644 ms 1.575 ms 1.549 ms 7 * * * 8 * * *
*Show Level 3 (Dallas, TX) Traceroute to 205.158.160.76*
1ae-1-60.edge2.Dallas3.Level3.net <http://ae-1-60.edge2.Dallas3.Level3.net> (4.69.145.12) 0 msec ae-2-70.edge2.Dallas3.Level3.net <http://ae-2-70.edge2.Dallas3.Level3.net> (4.69.145.76) 0 msec ae-4-90.edge2.Dallas3.Level3.net <http://ae-4-90.edge2.Dallas3.Level3.net> (4.69.145.204) 4 msec 2XO-Level3.edge2.Dallas3.Level3.net <http://XO-Level3.edge2.Dallas3.Level3.net> (4.68.127.150) 0 msec 0 msec 12 msec 3 * * * 4 * * *
LA over level3 to XO looks good but DFW still drops. From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On Behalf Of Chris Stone Sent: Wednesday, September 26, 2012 11:20 AM To: outages@outages.org Subject: Re: [outages] Level 3 -> XO No issues seen here in Denver via IPv4 (Cogent) or IPv6 (L3). Also looks fine via CenturyLink in Denver on both IPv4 and IPv6. Chris On 09/26/2012 10:12 AM, Jayson Navitsky wrote: We're showing intermittent (well -- now no) connectivity to XO from Level 3 in NYC (and from the looks of the L3 looking glass other areas of the US.) Anyone else seeing this? NYC to XO Web: traceroute to www.xo.com<http://www.xo.com> (205.158.160.76), 30 hops max, 60 byte packets 4 ge-9-23.car1.newyork1.level3.net<http://ge-9-23.car1.newyork1.level3.net> (4.26.32.89) 4.240 ms 4.210 ms 4.177 ms 5 ae-2-70.edge2.newyork1.level3.net<http://ae-2-70.edge2.newyork1.level3.net> (4.69.155.80) 2.200 ms 2.155 ms 1.895 ms 6 206.111.13.65.ptr.us.xo.net<http://206.111.13.65.ptr.us.xo.net> (206.111.13.65) 1.963 ms 1.667 ms 1.624 ms 7 * * * 8 * * * NYC to XO (NYC): traceroute to 207.86.XXX.x (207.86.XXX.x), 30 hops max, 60 byte packets 4 ge-9-23.car1.newyork1.level3.net<http://ge-9-23.car1.newyork1.level3.net> (4.26.32.89) 2.702 ms 2.678 ms 2.667 ms 5 ae-4-90.edge2.newyork1.level3.net<http://ae-4-90.edge2.newyork1.level3.net> (4.69.155.208) 1.607 ms 1.558 ms 1.538 ms 6 206.111.13.65.ptr.us.xo.net<http://206.111.13.65.ptr.us.xo.net> (206.111.13.65) 1.644 ms 1.575 ms 1.549 ms 7 * * * 8 * * * Show Level 3 (Dallas, TX) Traceroute to 205.158.160.76 1 ae-1-60.edge2.Dallas3.Level3.net<http://ae-1-60.edge2.Dallas3.Level3.net> (4.69.145.12) 0 msec ae-2-70.edge2.Dallas3.Level3.net<http://ae-2-70.edge2.Dallas3.Level3.net> (4.69.145.76) 0 msec ae-4-90.edge2.Dallas3.Level3.net<http://ae-4-90.edge2.Dallas3.Level3.net> (4.69.145.204) 4 msec 2 XO-Level3.edge2.Dallas3.Level3.net<http://XO-Level3.edge2.Dallas3.Level3.net> (4.68.127.150) 0 msec 0 msec 12 msec 3 * * * 4 * * *
We're also seeing that in Utah. We lost the link briefly. It's back now, but were still seeing odd problems resolving some sites. From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On Behalf Of Jayson Navitsky Sent: Wednesday, September 26, 2012 10:13 AM To: outages@outages.org Subject: [outages] Level 3 -> XO We're showing intermittent (well -- now no) connectivity to XO from Level 3 in NYC (and from the looks of the L3 looking glass other areas of the US.) Anyone else seeing this? NYC to XO Web: traceroute to www.xo.com<http://www.xo.com> (205.158.160.76), 30 hops max, 60 byte packets 4 ge-9-23.car1.newyork1.level3.net<http://ge-9-23.car1.newyork1.level3.net> (4.26.32.89) 4.240 ms 4.210 ms 4.177 ms 5 ae-2-70.edge2.newyork1.level3.net<http://ae-2-70.edge2.newyork1.level3.net> (4.69.155.80) 2.200 ms 2.155 ms 1.895 ms 6 206.111.13.65.ptr.us.xo.net<http://206.111.13.65.ptr.us.xo.net> (206.111.13.65) 1.963 ms 1.667 ms 1.624 ms 7 * * * 8 * * * NYC to XO (NYC): traceroute to 207.86.XXX.x (207.86.XXX.x), 30 hops max, 60 byte packets 4 ge-9-23.car1.newyork1.level3.net<http://ge-9-23.car1.newyork1.level3.net> (4.26.32.89) 2.702 ms 2.678 ms 2.667 ms 5 ae-4-90.edge2.newyork1.level3.net<http://ae-4-90.edge2.newyork1.level3.net> (4.69.155.208) 1.607 ms 1.558 ms 1.538 ms 6 206.111.13.65.ptr.us.xo.net<http://206.111.13.65.ptr.us.xo.net> (206.111.13.65) 1.644 ms 1.575 ms 1.549 ms 7 * * * 8 * * * Show Level 3 (Dallas, TX) Traceroute to 205.158.160.76 1 ae-1-60.edge2.Dallas3.Level3.net<http://ae-1-60.edge2.Dallas3.Level3.net> (4.69.145.12) 0 msec ae-2-70.edge2.Dallas3.Level3.net<http://ae-2-70.edge2.Dallas3.Level3.net> (4.69.145.76) 0 msec ae-4-90.edge2.Dallas3.Level3.net<http://ae-4-90.edge2.Dallas3.Level3.net> (4.69.145.204) 4 msec 2 XO-Level3.edge2.Dallas3.Level3.net<http://XO-Level3.edge2.Dallas3.Level3.net> (4.68.127.150) 0 msec 0 msec 12 msec 3 * * * 4 * * * -J
Seeing same from LAX & ATL on Level3, fine from CentLink. Started at 11:33 ET. Was unable to reach several sites at that time, however most have resolved. Also saw issues with L3 on net in Denver. Level3 stated there is an issue in Miami... FWIW. traceroute to 64.245.xx.xx (64.245.xx.xx) 30 hops max, 40 byte packets 1 192.168.9.1 (192.168.9.1) 0.250 ms 0.218 ms 0.206 ms 2 ae7-xxx.edge6.losangeles1.level3.net (4.30.196.xxx) 0.660 ms 1.805 ms 1.799 ms 3 ae-3-80.edge1.losangeles9.level3.net (4.69.144.138) 0.873 ms ae-2-70.edge1.losangeles9.level3.net (4.69.144.74) 0.756 ms 0.789 ms 4 xo-level3-xe.losangeles9.level3.net (4.53.228.10) 0.856 ms 0.839 ms 0.840 ms 5 * * * traceroute to 64.245.xx.xx (64.245.xx.xx), 30 hops max, 40 byte packets 1 * 192.168.20.1 (192.168.20.1) 0.387 ms * 2 ge-7-xx.car2.atlanta2.level3.net (4.71.255.xx) 1.345 ms 1.177 ms 1.433 ms 3 vlan52.ebr2.atlanta2.level3.net (4.69.150.126) 13.732 ms 13.841 ms 13.726 ms 4 ae-2-2.ebr2.miami1.level3.net (4.69.140.141) 13.730 ms 17.403 ms 17.428 ms 5 ae-2-52.edge1.miami2.level3.net (4.69.138.107) 14.082 ms 18.521 ms 18.467 ms 6 * * * 7 * * * From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On Behalf Of Joel Woodbury Sent: Wednesday, September 26, 2012 12:22 PM To: Jayson Navitsky; outages@outages.org Subject: Re: [outages] Level 3 -> XO We're also seeing that in Utah. We lost the link briefly. It's back now, but were still seeing odd problems resolving some sites. From: outages-bounces@outages.org<mailto:outages-bounces@outages.org> [mailto:outages-bounces@outages.org]<mailto:[mailto:outages-bounces@outages.org]> On Behalf Of Jayson Navitsky Sent: Wednesday, September 26, 2012 10:13 AM To: outages@outages.org<mailto:outages@outages.org> Subject: [outages] Level 3 -> XO We're showing intermittent (well -- now no) connectivity to XO from Level 3 in NYC (and from the looks of the L3 looking glass other areas of the US.) Anyone else seeing this? NYC to XO Web: traceroute to www.xo.com<http://www.xo.com> (205.158.160.76), 30 hops max, 60 byte packets 4 ge-9-23.car1.newyork1.level3.net<http://ge-9-23.car1.newyork1.level3.net> (4.26.32.89) 4.240 ms 4.210 ms 4.177 ms 5 ae-2-70.edge2.newyork1.level3.net<http://ae-2-70.edge2.newyork1.level3.net> (4.69.155.80) 2.200 ms 2.155 ms 1.895 ms 6 206.111.13.65.ptr.us.xo.net<http://206.111.13.65.ptr.us.xo.net> (206.111.13.65) 1.963 ms 1.667 ms 1.624 ms 7 * * * 8 * * * NYC to XO (NYC): traceroute to 207.86.XXX.x (207.86.XXX.x), 30 hops max, 60 byte packets 4 ge-9-23.car1.newyork1.level3.net<http://ge-9-23.car1.newyork1.level3.net> (4.26.32.89) 2.702 ms 2.678 ms 2.667 ms 5 ae-4-90.edge2.newyork1.level3.net<http://ae-4-90.edge2.newyork1.level3.net> (4.69.155.208) 1.607 ms 1.558 ms 1.538 ms 6 206.111.13.65.ptr.us.xo.net<http://206.111.13.65.ptr.us.xo.net> (206.111.13.65) 1.644 ms 1.575 ms 1.549 ms 7 * * * 8 * * * Show Level 3 (Dallas, TX) Traceroute to 205.158.160.76 1 ae-1-60.edge2.Dallas3.Level3.net<http://ae-1-60.edge2.Dallas3.Level3.net> (4.69.145.12) 0 msec ae-2-70.edge2.Dallas3.Level3.net<http://ae-2-70.edge2.Dallas3.Level3.net> (4.69.145.76) 0 msec ae-4-90.edge2.Dallas3.Level3.net<http://ae-4-90.edge2.Dallas3.Level3.net> (4.69.145.204) 4 msec 2 XO-Level3.edge2.Dallas3.Level3.net<http://XO-Level3.edge2.Dallas3.Level3.net> (4.68.127.150) 0 msec 0 msec 12 msec 3 * * * 4 * * * -J
On Sep 26, 2012, at 11:12 AM, Jayson Navitsky <jnavitsky@gmail.com> wrote:
We're showing intermittent (well -- now no) connectivity to XO from Level 3 in NYC (and from the looks of the L3 looking glass other areas of the US.) Anyone else seeing this?
NYC to XO Web: traceroute to www.xo.com (205.158.160.76), 30 hops max, 60 byte packets 4 ge-9-23.car1.newyork1.level3.net (4.26.32.89) 4.240 ms 4.210 ms 4.177 ms 5 ae-2-70.edge2.newyork1.level3.net (4.69.155.80) 2.200 ms 2.155 ms 1.895 ms 6 206.111.13.65.ptr.us.xo.net (206.111.13.65) 1.963 ms 1.667 ms 1.624 ms 7 * * * 8 * * *
NYC to XO (NYC): traceroute to 207.86.XXX.x (207.86.XXX.x), 30 hops max, 60 byte packets 4 ge-9-23.car1.newyork1.level3.net (4.26.32.89) 2.702 ms 2.678 ms 2.667 ms 5 ae-4-90.edge2.newyork1.level3.net (4.69.155.208) 1.607 ms 1.558 ms 1.538 ms 6 206.111.13.65.ptr.us.xo.net (206.111.13.65) 1.644 ms 1.575 ms 1.549 ms 7 * * * 8 * * *
Show Level 3 (Dallas, TX) Traceroute to 205.158.160.76
1 ae-1-60.edge2.Dallas3.Level3.net (4.69.145.12) 0 msec ae-2-70.edge2.Dallas3.Level3.net (4.69.145.76) 0 msec ae-4-90.edge2.Dallas3.Level3.net (4.69.145.204) 4 msec 2 XO-Level3.edge2.Dallas3.Level3.net (4.68.127.150) 0 msec 0 msec 12 msec 3 * * * 4 * * *
I'm seeing the same in Chicago. admin@core1-chi> traceroute 4.2.2.4 traceroute to 4.2.2.4 (4.2.2.4), 30 hops max, 40 byte packets 1 209.117.14.233 1.864 ms 1.730 ms 1.679 ms 2 * * * 3 * * * 4 * * * 5 * * * Ticket opened with XO, no response yet.
Things look ok from L3 looking glass http://lg.level3.net/traceroute/traceroute.cgi?site=nyc1&target=209.117.14.2... --Karl Karl Putland Senior VoIP Engineer *SimpleSignal* 3600 S Yosemite, Suite 150 Denver, CO 80237 One Number Rings All My Phones: 303-242-8608 SimpleSignal.com <http://www.simplesignal.com/> | Blog<http://www.simplesignal.com/blog> | Facebook <http://www.facebook.com/SimpleSignal?ref=ts> | Twitter<http://twitter.com/simplesignal> On Wed, Sep 26, 2012 at 10:22 AM, Kevin Day <toasty@dragondata.com> wrote:
On Sep 26, 2012, at 11:12 AM, Jayson Navitsky <jnavitsky@gmail.com> wrote:
We're showing intermittent (well -- now no) connectivity to XO from Level 3 in NYC (and from the looks of the L3 looking glass other areas of the US.) Anyone else seeing this?
NYC to XO Web: traceroute to www.xo.com (205.158.160.76), 30 hops max, 60 byte packets 4 ge-9-23.car1.newyork1.level3.net (4.26.32.89) 4.240 ms 4.210 ms 4.177 ms 5 ae-2-70.edge2.newyork1.level3.net (4.69.155.80) 2.200 ms 2.155 ms 1.895 ms 6 206.111.13.65.ptr.us.xo.net (206.111.13.65) 1.963 ms 1.667 ms 1.624 ms 7 * * * 8 * * *
NYC to XO (NYC): traceroute to 207.86.XXX.x (207.86.XXX.x), 30 hops max, 60 byte packets 4 ge-9-23.car1.newyork1.level3.net (4.26.32.89) 2.702 ms 2.678 ms 2.667 ms 5 ae-4-90.edge2.newyork1.level3.net (4.69.155.208) 1.607 ms 1.558 ms 1.538 ms 6 206.111.13.65.ptr.us.xo.net (206.111.13.65) 1.644 ms 1.575 ms 1.549 ms 7 * * * 8 * * *
*Show Level 3 (Dallas, TX) Traceroute to 205.158.160.76*
1 ae-1-60.edge2.Dallas3.Level3.net <http://ae-1-60.edge2.dallas3.level3.net/> (4.69.145.12) 0 msec ae-2-70.edge2.Dallas3.Level3.net <http://ae-2-70.edge2.dallas3.level3.net/> (4.69.145.76) 0 msec ae-4-90.edge2.Dallas3.Level3.net <http://ae-4-90.edge2.dallas3.level3.net/> (4.69.145.204) 4 msec 2 XO-Level3.edge2.Dallas3.Level3.net <http://xo-level3.edge2.dallas3.level3.net/> (4.68.127.150) 0 msec 0 msec 12 msec 3 * * * 4 * * *
I'm seeing the same in Chicago.
admin@core1-chi> traceroute 4.2.2.4 traceroute to 4.2.2.4 (4.2.2.4), 30 hops max, 40 byte packets 1 209.117.14.233 1.864 ms 1.730 ms 1.679 ms 2 * * * 3 * * * 4 * * * 5 * * *
Ticket opened with XO, no response yet.
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
I am seeing this as well. My trace from Atlanta to xo.com 5 vlan52.ebr2.atlanta2.level3.net (4.69.150.126) 14.096 ms 14.043 ms 14.050 ms 6 ae-2-2.ebr2.miami1.level3.net (4.69.140.141) 31.075 ms 28.708 ms 28.712 ms 7 ae-2-52.edge1.miami2.level3.net (4.69.138.107) 90.309 ms 90.277 ms 28.648 ms 8 xo-communic.edge1.miami2.level3.net (4.59.80.42) 28.656 ms 21.010 ms 20.961 ms 9 * * * 10 * * * From a different site that has XO as the uplink, cannot ping 4.2.2.2, and trace shows: 2 66.236.240.89.ptr.us.xo.net 8 msec * 8 msec 3 216.156.9.197.ptr.us.xo.net 20 msec 12 msec 12 msec 4 * * * 5 * * * 6 * * * --John On 09/26/2012 11:12 AM, Jayson Navitsky wrote:
We're showing intermittent (well -- now no) connectivity to XO from Level 3 in NYC (and from the looks of the L3 looking glass other areas of the US.) Anyone else seeing this?
NYC to XO Web: traceroute to www.xo.com <http://www.xo.com> (205.158.160.76), 30 hops max, 60 byte packets 4 ge-9-23.car1.newyork1.level3.net <http://ge-9-23.car1.newyork1.level3.net> (4.26.32.89) 4.240 ms 4.210 ms 4.177 ms 5 ae-2-70.edge2.newyork1.level3.net <http://ae-2-70.edge2.newyork1.level3.net> (4.69.155.80) 2.200 ms 2.155 ms 1.895 ms 6 206.111.13.65.ptr.us.xo.net <http://206.111.13.65.ptr.us.xo.net> (206.111.13.65) 1.963 ms 1.667 ms 1.624 ms 7 * * * 8 * * *
NYC to XO (NYC): traceroute to 207.86.XXX.x (207.86.XXX.x), 30 hops max, 60 byte packets 4 ge-9-23.car1.newyork1.level3.net <http://ge-9-23.car1.newyork1.level3.net> (4.26.32.89) 2.702 ms 2.678 ms 2.667 ms 5 ae-4-90.edge2.newyork1.level3.net <http://ae-4-90.edge2.newyork1.level3.net> (4.69.155.208) 1.607 ms 1.558 ms 1.538 ms 6 206.111.13.65.ptr.us.xo.net <http://206.111.13.65.ptr.us.xo.net> (206.111.13.65) 1.644 ms 1.575 ms 1.549 ms 7 * * * 8 * * *
*Show Level 3 (Dallas, TX) Traceroute to 205.158.160.76*
1ae-1-60.edge2.Dallas3.Level3.net <http://ae-1-60.edge2.Dallas3.Level3.net> (4.69.145.12) 0 msec ae-2-70.edge2.Dallas3.Level3.net <http://ae-2-70.edge2.Dallas3.Level3.net> (4.69.145.76) 0 msec ae-4-90.edge2.Dallas3.Level3.net <http://ae-4-90.edge2.Dallas3.Level3.net> (4.69.145.204) 4 msec 2XO-Level3.edge2.Dallas3.Level3.net <http://XO-Level3.edge2.Dallas3.Level3.net> (4.68.127.150) 0 msec 0 msec 12 msec 3 * * * 4 * * *
-J
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Looks like things are starting to come back in NYC and Chicago we can reach various sites in both cities now. -J On Wed, Sep 26, 2012 at 12:23 PM, John Barbieri <tenpin784@gmail.com> wrote:
I am seeing this as well.
My trace from Atlanta to xo.com
5 vlan52.ebr2.atlanta2.level3.net (4.69.150.126) 14.096 ms 14.043 ms 14.050 ms 6 ae-2-2.ebr2.miami1.level3.net (4.69.140.141) 31.075 ms 28.708 ms 28.712 ms 7 ae-2-52.edge1.miami2.level3.net (4.69.138.107) 90.309 ms 90.277 ms 28.648 ms 8 xo-communic.edge1.miami2.level3.net (4.59.80.42) 28.656 ms 21.010 ms 20.961 ms 9 * * * 10 * * *
From a different site that has XO as the uplink, cannot ping 4.2.2.2, and trace shows:
2 66.236.240.89.ptr.us.xo.net 8 msec * 8 msec 3 216.156.9.197.ptr.us.xo.net 20 msec 12 msec 12 msec
4 * * * 5 * * * 6 * * *
--John
On 09/26/2012 11:12 AM, Jayson Navitsky wrote:
We're showing intermittent (well -- now no) connectivity to XO from Level 3 in NYC (and from the looks of the L3 looking glass other areas of the US.) Anyone else seeing this?
NYC to XO Web: traceroute to www.xo.com (205.158.160.76), 30 hops max, 60 byte packets 4 ge-9-23.car1.newyork1.level3.net (4.26.32.89) 4.240 ms 4.210 ms 4.177 ms 5 ae-2-70.edge2.newyork1.level3.net (4.69.155.80) 2.200 ms 2.155 ms 1.895 ms 6 206.111.13.65.ptr.us.xo.net (206.111.13.65) 1.963 ms 1.667 ms 1.624 ms 7 * * * 8 * * *
NYC to XO (NYC): traceroute to 207.86.XXX.x (207.86.XXX.x), 30 hops max, 60 byte packets 4 ge-9-23.car1.newyork1.level3.net (4.26.32.89) 2.702 ms 2.678 ms 2.667 ms 5 ae-4-90.edge2.newyork1.level3.net (4.69.155.208) 1.607 ms 1.558 ms 1.538 ms 6 206.111.13.65.ptr.us.xo.net (206.111.13.65) 1.644 ms 1.575 ms 1.549 ms 7 * * * 8 * * *
*Show Level 3 (Dallas, TX) Traceroute to 205.158.160.76*
1 ae-1-60.edge2.Dallas3.Level3.net (4.69.145.12) 0 msec ae-2-70.edge2.Dallas3.Level3.net (4.69.145.76) 0 msec ae-4-90.edge2.Dallas3.Level3.net (4.69.145.204) 4 msec 2 XO-Level3.edge2.Dallas3.Level3.net (4.68.127.150) 0 msec 0 msec 12 msec 3 * * * 4 * * *
-J
_______________________________________________ Outages mailing listOutages@outages.orghttps://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
XO has confirmed with me that they are having a major issue which is also effecting Level 3 at this time. Turning down 2828 has resolved our problems here. http://internetpulse.net/ Regards, Bradley D. Bopp Director Of Engineering http://www.nationalnet.com http://as22384.peeringdb.net NationalNet is committed to the highest level of Customer Service available in the Web Hosting business. If you have any questions, comments or concerns feel free to contact us at 678-247-7000 ext 1 (or toll-free, 888-4-NATNET). On Sep 26, 2012, at Sep 26, 2012, 12:23 PM, John Barbieri <tenpin784@gmail.com> wrote:
I am seeing this as well.
My trace from Atlanta to xo.com
5 vlan52.ebr2.atlanta2.level3.net (4.69.150.126) 14.096 ms 14.043 ms 14.050 ms 6 ae-2-2.ebr2.miami1.level3.net (4.69.140.141) 31.075 ms 28.708 ms 28.712 ms 7 ae-2-52.edge1.miami2.level3.net (4.69.138.107) 90.309 ms 90.277 ms 28.648 ms 8 xo-communic.edge1.miami2.level3.net (4.59.80.42) 28.656 ms 21.010 ms 20.961 ms 9 * * * 10 * * *
From a different site that has XO as the uplink, cannot ping 4.2.2.2, and trace shows:
2 66.236.240.89.ptr.us.xo.net 8 msec * 8 msec 3 216.156.9.197.ptr.us.xo.net 20 msec 12 msec 12 msec 4 * * * 5 * * * 6 * * *
--John
On 09/26/2012 11:12 AM, Jayson Navitsky wrote:
We're showing intermittent (well -- now no) connectivity to XO from Level 3 in NYC (and from the looks of the L3 looking glass other areas of the US.) Anyone else seeing this?
NYC to XO Web: traceroute to www.xo.com (205.158.160.76), 30 hops max, 60 byte packets 4 ge-9-23.car1.newyork1.level3.net (4.26.32.89) 4.240 ms 4.210 ms 4.177 ms 5 ae-2-70.edge2.newyork1.level3.net (4.69.155.80) 2.200 ms 2.155 ms 1.895 ms 6 206.111.13.65.ptr.us.xo.net (206.111.13.65) 1.963 ms 1.667 ms 1.624 ms 7 * * * 8 * * *
NYC to XO (NYC): traceroute to 207.86.XXX.x (207.86.XXX.x), 30 hops max, 60 byte packets 4 ge-9-23.car1.newyork1.level3.net (4.26.32.89) 2.702 ms 2.678 ms 2.667 ms 5 ae-4-90.edge2.newyork1.level3.net (4.69.155.208) 1.607 ms 1.558 ms 1.538 ms 6 206.111.13.65.ptr.us.xo.net (206.111.13.65) 1.644 ms 1.575 ms 1.549 ms 7 * * * 8 * * *
Show Level 3 (Dallas, TX) Traceroute to 205.158.160.76
1 ae-1-60.edge2.Dallas3.Level3.net (4.69.145.12) 0 msec
ae-2-70.edge2.Dallas3.Level3.net (4.69.145.76) 0 msec
ae-4-90.edge2.Dallas3.Level3.net (4.69.145.204) 4 msec 2 XO-Level3.edge2.Dallas3.Level3.net (4.68.127.150) 0 msec 0 msec 12 msec 3 * * * 4 * * *
-J
_______________________________________________ Outages mailing list
Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
We are seeing basically no connectivity across XO from multiple markets (LA, SFO, ATL, CHI). DJ Conley Resolution Center (Tier 3) 770-874-4856 O 404-395-7861 M dj.conley@cbeyond.net<mailto:dj.conley@cbeyond.net> CBEYOND From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On Behalf Of Jayson Navitsky Sent: Wednesday, September 26, 2012 12:13 PM To: outages@outages.org Subject: [outages] Level 3 -> XO We're showing intermittent (well -- now no) connectivity to XO from Level 3 in NYC (and from the looks of the L3 looking glass other areas of the US.) Anyone else seeing this? NYC to XO Web: traceroute to www.xo.com<http://www.xo.com> (205.158.160.76), 30 hops max, 60 byte packets 4 ge-9-23.car1.newyork1.level3.net<http://ge-9-23.car1.newyork1.level3.net> (4.26.32.89) 4.240 ms 4.210 ms 4.177 ms 5 ae-2-70.edge2.newyork1.level3.net<http://ae-2-70.edge2.newyork1.level3.net> (4.69.155.80) 2.200 ms 2.155 ms 1.895 ms 6 206.111.13.65.ptr.us.xo.net<http://206.111.13.65.ptr.us.xo.net> (206.111.13.65) 1.963 ms 1.667 ms 1.624 ms 7 * * * 8 * * * NYC to XO (NYC): traceroute to 207.86.XXX.x (207.86.XXX.x), 30 hops max, 60 byte packets 4 ge-9-23.car1.newyork1.level3.net<http://ge-9-23.car1.newyork1.level3.net> (4.26.32.89) 2.702 ms 2.678 ms 2.667 ms 5 ae-4-90.edge2.newyork1.level3.net<http://ae-4-90.edge2.newyork1.level3.net> (4.69.155.208) 1.607 ms 1.558 ms 1.538 ms 6 206.111.13.65.ptr.us.xo.net<http://206.111.13.65.ptr.us.xo.net> (206.111.13.65) 1.644 ms 1.575 ms 1.549 ms 7 * * * 8 * * * Show Level 3 (Dallas, TX) Traceroute to 205.158.160.76 1 ae-1-60.edge2.Dallas3.Level3.net<http://ae-1-60.edge2.Dallas3.Level3.net> (4.69.145.12) 0 msec ae-2-70.edge2.Dallas3.Level3.net<http://ae-2-70.edge2.Dallas3.Level3.net> (4.69.145.76) 0 msec ae-4-90.edge2.Dallas3.Level3.net<http://ae-4-90.edge2.Dallas3.Level3.net> (4.69.145.204) 4 msec 2 XO-Level3.edge2.Dallas3.Level3.net<http://XO-Level3.edge2.Dallas3.Level3.net> (4.68.127.150) 0 msec 0 msec 12 msec 3 * * * 4 * * * -J
Yes, we are a customer of XO and there is intermittent connectivity. Possibly a peering issue similar to one that they had back at the first of the year. Notified XO at 955 am MST and there were barely aware of the issue. Reed Glauser Systems Engineer [cid:0EDB9A43-3C86-4CEA-9330-E229B151B3EF] PH. 801.765.9245 x477 FAX. 801.765.9246 @. rglauser@bidsync.com<mailto:rglauser@bidsync.com> Leave previous communications attached on all replies. This electronic message is confidential and is intended only for the use of the individual to whom it is addressed. If you are not the intended recipient, please immediately notify me by replying to this e-mail and delete the message from your system. Thank you. From: Jayson Navitsky <jnavitsky@gmail.com<mailto:jnavitsky@gmail.com>> Date: Wednesday, September 26, 2012 10:12 AM To: "outages@outages.org<mailto:outages@outages.org>" <outages@outages.org<mailto:outages@outages.org>> Subject: [outages] Level 3 -> XO We're showing intermittent (well -- now no) connectivity to XO from Level 3 in NYC (and from the looks of the L3 looking glass other areas of the US.) Anyone else seeing this? NYC to XO Web: traceroute to www.xo.com<http://www.xo.com> (205.158.160.76), 30 hops max, 60 byte packets 4 ge-9-23.car1.newyork1.level3.net<http://ge-9-23.car1.newyork1.level3.net> (4.26.32.89) 4.240 ms 4.210 ms 4.177 ms 5 ae-2-70.edge2.newyork1.level3.net<http://ae-2-70.edge2.newyork1.level3.net> (4.69.155.80) 2.200 ms 2.155 ms 1.895 ms 6 206.111.13.65.ptr.us.xo.net<http://206.111.13.65.ptr.us.xo.net> (206.111.13.65) 1.963 ms 1.667 ms 1.624 ms 7 * * * 8 * * * NYC to XO (NYC): traceroute to 207.86.XXX.x (207.86.XXX.x), 30 hops max, 60 byte packets 4 ge-9-23.car1.newyork1.level3.net<http://ge-9-23.car1.newyork1.level3.net> (4.26.32.89) 2.702 ms 2.678 ms 2.667 ms 5 ae-4-90.edge2.newyork1.level3.net<http://ae-4-90.edge2.newyork1.level3.net> (4.69.155.208) 1.607 ms 1.558 ms 1.538 ms 6 206.111.13.65.ptr.us.xo.net<http://206.111.13.65.ptr.us.xo.net> (206.111.13.65) 1.644 ms 1.575 ms 1.549 ms 7 * * * 8 * * * Show Level 3 (Dallas, TX) Traceroute to 205.158.160.76 1 ae-1-60.edge2.Dallas3.Level3.net<http://ae-1-60.edge2.Dallas3.Level3.net> (4.69.145.12) 0 msec ae-2-70.edge2.Dallas3.Level3.net<http://ae-2-70.edge2.Dallas3.Level3.net> (4.69.145.76) 0 msec ae-4-90.edge2.Dallas3.Level3.net<http://ae-4-90.edge2.Dallas3.Level3.net> (4.69.145.204) 4 msec 2 XO-Level3.edge2.Dallas3.Level3.net<http://XO-Level3.edge2.Dallas3.Level3.net> (4.68.127.150) 0 msec 0 msec 12 msec 3 * * * 4 * * * -J
Sites are starting to respond now. From ATL to xo.com 5 vlan52.ebr2.atlanta2.level3.net (4.69.150.126) 20.147 ms 11.744 ms 20.110 ms 6 ae-2-2.ebr2.miami1.level3.net (4.69.140.141) 31.091 ms 30.005 ms 22.271 ms 7 ae-2-52.edge1.miami2.level3.net (4.69.138.107) 21.719 ms 21.678 ms 21.691 ms 8 xo-communic.edge1.miami2.level3.net (4.59.80.42) 21.693 ms 21.687 ms 21.683 ms 9 vb2001.rar3.dallas-tx.us.xo.net (207.88.13.174) 42.574 ms 42.580 ms 42.562 ms 10 ae0d0.mcr1.dallas-tx.us.xo.net (216.156.0.82) 92.925 ms 92.904 ms 92.907 ms 11 216.55.13.100 (216.55.13.100) 40.185 ms 42.484 ms 42.435 ms 12 txplan01-fw01a-eth1.dc.xo.com (205.158.160.201) 41.171 ms 41.126 ms 41.103 ms 13 xonlbvip.pla.dc.xo.com (205.158.160.76) 41.108 ms 41.094 ms 41.045 ms --John On 09/26/2012 11:25 AM, Reed Glauser wrote:
Yes, we are a customer of XO and there is intermittent connectivity. Possibly a peering issue similar to one that they had back at the first of the year. Notified XO at 955 am MST and there were barely aware of the issue.
*/ /*
*/ /*
*/ Reed Glauser/*
http://www.bidsync.com/images/marketing/dark-blue-hr.gif Systems Engineer
*PH.* 801.765.9245 x477 *FAX.* 801.765.9246 *@.* rglauser@bidsync.com <mailto:rglauser@bidsync.com>
Leave previous communications attached on all replies. This electronic message is confidential and is intended only for the use of the individual to whom it is addressed. If you are not the intended recipient, please immediately notify me by replying to this e-mail and delete the message from your system. Thank you.
From: Jayson Navitsky <jnavitsky@gmail.com <mailto:jnavitsky@gmail.com>> Date: Wednesday, September 26, 2012 10:12 AM To: "outages@outages.org <mailto:outages@outages.org>" <outages@outages.org <mailto:outages@outages.org>> Subject: [outages] Level 3 -> XO
We're showing intermittent (well -- now no) connectivity to XO from Level 3 in NYC (and from the looks of the L3 looking glass other areas of the US.) Anyone else seeing this?
NYC to XO Web: traceroute to www.xo.com <http://www.xo.com> (205.158.160.76), 30 hops max, 60 byte packets 4 ge-9-23.car1.newyork1.level3.net <http://ge-9-23.car1.newyork1.level3.net> (4.26.32.89) 4.240 ms 4.210 ms 4.177 ms 5 ae-2-70.edge2.newyork1.level3.net <http://ae-2-70.edge2.newyork1.level3.net> (4.69.155.80) 2.200 ms 2.155 ms 1.895 ms 6 206.111.13.65.ptr.us.xo.net <http://206.111.13.65.ptr.us.xo.net> (206.111.13.65) 1.963 ms 1.667 ms 1.624 ms 7 * * * 8 * * *
NYC to XO (NYC): traceroute to 207.86.XXX.x (207.86.XXX.x), 30 hops max, 60 byte packets 4 ge-9-23.car1.newyork1.level3.net <http://ge-9-23.car1.newyork1.level3.net> (4.26.32.89) 2.702 ms 2.678 ms 2.667 ms 5 ae-4-90.edge2.newyork1.level3.net <http://ae-4-90.edge2.newyork1.level3.net> (4.69.155.208) 1.607 ms 1.558 ms 1.538 ms 6 206.111.13.65.ptr.us.xo.net <http://206.111.13.65.ptr.us.xo.net> (206.111.13.65) 1.644 ms 1.575 ms 1.549 ms 7 * * * 8 * * *
*Show Level 3 (Dallas, TX) Traceroute to 205.158.160.76* 1ae-1-60.edge2.Dallas3.Level3.net <http://ae-1-60.edge2.Dallas3.Level3.net> (4.69.145.12) 0 msec ae-2-70.edge2.Dallas3.Level3.net <http://ae-2-70.edge2.Dallas3.Level3.net> (4.69.145.76) 0 msec ae-4-90.edge2.Dallas3.Level3.net <http://ae-4-90.edge2.Dallas3.Level3.net> (4.69.145.204) 4 msec 2XO-Level3.edge2.Dallas3.Level3.net <http://XO-Level3.edge2.Dallas3.Level3.net> (4.68.127.150) 0 msec 0 msec 12 msec 3 * * * 4 * * *
-J
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Its fixed now. On Wed, Sep 26, 2012 at 10:12 AM, Jayson Navitsky <jnavitsky@gmail.com>wrote:
We're showing intermittent (well -- now no) connectivity to XO from Level 3 in NYC (and from the looks of the L3 looking glass other areas of the US.) Anyone else seeing this?
NYC to XO Web: traceroute to www.xo.com (205.158.160.76), 30 hops max, 60 byte packets 4 ge-9-23.car1.newyork1.level3.net (4.26.32.89) 4.240 ms 4.210 ms 4.177 ms 5 ae-2-70.edge2.newyork1.level3.net (4.69.155.80) 2.200 ms 2.155 ms 1.895 ms 6 206.111.13.65.ptr.us.xo.net (206.111.13.65) 1.963 ms 1.667 ms 1.624 ms 7 * * * 8 * * *
NYC to XO (NYC): traceroute to 207.86.XXX.x (207.86.XXX.x), 30 hops max, 60 byte packets 4 ge-9-23.car1.newyork1.level3.net (4.26.32.89) 2.702 ms 2.678 ms 2.667 ms 5 ae-4-90.edge2.newyork1.level3.net (4.69.155.208) 1.607 ms 1.558 ms 1.538 ms 6 206.111.13.65.ptr.us.xo.net (206.111.13.65) 1.644 ms 1.575 ms 1.549 ms 7 * * * 8 * * *
*Show Level 3 (Dallas, TX) Traceroute to 205.158.160.76*
1 ae-1-60.edge2.Dallas3.Level3.net (4.69.145.12) 0 msec ae-2-70.edge2.Dallas3.Level3.net (4.69.145.76) 0 msec ae-4-90.edge2.Dallas3.Level3.net (4.69.145.204) 4 msec 2 XO-Level3.edge2.Dallas3.Level3.net (4.68.127.150) 0 msec 0 msec 12 msec 3 * * * 4 * * *
-J
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
-- Mauricio Lizano 8821-6575 http://cr.linkedin.com/pub/mauricio-lizano/14/162/5b5
On 26-Sep-12 12:12 PM, Jayson Navitsky wrote:
We're showing intermittent (well -- now no) connectivity to XO from Level 3 in NYC (and from the looks of the L3 looking glass other areas of the US.) Anyone else seeing this?
While on-hold with XO, a robotic voice broke in and stated that they have identified and corrected the problem affecting connectivity across their network and to please try again. Still seeing some issues, but we are reaching out to customers to see if connectivity is restored. No further information from XO. -- Devon
They have updated twitter with a confirmation: twitter.com/XOCare "There is currently an outage related to a peering problem with Level 3 communications. At this time we do not have an ETR" I'm currently seeing traffic pass again between L3 and XO in NY.
participants (15)
-
Aubrey Wells -
Bradley Bopp -
Chris Stone -
Devon True -
DJ Conley -
Eric Tykwinski -
Jayson Navitsky -
Joel Woodbury -
John Barbieri -
Joseph Jackson -
Karl Putland -
Kevin Day -
Marco Prechel -
Mauricio Lizano -
Reed Glauser