This could be an NNI or peering disconnect between light tower and Level3. Light tower has been doing maintenance on different nodes... They have been know to forget to notify their peers . Regards, Joe Sanchez ( please excuse the brevity of this email as it was sent via a mobile device. Please excuse misspelled words or sentence structure.) On Oct 4, 2012, at 3:01 PM, John Barbieri <tenpin784@gmail.com> wrote:

This is my trace from Summer St. in Boston:

traceroute 192.246.235.1

Type escape sequence to abort. Tracing the route to hyphen.bard.edu (192.246.235.1)

2 ae-2-7.bar1.Boston1.Level3.net (4.69.132.242) [AS 3356] [MPLS: Label 302944 Exp 0] 0 msec 0 msec 4 msec 3 ae-0-11.bar2.Boston1.Level3.net (4.69.140.90) [AS 3356] [MPLS: Label 302944 Exp 0] 0 msec 0 msec 4 msec 4 ae-8-8.ebr1.NewYork1.Level3.net (4.69.140.98) [AS 3356] [MPLS: Label 1268 Exp 0] 4 msec 4 msec 24 msec 5 ae-81-81.csw3.NewYork1.Level3.net (4.69.134.74) [AS 3356] [MPLS: Label 1325 Exp 0] 8 msec ae-71-71.csw2.NewYork1.Level3.net (4.69.134.70) [AS 3356] [MPLS: Label 1325 Exp 0] 4 msec 8 msec 6 ae-1-60.edge4.NewYork1.Level3.net (4.69.155.20) [AS 3356] 60 msec 8 msec ae-2-70.edge4.NewYork1.Level3.net (4.69.155.84) [AS 3356] 4 msec 7 4.68.111.242 [AS 3356] 4 msec 64.215.195.229 [AS 3549] 4 msec 4 msec 8 HURRICANE-ELECTRIC-LLC-New-York.TenGigabitEthernet1-3.ar5.NYC1.gblx.net (64.209.92.98) [AS 3549] 24 msec 4 msec 4 msec 9 10gigabitethernet4-1.core1.nyc5.he.net (184.105.213.218) [AS 6939] 8 msec 4 msec 8 msec 10 lightower-fiber-networks.10gigabitethernet3-2.core1.nyc5.he.net (216.66.50.106) [AS 6939] 8 msec 4 msec 8 msec 11 xe-0-3-0.pghknyshj42.lightower.net (72.22.160.150) [AS 46887] 28 msec 12 msec 8 msec 12 ae0-pghknyshj91.lightower.net (72.22.160.158) [AS 46887] 8 msec 8 msec 8 msec 13 xe-7-2-0.kgtnnykgj91.lightower.net (72.22.160.107) [AS 46887] 28 msec 8 msec 8 msec 14 BardCollege-cust.customer.hvdata.net (64.72.66.234) [AS 46887] 8 msec * 284 msec

I don't see it going down to DC or ATL or Miami.

--John On 10/04/2012 01:35 PM, Steve Bohrer wrote:

...

Hi,

Bard College at Simon's Rock (simons-rock.edu) is a small college in Southwestern Mass, with our parent campus, Bard (bard.edu), in the Hudson Valley near Rhinebeck, NY.

We use a database hosted on the main campus, and poor database performance here is often correlated with Level3 / Global Crossing issues in New York city. Today my users are complaining about slow database response, and I find that traceroutes from here, and from Level3's Looking Glass Boston page now go from NYC to Poughkeepsie via Washington, Atlanta, and Miami. This is a new routing; traces from past months went directly from NYC to Poughkeepsie.

I've reported this to our provider, who has presumably passed it along to Level3, but I'm wondering if these goofy routes are part of a work-around of some bigger issue in NYC, or just a screw up, or perhaps there is some valid business reason for this routing change?

Anyone else seeing strangeness via Level3 from NYC to the Hudson?

Here's Level3's traceroute, from http://lg.level3.net/traceroute/traceroute.cgi?site=bos1&target=hyphen.bard.... :

Show Level 3 (Boston, MA) Traceroute to hyphen.bard.edu (192.246.235.1)

1 ae-2-7.bar1.Boston1.Level3.net (4.69.132.242) 0 msec 0 msec 0 msec 2 ae-0-11.bar2.Boston1.Level3.net (4.69.140.90) 0 msec 0 msec 4 msec 3 ae-8-8.ebr1.NewYork1.Level3.net (4.69.140.98) 32 msec 32 msec 36 msec 4 ae-10-10.ebr2.Washington12.Level3.net (4.69.148.50) 32 msec 32 msec 36 msec 5 ae-1-100.ebr1.Washington12.Level3.net (4.69.143.213) 36 msec 44 msec 44 msec 6 ae-6-6.ebr1.Atlanta2.Level3.net (4.69.148.105) 72 msec 36 msec 36 msec 7 ae-1-100.ebr2.Atlanta2.Level3.net (4.69.132.34) 32 msec 36 msec 44 msec 8 ae-2-2.ebr2.Miami1.Level3.net (4.69.140.141) 36 msec 40 msec 36 msec 9 ae-2-52.edge1.Miami2.Level3.net (4.69.138.107) 32 msec 36 msec 36 msec 10 64.209.106.214 36 msec 36 msec 36 msec 11 10gigabitethernet1-1.core1.atl1.he.net (72.52.92.53) [AS6939 {HURRICANE}] 40 msec 36 msec 40 msec 12 10gigabitethernet6-4.core1.ash1.he.net (184.105.213.109) [AS6939 {HURRICANE}] 40 msec 40 msec 52 msec 13 10gigabitethernet1-2.core1.nyc4.he.net (72.52.92.85) [AS6939 {HURRICANE}] 40 msec 48 msec 48 msec 14 10gigabitethernet4-1.core1.nyc5.he.net (184.105.213.218) [AS6939 {HURRICANE}] 40 msec 48 msec 48 msec 15 * lightower-fiber-networks.10gigabitethernet3-2.core1.nyc5.he.net (216.66.50.106) [AS6939 {HURRICANE}] 208 msec 204 msec 16 xe-0-3-0.pghknyshj42.lightower.net (72.22.160.150) [AS46887 {LIGHTOWER}] 208 msec * 220 msec 17 ae0-pghknyshj91.lightower.net (72.22.160.158) [AS46887 {LIGHTOWER}] 92 msec 64 msec 44 msec 18 xe-7-2-0.kgtnnykgj91.lightower.net (72.22.160.107) [AS46887 {LIGHTOWER}] 196 msec 128 msec 312 msec 19 BardCollege-cust.customer.hvdata.net (64.72.66.234) [AS46887 {LIGHTOWER}] 164 msec * *

(In the past, we sometimes got a nice short route directly from Level3 in Boston to Lightower.net in Boston, as Lightower seems to have a direct link Poughkeepsie to Boston. However, when Level3 merged with Global Crossing, our route to Bard started going through NYC to reach Lightower, which has been more troublesome.)

Thanks for any hints you can offer.

Steve Bohrer Network Admin Bard College at Simon's Rock 413-528-7645

_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages

_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages

Thanks Jeremy, I saw your post last week, and found it very useful. In fact the route is asymmetrical: traces from Bard back to us go from Lightower's Poughkeepsie location directly to Boston, and skip NYC altogether -- I wish we could get that route both directions! My concern is that the traceroute from here to Bard was suddenly much longer than in past months, and I got the same results directly from Level3, so that made me think it wasn't something goofy with our last mile. But, as John noted above, now it works as expected now from Level3's page, and also from our site. I don't know if the timing of this fix shortly after my post was just coincidental, or if the ticket I filed with my provider at 12:30 finally trickled through the system, or if perhaps the act of posting to Outages got someone at Level3 to notice. In any case, it works again now the way it did yesterday, so I'm all set. Steve Bohrer ITS, Bard College at Simon's Rock 413-528-7645 On Oct 4, 2012, at 3:02 PM, Jeremy Chadwick wrote:

Steve,

And what about a traceroute from hyphen.bard.edu to wherever that L3 looking glass is? Since the source (of the looking glass) doesn't appear to be shown, I would say a traceroute from hyphen.bard.edu to 4.69.132.242 would be the 2nd best thing to have (and ignore any packet loss you see at that destination; it's a router not a machine).

Just last week on the mailing list I covered the importance of having traceroutes from both directions, re: asymmetrical routing. The PDF I've linked should be read in full:

https://puck.nether.net/pipermail/outages/2012-September/004466.html

-- | Jeremy Chadwick jdc@koitsu.org | | UNIX Systems Administrator http://jdc.koitsu.org/ | | Mountain View, CA, US | | Making life hard for others since 1977. PGP 4BD6C0CB |

On Thu, Oct 04, 2012 at 02:35:07PM -0400, Steve Bohrer wrote:

...

Hi,

Bard College at Simon's Rock (simons-rock.edu) is a small college in Southwestern Mass, with our parent campus, Bard (bard.edu), in the Hudson Valley near Rhinebeck, NY.

We use a database hosted on the main campus, and poor database performance here is often correlated with Level3 / Global Crossing issues in New York city. Today my users are complaining about slow database response, and I find that traceroutes from here, and from Level3's Looking Glass Boston page now go from NYC to Poughkeepsie via Washington, Atlanta, and Miami. This is a new routing; traces from past months went directly from NYC to Poughkeepsie.

I've reported this to our provider, who has presumably passed it along to Level3, but I'm wondering if these goofy routes are part of a work-around of some bigger issue in NYC, or just a screw up, or perhaps there is some valid business reason for this routing change?

Anyone else seeing strangeness via Level3 from NYC to the Hudson?

Here's Level3's traceroute, from http://lg.level3.net/traceroute/traceroute.cgi?site=bos1&target=hyphen.bard.... :

Show Level 3 (Boston, MA) Traceroute to hyphen.bard.edu (192.246.235.1)

1 ae-2-7.bar1.Boston1.Level3.net (4.69.132.242) 0 msec 0 msec 0 msec 2 ae-0-11.bar2.Boston1.Level3.net (4.69.140.90) 0 msec 0 msec 4 msec 3 ae-8-8.ebr1.NewYork1.Level3.net (4.69.140.98) 32 msec 32 msec 36 msec 4 ae-10-10.ebr2.Washington12.Level3.net (4.69.148.50) 32 msec 32 msec 36 msec 5 ae-1-100.ebr1.Washington12.Level3.net (4.69.143.213) 36 msec 44 msec 44 msec 6 ae-6-6.ebr1.Atlanta2.Level3.net (4.69.148.105) 72 msec 36 msec 36 msec 7 ae-1-100.ebr2.Atlanta2.Level3.net (4.69.132.34) 32 msec 36 msec 44 msec 8 ae-2-2.ebr2.Miami1.Level3.net (4.69.140.141) 36 msec 40 msec 36 msec 9 ae-2-52.edge1.Miami2.Level3.net (4.69.138.107) 32 msec 36 msec 36 msec 10 64.209.106.214 36 msec 36 msec 36 msec 11 10gigabitethernet1-1.core1.atl1.he.net (72.52.92.53) [AS6939 {HURRICANE}] 40 msec 36 msec 40 msec 12 10gigabitethernet6-4.core1.ash1.he.net (184.105.213.109) [AS6939 {HURRICANE}] 40 msec 40 msec 52 msec 13 10gigabitethernet1-2.core1.nyc4.he.net (72.52.92.85) [AS6939 {HURRICANE}] 40 msec 48 msec 48 msec 14 10gigabitethernet4-1.core1.nyc5.he.net (184.105.213.218) [AS6939 {HURRICANE}] 40 msec 48 msec 48 msec 15 * lightower-fiber-networks.10gigabitethernet3-2.core1.nyc5.he.net (216.66.50.106) [AS6939 {HURRICANE}] 208 msec 204 msec 16 xe-0-3-0.pghknyshj42.lightower.net (72.22.160.150) [AS46887 {LIGHTOWER}] 208 msec * 220 msec 17 ae0-pghknyshj91.lightower.net (72.22.160.158) [AS46887 {LIGHTOWER}] 92 msec 64 msec 44 msec 18 xe-7-2-0.kgtnnykgj91.lightower.net (72.22.160.107) [AS46887 {LIGHTOWER}] 196 msec 128 msec 312 msec 19 BardCollege-cust.customer.hvdata.net (64.72.66.234) [AS46887 {LIGHTOWER}] 164 msec * *

(In the past, we sometimes got a nice short route directly from Level3 in Boston to Lightower.net in Boston, as Lightower seems to have a direct link Poughkeepsie to Boston. However, when Level3 merged with Global Crossing, our route to Bard started going through NYC to reach Lightower, which has been more troublesome.)

Thanks for any hints you can offer.

Steve Bohrer Network Admin Bard College at Simon's Rock 413-528-7645

_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages