It does seem to be limited XO/ATT. All looks good to XO via Level3. 4. crr02aldlmi-bue-4.aldl.mi.charter.com 0.0% 10 11.1 10.2 5.3 16.2 3.6 5. bbr01aldlmi-bue-2.aldl.mi.charter.com 0.0% 10 6.2 8.3 5.1 11.9 2.1 6. lag-20.ear1.Chicago2.Level3.net 50.0% 10 9.8 9.9 9.8 10.1 0.1 7. ae-1-51.edge3.Chicago3.Level3.net 0.0% 10 9.7 17.2 9.7 84.6 23.7 8. XO-Level3.edge3.Chicago3.Level3.net 0.0% 10 9.9 9.8 9.7 9.9 0.0 9. 207.88.14.193.ptr.us.xo.net 0.0% 10 17.8 17.7 15.9 19.1 1.1 10. ae1d0.mcr1.southfield-mi.us.xo.net 0.0% 10 15.7 15.9 15.7 16.4 0.2 - - Mitchell On Thu, Sep 18, 2014 at 1:01 PM, Breeden, James W. via Outages <outages@outages.org> wrote:

I don't think its only in Chicago. We normally go ATT AS7018 via XO out of Dallas, but I can't get there right now.

Originating from AS19754 BGP against XO in Dallas Equinix.

C:\Users\jwb_000>tracert cr2.cgcil.ip.att.net

Tracing route to cr2.cgcil.ip.att.net [12.122.8.4] over a maximum of 30 hops:

1 <1 ms <1 ms <1 ms 172.24.130.1 2 5 ms 5 ms 4 ms 172.24.128.1 3 5 ms 5 ms 12 ms 1825a-cloud-cv60-gw.1825a-rtr0.austin-tx.as19754.liveair.net [199.66.152.129] 4 10 ms 10 ms 10 ms 1825a-q303-10g.1950stemmons-rtr0.dallas-tx.as19754.liveair.net [199.66.153.86] 5 10 ms 11 ms 11 ms ge5-0-1d0.cir1.dallas2-tx.us.xo.net [65.47.204.17] 6 * * * Request timed out. 7 * * * Request timed out. 8 * * * Request timed out. 9 * ^C C:\Users\jwb_000>tracert 216.156.0.185

Tracing route to vb1730.rar3.chicago-il.us.xo.net [216.156.0.185] over a maximum of 30 hops:

1 <1 ms <1 ms <1 ms 172.24.130.1 2 5 ms 5 ms 4 ms 172.24.128.1 3 10 ms 16 ms 5 ms 1825a-cloud-cv60-gw.1825a-rtr0.austin-tx.as19754.liveair.net [199.66.152.129] 4 9 ms 10 ms 13 ms 1825a-q303-10g.1950stemmons-rtr0.dallas-tx.as19754.liveair.net [199.66.153.86] 5 10 ms 12 ms 10 ms ge5-0-1d0.cir1.dallas2-tx.us.xo.net [65.47.204.17] 6 39 ms 37 ms 37 ms vb1730.rar3.chicago-il.us.xo.net [216.156.0.185]

Trace complete.

C:\Users\jwb_000>tracert 12.122.133.34

Tracing route to cr1.cgcil.ip.att.net [12.122.133.34]over a maximum of 30 hops:

1 <1 ms <1 ms <1 ms 172.24.130.1 2 5 ms 5 ms 4 ms 172.24.128.1 3 5 ms 10 ms 5 ms 1825a-cloud-cv60-gw.1825a-rtr0.austin-tx.as19754.liveair.net [199.66.152.129] 4 10 ms 9 ms 10 ms 1825a-q303-10g.1950stemmons-rtr0.dallas-tx.as19754.liveair.net [199.66.153.86] 5 9 ms 10 ms 9 ms ge5-0-1d0.cir1.dallas2-tx.us.xo.net [65.47.204.17] 6 * * * Request timed out. 7 * * * Request timed out. 8 * * * Request timed out. 9 * * * Request timed out. 10 ^C

jwb -------------- James W. Breeden | CEO | LiveAir Networks | it's the way we think Smithville West Operations Center | 125 Kellar Rd, Suite C2 | Smithville, TX 78957 email/sip: jwb@liveair.net | o: 512.360.4273 x7011 | tf: 877.360.4273 | c: 512.304.0745 | f: 866.396.8990 | www.liveair.net social net: me - twitter.com/meltedfire | facebook.com/meltedfire | linkedin.com/in/meltedfire social net: liveair - twitter.com/liveair | facebook.com/liveair

internet and data services . infosys building blocks . digital phone . managed services . wholesale LiveAir Networks: Proven Better Value. More Reliable Service. Enhanced Local Support. Shop Local.

...

Be environmentally responsible. Only print this email when necessary.

“Here's to the crazy ones, the misfits, the rebels, the troublemakers, the round pegs in the square holes... the ones who see things differently -- they're not fond of rules... You can quote them, disagree with them, glorify or vilify them, but the only thing you can't do is ignore them because they change things... they push the human race forward, and while some may see them as the crazy ones, we see genius, because the ones who are crazy enough to think that they can change the world, are the ones who do.” – Steve Jobs

-----Original Message----- From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Mitchell Kuch via Outages Sent: Thursday, September 18, 2014 11:54 AM To: outages@outages.org Subject: Re: [outages] Possible XO/ATT peering issue in Chicago

For what it is worth, here is the other direction.

Host Loss% Last Avg Best Wrst StDev 4. cr2.cgcil.ip.att.net 0.0% 7.8 8.4 6.4 13.7 1.3 5. gar13.cgcil.ip.att.net 0.0% 8.5 8.7 6.6 33.6 2.2 6. 206.111.2.89.ptr.us.xo.net 17.9% 38.0 38.3 37.3 75.4 3.0 7. 207.88.14.193.ptr.us.xo.net 13.7% 44.7 45.7 43.2 48.8 1.2 8. ae1d0.mcr1.southfield-mi.us.xo.net 13.7% 44.5 46.5 43.2 104.6 8.2

- - Mitchell

On Thu, Sep 18, 2014 at 12:34 PM, Bill Wichers via Outages <outages@outages.org> wrote:

...

Is anyone else seeing excessive packet loss between ATT and XO in Chicago? We started seeing this this morning, and the problem looks like it might actually be within XO’s network in Chicago but I’m not 100% certain. I have no info from either carrier as of yet.

Hop #6 is 216.156.0.185

Hop #9 is 12.122.133.34

Hop #10 in the lower mtr output is 12.122.2.206

Packets Pings

Host Loss% Last Avg Best Wrst StDev

5. ae1d0.mcr1.southfield-mi.us.xo.net 0.0% 8.0 10.8 6.6 48.9 7.0

6. vb1730.rar3.chicago-il.us.xo.net 0.0% 9.8 10.0 6.6 14.7 1.4

7. 207.88.14.194.ptr.us.xo.net 46.4% 7.9 8.4 6.5 44.9 3.7

8. 206.111.2.90.ptr.us.xo.net 12.9% 39.5 41.8 38.4 60.6 1.9

9. cr1.cgcil.ip.att.net 15.5% 49.7 49.6 46.3 52.8 1.5

10. 12.122.99.26 17.8% 49.9 48.4 45.0 51.5 1.4

And

5. ae1d0.mcr1.southfield-mi.us.xo.net 0.0% 7.7 9.6 6.5 80.1 7.3

6. vb1730.rar3.chicago-il.us.xo.net 0.0% 10.4 9.0 6.6 11.6 1.3

7. 207.88.14.194.ptr.us.xo.net 44.4% 6.6 7.5 6.5 32.8 2.9

8. 206.111.2.90.ptr.us.xo.net 12.1% 41.5 41.4 38.1 64.2 2.1

9. cr1.cgcil.ip.att.net 14.2% 65.4 66.2 63.8 76.0 1.5

10. cr1.cl2oh.ip.att.net 15.2% 67.5 67.0 64.0 123.6 4.7

11. cr2.nsvtn.ip.att.net 15.2% 69.3 68.6 64.3 163.8 9.3

-Bill