Started seeing High Latency and Packetloss on a Comcast Business Circuit in the far Western Chicago Suburbs. Started around 11:50am CST. Looks like it might be at a cross connect between ATT and Comcast downtown Chicago. Thanks, Nate Burke Blast Communications Tracing route to 70-91-246-197-BusName-Illinois.hfc.comcastbusiness.net [70.91.246.197] over a maximum of 30 hops: 1 <1 ms <1 ms <1 ms fw1.border.core.router.blastcomm.com [66.151.17.2] 2 <1 ms <1 ms <1 ms 216.152.48.61 3 <1 ms <1 ms <1 ms noc-multihome.core1.blastcomm.com [66.151.18.109] 4 <1 ms 1 ms 2 ms gi03.border1.blastcomm.com [66.151.18.245] 5 3 ms 3 ms 3 ms border6.ge2-9.blast-10.chg.pnap.net [64.74.108.213] 6 4 ms 3 ms 3 ms core5.te2-1-bbnet1.chg.pnap.net [64.94.32.14] 7 3 ms 3 ms 4 ms 12.94.24.9 8 7 ms 8 ms 8 ms cr2.cgcil.ip.att.net [12.122.132.246] 9 4 ms 3 ms 3 ms gar1.clboh.ip.att.net [12.122.133.105] 10 149 ms * 150 ms 192.205.37.22 11 152 ms 152 ms 151 ms pos-2-2-0-0-cr01.350ecermak.il.ibone.comcast.net [68.86.86.77] 12 149 ms 150 ms 149 ms pos-0-14-0-0-ar01.area4.il.chicago.comcast.net [68.86.90.46] 13 152 ms 153 ms 152 ms te-2-3-ur04.wchicago.il.chicago.comcast.net [68.87.210.146] 14 153 ms 153 ms 153 ms te-3-4-sr01.dekalb.il.chicago.comcast.net [68.87.210.153] 15 * 159 ms * 68.86.115.134 16 167 ms 175 ms 163 ms 73.50.3.199 17 * 163 ms 161 ms 70-91-246-197-BusName-Illinois.hfc.comcastbusiness.net [70.91.246.197] Trace complete. Reverse Trace ADDRESS STATUS 1 70.91.246.198 1ms 1ms 1ms 2 73.49.236.1 17ms 24ms 20ms 3 68.86.115.141 19ms 9ms 9ms 4 68.87.210.154 9ms 8ms 10ms 5 68.87.210.145 12ms 40ms 12ms 6 68.86.93.181 22ms 17ms 16ms 7 68.86.88.93 12ms 12ms 12ms 8 68.86.87.114 34ms 33ms 33ms 9 68.86.88.218 64ms 53ms 50ms 10 68.86.88.98 47ms 53ms 49ms 11 66.110.9.101 156ms 161ms timeout 12 204.255.168.9 156ms 153ms 154ms 13 152.63.85.85 185ms 157ms 156ms 14 152.63.64.218 timeout 165ms 168ms 15 152.63.66.110 171ms 157ms 159ms 16 63.84.96.162 158ms 158ms timeout 17 64.94.32.11 timeout 159ms 158ms 18 64.74.108.214 161ms 161ms 161ms 19 66.151.16.1 161ms timeout timeout
On Mon, Dec 19, 2011 at 1:15 PM, Nate Burke <nate@blastcomm.com> wrote:
Started seeing High Latency and Packetloss on a Comcast Business Circuit in the far Western Chicago Suburbs. Started around 11:50am CST. Looks like it might be at a cross connect between ATT and Comcast downtown Chicago.
I'm seeing this here in Lebanon NH, affecting cross-town traffic between Fairpoint and Comcast: traceroute to 173.9.115.9 (173.9.115.9), 30 hops max, 60 byte packets 1 64.222.216.1 (64.222.216.1) 20.381 ms 21.252 ms 22.527 ms 2 pool-64-222-213-23.port.east.myfairpoint.net (64.222.213.23) 24.186 ms 25.071 ms 26.528 ms 3 xe-0-2-0.mpr4.bos2.us.above.net (64.124.202.29) 31.227 ms 32.431 ms 33.667 ms 4 xe-0-0-0.mpr3.bos2.us.above.net (64.125.25.61) 35.177 ms 36.522 ms 37.866 ms 5 xe-5-3-0.cr1.lga5.us.above.net (64.125.24.110) 45.022 ms 46.045 ms 46.985 ms 6 xe-0-0-0.cr2.lga5.us.above.net (64.125.29.42) 48.062 ms 29.675 ms 29.242 ms 7 xe-0-2-1.er2.lga5.us.above.net (64.125.30.210) 29.931 ms 30.970 ms 32.314 ms 8 above-comcast.lga5.us.above.net (64.125.12.210) 35.957 ms 36.227 ms 37.505 ms 9 pos-1-5-0-0-cr01.newyork.ny.ibone.comcast.net (68.86.85.17) 39.783 ms 43.557 ms 43.231 ms 10 pos-0-6-0-0-cr01.350ecermak.il.ibone.comcast.net (68.86.88.154) 62.634 ms 66.689 ms 66.438 ms 11 pos-1-6-0-0-cr01.chicago.il.ibone.comcast.net (68.86.88.41) 66.540 ms 67.105 ms 53.298 ms 12 * * * ... My pager didn't ring until 12:06, and my logs are on the other side of the problem, but wow, I'm still seeing this now, going on 40 minutes by your time schedule. -Bill
I had my provider (Internap) test from their 111 N. Canal facility in Chicago, and latency did not exist from that NAP, only at 350 Ceramc, on all upstreams that Internap connects to at 350 Cermac. Looks like it might be a busy router at 350 Cermac on Comcast. Nate On 12/19/2011 12:28 PM, Bill McGonigle wrote:
On Mon, Dec 19, 2011 at 1:15 PM, Nate Burke<nate@blastcomm.com> wrote:
Started seeing High Latency and Packetloss on a Comcast Business Circuit in the far Western Chicago Suburbs. Started around 11:50am CST. Looks like it might be at a cross connect between ATT and Comcast downtown Chicago. I'm seeing this here in Lebanon NH, affecting cross-town traffic between Fairpoint and Comcast:
traceroute to 173.9.115.9 (173.9.115.9), 30 hops max, 60 byte packets 1 64.222.216.1 (64.222.216.1) 20.381 ms 21.252 ms 22.527 ms 2 pool-64-222-213-23.port.east.myfairpoint.net (64.222.213.23) 24.186 ms 25.071 ms 26.528 ms 3 xe-0-2-0.mpr4.bos2.us.above.net (64.124.202.29) 31.227 ms 32.431 ms 33.667 ms 4 xe-0-0-0.mpr3.bos2.us.above.net (64.125.25.61) 35.177 ms 36.522 ms 37.866 ms 5 xe-5-3-0.cr1.lga5.us.above.net (64.125.24.110) 45.022 ms 46.045 ms 46.985 ms 6 xe-0-0-0.cr2.lga5.us.above.net (64.125.29.42) 48.062 ms 29.675 ms 29.242 ms 7 xe-0-2-1.er2.lga5.us.above.net (64.125.30.210) 29.931 ms 30.970 ms 32.314 ms 8 above-comcast.lga5.us.above.net (64.125.12.210) 35.957 ms 36.227 ms 37.505 ms 9 pos-1-5-0-0-cr01.newyork.ny.ibone.comcast.net (68.86.85.17) 39.783 ms 43.557 ms 43.231 ms 10 pos-0-6-0-0-cr01.350ecermak.il.ibone.comcast.net (68.86.88.154) 62.634 ms 66.689 ms 66.438 ms 11 pos-1-6-0-0-cr01.chicago.il.ibone.comcast.net (68.86.88.41) 66.540 ms 67.105 ms 53.298 ms 12 * * * ...
My pager didn't ring until 12:06, and my logs are on the other side of the problem, but wow, I'm still seeing this now, going on 40 minutes by your time schedule.
-Bill
I am seeing the same thing in Denver. upload speeds keep timing out. -Michael On Dec 19, 2011, at 11:47 AM, Nate Burke wrote:
I had my provider (Internap) test from their 111 N. Canal facility in Chicago, and latency did not exist from that NAP, only at 350 Ceramc, on all upstreams that Internap connects to at 350 Cermac. Looks like it might be a busy router at 350 Cermac on Comcast.
Nate
On 12/19/2011 12:28 PM, Bill McGonigle wrote:
On Mon, Dec 19, 2011 at 1:15 PM, Nate Burke<nate@blastcomm.com> wrote:
Started seeing High Latency and Packetloss on a Comcast Business Circuit in the far Western Chicago Suburbs. Started around 11:50am CST. Looks like it might be at a cross connect between ATT and Comcast downtown Chicago. I'm seeing this here in Lebanon NH, affecting cross-town traffic between Fairpoint and Comcast:
traceroute to 173.9.115.9 (173.9.115.9), 30 hops max, 60 byte packets 1 64.222.216.1 (64.222.216.1) 20.381 ms 21.252 ms 22.527 ms 2 pool-64-222-213-23.port.east.myfairpoint.net (64.222.213.23) 24.186 ms 25.071 ms 26.528 ms 3 xe-0-2-0.mpr4.bos2.us.above.net (64.124.202.29) 31.227 ms 32.431 ms 33.667 ms 4 xe-0-0-0.mpr3.bos2.us.above.net (64.125.25.61) 35.177 ms 36.522 ms 37.866 ms 5 xe-5-3-0.cr1.lga5.us.above.net (64.125.24.110) 45.022 ms 46.045 ms 46.985 ms 6 xe-0-0-0.cr2.lga5.us.above.net (64.125.29.42) 48.062 ms 29.675 ms 29.242 ms 7 xe-0-2-1.er2.lga5.us.above.net (64.125.30.210) 29.931 ms 30.970 ms 32.314 ms 8 above-comcast.lga5.us.above.net (64.125.12.210) 35.957 ms 36.227 ms 37.505 ms 9 pos-1-5-0-0-cr01.newyork.ny.ibone.comcast.net (68.86.85.17) 39.783 ms 43.557 ms 43.231 ms 10 pos-0-6-0-0-cr01.350ecermak.il.ibone.comcast.net (68.86.88.154) 62.634 ms 66.689 ms 66.438 ms 11 pos-1-6-0-0-cr01.chicago.il.ibone.comcast.net (68.86.88.41) 66.540 ms 67.105 ms 53.298 ms 12 * * * ...
My pager didn't ring until 12:06, and my logs are on the other side of the problem, but wow, I'm still seeing this now, going on 40 minutes by your time schedule.
-Bill
Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
On 12/19/2011 11:53 AM, michael sterl wrote:
I am seeing the same thing in Denver. upload speeds keep timing out.
Also seeing issues getting to a customer in Philly (on Comcast) from our servers in Denver. on Cogent: [root@pegasus sigs]# mtr --report --report-cycles=10 --report-wide 50.73.191.41 HOST: pegasus.spameater.net Loss% Snt Last Avg Best Wrst StDev 1. ai-core.axint.net 0.0% 10 1.3 0.8 0.4 1.3 0.3 2. fa0-2.na01.b009854-0.den01.atlas.cogentco.com 0.0% 10 1.3 1.6 0.9 4.0 0.9 3. vl3509.mag02.den01.atlas.cogentco.com 0.0% 10 175.7 37.6 0.8 182.0 74.5 4. te7-4.mpd02.den01.atlas.cogentco.com 0.0% 10 1.8 1.6 0.8 4.9 1.2 5. te3-1.ccr01.den03.atlas.cogentco.com 0.0% 10 171.9 61.4 2.2 193.3 80.4 6. comcast.den03.atlas.cogentco.com 0.0% 10 1.5 2.7 1.5 7.9 1.9 7. pos-0-10-0-0-cr01.chicago.il.ibone.comcast.net 0.0% 10 27.7 27.5 26.9 28.1 0.4 8. pos-1-10-0-0-cr01.350ecermak.il.ibone.comcast.net 0.0% 10 30.1 31.2 28.2 43.4 4.4 9. pos-4-10-0-0-cr01.newyork.ny.ibone.comcast.net 0.0% 10 50.4 48.4 46.2 50.4 1.5 10. pos-0-15-0-0-ar03.pittsburgh.pa.pitt.comcast.net 0.0% 10 56.8 56.9 54.9 59.0 1.2 11. te-3-3-ur01.pittsburgh.pa.pitt.comcast.net 0.0% 10 55.6 55.0 54.3 55.6 0.5 12. ge-3-1-0-ten05.pittsburgh.pa.pitt.comcast.net 0.0% 10 53.0 53.5 52.8 55.7 0.9 13. c-98-219-185-46.hsd1.pa.comcast.net 0.0% 10 65.5 64.7 59.8 74.5 5.2 14. ??? 100.0 10 0.0 0.0 0.0 0.0 0.0 Chris
I'm going to guess they lost some capacity, as evidenced by these round trip times between New York and Hartford: http://i.imgur.com/j80xE.png and the lack of direct path taken here: [...] 6. vlan90.csw4.NewYork1.Level3.net 0.0% 226 2.7 6.6 0.9 14.3 4.5 7. ae-92-92.ebr2.NewYork1.Level3.net 0.0% 226 1.5 1.8 1.1 2.6 0.3 8. ae-6-6.ebr2.NewYork2.Level3.net 0.0% 226 1.4 1.6 0.9 11.5 0.8 9. ae-2-52.edge1.NewYork2.Level3.net 0.0% 226 1.5 3.4 1.1 83.7 8.8 10. COMCAST-IP.edge1.NewYork2.Level3.net 0.0% 226 5.3 5.2 2.5 21.7 2.4 11. pos-0-10-0-0-cr01.350ecermak.il.ibone.comcast.net 0.0% 226 23.7 24.2 21.6 110.7 5.9 12. pos-0-3-0-0-cr01.chicago.il.ibone.comcast.net 0.0% 226 21.2 21.3 20.7 22.3 0.3 13. pos-3-15-0-0-ar01.woburn.ma.boston.comcast.net 0.0% 226 52.2 52.3 51.4 53.5 0.3 14. pos-0-0-0-0-ar01.needham.ma.boston.comcast.net 0.0% 226 53.0 53.0 52.1 71.6 1.4 [...] HTH, -a
On 12/19/2011 01:28 PM, Bill McGonigle wrote:
I'm seeing this here in Lebanon NH, affecting cross-town traffic between Fairpoint and Comcast:
Mine came back around 13:49 Eastern. The link from Chicago to Mass. came back up: Packets Pings 11. pos-1-6-0-0-cr01.chicago.il.ibone.comcast.net 1.2% 1031 52.4 54.3 51.4 228.5 12.9 12. pos-3-15-0-0-ar01.woburn.ma.boston.comcast.net 95.8% 1031 82.3 87.8 81.5 203.4 21.6 -Bill
This looks resolved from my prospectives as well. I'd imagine the Comcast folk on this list will be able to provide a more detailed RFO once the dust has settled and a root cause is known. -a
From Within Chicago, still seeing issues. More specifically any comcast destined traffic going through 350 Cermac, latency stays constant at 160ms, but packetloss has significantly increased in last 30 min. Nate On 12/19/2011 2:39 PM, Adam Rothschild wrote:
This looks resolved from my prospectives as well. I'd imagine the Comcast folk on this list will be able to provide a more detailed RFO once the dust has settled and a root cause is known.
-a _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
I'm at a small college in the South West corner of Mass. We lost our main link from campus to Boston this morning, with service restored at 13:26 Eastern. Our local provider described it as "fiber cut due to a fire at our carrier's location between Pittsfield, MA and Springfield, MA". Our monitoring showed the link going down between 06:14 and 06:19 Eastern. I'm guessing based on the restore timing and location that your Fairpoint/Comcast link might have been caused by the same break, although our traceroutes show Global Crossing as our carrier from here to Boston. Not sure how many of them use that same fiber... Steve Bohrer Network Admin Bard College at Simon's Rock 413-528-7645 On Dec 19, 2011, at 2:57 PM, Bill McGonigle wrote:
On 12/19/2011 01:28 PM, Bill McGonigle wrote:
I'm seeing this here in Lebanon NH, affecting cross-town traffic between Fairpoint and Comcast:
Mine came back around 13:49 Eastern. The link from Chicago to Mass. came back up: Packets Pings 11. pos-1-6-0-0- cr01 .chicago .il.ibone.comcast.net 1.2% 1031 52.4 54.3 51.4 228.5 12.9 12. pos-3-15-0-0- ar01 .woburn .ma.boston.comcast.net 95.8% 1031 82.3 87.8 81.5 203.4 21.6
-Bill _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
A lot of Global Crossing and at least some of the Comcast long haul fiber rides IRUs in Level(3)'s physical cable... -Bill
I'm guessing based on the restore timing and location that your Fairpoint/Comcast link might have been caused by the same break, although our traceroutes show Global Crossing as our carrier from here to Boston. Not sure how many of them use that same fiber...
I'm seeing weird topology from comcast, primarily taking traffic from Chicago and carrying it to Miami while the return path stays something short. I'm thinking there is something wrong here as Miami is certainly the wrong place to take traffic to divert from Chicago. NYC/Ashburn would seem to be a better fit, or even Dallas. - Jared On Dec 19, 2011, at 1:15 PM, Nate Burke wrote:
Started seeing High Latency and Packetloss on a Comcast Business Circuit in the far Western Chicago Suburbs. Started around 11:50am CST. Looks like it might be at a cross connect between ATT and Comcast downtown Chicago.
Thanks, Nate Burke Blast Communications
Tracing route to 70-91-246-197-BusName-Illinois.hfc.comcastbusiness.net [70.91.246.197] over a maximum of 30 hops:
1 <1 ms <1 ms <1 ms fw1.border.core.router.blastcomm.com [66.151.17.2] 2 <1 ms <1 ms <1 ms 216.152.48.61 3 <1 ms <1 ms <1 ms noc-multihome.core1.blastcomm.com [66.151.18.109] 4 <1 ms 1 ms 2 ms gi03.border1.blastcomm.com [66.151.18.245] 5 3 ms 3 ms 3 ms border6.ge2-9.blast-10.chg.pnap.net [64.74.108.213] 6 4 ms 3 ms 3 ms core5.te2-1-bbnet1.chg.pnap.net [64.94.32.14] 7 3 ms 3 ms 4 ms 12.94.24.9 8 7 ms 8 ms 8 ms cr2.cgcil.ip.att.net [12.122.132.246] 9 4 ms 3 ms 3 ms gar1.clboh.ip.att.net [12.122.133.105] 10 149 ms * 150 ms 192.205.37.22 11 152 ms 152 ms 151 ms pos-2-2-0-0-cr01.350ecermak.il.ibone.comcast.net [68.86.86.77] 12 149 ms 150 ms 149 ms pos-0-14-0-0-ar01.area4.il.chicago.comcast.net [68.86.90.46] 13 152 ms 153 ms 152 ms te-2-3-ur04.wchicago.il.chicago.comcast.net [68.87.210.146] 14 153 ms 153 ms 153 ms te-3-4-sr01.dekalb.il.chicago.comcast.net [68.87.210.153] 15 * 159 ms * 68.86.115.134 16 167 ms 175 ms 163 ms 73.50.3.199 17 * 163 ms 161 ms 70-91-246-197-BusName-Illinois.hfc.comcastbusiness.net [70.91.246.197]
Trace complete.
Reverse Trace
ADDRESS STATUS 1 70.91.246.198 1ms 1ms 1ms 2 73.49.236.1 17ms 24ms 20ms 3 68.86.115.141 19ms 9ms 9ms 4 68.87.210.154 9ms 8ms 10ms 5 68.87.210.145 12ms 40ms 12ms 6 68.86.93.181 22ms 17ms 16ms 7 68.86.88.93 12ms 12ms 12ms 8 68.86.87.114 34ms 33ms 33ms 9 68.86.88.218 64ms 53ms 50ms 10 68.86.88.98 47ms 53ms 49ms 11 66.110.9.101 156ms 161ms timeout 12 204.255.168.9 156ms 153ms 154ms 13 152.63.85.85 185ms 157ms 156ms 14 152.63.64.218 timeout 165ms 168ms 15 152.63.66.110 171ms 157ms 159ms 16 63.84.96.162 158ms 158ms timeout 17 64.94.32.11 timeout 159ms 158ms 18 64.74.108.214 161ms 161ms 161ms 19 66.151.16.1 161ms timeout timeout
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
participants (8)
-
Adam Rothschild -
Bill McGonigle -
Bill Wichers -
Chris Stone -
Jared Mauch -
michael sterl -
Nate Burke -
Steve Bohrer