January 2013 - Outages - Outages Lists

XO - Dallas
by Mauricio Lizano 23 Jan '13

23 Jan '13

Did XO just blow up their network again? I am seeing tons on stuff down, cant hit xo.com and their 800 number is ringing and ringing, no IVR. Thanks, -- Mauricio Lizano 8821-6575 http://cr.linkedin.com/pub/mauricio-lizano/14/162/5b5

5 4

Level3 from LA to Korea down?
by Chris Stone 23 Jan '13

23 Jan '13

Have a customer that does a lot of email to Korea and saw their mail backing up this morning due to 'host not found' errors to a domain they email frequently. Figured they let their domain expire, but in checking it out, I find I cannot query the name servers for their domain, which are ns[1234].whoisdomain.kr. Traffic to them dies on Level3 in LA: traceroute to 211.206.125.155 (211.206.125.155), 30 hops max, 60 byte packets 1 ge-6-24-576.car1.Denver1.Level3.net (4.28.16.137) 85.727 ms 85.724 ms 85.799 ms 2 vlan51.ebr1.Denver1.Level3.net (4.69.147.94) 35.938 ms 35.952 ms 35.948 ms 3 ae-3-3.ebr2.SanJose1.Level3.net (4.69.132.57) 36.016 ms 35.953 ms 35.963 ms 4 ae-2-2.ebr2.SanJose5.Level3.net (4.69.148.141) 36.122 ms * 40.685 ms 5 ae-6-6.ebr2.LosAngeles1.Level3.net (4.69.148.201) 36.202 ms 36.222 ms 36.203 ms 6 ae-62-62.csw1.LosAngeles1.Level3.net (4.69.137.18) 36.225 ms ae-82-82.csw3.LosAngeles1.Level3.net (4.69.137.26) 36.263 ms ae-92-92.csw4.LosAngeles1.Level3.net (4.69.137.30) 35.944 ms 7 ae-2-70.edge2.LosAngeles9.Level3.net (4.69.144.79) 35.992 ms ae-3-80.edge2.LosAngeles9.Level3.net (4.69.144.143) 36.094 ms 36.105 ms 8 * * * 9 * * * 10 * * * 11 * * * 12 * * * 13 * * * 14 * * * 15 * * * 16 * * * 17 * * * 18 * * * 19 * * * 20 * * * 21 * * * 22 * * * 23 * * * 24 * * * 25 * * * 26 * * * 27 * * * 28 * * * 29 * * * 30 * * * traceroute to 211.206.125.155 (211.206.125.155), 30 hops max, 60 byte packets 1 gateway.tbie.net (10.10.100.1) 4.191 ms 4.195 ms 4.203 ms 2 hlrn-dsl-gw01.hlrn.qwest.net (207.225.112.1) 44.353 ms 45.995 ms 47.282 ms 3 hlrn-agw1.inet.qwest.net (71.217.188.1) 48.971 ms 49.917 ms 51.637 ms 4 dvr-brdr-02.inet.qwest.net (67.14.24.118) 53.619 ms 55.320 ms 56.269 ms 5 63.146.26.134 (63.146.26.134) 77.074 ms 78.090 ms 79.713 ms 6 vlan51.ebr1.Denver1.Level3.net (4.69.147.94) 99.612 ms 95.310 ms 95.268 ms 7 ae-3-3.ebr2.SanJose1.Level3.net (4.69.132.57) 97.331 ms 75.825 ms 76.472 ms 8 ae-2-2.ebr2.SanJose5.Level3.net (4.69.148.141) 80.746 ms 78.862 ms 76.726 ms 9 ae-6-6.ebr2.LosAngeles1.Level3.net (4.69.148.201) 76.382 ms 77.211 ms 76.548 ms 10 ae-72-72.csw2.LosAngeles1.Level3.net (4.69.137.22) 76.767 ms ae-62-62.csw1.LosAngeles1.Level3.net (4.69.137.18) 76.151 ms ae-82-82.csw3.LosAngeles1.Level3.net (4.69.137.26) 77.360 ms 11 ae-1-60.edge2.LosAngeles9.Level3.net (4.69.144.15) 75.848 ms ae-2-70.edge2.LosAngeles9.Level3.net (4.69.144.79) 75.706 ms ae-4-90.edge2.LosAngeles9.Level3.net (4.69.144.207) 77.386 ms 12 * * * 13 * * * 14 * * * 15 * * * 16 * * * 17 * * * 18 * * * 19 * * * 20 * * * 21 * * * 22 * * * 23 * * * 24 * * * 25 * * * 26 * * * 27 * * * 28 * * * 29 * * * 30 * * * Anyone aware of any issues on that path to Asia? Thanks Chris -- Chris Stone, MCSE Vice President, CTO AxisInternet, Inc. http://www.axint.net DSL, VoIP, hosting, email filtering, co-location, online backup & more **************************************************************** Authorized ScopTel IP PBX US Reseller - www.scopserv.com The most complete and comprehensive GUI on the market for telephony server management interface based on Voice over Internet Protocol (VoIP) ****************************************************************

8 9

Re: [outages] Charter <-> Cogent issues @ Chicago?
by Chris Wopat 23 Jan '13

23 Jan '13

> > Anyone seeing issues between Charter and Cogent in the Chicago area? I've narrowed a severe packet loss issue down to a handoff there using multiple route servers and looking glasses. > > traceroute to 68.115.40.226 (68.115.40.226), 30 hops max, 60 byte packets > 1 vl3.mag02.dfw01.atlas.cogentco.com (66.250.250.81) 32.175 ms 32.237 ms > 2 te0-2-0-0.ccr21.dfw01.atlas.cogentco.com (154.54.46.173) 0.612 ms 0.620 ms > 3 te0-3-0-0.mpd22.mci01.atlas.cogentco.com (154.54.25.206) 10.548 ms te0-3-0-6.mpd21.mci01.atlas.cogentco.com (154.54.46.197) 10.596 ms > 4 te0-3-0-2.mpd21.ord01.atlas.cogentco.com (154.54.7.137) 22.827 ms te0-1-0-0.ccr21.mci01.atlas.cogentco.com (154.54.29.230) 10.620 ms > 5 te0-2-0-0.ccr21.ord01.atlas.cogentco.com (154.54.25.177) 22.731 ms te0-1-0-0.ccr22.ord01.atlas.cogentco.com (154.54.6.178) 22.627 ms > 6 38.122.180.26 (38.122.180.26) 22.933 ms 22.960 ms > 7 bbr01chcgil-bue-6.chcg.il.charter.com (96.34.0.66) 31.500 ms * > 8 bbr02euclwi-bue-1.eucl.wi.charter.com (96.34.0.8) 36.465 ms * > 9 crr02euclwi-tge-0-2-0-6.eucl.wi.charter.com (96.34.2.147) 33.725 ms * > 10 * crr01euclwi-tge-0-1-0-2.eucl.wi.charter.com (96.34.16.6) 31.081 ms > 11 crr01onlswi-tge-0-1-0-0.onls.wi.charter.com (96.34.17.207) 34.721 ms * > 12 dtr01onlswi-tge-0-4-1-3.onls.wi.charter.com (96.34.19.167) 33.614 ms * > 13 * cts02onlswi-tge-1-0-0.onls.wi.charter.com (96.34.22.47) 33.114 ms > 14 10.134.188.87 (10.134.188.87) 38.558 ms 38.421 ms > 15 * We peer with charter and also see issues. They just sent us this: ======================== Issue Description / Summary : Problems in the WI / MN / MI area reaching certain sites on the internet. Charter Communications has opened a ticket with upstream provider to investigate routing issues. Start Date & Time: 1:00 PM ET Parent Ticket Number: 4275241 Type of Outage and Severity Level: Degraded No ETR available ======================== --Chris

1 0

Charter <-> Cogent issues @ Chicago?
by Vinny_Abello＠Dell.com 23 Jan '13

23 Jan '13

Anyone seeing issues between Charter and Cogent in the Chicago area? I've narrowed a severe packet loss issue down to a handoff there using multiple route servers and looking glasses. traceroute to 68.115.40.226 (68.115.40.226), 30 hops max, 60 byte packets 1 vl3.mag02.dfw01.atlas.cogentco.com (66.250.250.81) 32.175 ms 32.237 ms 2 te0-2-0-0.ccr21.dfw01.atlas.cogentco.com (154.54.46.173) 0.612 ms 0.620 ms 3 te0-3-0-0.mpd22.mci01.atlas.cogentco.com (154.54.25.206) 10.548 ms te0-3-0-6.mpd21.mci01.atlas.cogentco.com (154.54.46.197) 10.596 ms 4 te0-3-0-2.mpd21.ord01.atlas.cogentco.com (154.54.7.137) 22.827 ms te0-1-0-0.ccr21.mci01.atlas.cogentco.com (154.54.29.230) 10.620 ms 5 te0-2-0-0.ccr21.ord01.atlas.cogentco.com (154.54.25.177) 22.731 ms te0-1-0-0.ccr22.ord01.atlas.cogentco.com (154.54.6.178) 22.627 ms 6 38.122.180.26 (38.122.180.26) 22.933 ms 22.960 ms 7 bbr01chcgil-bue-6.chcg.il.charter.com (96.34.0.66) 31.500 ms * 8 bbr02euclwi-bue-1.eucl.wi.charter.com (96.34.0.8) 36.465 ms * 9 crr02euclwi-tge-0-2-0-6.eucl.wi.charter.com (96.34.2.147) 33.725 ms * 10 * crr01euclwi-tge-0-1-0-2.eucl.wi.charter.com (96.34.16.6) 31.081 ms 11 crr01onlswi-tge-0-1-0-0.onls.wi.charter.com (96.34.17.207) 34.721 ms * 12 dtr01onlswi-tge-0-4-1-3.onls.wi.charter.com (96.34.19.167) 33.614 ms * 13 * cts02onlswi-tge-1-0-0.onls.wi.charter.com (96.34.22.47) 33.114 ms 14 10.134.188.87 (10.134.188.87) 38.558 ms 38.421 ms 15 * * -Vinny

3 2

Level 3 issues
by Jeremie Chism 23 Jan '13

23 Jan '13

Seeing intermittent issues with level 3 router at 64.208.27.65 Hop (ms) (ms) (ms) IP Address Host name 1 0 0 0 8.9.232.73 xe-5-3-0.edge3.dallas1.level3.net 2 0 0 0 4.69.145.141 ae-3-80.edge4.dallas3.level3.net 3 8 0 0 4.68.63.146 gblx-level3-40g.dallas3.level3.net 4 0 0 0 67.16.150.190 ae2-80g.csr1.dal2.gblx.net 5 0 0 0 64.208.27.65 te1-3-10g.ar3.dal2.gblx.net Hop (ms) (ms) (ms) IP Address Host name 1 1 0 0 8.9.232.73 xe-5-3-0.edge3.dallas1.level3.net 2 0 0 0 4.69.145.141 ae-3-80.edge4.dallas3.level3.net 3 0 0 0 4.68.63.146 gblx-level3-40g.dallas3.level3.net 4 0 0 0 67.16.150.190 ae2-80g.csr1.dal2.gblx.net 5 7 1 208 64.208.27.65 te1-3-10g.ar3.dal2.gblx.net Trace complete Ping times from 8ms to 330ms. -- Jeremie Chism Triton Communications

7 6

Wikipedia Migration Complete
by Jay Ashworth 23 Jan '13

23 Jan '13

They got it all done last night; cancelling tonight and tomorrow night's windows. Not bad for the #5 site on the web. Cheers, -- jra ----- Forwarded Message ----- > From: "Ct Woo" <ctwoo(a)wikimedia.org> > To: "Wikimedia developers" <wikitech-l(a)lists.wikimedia.org>, "Wikimedia Engineering" <engineering(a)lists.wikimedia.org> > Sent: Tuesday, January 22, 2013 5:53:42 PM > Subject: [Wikitech-l] Completed ! Re: More Update on Ashburn data center switchover / migration – target date is week > of 1/22/13 > All, > > The switchover work is done. > > The site was was available to readers throughout the migration work > though > it was in read-only mode for about 32 minutes, when Asher and Mark had > to > migrate the database masters over from Tampa to Ashburn. > > We will cancel the reminding maintenance windows. > > Thank you all for your patience and understanding. > > Regards, > CT Woo > > On Sat, Jan 19, 2013 at 10:57 AM, Ct Woo <ctwoo(a)wikimedia.org> wrote: > > > All, > > > > We will be proceeding with the datacenter switchover plan this > > coming > > Tuesday (Jan 22, 2013), unless we discover some unexpected and > > insurmountable issues in our tests between now and then. > > > > During the 8-hour migration window on the 22nd, 23rd and 24th (from > > 17:00 > > UTC to 01:00 UTC hours / 9am to 5pm PST), there would be times > > (lasting > > about 30 minutes) where the site would be set to "read-only" mode, > > to > > facilitate master database switchovers from one datacenter to > > another. > > While the site should be available to readers, no new contents could > > be > > created, edited or uploaded. > > > > We are aware of the inconvenience and we have put together plans to > > minimize such annoyances, e.g., automating much of the procedures, > > mitigating known risks, and performing tests to identify issues > > prior to > > deployment. Given the scale and complexity of this migration, we do > > realize > > not all operational impact is predictable. Some users could > > experience > > intermittent site unavailability and/or performance issues > > unfortunately. > > > > You can follow the migration on > > chat.freenode.net<http://irc.freenode.net> > > (and not irc.freenode.org as mentioned in previous email) in the > > #wikimedia-operations channel. > > > > Thanks, > > CT Woo > > > > ---------- Forwarded message ---------- > > From: Ct Woo <ctwoo(a)wikimedia.org> > > Date: Fri, Jan 11, 2013 at 12:07 PM > > Subject: Update on Ashburn data center switchover / migration – > > target > > date is week of 1/22/13 > > To: Wikimedia developers <wikitech-l(a)lists.wikimedia.org>, > > Development > > and Operations Engineers <engineering(a)lists.wikimedia.org> > > > > > > All, > > > > The Migration team is in the last lap on completing the remaining > > tasks to > > ready our software stack and Ashburn infrastructure for the big > > switchover day. > > > > Per my last > > update,<http://lists.wikimedia.org/pipermail/wikitech-l/2012-October/063668.html> > > with the Fundraising activity behind us now, the team has scheduled > > the *week > > of 22nd January*, 2013 to perform the switchover. We are going to > > block a > > 8-hour migration window on the *22nd, 23rd and 24**th*. During those > > periods, *17:00 UTC to 01:00 UTC hours (9am to 5pm PST*), there will > > be > > intermittent blackouts and they will be treated as 'planned' > > outages. You > > can follow the migration on irc.freenode.org in the > > #wikimedia-operations > > channel. > > > > The team is putting the finishing touches to the last few tasks and > > we > > will make the final Go/No decision on 18th Jan, 2013. An update will > > send > > out then. For those interested in tracking the progress, the meeting > > notes > > are captured on this wikitech > > page<http://wikitech.wikimedia.org/view/Eqiad_Migration_Planning#Improving_Switc…> > > . > > > > *Please note that we will be restricting code deployment during that > > week, allowing only emergency and critical ones only.* > > > > Thanks. > > > > CT Woo > > > > > > > > > > > _______________________________________________ > Wikitech-l mailing list > Wikitech-l(a)lists.wikimedia.org > https://lists.wikimedia.org/mailman/listinfo/wikitech-l -- Jay R. Ashworth Baylink jra(a)baylink.com Designer The Things I Think RFC 2100 Ashworth & Associates http://baylink.pitas.com 2000 Land Rover DII St Petersburg FL USA #natog +1 727 647 1274

1 0

PacWest Telecom - one-way audio
by Jay Hennigan 22 Jan '13

22 Jan '13

Seems to be affecting multiple trunks in California. Can't even call their NOC due to the problem. Problem since 8:30 AM PST this morning. -- Jay Hennigan - CCIE #7880 - Network Engineering - jay(a)impulse.net Impulse Internet Service - http://www.impulse.net/ Your local telephone and internet company - 805 884-6323 - WB6RDV

3 2

Macquarie Telecom SAN? issues.
by Martin Hepworth 22 Jan '13

22 Jan '13

All currently seeing a big outage down at MT datacentre1 in Sydney. Master ticket 604473. Seems to be a SAN fabric or switch issue that's affecting lots of customers (again). -- Martin Hepworth, CISSP Oxford, UK

1 0

Re: [outages] Bay Area packet loss at Comcast (gblx.net, he.net)
by Constantine A. Murenin 19 Jan '13

19 Jan '13

On Sat, Jan 19, 2013 at 11:52:04AM -0800, Constantine A. Murenin wrote: > On Sat, 19 Jan 2013 at 11:04:14 -0800, Dylan N wrote: > > I get packetloss on te-1-12-0-0-ar01.sfsutro.ca.sfba.comcast.net to my > > linode @ fmt at weekends and at nights, looks more like just too much > > load rather than an outage > > In my case, all packet loss seem to be within gblx.net > (e.g. no packet loss between Comcast and hetzner.de). > > What I find strange is that the latency is still very good, > even when there is a significant packet loss. > > But my ssh connection between Comcast and Linode/Fremont is completely > unusable for several seconds at a time during these events. > > > The loss seems to have continued throughout last night: > > 2013-01-19T01:50:04-0800 > HOST: li163-### Snt Rcv Loss% Best Avg Wrst StDev > 1. router1-fmt.linode.com 600 600 0.0% 0.5 6.6 268.8 18.5 > 2. 10gigabitethernet2-3.core1.fmt1.he.net 600 600 0.0% 0.5 12.8 1385. 66.3 > 3. 10gigabitethernet1-2.core1.sjc2.he.net 600 600 0.0% 0.7 4.4 28.9 5.1 > 4. Port-channel100.ar3.SJC2.gblx.net 600 600 0.0% 0.8 28.2 234.0 54.8 > 5. po6-20G.ar4.SJC2.gblx.net 600 579 3.5% 0.9 32.1 888.1 66.0 > 6. lag8.ar5.SJC2.gblx.net 600 570 5.0% 0.9 4.4 20.9 5.1 > 7. 208.178.58.2 600 572 4.7% 1.8 4.1 14.4 1.3 > 8. pos-2-7-0-0-cr01.sanjose.ca.ibone.comcast.net 600 575 4.2% 4.8 7.7 43.1 2.9 > 9. pos-1-4-0-0-cr01.sacramento.ca.ibone.comcast.net 600 574 4.3% 7.1 9.3 20.2 1.5 > 10. te-7-3-ar02.saltlakecity.ut.utah.comcast.net 600 573 4.5% 7.3 11.9 115.7 17.6 > 11. te-3-1-ur08.sacramento.ca.ccal.comcast.net 600 574 4.3% 7.5 8.3 155.6 6.6 > 12. te-1-0-0-ten05.sacramento.ca.ccal.comcast.net 600 581 3.2% 14.1 14.3 25.1 1.3 > 13. 74-93-180-##-Sacramento.hfc.comcastbusiness.net 600 572 4.7% 14.1 20.5 83.9 8.8 > 2013-01-19T02:00:04-0800 > > 2013-01-19T01:50:44-0800 > HOST: hype.####.########## Snt Rcv Loss% Best Avg Wrst StDev > 1. logout.lan 60 60 0.0% 2.1 7.1 35.2 6.5 > 2. 192.168.1.1 60 60 0.0% 2.5 8.6 41.3 6.7 > 3. 10.1.10.1 60 60 0.0% 3.1 10.3 49.8 8.4 > 4. 96.120.14.165 60 60 0.0% 11.3 22.7 75.3 11.7 > 5. te-6-5-ur08.sacramento.ca.ccal.comcast.net 60 60 0.0% 12.3 22.8 66.1 12.1 > 6. te-0-2-0-2-ar03.sacramento.ca.sacra.comcast.net 60 60 0.0% 11.8 26.3 101.6 18.3 > 7. pos-0-3-0-0-cr01.sacramento.ca.ibone.comcast.net 60 60 0.0% 14.8 26.8 85.3 14.1 > 8. pos-0-3-0-0-cr01.sanjose.ca.ibone.comcast.net 60 60 0.0% 15.6 28.2 71.6 10.3 > 9. pos-0-1-0-0-pe01.11greatoaks.ca.ibone.comcast.net 60 60 0.0% 20.8 30.6 92.6 12.2 > 10. 208.178.58.1 60 60 0.0% 18.6 34.8 98.6 16.6 > 11. po4-20G.ar2.SJC2.gblx.net 60 60 0.0% 17.5 61.6 273.3 66.9 > 12. po1-20G.ar3.SJC2.gblx.net 60 47 21.7% 17.8 48.2 238.7 51.7 > 13. Hurrican-Electric-LLC.Port-channel100.ar3.SJC2.gblx.net 60 44 26.7% 17.1 30.5 69.8 11.6 > 14. 10gigabitethernet1-1.core1.fmt1.he.net 60 43 28.3% 21.5 55.2 598.8 87.9 > 15. router1-fmt.linode.com 60 44 26.7% 18.3 44.0 187.1 34.4 > 16. li163-###.members.linode.com 60 43 28.3% 17.8 29.0 68.1 8.6 > 2013-01-19T01:51:55-0800 It's been a 61% packet loss in the last minute... Ridiculous. 2013-01-19T12:15:15-0800 HOST: hype.####.########## Snt Rcv Loss% Best Avg Wrst StDev 1. logout.lan 60 60 0.0% 2.1 18.8 222.7 50.1 2. 192.168.1.1 60 60 0.0% 2.3 20.4 421.0 65.5 3. 10.1.10.1 60 60 0.0% 3.0 22.9 357.1 57.2 4. 96.120.14.165 60 60 0.0% 9.6 29.3 306.7 54.4 5. te-6-5-ur08.sacramento.ca.ccal.comcast.net 60 60 0.0% 10.3 34.2 265.3 51.9 6. te-0-2-0-2-ar03.sacramento.ca.sacra.comcast.net 60 60 0.0% 9.8 33.1 185.6 43.3 7. pos-0-4-0-0-cr01.sacramento.ca.ibone.comcast.net 60 60 0.0% 11.8 32.3 234.3 42.8 8. pos-0-3-0-0-cr01.sanjose.ca.ibone.comcast.net 60 60 0.0% 15.6 31.7 282.1 41.5 9. pos-0-1-0-0-pe01.11greatoaks.ca.ibone.comcast.net 60 60 0.0% 18.1 36.0 221.3 38.0 10. 208.178.58.1 60 60 0.0% 17.1 40.8 366.8 51.5 11. po4-20G.ar2.SJC2.gblx.net 60 59 1.7% 16.8 58.3 291.6 68.0 12. po1-20G.ar3.SJC2.gblx.net 60 46 23.3% 16.8 67.3 258.6 66.5 13. Hurrican-Electric-LLC.Port-channel100.ar3.SJC2.gblx.net 60 48 20.0% 17.3 34.9 155.8 32.0 14. 10gigabitethernet1-1.core1.fmt1.he.net 60 19 68.3% 18.3 50.4 117.8 32.2 15. router1-fmt.linode.com 60 22 63.3% 19.1 66.3 287.3 64.2 16. li163-###.members.linode.com 60 23 61.7% 18.3 42.5 213.8 48.7 2013-01-19T12:16:27-0800 Anyone from he.net / gblx.net / comcast.net to comment? Best regards, Constantine.

2 1

Bay Area packet loss at Comcast (gblx.net, he.net)
by Constantine A. Murenin 19 Jan '13

19 Jan '13

I think this happens practically every evening recently, for traffic between Comcast in NorCal and Linode at he.net in Fremont, around gblx.net. Today I have been monitoring through a crontab script since 16:00 PT, with a 10-minute mtr runs in a loop. The traffic loss has started in the 20:50/21:00 timeframe, and is still ongoing (22:20). There weren't a 10-minute interval between 20:50 and 22:20 without at least a 4% packet loss (often more than that). 2013-01-18T20:50:09-0800 HOST: li163-### Loss% Snt Last Avg Best Wrst StDev 1. router1-fmt.linode.com 0.0% 600 47.5 29.2 0.5 304.2 38.4 2. 10gigabitethernet2-3.core1.fmt1.he.net 0.0% 600 0.6 16.3 0.6 929.0 64.5 3. 10gigabitethernet1-2.core1.sjc2.he.net 0.5% 600 2.4 4.1 0.7 41.9 6.4 4. Port-channel100.ar3.SJC2.gblx.net 2.2% 600 75.7 31.8 0.8 1160. 76.7 5. po6-20G.ar4.SJC2.gblx.net 1.3% 600 19.9 33.4 0.9 917.4 72.8 6. lag8.ar5.SJC2.gblx.net 3.3% 600 1.1 4.4 0.9 22.3 5.4 7. 208.178.58.2 4.2% 600 4.2 4.2 1.5 39.7 2.1 8. pos-2-7-0-0-cr01.sanjose.ca.ibone.comcast.net 6.8% 600 10.6 7.2 4.7 43.8 2.9 9. pos-1-4-0-0-cr01.sacramento.ca.ibone.comcast.net 3.7% 600 20.7 9.5 7.0 26.7 1.9 10. te-7-3-ar02.saltlakecity.ut.utah.comcast.net 4.7% 600 92.1 12.4 7.4 129.2 19.1 11. 69.139.222.50 3.3% 600 7.9 8.1 7.6 59.3 2.6 12. te-1-0-0-ten05.sacramento.ca.ccal.comcast.net 3.2% 600 14.3 14.4 14.1 26.0 1.6 13. 74-93-180-##-Sacramento.hfc.comcastbusiness.net 4.2% 600 20.9 21.7 14.2 193.0 13.6 2013-01-18T21:00:10-0800 2013-01-18T21:30:04-0800 HOST: li163-### Loss% Snt Last Avg Best Wrst StDev 1. router1-fmt.linode.com 0.0% 600 0.6 11.0 0.5 202.3 21.2 2. 10gigabitethernet2-3.core1.fmt1.he.net 0.0% 600 4.5 20.2 0.5 315.3 46.0 3. 10gigabitethernet1-2.core1.sjc2.he.net 0.0% 600 11.5 3.4 0.7 25.9 3.7 4. Port-channel100.ar3.SJC2.gblx.net 6.2% 600 1.0 28.5 0.8 845.3 65.5 5. po6-20G.ar4.SJC2.gblx.net 9.8% 600 1.2 32.6 0.9 689.9 68.5 6. lag8.ar5.SJC2.gblx.net 12.2% 600 1.0 4.8 0.9 26.9 5.4 7. 208.178.58.2 12.0% 600 5.2 4.4 1.6 39.0 2.2 8. pos-2-7-0-0-cr01.sanjose.ca.ibone.comcast.net 13.0% 600 7.6 7.3 4.7 45.0 2.8 9. pos-1-4-0-0-cr01.sacramento.ca.ibone.comcast.net 11.5% 600 7.6 9.6 7.2 24.1 2.1 10. te-7-3-ar02.saltlakecity.ut.utah.comcast.net 12.3% 600 7.6 12.6 7.4 162.7 19.7 11. 69.139.222.50 12.2% 600 8.0 8.5 7.6 157.5 6.9 12. te-1-0-0-ten05.sacramento.ca.ccal.comcast.net 11.0% 600 14.3 14.8 14.1 29.2 2.3 13. 74-93-180-##-Sacramento.hfc.comcastbusiness.net 12.5% 600 15.9 26.3 14.4 417.3 33.3 2013-01-18T21:40:04-0800 2013-01-18T22:00:01-0800 HOST: li163-### Loss% Snt Last Avg Best Wrst StDev 1. router1-fmt.linode.com 0.0% 600 17.2 19.9 0.5 195.7 27.4 2. 10gigabitethernet2-3.core1.fmt1.he.net 0.0% 600 1.1 31.0 0.5 1666. 109.0 3. 10gigabitethernet1-2.core1.sjc2.he.net 0.0% 600 0.8 3.3 0.7 13.2 3.6 4. Port-channel100.ar3.SJC2.gblx.net 5.5% 600 2.0 27.7 0.9 290.3 56.5 5. po6-20G.ar4.SJC2.gblx.net 13.3% 600 1.0 33.5 0.9 770.0 71.0 6. lag8.ar5.SJC2.gblx.net 16.8% 600 15.0 5.7 0.8 30.1 6.0 7. 208.178.58.2 17.2% 600 5.9 4.5 1.5 30.2 2.4 8. pos-2-7-0-0-cr01.sanjose.ca.ibone.comcast.net 18.5% 600 7.2 7.3 4.9 34.8 2.4 9. pos-1-4-0-0-cr01.sacramento.ca.ibone.comcast.net 17.8% 600 9.8 9.6 7.1 24.0 2.1 10. te-7-3-ar02.saltlakecity.ut.utah.comcast.net 15.8% 600 7.7 12.8 7.4 177.3 19.8 11. 69.139.222.90 17.0% 600 7.9 8.9 7.6 164.0 8.3 12. te-1-0-0-ten05.sacramento.ca.ccal.comcast.net 13.3% 600 14.3 15.1 14.1 27.9 3.0 13. 74-93-180-##-Sacramento.hfc.comcastbusiness.net 17.5% 599 21.4 20.5 14.3 147.7 9.5 2013-01-18T22:10:07-0800 Anyone else has noticed? ssh is completely unusable for a couple of seconds every now and again. It seems to have subdued back to a 0% loss as of 22:20/22:30 window, but I bet it'll be back. C.

3 2