Just confirmed with the staff @ HE.net they are experiencing packet loss and latency from the Columbus Coligix POP. I have a feeling it is more since we have customers in Indianapolis as well as Chicago Markets now reporting issues. #tool traceroute address=hulu.com use-dns=yes # ADDRESS LOSS SENT LAST AVG BEST WORST STD-DEV STATUS 1 peering_fb.horizon-ce.10ge-1-... 0% 492 4.4ms 4.4 4.3 5.3 0.1 2 edge.hurricianelectric-net.10... 0% 492 4.4ms 4.5 4.4 5.7 0.1 3 v212.core1.cmh1.he.net 0% 492 4.7ms 5.3 4.6 32.9 2.8 4 10ge4-1.core1.ind1.he.net 0% 492 11.5ms 11.4 11.1 14.5 0.4 5 100ge16-2.core1.chi1.he.net 7.5% 492 43.7ms 41.9 37.4 61.9 40.8 6 chi-b21-link.telia.net 8.7% 492 42.1ms 42.1 38.9 49.6 41.3 7 ix-ae-24-0.tcore1.ct8-chicago... 6.9% 492 41.7ms 42.1 37.9 75.3 40.6 8 if-ae-26-2.tcore2.nto-new-yor... 0.6% 492 118.5ms 119.2 118.2 156.1 119.2 <MPLS:L=334955,E=0> 9 if-ae-32-2.tcore2.ldn-london.... 0% 492 109.9ms 110.7 109.8 153.3 110.7 10 80.231.20.42 0% 492 95.2ms 95.7 95.1 122.4 95.1 11 a23-214-189-19.deploy.static.... 0% 492 93.4ms 93.5 93.2 99.4 93.2 -- [Q quit|D dump|C-z pause] tool traceroute address=amazon.com use-dns=yes # ADDRESS LOSS SENT LAST AVG BEST WORST STD-DEV STATUS 1 peering_fb.horizon-ce.10ge-1-... 0% 6 4.3ms 4.5 4.3 4.5 0.1 2 edge.hurricianelectric-net.10... 0% 6 4.6ms 4.5 4.4 4.6 0.1 3 v212.core1.cmh1.he.net 0% 6 4.8ms 4.7 4.7 4.8 0 4 10ge4-1.core1.ind1.he.net 0% 6 11.3ms 11.5 11.3 12 0.2 5 100ge16-2.core1.chi1.he.net 16.. 6 43.3ms 42 40.1 43.3 1.2 6 100ge2-2.core2.chi1.he.net 33.. 6 42.2ms 42.7 40 44.7 1.8 7 equinix01-chi2.amazon.com 0% 6 41.8ms 42.8 40.3 45 1.5 8 52.95.62.68 0% 6 27.5ms 27.8 27.5 28.4 0.3 <MPLS:L=489056,E=0 L=2807,E=0,T=1> 9 52.95.62.81 0% 6 29.6ms 27.7 27.1 29.6 0.9 <MPLS:L=2807,E=0> 10 100% 6 timeout 11 54.239.44.77 0% 6 32.4ms 29.2 28.5 32.4 1.5 <MPLS:L=516692,E=0> 12 100% 6 timeout 13 100% 6 timeout 14 100% 6 timeout 15 100% 5 timeout (this network on the Amazon Cloud blocks ICMP - so the results before the timeouts are what we are actually looking @ since it shows drops on the HE backbone) tool traceroute address=isup.me use-dns=yes # ADDRESS LOSS SENT LAST AVG BEST WORST STD-DEV STATUS 1 peering_fb.horizon-ce.10ge-1-... 0% 6 4.5ms 4.5 4.4 4.6 0.1 2 edge.hurricianelectric-net.10... 0% 6 4.6ms 4.5 4.4 4.6 0.1 3 v212.core1.cmh1.he.net 0% 6 4.7ms 7.1 4.6 19.1 5.4 4 10ge4-1.core1.ind1.he.net 0% 6 11.2ms 11.2 11.2 11.3 0 5 100ge16-2.core1.chi1.he.net 0% 6 40.2ms 41.9 40.2 43.9 1.2 6 100ge2-2.core2.chi1.he.net 0% 6 41.1ms 41.6 39.6 44.4 1.6 7 100ge14-2.core1.msp1.he.net 0% 6 48ms 50.7 48 52.5 1.7 8 100ge10-1.core1.sea1.he.net 16.. 6 93.3ms 85 81.3 93.3 4.3 9 six01-sea4.amazon.com 0% 6 83.7ms 83 81.2 84.1 1.2 10 52.95.52.86 16.. 6 81.7ms 83.6 81.2 85.7 1.8 <MPLS:L=656224,E=0> 11 52.95.52.97 0% 6 84.8ms 83.4 80.2 84.8 1.5 12 100% 6 timeout 13 54.239.45.119 0% 6 87.1ms 89.3 87.1 91.1 1.4 <MPLS:L=361254,E=0> 14 100% 6 timeout 15 52.93.12.188 40% 5 timeout 98.8 90.1 114.6 11.2 <MPLS:L=347944,E=0> 16 52.93.12.209 0% 5 89.8ms 89.1 85.9 91.1 1.9 17 52.93.240.53 0% 5 88.8ms 89.6 88.8 91.1 0.9 18 100% 5 timeout 19 100% 5 timeout (again end site blocks ICMP we expect those timeouts - the latency in the HE.network is what is of concern) tool traceroute address=gmail.com use-dns=yes # ADDRESS LOSS SENT LAST AVG BEST WORST STD-DEV STATUS 1 peering_fb.horizon-ce.10ge-1-... 0% 25 4.5ms 4.4 4.4 4.6 0.1 2 edge.hurricianelectric-net.10... 0% 25 4.5ms 4.5 4.4 4.6 0.1 3 v212.core1.cmh1.he.net 0% 25 4.8ms 4.8 4.7 4.9 0.1 4 10ge4-1.core1.ind1.he.net 0% 25 11.1ms 11.2 11.1 11.8 0.1 5 100ge16-2.core1.chi1.he.net 8% 25 41ms 41.2 38.8 43.3 1.4 6 100ge2-2.core2.chi1.he.net 16% 25 39ms 41.3 39 44.2 1.3 7 eqix-ch-100g.google.com 8% 25 41ms 42.2 38.9 53 2.7 8 108.170.243.225 8% 25 39.6ms 43 39.6 47.7 1.8 9 209.85.249.73 0% 25 15.2ms 15.2 15.1 15.4 0.1 10 ord31s22-in-f229.1e100.net 4% 25 40.9ms 41.5 39.1 43.3 1.2 tool traceroute address=outlook.com use-dns=yes # ADDRESS LOSS SENT LAST AVG BEST WORST STD-DEV STATUS 1 peering_fb.horizon-ce.10ge-1-... 0% 10 4.4ms 4.5 4.3 4.5 0.1 2 edge.hurricianelectric-net.10... 0% 10 4.6ms 4.5 4.4 4.6 0.1 3 v212.core1.cmh1.he.net 0% 10 5.2ms 4.7 4.6 5.2 0.2 4 10ge4-1.core1.ind1.he.net 0% 10 11.3ms 11.7 11.2 15.5 1.3 5 100ge16-2.core1.chi1.he.net 0% 10 39ms 40.8 39 43.7 1.4 6 100ge2-2.core2.chi1.he.net 10% 10 41.9ms 41.3 39.9 42.3 0.8 7 206.108.115.47 10% 10 40ms 41.4 40 43.6 1.1 8 ae4-0.ch1-96c-2b.ntwk.msn.net 0% 10 42.3ms 46.2 41.1 62.3 6.3 <MPLS:L=16841,E=0> 9 be-64-0.ibr01.ch1.ntwk.msn.net 0% 10 41.5ms 41.8 40.8 43 0.7 10 be-5-0.ibr01.ch2.ntwk.msn.net 0% 10 40.6ms 40.7 40.3 41 0.2 11 be-6-0.ibr01.cnr02.dsm05.ntwk... 0% 10 40.6ms 41.7 40.6 43.1 0.8 12 be-3-0.ibr01.cnr03.dsm05.ntwk... 0% 10 40.9ms 41.5 40.9 42 0.4 13 be-5-0.ibr02.den07.ntwk.msn.net 0% 10 41.7ms 41.5 40.7 42.4 0.5 14 be-1-0.ibr01.den07.ntwk.msn.net 0% 10 41.9ms 41.5 40.8 42.2 0.5 15 be-5-0.ibr02.cys04.ntwk.msn.net 10% 10 40.4ms 40.8 40.4 42 0.5 16 ae162-0.icr02.cys04.ntwk.msn.net 0% 10 40.3ms 40.2 39.9 41 0.3 17 100% 10 timeout 18 100% 10 timeout tool traceroute address=pornhub.com use-dns=yes # ADDRESS LOSS SENT LAST AVG BEST WORST STD-DEV STATUS 1 peering_fb.horizon-ce.10ge-1-... 0% 42 4.5ms 4.5 4.4 4.6 0.1 2 edge.hurricianelectric-net.10... 0% 42 4.4ms 4.5 4.4 4.7 0.1 3 v212.core1.cmh1.he.net 0% 42 4.8ms 4.9 4.7 9.6 0.8 4 10ge4-1.core1.ind1.he.net 0% 42 11.4ms 11.3 11.1 11.6 0.1 5 100ge16-2.core1.chi1.he.net 2.4% 42 41ms 41.9 38.6 57.3 3 6 chi-b21-link.telia.net 7.1% 42 41.8ms 42 39.5 48.4 1.6 7 haproxy-ic-332713-chi-b21.c.t... 14.. 42 46.8ms 44 39.4 64.8 4.8 8 216.18.168.16 11.. 42 42ms 41.7 38.5 45.6 1.6 tool traceroute address=microsoft.com use-dns=yes # ADDRESS LOSS SENT LAST AVG BEST WORST STD-DEV STATUS 1 peering_fb.horizon-ce.10ge-1-... 0% 5 4.4ms 4.5 4.4 4.6 0.1 2 edge.hurricianelectric-net.10... 0% 5 4.5ms 4.5 4.5 4.6 0 3 v212.core1.cmh1.he.net 0% 5 4.6ms 4.7 4.6 4.8 0.1 4 10ge4-1.core1.ind1.he.net 0% 5 11.3ms 11.3 11.2 11.4 0.1 5 100ge16-2.core1.chi1.he.net 20% 5 42.5ms 43.1 42.2 44.3 0.8 6 100ge2-2.core2.chi1.he.net 0% 5 42.7ms 44.3 39.6 52 4.4 7 chi-8075.msn.net 20% 5 40.9ms 42.9 40.9 46.5 2.1 8 ae12-0.ch1-96c-2a.ntwk.msn.net 20% 5 42.5ms 44.3 41.8 46.7 2.1 <MPLS:L=16835,E=0> 9 be-63-0.ibr01.ch1.ntwk.msn.net 0% 5 27.1ms 28.1 27.1 29.1 0.8 10 be-5-0.ibr01.ch2.ntwk.msn.net 0% 5 27.1ms 27.7 27.1 28.9 0.6 11 be-3-0.ibr01.was02.ntwk.msn.net 0% 5 28.9ms 28.4 27.5 29.3 0.6 12 be-5-0.ibr03.bl7.ntwk.msn.net 0% 5 27.3ms 27.5 27.3 27.8 0.2 13 ae142-0.icr02.bl7.ntwk.msn.net 0% 5 31.6ms 30.4 27 37.9 4.1 14 100% 5 timeout 15 100% 5 timeout 16 100% 5 timeout 17 100% 4 timeout 18 100% 4 timeout We ran most of these for minutes on end - just captured the snapshots briefly however -- Glenn Kelley www.Connectivity.Engineer EMT-P, CCNA, MTCINE, MTCRE,CFOS/DC, CFOS/A ------------------------------------------------------------------------------------------------- Connectivity.Engineer is a family of companies including Hop Off A Cloud Networks, Typo3USA PasturePress, White Lab Studios & the CARM Network