Hi Glenn, I saw the same thing this morning and had to shift all our traffic away. They emailed shortly after 5:30pm to let me know it was resolved. Their 100GigE wave from Columbus to Chicago was down. On an unrelated note, are you interested in joining Ohio IX? We (Amplex Internet) just joined, and I'm working with them to onboard some of the larger networks (Amazon, Google, Netflix). More info: http://ohioix.net/ https://www.peeringdb.com/ix/869 -Adam On Wed, Jan 23, 2019 at 1:50 PM Glenn Kelley via Outages < outages@outages.org> wrote:

Just confirmed with the staff @ HE.net they are experiencing packet loss and latency from the Columbus Coligix POP.

I have a feeling it is more since we have customers in Indianapolis as well as Chicago Markets now reporting issues.

#tool traceroute address=hulu.com use-dns=yes # ADDRESS LOSS SENT LAST AVG BEST WORST STD-DEV STATUS 1 peering_fb.horizon-ce.10ge-1-... 0% 492 4.4ms 4.4 4.3 5.3 0.1 2 edge.hurricianelectric-net.10... 0% 492 4.4ms 4.5 4.4 5.7 0.1 3 v212.core1.cmh1.he.net 0% 492 4.7ms 5.3 4.6 32.9 2.8 4 10ge4-1.core1.ind1.he.net 0% 492 11.5ms 11.4 11.1 14.5 0.4 5 100ge16-2.core1.chi1.he.net 7.5% 492 43.7ms 41.9 37.4 61.9 40.8 6 chi-b21-link.telia.net 8.7% 492 42.1ms 42.1 38.9 49.6 41.3 7 ix-ae-24-0.tcore1.ct8-chicago... 6.9% 492 41.7ms 42.1 37.9 75.3 40.6 8 if-ae-26-2.tcore2.nto-new-yor... 0.6% 492 118.5ms 119.2 118.2 156.1 119.2 <MPLS:L=334955,E=0> 9 if-ae-32-2.tcore2.ldn-london.... 0% 492 109.9ms 110.7 109.8 153.3 110.7 10 80.231.20.42 0% 492 95.2ms 95.7 95.1 122.4 95.1 11 a23-214-189-19.deploy.static.... 0% 492 93.4ms 93.5 93.2 99.4 93.2 -- [Q quit|D dump|C-z pause]

tool traceroute address=amazon.com use-dns=yes # ADDRESS LOSS SENT LAST AVG BEST WORST STD-DEV STATUS 1 peering_fb.horizon-ce.10ge-1-... 0% 6 4.3ms 4.5 4.3 4.5 0.1 2 edge.hurricianelectric-net.10... 0% 6 4.6ms 4.5 4.4 4.6 0.1 3 v212.core1.cmh1.he.net 0% 6 4.8ms 4.7 4.7 4.8 0 4 10ge4-1.core1.ind1.he.net 0% 6 11.3ms 11.5 11.3 12 0.2 5 100ge16-2.core1.chi1.he.net 16.. 6 43.3ms 42 40.1 43.3 1.2 6 100ge2-2.core2.chi1.he.net 33.. 6 42.2ms 42.7 40 44.7 1.8 7 equinix01-chi2.amazon.com 0% 6 41.8ms 42.8 40.3 45 1.5 8 52.95.62.68 0% 6 27.5ms 27.8 27.5 28.4 0.3 <MPLS:L=489056,E=0 L=2807,E=0,T=1> 9 52.95.62.81 0% 6 29.6ms 27.7 27.1 29.6 0.9 <MPLS:L=2807,E=0> 10 100% 6 timeout

11 54.239.44.77 0% 6 32.4ms 29.2 28.5 32.4 1.5 <MPLS:L=516692,E=0> 12 100% 6 timeout

13 100% 6 timeout

14 100% 6 timeout

15 100% 5 timeout

(this network on the Amazon Cloud blocks ICMP - so the results before the timeouts are what we are actually looking @ since it shows drops on the HE backbone)

tool traceroute address=isup.me use-dns=yes # ADDRESS LOSS SENT LAST AVG BEST WORST STD-DEV STATUS 1 peering_fb.horizon-ce.10ge-1-... 0% 6 4.5ms 4.5 4.4 4.6 0.1 2 edge.hurricianelectric-net.10... 0% 6 4.6ms 4.5 4.4 4.6 0.1 3 v212.core1.cmh1.he.net 0% 6 4.7ms 7.1 4.6 19.1 5.4 4 10ge4-1.core1.ind1.he.net 0% 6 11.2ms 11.2 11.2 11.3 0 5 100ge16-2.core1.chi1.he.net 0% 6 40.2ms 41.9 40.2 43.9 1.2 6 100ge2-2.core2.chi1.he.net 0% 6 41.1ms 41.6 39.6 44.4 1.6 7 100ge14-2.core1.msp1.he.net 0% 6 48ms 50.7 48 52.5 1.7 8 100ge10-1.core1.sea1.he.net 16.. 6 93.3ms 85 81.3 93.3 4.3 9 six01-sea4.amazon.com 0% 6 83.7ms 83 81.2 84.1 1.2 10 52.95.52.86 16.. 6 81.7ms 83.6 81.2 85.7 1.8 <MPLS:L=656224,E=0> 11 52.95.52.97 0% 6 84.8ms 83.4 80.2 84.8 1.5 12 100% 6 timeout

13 54.239.45.119 0% 6 87.1ms 89.3 87.1 91.1 1.4 <MPLS:L=361254,E=0> 14 100% 6 timeout

15 52.93.12.188 40% 5 timeout 98.8 90.1 114.6 11.2 <MPLS:L=347944,E=0> 16 52.93.12.209 0% 5 89.8ms 89.1 85.9 91.1 1.9 17 52.93.240.53 0% 5 88.8ms 89.6 88.8 91.1 0.9 18 100% 5 timeout

19 100% 5 timeout

(again end site blocks ICMP we expect those timeouts - the latency in the HE.network is what is of concern)

tool traceroute address=gmail.com use-dns=yes # ADDRESS LOSS SENT LAST AVG BEST WORST STD-DEV STATUS 1 peering_fb.horizon-ce.10ge-1-... 0% 25 4.5ms 4.4 4.4 4.6 0.1 2 edge.hurricianelectric-net.10... 0% 25 4.5ms 4.5 4.4 4.6 0.1 3 v212.core1.cmh1.he.net 0% 25 4.8ms 4.8 4.7 4.9 0.1 4 10ge4-1.core1.ind1.he.net 0% 25 11.1ms 11.2 11.1 11.8 0.1 5 100ge16-2.core1.chi1.he.net 8% 25 41ms 41.2 38.8 43.3 1.4 6 100ge2-2.core2.chi1.he.net 16% 25 39ms 41.3 39 44.2 1.3 7 eqix-ch-100g.google.com 8% 25 41ms 42.2 38.9 53 2.7 8 108.170.243.225 8% 25 39.6ms 43 39.6 47.7 1.8 9 209.85.249.73 0% 25 15.2ms 15.2 15.1 15.4 0.1 10 ord31s22-in-f229.1e100.net 4% 25 40.9ms 41.5 39.1 43.3 1.2

tool traceroute address=outlook.com use-dns=yes # ADDRESS LOSS SENT LAST AVG BEST WORST STD-DEV STATUS

1 peering_fb.horizon-ce.10ge-1-... 0% 10 4.4ms 4.5 4.3 4.5 0.1

2 edge.hurricianelectric-net.10... 0% 10 4.6ms 4.5 4.4 4.6 0.1

3 v212.core1.cmh1.he.net 0% 10 5.2ms 4.7 4.6 5.2 0.2

4 10ge4-1.core1.ind1.he.net 0% 10 11.3ms 11.7 11.2 15.5 1.3

5 100ge16-2.core1.chi1.he.net 0% 10 39ms 40.8 39 43.7 1.4

6 100ge2-2.core2.chi1.he.net 10% 10 41.9ms 41.3 39.9 42.3 0.8

7 206.108.115.47 10% 10 40ms 41.4 40 43.6 1.1

8 ae4-0.ch1-96c-2b.ntwk.msn.net 0% 10 42.3ms 46.2 41.1 62.3 6.3 <MPLS:L=16841,E=0>

9 be-64-0.ibr01.ch1.ntwk.msn.net 0% 10 41.5ms 41.8 40.8 43 0.7

10 be-5-0.ibr01.ch2.ntwk.msn.net 0% 10 40.6ms 40.7 40.3 41 0.2

11 be-6-0.ibr01.cnr02.dsm05.ntwk... 0% 10 40.6ms 41.7 40.6 43.1 0.8

12 be-3-0.ibr01.cnr03.dsm05.ntwk... 0% 10 40.9ms 41.5 40.9 42 0.4

13 be-5-0.ibr02.den07.ntwk.msn.net 0% 10 41.7ms 41.5 40.7 42.4 0.5

14 be-1-0.ibr01.den07.ntwk.msn.net 0% 10 41.9ms 41.5 40.8 42.2 0.5

15 be-5-0.ibr02.cys04.ntwk.msn.net 10% 10 40.4ms 40.8 40.4 42 0.5

16 ae162-0.icr02.cys04.ntwk.msn.net 0% 10 40.3ms 40.2 39.9 41 0.3

17 100% 10 timeout

18 100% 10 timeout

tool traceroute address=pornhub.com use-dns=yes # ADDRESS LOSS SENT LAST AVG BEST WORST STD-DEV STATUS

1 peering_fb.horizon-ce.10ge-1-... 0% 42 4.5ms 4.5 4.4 4.6 0.1

2 edge.hurricianelectric-net.10... 0% 42 4.4ms 4.5 4.4 4.7 0.1

3 v212.core1.cmh1.he.net 0% 42 4.8ms 4.9 4.7 9.6 0.8

4 10ge4-1.core1.ind1.he.net 0% 42 11.4ms 11.3 11.1 11.6 0.1

5 100ge16-2.core1.chi1.he.net 2.4% 42 41ms 41.9 38.6 57.3 3

6 chi-b21-link.telia.net 7.1% 42 41.8ms 42 39.5 48.4 1.6

7 haproxy-ic-332713-chi-b21.c.t... 14.. 42 46.8ms 44 39.4 64.8 4.8

8 216.18.168.16 11.. 42 42ms 41.7 38.5 45.6 1.6

tool traceroute address=microsoft.com use-dns=yes # ADDRESS LOSS SENT LAST AVG BEST WORST STD-DEV STATUS

1 peering_fb.horizon-ce.10ge-1-... 0% 5 4.4ms 4.5 4.4 4.6 0.1

2 edge.hurricianelectric-net.10... 0% 5 4.5ms 4.5 4.5 4.6 0

3 v212.core1.cmh1.he.net 0% 5 4.6ms 4.7 4.6 4.8 0.1

4 10ge4-1.core1.ind1.he.net 0% 5 11.3ms 11.3 11.2 11.4 0.1

5 100ge16-2.core1.chi1.he.net 20% 5 42.5ms 43.1 42.2 44.3 0.8

6 100ge2-2.core2.chi1.he.net 0% 5 42.7ms 44.3 39.6 52 4.4

7 chi-8075.msn.net 20% 5 40.9ms 42.9 40.9 46.5 2.1

8 ae12-0.ch1-96c-2a.ntwk.msn.net 20% 5 42.5ms 44.3 41.8 46.7 2.1 <MPLS:L=16835,E=0>

9 be-63-0.ibr01.ch1.ntwk.msn.net 0% 5 27.1ms 28.1 27.1 29.1 0.8

10 be-5-0.ibr01.ch2.ntwk.msn.net 0% 5 27.1ms 27.7 27.1 28.9 0.6

11 be-3-0.ibr01.was02.ntwk.msn.net 0% 5 28.9ms 28.4 27.5 29.3 0.6

12 be-5-0.ibr03.bl7.ntwk.msn.net 0% 5 27.3ms 27.5 27.3 27.8 0.2

13 ae142-0.icr02.bl7.ntwk.msn.net 0% 5 31.6ms 30.4 27 37.9 4.1

14 100% 5 timeout

15 100% 5 timeout

16 100% 5 timeout

17 100% 4 timeout

18 100% 4 timeout

We ran most of these for minutes on end - just captured the snapshots briefly however

-- Glenn Kelley www.Connectivity.Engineer EMT-P, CCNA, MTCINE, MTCRE,CFOS/DC, CFOS/A

------------------------------------------------------------------------------------------------- Connectivity.Engineer is a family of companies including Hop Off A Cloud Networks, Typo3USA PasturePress, White Lab Studios & the CARM Network _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages