Issue appears to have started as early as 20:00 PDT but I need to review more logs to see if occasional trickles of problems began even earlier. mtrs from both direction are provided and src/dst IPs. These are ICMP echo/echo-reply with a 512-byte ICMP payload (vs. default 56). Issue continues as I write this. src IP: 69.181.142.213 dst IP: 104.238.183.73 === Wed Jun 8 22:04:00 PDT 2016 (1465448640) Start: Wed Jun 8 22:04:00 2016 HOST: icarus.home.lan Loss% Snt Rcv Last Avg Best Wrst 1.|-- gw.home.lan (192.168.1.1) 0.0% 30 30 0.3 0.3 0.2 0.4 2.|-- 96.120.89.145 0.0% 30 30 9.2 9.6 8.7 12.4 3.|-- be-20003-sur04.santaclara.ca.sfba.comcast.net (68.86.249.249) 0.0% 30 30 9.7 9.8 8.9 12.6 4.|-- hu-0-3-0-6-ar01.hayward.ca.sfba.comcast.net (68.87.192.249) 3.3% 30 29 10.8 10.9 10.3 15.8 5.|-- hu-0-2-0-1-ar01.santaclara.ca.sfba.comcast.net (68.85.57.145) 3.3% 30 29 11.2 11.1 10.4 13.5 6.|-- be-33651-cr01.sunnyvale.ca.ibone.comcast.net (68.86.90.93) 3.3% 30 29 11.1 11.2 10.3 15.3 7.|-- be-10925-cr01.9greatoaks.ca.ibone.comcast.net (68.86.87.158) 0.0% 30 30 14.6 15.7 12.9 36.3 8.|-- hu-0-11-0-0-pe03.11greatoaks.ca.ibone.comcast.net (68.86.85.238) 33.3% 30 20 13.2 13.5 12.1 21.9 9.|-- ae12.sjc12.ip4.gtt.net (173.205.58.169) 36.7% 30 19 12.7 15.5 11.9 64.7 10.|-- xe-1-1-2.sjc20.ip4.gtt.net (89.149.184.246) 36.7% 30 19 12.5 17.7 11.8 87.9 11.|-- as20473.xe-1-2-3.cr1.sjc2.us.as4436.gtt.net (69.22.130.90) 40.0% 30 18 19.8 16.4 12.3 66.1 12.|-- ??? 100.0 30 0 0.0 0.0 0.0 0.0 13.|-- mambo.koitsu.org (104.238.183.73) 41.4% 29 17 13.2 13.3 12.5 19.9 === END === Wed Jun 8 22:05:00 PDT 2016 (1465448700) Start: Wed Jun 8 22:05:00 2016 HOST: icarus.home.lan Loss% Snt Rcv Last Avg Best Wrst 1.|-- gw.home.lan (192.168.1.1) 0.0% 30 30 0.4 0.3 0.3 0.4 2.|-- 96.120.89.145 0.0% 30 30 9.0 9.5 8.5 12.5 3.|-- be-20003-sur04.santaclara.ca.sfba.comcast.net (68.86.249.249) 0.0% 30 30 9.3 9.7 9.0 12.3 4.|-- hu-0-3-0-6-ar01.hayward.ca.sfba.comcast.net (68.87.192.249) 0.0% 30 30 16.9 11.6 10.3 20.8 5.|-- hu-0-2-0-1-ar01.santaclara.ca.sfba.comcast.net (68.85.57.145) 0.0% 30 30 10.6 11.3 10.3 14.1 6.|-- be-33651-cr01.sunnyvale.ca.ibone.comcast.net (68.86.90.93) 33.3% 30 20 18.3 12.6 10.2 38.7 7.|-- be-10925-cr01.9greatoaks.ca.ibone.comcast.net (68.86.87.158) 0.0% 30 30 13.7 14.2 12.6 31.1 8.|-- hu-0-11-0-0-pe03.11greatoaks.ca.ibone.comcast.net (68.86.85.238) 40.0% 30 18 12.6 13.4 11.9 24.4 9.|-- ae12.sjc12.ip4.gtt.net (173.205.58.169) 50.0% 30 15 12.5 12.6 11.8 15.3 10.|-- xe-1-1-2.sjc20.ip4.gtt.net (89.149.184.246) 50.0% 30 15 12.7 12.5 12.1 13.4 11.|-- as20473.xe-1-2-3.cr1.sjc2.us.as4436.gtt.net (69.22.130.90) 43.3% 30 17 12.4 13.3 12.1 17.6 12.|-- ??? 100.0 30 0 0.0 0.0 0.0 0.0 13.|-- mambo.koitsu.org (104.238.183.73) 46.7% 30 16 12.8 14.5 12.4 20.4 === END === Wed Jun 8 22:06:00 PDT 2016 (1465448760) Start: Wed Jun 8 22:06:00 2016 HOST: icarus.home.lan Loss% Snt Rcv Last Avg Best Wrst 1.|-- gw.home.lan (192.168.1.1) 0.0% 30 30 0.4 0.3 0.2 0.4 2.|-- 96.120.89.145 0.0% 30 30 9.0 9.5 8.6 15.6 3.|-- be-20003-sur04.santaclara.ca.sfba.comcast.net (68.86.249.249) 0.0% 30 30 9.5 10.8 9.1 19.9 4.|-- hu-0-3-0-6-ar01.hayward.ca.sfba.comcast.net (68.87.192.249) 3.3% 30 29 10.9 11.0 10.2 14.2 5.|-- hu-0-2-0-1-ar01.santaclara.ca.sfba.comcast.net (68.85.57.145) 3.3% 30 29 11.7 12.2 10.3 27.8 6.|-- be-33651-cr01.sunnyvale.ca.ibone.comcast.net (68.86.90.93) 93.3% 30 2 12.9 12.2 11.5 12.9 7.|-- be-10925-cr01.9greatoaks.ca.ibone.comcast.net (68.86.87.158) 0.0% 30 30 12.5 14.2 12.1 18.1 8.|-- hu-0-11-0-0-pe03.11greatoaks.ca.ibone.comcast.net (68.86.85.238) 36.7% 30 19 12.2 15.0 12.1 50.8 9.|-- ae12.sjc12.ip4.gtt.net (173.205.58.169) 33.3% 30 20 12.1 12.8 11.8 19.4 10.|-- xe-1-1-2.sjc20.ip4.gtt.net (89.149.184.246) 43.3% 30 17 12.1 12.5 11.9 14.9 11.|-- as20473.xe-1-2-3.cr1.sjc2.us.as4436.gtt.net (69.22.130.90) 40.0% 30 18 12.3 19.7 12.2 74.3 12.|-- ??? 100.0 30 0 0.0 0.0 0.0 0.0 13.|-- mambo.koitsu.org (104.238.183.73) 40.0% 30 18 12.8 12.7 12.1 13.4 === END (Note: in the above, hop #6 packet loss is totally normal) src IP: 104.238.183.73 dst IP: 69.181.142.213 === Wed Jun 8 22:04:00 PDT 2016 (1465448640) Start: Wed Jun 8 22:04:01 2016 HOST: mambo.koitsu.org Loss% Snt Rcv Last Avg Best Wrst 1.|-- ??? 100.0 30 0 0.0 0.0 0.0 0.0 2.|-- 45.63.80.1 0.0% 30 30 0.4 1.6 0.3 32.3 3.|-- xe-1-2-3.cr1.sjc2.us.as4436.gtt.net (69.22.130.89) 0.0% 30 30 0.9 2.6 0.3 15.3 4.|-- xe-7-0-7.sjc12.ip4.gtt.net (89.149.184.237) 0.0% 30 30 0.7 0.8 0.6 2.6 5.|-- as7922.sjc12.ip4.gtt.net (173.205.58.170) 0.0% 30 30 1.2 1.4 1.0 6.4 6.|-- hu-0-4-0-5-cr01.9greatoaks.ca.ibone.comcast.net (68.86.88.205) 36.7% 30 19 2.7 2.6 1.3 4.8 7.|-- be-7922-ar01.hayward.ca.sfba.comcast.net (68.86.94.154) 30.0% 30 21 2.6 2.9 2.5 5.9 8.|-- hu-0-4-0-1-sur04.santaclara.ca.sfba.comcast.net (68.87.192.254) 16.7% 30 25 4.1 4.0 3.8 4.4 9.|-- te-7-0-acr03.santaclara.ca.sfba.comcast.net (68.86.249.250) 36.7% 30 19 4.7 2.9 2.4 6.7 10.|-- c-69-181-142-213.hsd1.ca.comcast.net (69.181.142.213) 40.0% 30 18 13.4 13.1 11.8 15.9 === END === Wed Jun 8 22:05:00 PDT 2016 (1465448700) Start: Wed Jun 8 22:05:00 2016 HOST: mambo.koitsu.org Loss% Snt Rcv Last Avg Best Wrst 1.|-- ??? 100.0 30 0 0.0 0.0 0.0 0.0 2.|-- 45.63.80.1 0.0% 30 30 0.4 0.9 0.3 6.2 3.|-- xe-1-2-3.cr1.sjc2.us.as4436.gtt.net (69.22.130.89) 0.0% 30 30 0.4 1.0 0.3 16.3 4.|-- xe-7-0-7.sjc12.ip4.gtt.net (89.149.184.237) 0.0% 30 30 3.5 1.2 0.6 3.7 5.|-- as7922.sjc12.ip4.gtt.net (173.205.58.170) 0.0% 30 30 1.2 1.4 1.0 2.9 6.|-- hu-0-3-0-5-cr01.9greatoaks.ca.ibone.comcast.net (68.86.83.129) 40.0% 30 18 2.1 2.5 1.3 3.5 7.|-- be-7922-ar01.hayward.ca.sfba.comcast.net (68.86.94.154) 40.0% 30 18 2.9 2.8 2.5 3.1 8.|-- hu-0-4-0-1-sur04.santaclara.ca.sfba.comcast.net (68.87.192.254) 43.3% 30 17 3.9 4.0 3.9 4.2 9.|-- te-7-0-acr03.santaclara.ca.sfba.comcast.net (68.86.249.250) 43.3% 30 17 2.8 2.6 2.4 3.3 10.|-- c-69-181-142-213.hsd1.ca.comcast.net (69.181.142.213) 46.7% 30 16 12.8 13.7 12.3 19.7 === END === Wed Jun 8 22:06:00 PDT 2016 (1465448760) Start: Wed Jun 8 22:06:00 2016 HOST: mambo.koitsu.org Loss% Snt Rcv Last Avg Best Wrst 1.|-- ??? 100.0 30 0 0.0 0.0 0.0 0.0 2.|-- 45.63.80.1 0.0% 30 30 0.8 0.7 0.3 5.3 3.|-- xe-1-2-3.cr1.sjc2.us.as4436.gtt.net (69.22.130.89) 0.0% 30 30 0.4 0.4 0.3 0.7 4.|-- xe-7-0-7.sjc12.ip4.gtt.net (89.149.184.237) 0.0% 30 30 0.7 0.9 0.6 2.5 5.|-- as7922.sjc12.ip4.gtt.net (173.205.58.170) 0.0% 30 30 1.2 1.2 1.0 1.7 6.|-- hu-0-4-0-0-cr01.9greatoaks.ca.ibone.comcast.net (68.86.86.209) 30.0% 30 21 1.5 2.7 1.4 5.7 7.|-- be-7922-ar01.hayward.ca.sfba.comcast.net (68.86.94.154) 40.0% 30 18 2.7 2.7 2.5 3.1 8.|-- hu-0-4-0-1-sur04.santaclara.ca.sfba.comcast.net (68.87.192.254) 36.7% 30 19 3.9 3.9 3.9 4.4 9.|-- te-7-0-acr03.santaclara.ca.sfba.comcast.net (68.86.249.250) 40.0% 30 18 2.5 2.5 2.4 2.8 10.|-- c-69-181-142-213.hsd1.ca.comcast.net (69.181.142.213) 40.0% 30 18 22.0 13.4 12.1 22.0 === END -- | Jeremy Chadwick jdc@koitsu.org | | UNIX Systems Administrator http://jdc.koitsu.org/ | | Making life hard for others since 1977. PGP 4BD6C0CB |