
For the past few days I've seen what may be a layer 1 problem between two of Comcast's locations: Sunnyvale, CA and "Great Oaks", CA (not sure if this is a street, town, or what). Was hoping John Neiberger et al might be able to look into this. Timestamps and durations are below, with the relevant mtr data included as well. Sadly, due to a DNS mistake in my mtr configs, I don't have return-path (VPS --> Comcast) mtrs. I've since fixed this mistake, so If it happens again I'll have them from both directions. I'll also note that the intermittent packet loss + latency seen at hop #5 is normal -- I see this even when things are working just fine. It's almost certainly ICMP prio, no biggie. #1 -- January 19, 13:17 to 13:20 PST (UTC-8) (~3 minutes) #2 -- January 19, 22:30 to 22:33 PST (UTC-8) (~3 minutes) #3 -- January 25, 14:30 to 14:35 PST (UTC-8) (~5 minutes) For all data shown: src = 69.181.142.213, dst = 104.238.183.73 Data #1: === Tue Jan 19 13:16:00 PST 2016 (1453238160) Start: Tue Jan 19 13:16:00 2016 HOST: icarus.home.lan Loss% Snt Rcv Last Avg Best Wrst 1.|-- gw.home.lan (192.168.1.1) 0.0% 30 30 0.2 0.2 0.2 0.4 2.|-- 69.181.136.1 0.0% 30 30 9.6 16.8 9.6 39.4 3.|-- te-0-2-0-5-sur03.santaclara.ca.sfba.comcast.net (68.86.249.69) 0.0% 30 30 11.5 18.1 10.1 86.1 4.|-- hu-0-3-0-4-ar01.santaclara.ca.sfba.comcast.net (68.87.192.177) 0.0% 30 30 15.9 17.7 9.8 54.1 5.|-- be-33651-cr01.sunnyvale.ca.ibone.comcast.net (68.86.90.93) 0.0% 30 30 18.4 24.7 9.6 114.9 6.|-- be-10925-cr01.9greatoaks.ca.ibone.comcast.net (68.86.87.158) 0.0% 30 30 15.6 18.5 11.4 35.3 7.|-- hu-0-11-0-0-pe03.11greatoaks.ca.ibone.comcast.net (68.86.85.238) 0.0% 30 30 11.5 19.2 11.3 78.0 8.|-- ae12.sjc12.ip4.gtt.net (173.205.58.169) 0.0% 30 30 28.8 21.2 11.7 57.5 9.|-- xe-1-0-1.sjc20.ip4.gtt.net (89.149.184.250) 0.0% 30 30 17.6 19.3 11.3 52.3 10.|-- as20473.xe-1-2-3.cr1.sjc2.us.as4436.gtt.net (69.22.130.90) 0.0% 30 30 21.0 17.3 11.4 24.8 11.|-- ??? 100.0 30 0 0.0 0.0 0.0 0.0 12.|-- mambo.koitsu.org (104.238.183.73) 0.0% 30 30 93.9 23.1 11.8 97.4 === END === Tue Jan 19 13:17:00 PST 2016 (1453238220) Start: Tue Jan 19 13:17:00 2016 HOST: icarus.home.lan Loss% Snt Rcv Last Avg Best Wrst 1.|-- gw.home.lan (192.168.1.1) 0.0% 30 30 0.3 0.2 0.2 0.3 2.|-- 69.181.136.1 0.0% 30 30 9.9 14.7 9.1 21.0 3.|-- te-0-2-0-5-sur03.santaclara.ca.sfba.comcast.net (68.86.249.69) 0.0% 30 30 15.1 16.1 9.8 44.7 4.|-- hu-0-3-0-4-ar01.santaclara.ca.sfba.comcast.net (68.87.192.177) 0.0% 30 30 16.8 15.8 9.3 43.2 5.|-- be-33651-cr01.sunnyvale.ca.ibone.comcast.net (68.86.90.93) 40.0% 30 18 11.6 64.8 11.2 406.3 6.|-- ??? 100.0 30 0 0.0 0.0 0.0 0.0 === END === Tue Jan 19 13:18:00 PST 2016 (1453238280) Start: Tue Jan 19 13:18:00 2016 HOST: icarus.home.lan Loss% Snt Rcv Last Avg Best Wrst 1.|-- gw.home.lan (192.168.1.1) 0.0% 30 30 0.2 0.2 0.2 0.3 2.|-- 69.181.136.1 0.0% 30 30 21.5 13.7 8.9 21.5 3.|-- te-0-2-0-5-sur03.santaclara.ca.sfba.comcast.net (68.86.249.69) 0.0% 30 30 12.9 13.5 9.0 19.5 4.|-- hu-0-3-0-4-ar01.santaclara.ca.sfba.comcast.net (68.87.192.177) 0.0% 30 30 9.0 13.8 9.0 19.6 5.|-- be-33651-cr01.sunnyvale.ca.ibone.comcast.net (68.86.90.93) 0.0% 30 30 36.7 14.7 9.5 36.7 6.|-- ??? 100.0 30 0 0.0 0.0 0.0 0.0 === END === Tue Jan 19 13:19:00 PST 2016 (1453238340) Start: Tue Jan 19 13:19:00 2016 HOST: icarus.home.lan Loss% Snt Rcv Last Avg Best Wrst 1.|-- gw.home.lan (192.168.1.1) 0.0% 30 30 0.3 0.2 0.2 0.4 2.|-- 69.181.136.1 0.0% 30 30 16.8 19.7 9.1 111.4 3.|-- te-0-2-0-5-sur03.santaclara.ca.sfba.comcast.net (68.86.249.69) 0.0% 30 30 12.4 18.1 9.6 95.8 4.|-- hu-0-3-0-4-ar01.santaclara.ca.sfba.comcast.net (68.87.192.177) 0.0% 30 30 13.7 14.8 10.0 26.1 5.|-- be-33651-cr01.sunnyvale.ca.ibone.comcast.net (68.86.90.93) 0.0% 30 30 14.5 14.4 9.5 22.6 6.|-- ??? 100.0 30 0 0.0 0.0 0.0 0.0 === END === Tue Jan 19 13:20:00 PST 2016 (1453238400) Start: Tue Jan 19 13:20:00 2016 HOST: icarus.home.lan Loss% Snt Rcv Last Avg Best Wrst 1.|-- gw.home.lan (192.168.1.1) 0.0% 30 30 0.2 0.2 0.2 0.3 2.|-- 69.181.136.1 0.0% 30 30 57.6 15.6 9.5 57.6 3.|-- te-0-2-0-5-sur03.santaclara.ca.sfba.comcast.net (68.86.249.69) 0.0% 30 30 18.1 19.0 9.3 80.9 4.|-- hu-0-3-0-4-ar01.santaclara.ca.sfba.comcast.net (68.87.192.177) 0.0% 30 30 12.4 14.7 9.2 55.8 5.|-- be-33651-cr01.sunnyvale.ca.ibone.comcast.net (68.86.90.93) 20.0% 30 24 10.3 701.5 9.9 2105.7 6.|-- be-10925-cr01.9greatoaks.ca.ibone.comcast.net (68.86.87.158) 0.0% 30 30 13.3 18.3 11.9 54.4 7.|-- hu-0-11-0-0-pe03.11greatoaks.ca.ibone.comcast.net (68.86.85.238) 0.0% 30 30 11.7 20.5 11.0 91.0 8.|-- ae12.sjc12.ip4.gtt.net (173.205.58.169) 0.0% 30 30 15.3 17.0 11.1 37.4 9.|-- xe-1-0-1.sjc20.ip4.gtt.net (89.149.184.250) 0.0% 30 30 12.9 16.0 11.2 32.8 10.|-- as20473.xe-1-2-3.cr1.sjc2.us.as4436.gtt.net (69.22.130.90) 0.0% 30 30 16.9 17.5 11.7 29.0 11.|-- ??? 100.0 30 0 0.0 0.0 0.0 0.0 12.|-- mambo.koitsu.org (104.238.183.73) 0.0% 30 30 16.8 17.1 11.1 27.3 === END Data #2: === Tue Jan 19 22:29:00 PST 2016 (1453271340) Start: Tue Jan 19 22:29:00 2016 HOST: icarus.home.lan Loss% Snt Rcv Last Avg Best Wrst 1.|-- gw.home.lan (192.168.1.1) 0.0% 30 30 0.2 0.2 0.2 0.3 2.|-- 69.181.136.1 0.0% 30 30 8.8 13.7 8.7 28.4 3.|-- te-0-2-0-5-sur03.santaclara.ca.sfba.comcast.net (68.86.249.69) 0.0% 30 30 10.7 13.9 9.4 32.6 4.|-- hu-0-3-0-4-ar01.santaclara.ca.sfba.comcast.net (68.87.192.177) 0.0% 30 30 65.9 15.8 9.7 65.9 5.|-- be-33651-cr01.sunnyvale.ca.ibone.comcast.net (68.86.90.93) 26.7% 30 22 8344. 10013 8097. 11372.1 6.|-- be-10925-cr01.9greatoaks.ca.ibone.comcast.net (68.86.87.158) 0.0% 30 30 15.3 19.0 12.2 83.3 7.|-- hu-0-11-0-0-pe03.11greatoaks.ca.ibone.comcast.net (68.86.85.238) 0.0% 30 30 17.5 16.5 11.5 21.8 8.|-- ae12.sjc12.ip4.gtt.net (173.205.58.169) 0.0% 30 30 17.3 18.9 13.3 28.4 9.|-- xe-1-1-2.sjc20.ip4.gtt.net (89.149.184.246) 0.0% 30 30 22.0 18.7 14.5 26.0 10.|-- as20473.xe-1-2-3.cr1.sjc2.us.as4436.gtt.net (69.22.130.90) 0.0% 30 30 18.7 19.5 13.6 40.2 11.|-- ??? 100.0 30 0 0.0 0.0 0.0 0.0 12.|-- mambo.koitsu.org (104.238.183.73) 0.0% 30 30 20.7 21.4 13.3 91.0 === END === Tue Jan 19 22:30:00 PST 2016 (1453271400) Start: Tue Jan 19 22:30:00 2016 HOST: icarus.home.lan Loss% Snt Rcv Last Avg Best Wrst 1.|-- gw.home.lan (192.168.1.1) 0.0% 30 30 0.2 0.2 0.2 0.3 2.|-- 69.181.136.1 0.0% 30 30 16.2 16.4 9.7 50.5 3.|-- te-0-2-0-5-sur03.santaclara.ca.sfba.comcast.net (68.86.249.69) 0.0% 30 30 11.6 15.0 9.2 74.6 4.|-- hu-0-3-0-4-ar01.santaclara.ca.sfba.comcast.net (68.87.192.177) 0.0% 30 30 11.0 14.1 9.4 34.7 5.|-- be-33651-cr01.sunnyvale.ca.ibone.comcast.net (68.86.90.93) 6.7% 30 28 6042. 5278. 4875. 6044.0 6.|-- ??? 100.0 30 0 0.0 0.0 0.0 0.0 === END === Tue Jan 19 22:31:00 PST 2016 (1453271460) Start: Tue Jan 19 22:31:00 2016 HOST: icarus.home.lan Loss% Snt Rcv Last Avg Best Wrst 1.|-- gw.home.lan (192.168.1.1) 0.0% 30 30 0.3 0.2 0.2 0.3 2.|-- 69.181.136.1 0.0% 30 30 42.3 17.0 9.5 60.0 3.|-- te-0-2-0-5-sur03.santaclara.ca.sfba.comcast.net (68.86.249.69) 0.0% 30 30 17.7 18.6 9.9 81.3 4.|-- hu-0-3-0-4-ar01.santaclara.ca.sfba.comcast.net (68.87.192.177) 0.0% 30 30 12.7 14.8 9.2 39.2 5.|-- be-33651-cr01.sunnyvale.ca.ibone.comcast.net (68.86.90.93) 56.7% 30 13 114.9 49.6 9.3 199.0 6.|-- ??? 100.0 30 0 0.0 0.0 0.0 0.0 === END === Tue Jan 19 22:32:00 PST 2016 (1453271520) Start: Tue Jan 19 22:32:00 2016 HOST: icarus.home.lan Loss% Snt Rcv Last Avg Best Wrst 1.|-- gw.home.lan (192.168.1.1) 0.0% 30 30 0.3 0.2 0.2 0.3 2.|-- 69.181.136.1 0.0% 30 30 12.1 16.5 8.9 57.1 3.|-- te-0-2-0-5-sur03.santaclara.ca.sfba.comcast.net (68.86.249.69) 0.0% 30 30 11.0 15.0 9.5 53.6 4.|-- hu-0-3-0-4-ar01.santaclara.ca.sfba.comcast.net (68.87.192.177) 0.0% 30 30 13.2 14.0 9.4 33.6 5.|-- be-33651-cr01.sunnyvale.ca.ibone.comcast.net (68.86.90.93) 43.3% 30 17 132.7 25.9 9.2 132.7 6.|-- be-10925-cr01.9greatoaks.ca.ibone.comcast.net (68.86.87.158) 93.3% 30 2 20.5 17.3 14.2 20.5 7.|-- hu-0-11-0-0-pe03.11greatoaks.ca.ibone.comcast.net (68.86.85.238) 93.3% 30 2 19.9 16.0 12.0 19.9 8.|-- ae12.sjc12.ip4.gtt.net (173.205.58.169) 93.3% 30 2 13.4 13.5 13.4 13.7 9.|-- xe-1-1-2.sjc20.ip4.gtt.net (89.149.184.246) 93.3% 30 2 15.4 14.4 13.4 15.4 10.|-- as20473.xe-1-2-3.cr1.sjc2.us.as4436.gtt.net (69.22.130.90) 93.3% 30 2 12.8 17.2 12.8 21.7 11.|-- ??? 100.0 30 0 0.0 0.0 0.0 0.0 12.|-- mambo.koitsu.org (104.238.183.73) 93.3% 30 2 12.9 12.8 12.7 12.9 === END === Tue Jan 19 22:33:00 PST 2016 (1453271580) Start: Tue Jan 19 22:33:00 2016 HOST: icarus.home.lan Loss% Snt Rcv Last Avg Best Wrst 1.|-- gw.home.lan (192.168.1.1) 0.0% 30 30 0.3 0.2 0.2 0.3 2.|-- 69.181.136.1 0.0% 30 30 16.9 16.1 9.6 71.2 3.|-- te-0-2-0-5-sur03.santaclara.ca.sfba.comcast.net (68.86.249.69) 0.0% 30 30 12.3 15.0 9.4 52.6 4.|-- hu-0-3-0-4-ar01.santaclara.ca.sfba.comcast.net (68.87.192.177) 0.0% 30 30 14.5 14.3 9.2 35.6 5.|-- be-33651-cr01.sunnyvale.ca.ibone.comcast.net (68.86.90.93) 56.7% 30 13 13553 11910 9297. 13553.6 6.|-- be-10925-cr01.9greatoaks.ca.ibone.comcast.net (68.86.87.158) 0.0% 30 30 12.8 17.4 11.3 26.6 7.|-- hu-0-11-0-0-pe03.11greatoaks.ca.ibone.comcast.net (68.86.85.238) 0.0% 30 30 11.6 16.7 11.6 29.0 8.|-- ae12.sjc12.ip4.gtt.net (173.205.58.169) 0.0% 30 30 12.3 17.9 11.4 54.7 9.|-- xe-1-0-1.sjc20.ip4.gtt.net (89.149.184.250) 0.0% 30 30 18.4 16.7 11.2 35.0 10.|-- as20473.xe-1-2-3.cr1.sjc2.us.as4436.gtt.net (69.22.130.90) 0.0% 30 30 13.7 16.9 12.0 26.8 11.|-- ??? 100.0 30 0 0.0 0.0 0.0 0.0 12.|-- mambo.koitsu.org (104.238.183.73) 0.0% 30 30 17.6 18.7 12.3 64.6 === END Data #3: === Mon Jan 25 14:30:00 PST 2016 (1453761000) Start: Mon Jan 25 14:30:00 2016 HOST: icarus.home.lan Loss% Snt Rcv Last Avg Best Wrst 1.|-- gw.home.lan (192.168.1.1) 0.0% 30 30 0.3 0.2 0.2 0.3 2.|-- 69.181.136.1 0.0% 30 30 14.7 13.7 8.4 23.3 3.|-- te-0-2-0-5-sur03.santaclara.ca.sfba.comcast.net (68.86.249.69) 0.0% 30 30 10.1 16.6 9.2 100.5 4.|-- hu-0-3-0-4-ar01.santaclara.ca.sfba.comcast.net (68.87.192.177) 0.0% 30 30 15.3 15.1 9.1 38.5 5.|-- be-33651-cr01.sunnyvale.ca.ibone.comcast.net (68.86.90.93) 10.0% 30 27 10.7 355.4 9.4 1251.7 6.|-- be-10925-cr01.9greatoaks.ca.ibone.comcast.net (68.86.87.158) 0.0% 30 30 18.5 17.7 11.7 31.3 7.|-- hu-0-11-0-0-pe03.11greatoaks.ca.ibone.comcast.net (68.86.85.238) 0.0% 30 30 13.4 17.9 11.1 68.7 8.|-- ae12.sjc12.ip4.gtt.net (173.205.58.169) 0.0% 30 30 14.1 15.8 10.5 56.3 9.|-- xe-1-1-2.sjc20.ip4.gtt.net (89.149.184.246) 0.0% 30 30 14.6 15.0 11.0 20.9 10.|-- as20473.xe-1-2-3.cr1.sjc2.us.as4436.gtt.net (69.22.130.90) 0.0% 30 30 19.6 15.2 10.9 21.0 11.|-- ??? 100.0 30 0 0.0 0.0 0.0 0.0 12.|-- mambo.koitsu.org (104.238.183.73) 0.0% 30 30 16.5 16.1 11.1 41.9 === END === Mon Jan 25 14:31:00 PST 2016 (1453761060) Start: Mon Jan 25 14:31:00 2016 HOST: icarus.home.lan Loss% Snt Rcv Last Avg Best Wrst 1.|-- gw.home.lan (192.168.1.1) 0.0% 30 30 0.3 0.3 0.2 0.3 2.|-- 69.181.136.1 0.0% 30 30 14.9 15.0 8.9 56.8 3.|-- te-0-2-0-5-sur03.santaclara.ca.sfba.comcast.net (68.86.249.69) 0.0% 30 30 11.0 14.4 9.1 23.2 4.|-- hu-0-3-0-4-ar01.santaclara.ca.sfba.comcast.net (68.87.192.177) 0.0% 30 30 16.9 16.3 9.1 89.2 5.|-- be-33651-cr01.sunnyvale.ca.ibone.comcast.net (68.86.90.93) 10.0% 30 27 13.0 15.9 9.1 45.8 6.|-- ??? 100.0 30 0 0.0 0.0 0.0 0.0 === END === Mon Jan 25 14:32:00 PST 2016 (1453761120) Start: Mon Jan 25 14:32:00 2016 HOST: icarus.home.lan Loss% Snt Rcv Last Avg Best Wrst 1.|-- gw.home.lan (192.168.1.1) 0.0% 30 30 0.3 0.2 0.2 0.4 2.|-- 69.181.136.1 0.0% 30 30 17.1 13.7 9.3 31.9 3.|-- te-0-2-0-5-sur03.santaclara.ca.sfba.comcast.net (68.86.249.69) 0.0% 30 30 12.7 14.9 9.1 46.5 4.|-- hu-0-3-0-4-ar01.santaclara.ca.sfba.comcast.net (68.87.192.177) 0.0% 30 30 19.0 16.6 8.9 93.7 5.|-- be-33651-cr01.sunnyvale.ca.ibone.comcast.net (68.86.90.93) 10.0% 30 27 12.0 17.7 9.3 101.0 6.|-- ??? 100.0 30 0 0.0 0.0 0.0 0.0 === END === Mon Jan 25 14:33:00 PST 2016 (1453761180) Start: Mon Jan 25 14:33:00 2016 HOST: icarus.home.lan Loss% Snt Rcv Last Avg Best Wrst 1.|-- gw.home.lan (192.168.1.1) 0.0% 30 30 0.3 0.2 0.2 0.3 2.|-- 69.181.136.1 0.0% 30 30 10.2 23.7 8.6 121.1 3.|-- te-0-2-0-5-sur03.santaclara.ca.sfba.comcast.net (68.86.249.69) 0.0% 30 30 9.7 15.7 9.4 62.2 4.|-- hu-0-3-0-4-ar01.santaclara.ca.sfba.comcast.net (68.87.192.177) 0.0% 30 30 16.4 14.1 9.6 21.0 5.|-- be-33651-cr01.sunnyvale.ca.ibone.comcast.net (68.86.90.93) 0.0% 30 30 12.0 43.9 9.8 423.8 6.|-- ??? 100.0 30 0 0.0 0.0 0.0 0.0 === END === Mon Jan 25 14:34:00 PST 2016 (1453761240) Start: Mon Jan 25 14:34:00 2016 HOST: icarus.home.lan Loss% Snt Rcv Last Avg Best Wrst 1.|-- gw.home.lan (192.168.1.1) 0.0% 30 30 0.3 0.2 0.2 0.4 2.|-- 69.181.136.1 0.0% 30 30 12.0 14.4 8.7 57.1 3.|-- te-0-2-0-5-sur03.santaclara.ca.sfba.comcast.net (68.86.249.69) 0.0% 30 30 10.5 15.8 9.4 80.0 4.|-- hu-0-3-0-4-ar01.santaclara.ca.sfba.comcast.net (68.87.192.177) 0.0% 30 30 13.7 14.5 9.2 23.3 5.|-- be-33651-cr01.sunnyvale.ca.ibone.comcast.net (68.86.90.93) 0.0% 30 30 166.4 31.3 9.2 166.4 6.|-- be-10925-cr01.9greatoaks.ca.ibone.comcast.net (68.86.87.158) 83.3% 30 5 17.6 17.4 16.4 18.7 7.|-- hu-0-11-0-0-pe03.11greatoaks.ca.ibone.comcast.net (68.86.85.238) 83.3% 30 5 11.6 14.9 11.6 19.6 8.|-- ae12.sjc12.ip4.gtt.net (173.205.58.169) 80.0% 30 6 17.2 20.5 17.2 26.2 9.|-- xe-1-1-2.sjc20.ip4.gtt.net (89.149.184.246) 80.0% 30 6 12.8 15.0 12.8 20.9 10.|-- as20473.xe-1-2-3.cr1.sjc2.us.as4436.gtt.net (69.22.130.90) 80.0% 30 6 19.0 16.9 15.1 19.4 11.|-- ??? 100.0 30 0 0.0 0.0 0.0 0.0 12.|-- mambo.koitsu.org (104.238.183.73) 80.0% 30 6 12.5 15.8 11.4 20.7 === END === Mon Jan 25 14:35:00 PST 2016 (1453761300) Start: Mon Jan 25 14:35:00 2016 HOST: icarus.home.lan Loss% Snt Rcv Last Avg Best Wrst 1.|-- gw.home.lan (192.168.1.1) 0.0% 30 30 0.2 0.2 0.2 0.3 2.|-- 69.181.136.1 0.0% 30 30 22.9 18.4 8.9 93.2 3.|-- te-0-2-0-5-sur03.santaclara.ca.sfba.comcast.net (68.86.249.69) 0.0% 30 30 11.2 13.3 9.5 19.1 4.|-- hu-0-3-0-4-ar01.santaclara.ca.sfba.comcast.net (68.87.192.177) 0.0% 30 30 16.2 14.0 9.0 46.1 5.|-- be-33651-cr01.sunnyvale.ca.ibone.comcast.net (68.86.90.93) 26.7% 30 22 1776. 3705. 1776. 4921.8 6.|-- be-10925-cr01.9greatoaks.ca.ibone.comcast.net (68.86.87.158) 0.0% 30 30 19.3 19.6 11.2 73.6 7.|-- hu-0-11-0-0-pe03.11greatoaks.ca.ibone.comcast.net (68.86.85.238) 0.0% 30 30 17.0 15.0 10.9 25.1 8.|-- ae12.sjc12.ip4.gtt.net (173.205.58.169) 0.0% 30 30 18.0 15.9 10.9 41.0 9.|-- xe-1-1-2.sjc20.ip4.gtt.net (89.149.184.246) 0.0% 30 30 14.3 18.4 10.6 79.6 10.|-- as20473.xe-1-2-3.cr1.sjc2.us.as4436.gtt.net (69.22.130.90) 0.0% 30 30 19.0 17.8 11.1 72.5 11.|-- ??? 100.0 30 0 0.0 0.0 0.0 0.0 12.|-- mambo.koitsu.org (104.238.183.73) 0.0% 30 30 20.1 15.9 11.3 22.8 === END -- | Jeremy Chadwick jdc@koitsu.org | | UNIX Systems Administrator http://jdc.koitsu.org/ | | Making life hard for others since 1977. PGP 4BD6C0CB |