Intermittent ICMP and DNS for 4.2.2.[123]

I've been noticing packet loss while pinging 4.2.2.1, 4.2.2.2, and 4.2.2.3 in the last few minutes, which are usually my go to IP addresses for testing external server connectivity. DNS also seems to be intermittent. -- Dan White

Rock solid at 26 ms for me -> traceroute 4.2.2.2 traceroute to 4.2.2.2 (4.2.2.2), 30 hops max, 40 byte packets 1 rrcs-74-218-208-193.central.biz.rr.com (74.218.208.193) 0.531 ms 0.485 ms 0.450 ms 2 rrcs-74-218-88-133.central.biz.rr.com (74.218.88.133) 0.418 ms 0.390 ms 0.589 ms 3 rrcs-74-218-88-129.central.biz.rr.com (74.218.88.129) 1.061 ms 2.046 ms 2.024 ms 4 vln864.dytnoh1-swt101.woh.rr.com (74.218.88.73) 3.000 ms 2.977 ms 2.943 ms 5 gig2-1-801.dytnoh1-rtr102.woh.rr.com (70.60.47.5) 2.910 ms 2.884 ms 2.858 ms 6 gig5-0.dytnoh1-rtr1.woh.rr.com (24.29.160.21) 2.840 ms 2.845 ms 1.922 ms 7 tge5-1-0.tr00.clmkohpe.mwrtn.rr.com (65.25.137.177) 8.852 ms 8.812 ms 8.788 ms 8 ae-4-0.cr0.chi30.tbone.rr.com (66.109.6.68) 18.744 ms 18.953 ms ae-9-0.cr0.chi30.tbone.rr.com (107.14.19.16) 18.692 ms 9 107.14.17.147 (107.14.17.147) 18.899 ms ae-1-0.pr0.chi10.tbone.rr.com (66.109.6.155) 18.874 ms 18.839 ms 10 xe-10-2-0.edge2.Chicago2.Level3.net (4.59.28.109) 25.573 ms xe-10-0-0.edge2.Chicago2.Level3.net (4.59.28.101) 25.549 ms xe-10-2-0.edge2.Chicago2.Level3.net (4.59.28.109) 25.507 ms 11 vlan51.ebr1.Chicago2.Level3.net (4.69.138.158) 25.482 ms 25.458 ms vlan52.ebr2.Chicago2.Level3.net (4.69.138.190) 25.432 ms 12 ae-6-6.ebr1.Chicago1.Level3.net (4.69.140.189) 41.152 ms 34.879 ms 35.936 ms 13 ae-11-51.car1.Chicago1.Level3.net (4.69.138.3) 35.880 ms ae-21-52.car1.Chicago1.Level3.net (4.69.138.35) 35.848 ms 35.822 ms 14 vnsc-bak.sys.gtei.net (4.2.2.2) 36.780 ms 36.755 ms Josh Luthman Office: 937-552-2340 Direct: 937-552-2343 1100 Wayne St Suite 1337 Troy, OH 45373 On Wed, Oct 19, 2011 at 11:59 AM, Dan White <dwhite@olp.net> wrote:
I've been noticing packet loss while pinging 4.2.2.1, 4.2.2.2, and 4.2.2.3 in the last few minutes, which are usually my go to IP addresses for testing external server connectivity.
DNS also seems to be intermittent.
-- Dan White _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages

Additional information (I'm still seeing the problem) ~$ traceroute 4.2.2.2 traceroute to 4.2.2.2 (4.2.2.2), 30 hops max, 60 byte packets 1 10.0.2.1 (10.0.2.1) 3.813 ms 3.779 ms 4.183 ms 2 olp-67-217-152-5.olp.net (67.217.152.5) 2.547 ms 2.582 ms 2.761 ms 3 router.olp.net (67.217.151.97) 2.520 ms 2.514 ms 2.509 ms 4 olp-67-217-152-34.olp.net (67.217.152.34) 2.678 ms 2.659 ms 2.582 ms 5 ge-6-10-183.car1.dallas1.level3.net (4.71.13.121) 11.008 ms 11.010 ms 11.002 ms 6 * * * 7 * * * 8 * * * 9 * * * 10 vnsc-bak.sys.gtei.net (4.2.2.2) 56.222 ms 61.235 ms 61.239 ms ~$ ping -c 10 4.2.2.2 PING 4.2.2.2 (4.2.2.2) 56(84) bytes of data. 64 bytes from 4.2.2.2: icmp_req=1 ttl=59 time=51.1 ms 64 bytes from 4.2.2.2: icmp_req=6 ttl=59 time=57.0 ms 64 bytes from 4.2.2.2: icmp_req=7 ttl=59 time=35.5 ms 64 bytes from 4.2.2.2: icmp_req=8 ttl=59 time=31.7 ms --- 4.2.2.2 ping statistics --- 10 packets transmitted, 4 received, 60% packet loss, time 9045ms rtt min/avg/max/mdev = 31.754/43.865/57.000/10.493 ms ~$ ping -c 10 4.71.13.121 PING 4.71.13.121 (4.71.13.121) 56(84) bytes of data. 64 bytes from 4.71.13.121: icmp_req=1 ttl=251 time=14.7 ms 64 bytes from 4.71.13.121: icmp_req=2 ttl=251 time=9.15 ms 64 bytes from 4.71.13.121: icmp_req=3 ttl=251 time=9.14 ms 64 bytes from 4.71.13.121: icmp_req=4 ttl=251 time=8.82 ms 64 bytes from 4.71.13.121: icmp_req=5 ttl=251 time=11.3 ms 64 bytes from 4.71.13.121: icmp_req=6 ttl=251 time=9.72 ms 64 bytes from 4.71.13.121: icmp_req=7 ttl=251 time=8.47 ms 64 bytes from 4.71.13.121: icmp_req=8 ttl=251 time=8.47 ms 64 bytes from 4.71.13.121: icmp_req=9 ttl=251 time=9.57 ms 64 bytes from 4.71.13.121: icmp_req=10 ttl=251 time=9.49 ms --- 4.71.13.121 ping statistics --- 10 packets transmitted, 10 received, 0% packet loss, time 9014ms rtt min/avg/max/mdev = 8.471/9.898/14.793/1.809 ms On 19/10/11 12:14 -0400, Josh Luthman wrote: ping -c 10 4.2.2
Rock solid at 26 ms for me
-> traceroute 4.2.2.2 traceroute to 4.2.2.2 (4.2.2.2), 30 hops max, 40 byte packets 1 rrcs-74-218-208-193.central.biz.rr.com (74.218.208.193) 0.531 ms 0.485 ms 0.450 ms 2 rrcs-74-218-88-133.central.biz.rr.com (74.218.88.133) 0.418 ms 0.390 ms 0.589 ms 3 rrcs-74-218-88-129.central.biz.rr.com (74.218.88.129) 1.061 ms 2.046 ms 2.024 ms 4 vln864.dytnoh1-swt101.woh.rr.com (74.218.88.73) 3.000 ms 2.977 ms 2.943 ms 5 gig2-1-801.dytnoh1-rtr102.woh.rr.com (70.60.47.5) 2.910 ms 2.884 ms 2.858 ms 6 gig5-0.dytnoh1-rtr1.woh.rr.com (24.29.160.21) 2.840 ms 2.845 ms 1.922 ms 7 tge5-1-0.tr00.clmkohpe.mwrtn.rr.com (65.25.137.177) 8.852 ms 8.812 ms 8.788 ms 8 ae-4-0.cr0.chi30.tbone.rr.com (66.109.6.68) 18.744 ms 18.953 ms ae-9-0.cr0.chi30.tbone.rr.com (107.14.19.16) 18.692 ms 9 107.14.17.147 (107.14.17.147) 18.899 ms ae-1-0.pr0.chi10.tbone.rr.com (66.109.6.155) 18.874 ms 18.839 ms 10 xe-10-2-0.edge2.Chicago2.Level3.net (4.59.28.109) 25.573 ms xe-10-0-0.edge2.Chicago2.Level3.net (4.59.28.101) 25.549 ms xe-10-2-0.edge2.Chicago2.Level3.net (4.59.28.109) 25.507 ms 11 vlan51.ebr1.Chicago2.Level3.net (4.69.138.158) 25.482 ms 25.458 ms vlan52.ebr2.Chicago2.Level3.net (4.69.138.190) 25.432 ms 12 ae-6-6.ebr1.Chicago1.Level3.net (4.69.140.189) 41.152 ms 34.879 ms 35.936 ms 13 ae-11-51.car1.Chicago1.Level3.net (4.69.138.3) 35.880 ms ae-21-52.car1.Chicago1.Level3.net (4.69.138.35) 35.848 ms 35.822 ms 14 vnsc-bak.sys.gtei.net (4.2.2.2) 36.780 ms 36.755 ms
Josh Luthman Office: 937-552-2340 Direct: 937-552-2343 1100 Wayne St Suite 1337 Troy, OH 45373
On Wed, Oct 19, 2011 at 11:59 AM, Dan White <dwhite@olp.net> wrote:
I've been noticing packet loss while pinging 4.2.2.1, 4.2.2.2, and 4.2.2.3 in the last few minutes, which are usually my go to IP addresses for testing external server connectivity.
DNS also seems to be intermittent.
-- Dan White _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
-- Dan White BTC Broadband Ph 918.366.0248 (direct) main: (918)366-8000 Fax 918.366.6610 email: dwhite@olp.net http://www.btcbroadband.com

If you're seeing packet loss on pings, please use a tool like mtr to show where it's happening. Frank -----Original Message----- From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On Behalf Of Dan White Sent: Wednesday, October 19, 2011 11:21 AM To: Josh Luthman Cc: outages@outages.org Subject: Re: [outages] Intermittent ICMP and DNS for 4.2.2.[123] Additional information (I'm still seeing the problem) ~$ traceroute 4.2.2.2 traceroute to 4.2.2.2 (4.2.2.2), 30 hops max, 60 byte packets 1 10.0.2.1 (10.0.2.1) 3.813 ms 3.779 ms 4.183 ms 2 olp-67-217-152-5.olp.net (67.217.152.5) 2.547 ms 2.582 ms 2.761 ms 3 router.olp.net (67.217.151.97) 2.520 ms 2.514 ms 2.509 ms 4 olp-67-217-152-34.olp.net (67.217.152.34) 2.678 ms 2.659 ms 2.582 ms 5 ge-6-10-183.car1.dallas1.level3.net (4.71.13.121) 11.008 ms 11.010 ms 11.002 ms 6 * * * 7 * * * 8 * * * 9 * * * 10 vnsc-bak.sys.gtei.net (4.2.2.2) 56.222 ms 61.235 ms 61.239 ms ~$ ping -c 10 4.2.2.2 PING 4.2.2.2 (4.2.2.2) 56(84) bytes of data. 64 bytes from 4.2.2.2: icmp_req=1 ttl=59 time=51.1 ms 64 bytes from 4.2.2.2: icmp_req=6 ttl=59 time=57.0 ms 64 bytes from 4.2.2.2: icmp_req=7 ttl=59 time=35.5 ms 64 bytes from 4.2.2.2: icmp_req=8 ttl=59 time=31.7 ms --- 4.2.2.2 ping statistics --- 10 packets transmitted, 4 received, 60% packet loss, time 9045ms rtt min/avg/max/mdev = 31.754/43.865/57.000/10.493 ms ~$ ping -c 10 4.71.13.121 PING 4.71.13.121 (4.71.13.121) 56(84) bytes of data. 64 bytes from 4.71.13.121: icmp_req=1 ttl=251 time=14.7 ms 64 bytes from 4.71.13.121: icmp_req=2 ttl=251 time=9.15 ms 64 bytes from 4.71.13.121: icmp_req=3 ttl=251 time=9.14 ms 64 bytes from 4.71.13.121: icmp_req=4 ttl=251 time=8.82 ms 64 bytes from 4.71.13.121: icmp_req=5 ttl=251 time=11.3 ms 64 bytes from 4.71.13.121: icmp_req=6 ttl=251 time=9.72 ms 64 bytes from 4.71.13.121: icmp_req=7 ttl=251 time=8.47 ms 64 bytes from 4.71.13.121: icmp_req=8 ttl=251 time=8.47 ms 64 bytes from 4.71.13.121: icmp_req=9 ttl=251 time=9.57 ms 64 bytes from 4.71.13.121: icmp_req=10 ttl=251 time=9.49 ms --- 4.71.13.121 ping statistics --- 10 packets transmitted, 10 received, 0% packet loss, time 9014ms rtt min/avg/max/mdev = 8.471/9.898/14.793/1.809 ms On 19/10/11 12:14 -0400, Josh Luthman wrote: ping -c 10 4.2.2
Rock solid at 26 ms for me
-> traceroute 4.2.2.2 traceroute to 4.2.2.2 (4.2.2.2), 30 hops max, 40 byte packets 1 rrcs-74-218-208-193.central.biz.rr.com (74.218.208.193) 0.531 ms 0.485 ms 0.450 ms 2 rrcs-74-218-88-133.central.biz.rr.com (74.218.88.133) 0.418 ms 0.390 ms 0.589 ms 3 rrcs-74-218-88-129.central.biz.rr.com (74.218.88.129) 1.061 ms 2.046 ms 2.024 ms 4 vln864.dytnoh1-swt101.woh.rr.com (74.218.88.73) 3.000 ms 2.977 ms 2.943 ms 5 gig2-1-801.dytnoh1-rtr102.woh.rr.com (70.60.47.5) 2.910 ms 2.884 ms 2.858 ms 6 gig5-0.dytnoh1-rtr1.woh.rr.com (24.29.160.21) 2.840 ms 2.845 ms 1.922 ms 7 tge5-1-0.tr00.clmkohpe.mwrtn.rr.com (65.25.137.177) 8.852 ms 8.812 ms 8.788 ms 8 ae-4-0.cr0.chi30.tbone.rr.com (66.109.6.68) 18.744 ms 18.953 ms ae-9-0.cr0.chi30.tbone.rr.com (107.14.19.16) 18.692 ms 9 107.14.17.147 (107.14.17.147) 18.899 ms ae-1-0.pr0.chi10.tbone.rr.com (66.109.6.155) 18.874 ms 18.839 ms 10 xe-10-2-0.edge2.Chicago2.Level3.net (4.59.28.109) 25.573 ms xe-10-0-0.edge2.Chicago2.Level3.net (4.59.28.101) 25.549 ms xe-10-2-0.edge2.Chicago2.Level3.net (4.59.28.109) 25.507 ms 11 vlan51.ebr1.Chicago2.Level3.net (4.69.138.158) 25.482 ms 25.458 ms vlan52.ebr2.Chicago2.Level3.net (4.69.138.190) 25.432 ms 12 ae-6-6.ebr1.Chicago1.Level3.net (4.69.140.189) 41.152 ms 34.879 ms 35.936 ms 13 ae-11-51.car1.Chicago1.Level3.net (4.69.138.3) 35.880 ms ae-21-52.car1.Chicago1.Level3.net (4.69.138.35) 35.848 ms 35.822 ms 14 vnsc-bak.sys.gtei.net (4.2.2.2) 36.780 ms 36.755 ms
Josh Luthman Office: 937-552-2340 Direct: 937-552-2343 1100 Wayne St Suite 1337 Troy, OH 45373
On Wed, Oct 19, 2011 at 11:59 AM, Dan White <dwhite@olp.net> wrote:
I've been noticing packet loss while pinging 4.2.2.1, 4.2.2.2, and 4.2.2.3 in the last few minutes, which are usually my go to IP addresses for testing external server connectivity.
DNS also seems to be intermittent.
-- Dan White _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
-- Dan White BTC Broadband Ph 918.366.0248 (direct) main: (918)366-8000 Fax 918.366.6610 email: dwhite@olp.net http://www.btcbroadband.com _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages

10-4: ~$ mtr -n -c 100 --report 4.2.2.2 HOST: quark Loss% Snt Last Avg Best Wrst StDev 1.|-- 10.0.2.1 0.0% 100 1.9 1.9 0.8 4.5 0.7 2.|-- 67.217.152.5 0.0% 100 1.5 2.4 0.8 10.3 1.8 3.|-- 67.217.151.97 0.0% 100 1.1 4.1 0.9 136.6 15.9 4.|-- 67.217.152.34 0.0% 100 1.5 1.6 1.5 2.4 0.1 5.|-- 4.71.13.121 0.0% 100 8.6 23.7 8.5 341.7 49.5 6.|-- 4.2.2.2 60.0% 100 47.4 45.0 40.0 54.7 3.2 Under normal circumstances, I typically see at least one more hop in front of 4.2.2.2, and much lower latency. From this, I assume the packet loss I'm seeing is due to a problem within Level 3's network. And may be limited to a Dallas link/router, based on others' reports. On 19/10/11 11:45 -0500, Frank Bulk wrote:
If you're seeing packet loss on pings, please use a tool like mtr to show where it's happening.
Frank
-----Original Message----- From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On Behalf Of Dan White Sent: Wednesday, October 19, 2011 11:21 AM To: Josh Luthman Cc: outages@outages.org Subject: Re: [outages] Intermittent ICMP and DNS for 4.2.2.[123]
Additional information (I'm still seeing the problem)
~$ traceroute 4.2.2.2 traceroute to 4.2.2.2 (4.2.2.2), 30 hops max, 60 byte packets 1 10.0.2.1 (10.0.2.1) 3.813 ms 3.779 ms 4.183 ms 2 olp-67-217-152-5.olp.net (67.217.152.5) 2.547 ms 2.582 ms 2.761 ms 3 router.olp.net (67.217.151.97) 2.520 ms 2.514 ms 2.509 ms 4 olp-67-217-152-34.olp.net (67.217.152.34) 2.678 ms 2.659 ms 2.582 ms 5 ge-6-10-183.car1.dallas1.level3.net (4.71.13.121) 11.008 ms 11.010 ms 11.002 ms 6 * * * 7 * * * 8 * * * 9 * * * 10 vnsc-bak.sys.gtei.net (4.2.2.2) 56.222 ms 61.235 ms 61.239 ms
~$ ping -c 10 4.2.2.2 PING 4.2.2.2 (4.2.2.2) 56(84) bytes of data. 64 bytes from 4.2.2.2: icmp_req=1 ttl=59 time=51.1 ms 64 bytes from 4.2.2.2: icmp_req=6 ttl=59 time=57.0 ms 64 bytes from 4.2.2.2: icmp_req=7 ttl=59 time=35.5 ms 64 bytes from 4.2.2.2: icmp_req=8 ttl=59 time=31.7 ms
--- 4.2.2.2 ping statistics --- 10 packets transmitted, 4 received, 60% packet loss, time 9045ms rtt min/avg/max/mdev = 31.754/43.865/57.000/10.493 ms
~$ ping -c 10 4.71.13.121 PING 4.71.13.121 (4.71.13.121) 56(84) bytes of data. 64 bytes from 4.71.13.121: icmp_req=1 ttl=251 time=14.7 ms 64 bytes from 4.71.13.121: icmp_req=2 ttl=251 time=9.15 ms 64 bytes from 4.71.13.121: icmp_req=3 ttl=251 time=9.14 ms 64 bytes from 4.71.13.121: icmp_req=4 ttl=251 time=8.82 ms 64 bytes from 4.71.13.121: icmp_req=5 ttl=251 time=11.3 ms 64 bytes from 4.71.13.121: icmp_req=6 ttl=251 time=9.72 ms 64 bytes from 4.71.13.121: icmp_req=7 ttl=251 time=8.47 ms 64 bytes from 4.71.13.121: icmp_req=8 ttl=251 time=8.47 ms 64 bytes from 4.71.13.121: icmp_req=9 ttl=251 time=9.57 ms 64 bytes from 4.71.13.121: icmp_req=10 ttl=251 time=9.49 ms
--- 4.71.13.121 ping statistics --- 10 packets transmitted, 10 received, 0% packet loss, time 9014ms rtt min/avg/max/mdev = 8.471/9.898/14.793/1.809 ms
On 19/10/11 12:14 -0400, Josh Luthman wrote: ping -c 10 4.2.2
Rock solid at 26 ms for me
-> traceroute 4.2.2.2 traceroute to 4.2.2.2 (4.2.2.2), 30 hops max, 40 byte packets 1 rrcs-74-218-208-193.central.biz.rr.com (74.218.208.193) 0.531 ms 0.485 ms 0.450 ms 2 rrcs-74-218-88-133.central.biz.rr.com (74.218.88.133) 0.418 ms 0.390 ms 0.589 ms 3 rrcs-74-218-88-129.central.biz.rr.com (74.218.88.129) 1.061 ms 2.046 ms 2.024 ms 4 vln864.dytnoh1-swt101.woh.rr.com (74.218.88.73) 3.000 ms 2.977 ms 2.943 ms 5 gig2-1-801.dytnoh1-rtr102.woh.rr.com (70.60.47.5) 2.910 ms 2.884 ms 2.858 ms 6 gig5-0.dytnoh1-rtr1.woh.rr.com (24.29.160.21) 2.840 ms 2.845 ms 1.922 ms 7 tge5-1-0.tr00.clmkohpe.mwrtn.rr.com (65.25.137.177) 8.852 ms 8.812 ms 8.788 ms 8 ae-4-0.cr0.chi30.tbone.rr.com (66.109.6.68) 18.744 ms 18.953 ms ae-9-0.cr0.chi30.tbone.rr.com (107.14.19.16) 18.692 ms 9 107.14.17.147 (107.14.17.147) 18.899 ms ae-1-0.pr0.chi10.tbone.rr.com (66.109.6.155) 18.874 ms 18.839 ms 10 xe-10-2-0.edge2.Chicago2.Level3.net (4.59.28.109) 25.573 ms xe-10-0-0.edge2.Chicago2.Level3.net (4.59.28.101) 25.549 ms xe-10-2-0.edge2.Chicago2.Level3.net (4.59.28.109) 25.507 ms 11 vlan51.ebr1.Chicago2.Level3.net (4.69.138.158) 25.482 ms 25.458 ms vlan52.ebr2.Chicago2.Level3.net (4.69.138.190) 25.432 ms 12 ae-6-6.ebr1.Chicago1.Level3.net (4.69.140.189) 41.152 ms 34.879 ms 35.936 ms 13 ae-11-51.car1.Chicago1.Level3.net (4.69.138.3) 35.880 ms ae-21-52.car1.Chicago1.Level3.net (4.69.138.35) 35.848 ms 35.822 ms 14 vnsc-bak.sys.gtei.net (4.2.2.2) 36.780 ms 36.755 ms
Josh Luthman Office: 937-552-2340 Direct: 937-552-2343 1100 Wayne St Suite 1337 Troy, OH 45373
On Wed, Oct 19, 2011 at 11:59 AM, Dan White <dwhite@olp.net> wrote:
I've been noticing packet loss while pinging 4.2.2.1, 4.2.2.2, and 4.2.2.3 in the last few minutes, which are usually my go to IP addresses for testing external server connectivity.
DNS also seems to be intermittent.
-- Dan White _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
-- Dan White BTC Broadband Ph 918.366.0248 (direct) main: (918)366-8000 Fax 918.366.6610 email: dwhite@olp.net http://www.btcbroadband.com _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
-- Dan White BTC Broadband Ph 918.366.0248 (direct) main: (918)366-8000 Fax 918.366.6610 email: dwhite@olp.net http://www.btcbroadband.com

Not to possibly point out the obvious, but 4.2.2.2 is an anycast address, so you'll be hitting all sorts of different destiantions when trying to trace to it, depending on where you are. e.g., I tested from a few shells I have around: LA: 6. ae-21-70.car1.LosAngeles1.Le 0.0% 5 10.8 11.8 9.3 18.2 3.6 7. vnsc-bak.sys.gtei.net 0.0% 5 9.7 11.0 9.7 12.6 1.2 TX: 6. xe-8-1-0.edge4.Dallas3.Level 0.0% 5 0.6 0.7 0.6 0.9 0.1 7. ae-31-80.car1.Dallas1.Level3 0.0% 5 163.2 157.8 123.5 181.6 22.2 8. vnsc-bak.sys.gtei.net 80.0% 5 30.0 30.0 30.0 30.0 0.0 Bay Area (Fremont): 6. ae-11-60.car1.SanJose1.Level 0.0% 5 2.1 6.0 1.9 22.0 9.0 7. vnsc-bak.sys.gtei.net 0.0% 5 2.0 2.0 1.9 2.1 0.1 So it seems that (at least) the one in TX might have issues). -T On Wed, Oct 19, 2011 at 12:08:11PM -0500, Dan White <dwhite@olp.net> wrote:
10-4:
~$ mtr -n -c 100 --report 4.2.2.2 HOST: quark Loss% Snt Last Avg Best Wrst StDev 1.|-- 10.0.2.1 0.0% 100 1.9 1.9 0.8 4.5 0.7 2.|-- 67.217.152.5 0.0% 100 1.5 2.4 0.8 10.3 1.8 3.|-- 67.217.151.97 0.0% 100 1.1 4.1 0.9 136.6 15.9 4.|-- 67.217.152.34 0.0% 100 1.5 1.6 1.5 2.4 0.1 5.|-- 4.71.13.121 0.0% 100 8.6 23.7 8.5 341.7 49.5 6.|-- 4.2.2.2 60.0% 100 47.4 45.0 40.0 54.7 3.2
Under normal circumstances, I typically see at least one more hop in front of 4.2.2.2, and much lower latency.
From this, I assume the packet loss I'm seeing is due to a problem within Level 3's network. And may be limited to a Dallas link/router, based on others' reports.
On 19/10/11Â 11:45Â -0500, Frank Bulk wrote:
If you're seeing packet loss on pings, please use a tool like mtr to show where it's happening.
Frank
-----Original Message----- From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On Behalf Of Dan White Sent: Wednesday, October 19, 2011 11:21 AM To: Josh Luthman Cc: outages@outages.org Subject: Re: [outages] Intermittent ICMP and DNS for 4.2.2.[123]
Additional information (I'm still seeing the problem)
~$ traceroute 4.2.2.2 traceroute to 4.2.2.2 (4.2.2.2), 30 hops max, 60 byte packets 1 10.0.2.1 (10.0.2.1) 3.813 ms 3.779 ms 4.183 ms 2 olp-67-217-152-5.olp.net (67.217.152.5) 2.547 ms 2.582 ms 2.761 ms 3 router.olp.net (67.217.151.97) 2.520 ms 2.514 ms 2.509 ms 4 olp-67-217-152-34.olp.net (67.217.152.34) 2.678 ms 2.659 ms 2.582 ms 5 ge-6-10-183.car1.dallas1.level3.net (4.71.13.121) 11.008 ms 11.010 ms 11.002 ms 6 * * * 7 * * * 8 * * * 9 * * * 10 vnsc-bak.sys.gtei.net (4.2.2.2) 56.222 ms 61.235 ms 61.239 ms
~$ ping -c 10 4.2.2.2 PING 4.2.2.2 (4.2.2.2) 56(84) bytes of data. 64 bytes from 4.2.2.2: icmp_req=1 ttl=59 time=51.1 ms 64 bytes from 4.2.2.2: icmp_req=6 ttl=59 time=57.0 ms 64 bytes from 4.2.2.2: icmp_req=7 ttl=59 time=35.5 ms 64 bytes from 4.2.2.2: icmp_req=8 ttl=59 time=31.7 ms
--- 4.2.2.2 ping statistics --- 10 packets transmitted, 4 received, 60% packet loss, time 9045ms rtt min/avg/max/mdev = 31.754/43.865/57.000/10.493 ms
~$ ping -c 10 4.71.13.121 PING 4.71.13.121 (4.71.13.121) 56(84) bytes of data. 64 bytes from 4.71.13.121: icmp_req=1 ttl=251 time=14.7 ms 64 bytes from 4.71.13.121: icmp_req=2 ttl=251 time=9.15 ms 64 bytes from 4.71.13.121: icmp_req=3 ttl=251 time=9.14 ms 64 bytes from 4.71.13.121: icmp_req=4 ttl=251 time=8.82 ms 64 bytes from 4.71.13.121: icmp_req=5 ttl=251 time=11.3 ms 64 bytes from 4.71.13.121: icmp_req=6 ttl=251 time=9.72 ms 64 bytes from 4.71.13.121: icmp_req=7 ttl=251 time=8.47 ms 64 bytes from 4.71.13.121: icmp_req=8 ttl=251 time=8.47 ms 64 bytes from 4.71.13.121: icmp_req=9 ttl=251 time=9.57 ms 64 bytes from 4.71.13.121: icmp_req=10 ttl=251 time=9.49 ms
--- 4.71.13.121 ping statistics --- 10 packets transmitted, 10 received, 0% packet loss, time 9014ms rtt min/avg/max/mdev = 8.471/9.898/14.793/1.809 ms
On 19/10/11Â 12:14Â -0400, Josh Luthman wrote: ping -c 10 4.2.2
Rock solid at 26 ms for me
-> traceroute 4.2.2.2 traceroute to 4.2.2.2 (4.2.2.2), 30 hops max, 40 byte packets 1 rrcs-74-218-208-193.central.biz.rr.com (74.218.208.193) 0.531 ms 0.485 ms 0.450 ms 2 rrcs-74-218-88-133.central.biz.rr.com (74.218.88.133) 0.418 ms 0.390 ms 0.589 ms 3 rrcs-74-218-88-129.central.biz.rr.com (74.218.88.129) 1.061 ms 2.046 ms 2.024 ms 4 vln864.dytnoh1-swt101.woh.rr.com (74.218.88.73) 3.000 ms 2.977 ms 2.943 ms 5 gig2-1-801.dytnoh1-rtr102.woh.rr.com (70.60.47.5) 2.910 ms 2.884 ms 2.858 ms 6 gig5-0.dytnoh1-rtr1.woh.rr.com (24.29.160.21) 2.840 ms 2.845 ms 1.922 ms 7 tge5-1-0.tr00.clmkohpe.mwrtn.rr.com (65.25.137.177) 8.852 ms 8.812 ms 8.788 ms 8 ae-4-0.cr0.chi30.tbone.rr.com (66.109.6.68) 18.744 ms 18.953 ms ae-9-0.cr0.chi30.tbone.rr.com (107.14.19.16) 18.692 ms 9 107.14.17.147 (107.14.17.147) 18.899 ms ae-1-0.pr0.chi10.tbone.rr.com (66.109.6.155) 18.874 ms 18.839 ms 10 xe-10-2-0.edge2.Chicago2.Level3.net (4.59.28.109) 25.573 ms xe-10-0-0.edge2.Chicago2.Level3.net (4.59.28.101) 25.549 ms xe-10-2-0.edge2.Chicago2.Level3.net (4.59.28.109) 25.507 ms 11 vlan51.ebr1.Chicago2.Level3.net (4.69.138.158) 25.482 ms 25.458 ms vlan52.ebr2.Chicago2.Level3.net (4.69.138.190) 25.432 ms 12 ae-6-6.ebr1.Chicago1.Level3.net (4.69.140.189) 41.152 ms 34.879 ms 35.936 ms 13 ae-11-51.car1.Chicago1.Level3.net (4.69.138.3) 35.880 ms ae-21-52.car1.Chicago1.Level3.net (4.69.138.35) 35.848 ms 35.822 ms 14 vnsc-bak.sys.gtei.net (4.2.2.2) 36.780 ms 36.755 ms
Josh Luthman Office: 937-552-2340 Direct: 937-552-2343 1100 Wayne St Suite 1337 Troy, OH 45373
On Wed, Oct 19, 2011 at 11:59 AM, Dan White <dwhite@olp.net> wrote:
I've been noticing packet loss while pinging 4.2.2.1, 4.2.2.2, and 4.2.2.3 in the last few minutes, which are usually my go to IP addresses for testing external server connectivity.
DNS also seems to be intermittent.
-- Dan White _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
-- Dan White BTC Broadband Ph 918.366.0248 (direct) main: (918)366-8000 Fax 918.366.6610 email: dwhite@olp.net http://www.btcbroadband.com _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
-- Dan White BTC Broadband Ph 918.366.0248 (direct) main: (918)366-8000 Fax 918.366.6610 email: dwhite@olp.net http://www.btcbroadband.com _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages

We're seeing the issue clearing up for us now: http://web.olp.net/dwhite/4.2.2.2.png On 19/10/11 12:08 -0500, Dan White wrote:
10-4:
~$ mtr -n -c 100 --report 4.2.2.2 HOST: quark Loss% Snt Last Avg Best Wrst StDev 1.|-- 10.0.2.1 0.0% 100 1.9 1.9 0.8 4.5 0.7 2.|-- 67.217.152.5 0.0% 100 1.5 2.4 0.8 10.3 1.8 3.|-- 67.217.151.97 0.0% 100 1.1 4.1 0.9 136.6 15.9 4.|-- 67.217.152.34 0.0% 100 1.5 1.6 1.5 2.4 0.1 5.|-- 4.71.13.121 0.0% 100 8.6 23.7 8.5 341.7 49.5 6.|-- 4.2.2.2 60.0% 100 47.4 45.0 40.0 54.7 3.2
Under normal circumstances, I typically see at least one more hop in front of 4.2.2.2, and much lower latency.
From this, I assume the packet loss I'm seeing is due to a problem within Level 3's network. And may be limited to a Dallas link/router, based on others' reports.
-- Dan White

I am seeing the same. Tracing route to vnsc-pri.sys.gtei.net [4.2.2.1] over a maximum of 30 hops: 1 1 ms <1 ms <1 ms 173-164-56-78-colorado.hfc.comcastbusiness.net [ 173.164.56.78] 2 52 ms 24 ms 29 ms 73.83.164.1 3 10 ms 15 ms 10 ms te-8-1-ur01.boulder.co.denver.comcast.net [68.85 .107.85] 4 8 ms 9 ms 9 ms te-7-4-ur02.boulder.co.denver.comcast.net [68.86 .103.122] 5 12 ms 15 ms 16 ms te-0-8-0-0-ar02.aurora.co.denver.comcast.net [68 .86.179.97] 6 27 ms 11 ms 14 ms pos-3-8-0-0-cr01.denver.co.ibone.comcast.net [68 .86.94.153] 7 52 ms 28 ms 11 ms xe-9-2-0.edge3.Denver1.Level3.net [4.28.20.5] 8 18 ms 11 ms 12 ms vlan51.ebr1.Denver1.Level3.net [4.69.147.94] 9 63 ms 27 ms 29 ms ae-2-2.ebr2.Dallas1.Level3.net [4.69.132.106] 10 34 ms 35 ms 26 ms ae-72-72.csw2.Dallas1.Level3.net [4.69.151.141] 11 79 ms 26 ms 25 ms ae-21-70.car1.Dallas1.Level3.net [4.69.145.67] 12 61 ms * * vnsc-pri.sys.gtei.net [4.2.2.1] 13 65 ms 68 ms * vnsc-pri.sys.gtei.net [4.2.2.1] 14 85 ms * * vnsc-pri.sys.gtei.net [4.2.2.1] 15 75 ms * * vnsc-pri.sys.gtei.net [4.2.2.1] 16 * * * Request timed out. 17 70 ms * 62 ms vnsc-pri.sys.gtei.net [4.2.2.1] Trace complete. Geoff MINA --------------------------------- CTO/Co-Founder Connect First Inc. CCI * VB * 911 P: 720.335.5924 T: 888.410.3071 F: 678.265.1158 E: gmina@connectfirst.com --------------------------------- www.connectfirst.com -----Original Message----- From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On Behalf Of Dan White Sent: Wednesday, October 19, 2011 10:21 AM To: Josh Luthman Cc: outages@outages.org Subject: Re: [outages] Intermittent ICMP and DNS for 4.2.2.[123] Additional information (I'm still seeing the problem) ~$ traceroute 4.2.2.2 traceroute to 4.2.2.2 (4.2.2.2), 30 hops max, 60 byte packets 1 10.0.2.1 (10.0.2.1) 3.813 ms 3.779 ms 4.183 ms 2 olp-67-217-152-5.olp.net (67.217.152.5) 2.547 ms 2.582 ms 2.761 ms 3 router.olp.net (67.217.151.97) 2.520 ms 2.514 ms 2.509 ms 4 olp-67-217-152-34.olp.net (67.217.152.34) 2.678 ms 2.659 ms 2.582 ms 5 ge-6-10-183.car1.dallas1.level3.net (4.71.13.121) 11.008 ms 11.010 ms 11.002 ms 6 * * * 7 * * * 8 * * * 9 * * * 10 vnsc-bak.sys.gtei.net (4.2.2.2) 56.222 ms 61.235 ms 61.239 ms ~$ ping -c 10 4.2.2.2 PING 4.2.2.2 (4.2.2.2) 56(84) bytes of data. 64 bytes from 4.2.2.2: icmp_req=1 ttl=59 time=51.1 ms 64 bytes from 4.2.2.2: icmp_req=6 ttl=59 time=57.0 ms 64 bytes from 4.2.2.2: icmp_req=7 ttl=59 time=35.5 ms 64 bytes from 4.2.2.2: icmp_req=8 ttl=59 time=31.7 ms --- 4.2.2.2 ping statistics --- 10 packets transmitted, 4 received, 60% packet loss, time 9045ms rtt min/avg/max/mdev = 31.754/43.865/57.000/10.493 ms ~$ ping -c 10 4.71.13.121 PING 4.71.13.121 (4.71.13.121) 56(84) bytes of data. 64 bytes from 4.71.13.121: icmp_req=1 ttl=251 time=14.7 ms 64 bytes from 4.71.13.121: icmp_req=2 ttl=251 time=9.15 ms 64 bytes from 4.71.13.121: icmp_req=3 ttl=251 time=9.14 ms 64 bytes from 4.71.13.121: icmp_req=4 ttl=251 time=8.82 ms 64 bytes from 4.71.13.121: icmp_req=5 ttl=251 time=11.3 ms 64 bytes from 4.71.13.121: icmp_req=6 ttl=251 time=9.72 ms 64 bytes from 4.71.13.121: icmp_req=7 ttl=251 time=8.47 ms 64 bytes from 4.71.13.121: icmp_req=8 ttl=251 time=8.47 ms 64 bytes from 4.71.13.121: icmp_req=9 ttl=251 time=9.57 ms 64 bytes from 4.71.13.121: icmp_req=10 ttl=251 time=9.49 ms --- 4.71.13.121 ping statistics --- 10 packets transmitted, 10 received, 0% packet loss, time 9014ms rtt min/avg/max/mdev = 8.471/9.898/14.793/1.809 ms On 19/10/11 12:14 -0400, Josh Luthman wrote: ping -c 10 4.2.2
Rock solid at 26 ms for me
-> traceroute 4.2.2.2 traceroute to 4.2.2.2 (4.2.2.2), 30 hops max, 40 byte packets 1 rrcs-74-218-208-193.central.biz.rr.com (74.218.208.193) 0.531 ms 0.485 ms 0.450 ms 2 rrcs-74-218-88-133.central.biz.rr.com (74.218.88.133) 0.418 ms 0.390 ms 0.589 ms 3 rrcs-74-218-88-129.central.biz.rr.com (74.218.88.129) 1.061 ms 2.046 ms 2.024 ms 4 vln864.dytnoh1-swt101.woh.rr.com (74.218.88.73) 3.000 ms 2.977 ms 2.943 ms 5 gig2-1-801.dytnoh1-rtr102.woh.rr.com (70.60.47.5) 2.910 ms 2.884 ms 2.858 ms 6 gig5-0.dytnoh1-rtr1.woh.rr.com (24.29.160.21) 2.840 ms 2.845 ms 1.922 ms 7 tge5-1-0.tr00.clmkohpe.mwrtn.rr.com (65.25.137.177) 8.852 ms 8.812 ms 8.788 ms 8 ae-4-0.cr0.chi30.tbone.rr.com (66.109.6.68) 18.744 ms 18.953 ms ae-9-0.cr0.chi30.tbone.rr.com (107.14.19.16) 18.692 ms 9 107.14.17.147 (107.14.17.147) 18.899 ms ae-1-0.pr0.chi10.tbone.rr.com (66.109.6.155) 18.874 ms 18.839 ms 10 xe-10-2-0.edge2.Chicago2.Level3.net (4.59.28.109) 25.573 ms xe-10-0-0.edge2.Chicago2.Level3.net (4.59.28.101) 25.549 ms xe-10-2-0.edge2.Chicago2.Level3.net (4.59.28.109) 25.507 ms 11 vlan51.ebr1.Chicago2.Level3.net (4.69.138.158) 25.482 ms 25.458 ms vlan52.ebr2.Chicago2.Level3.net (4.69.138.190) 25.432 ms 12 ae-6-6.ebr1.Chicago1.Level3.net (4.69.140.189) 41.152 ms 34.879 ms 35.936 ms 13 ae-11-51.car1.Chicago1.Level3.net (4.69.138.3) 35.880 ms ae-21-52.car1.Chicago1.Level3.net (4.69.138.35) 35.848 ms 35.822 ms 14 vnsc-bak.sys.gtei.net (4.2.2.2) 36.780 ms 36.755 ms
Josh Luthman Office: 937-552-2340 Direct: 937-552-2343 1100 Wayne St Suite 1337 Troy, OH 45373
On Wed, Oct 19, 2011 at 11:59 AM, Dan White <dwhite@olp.net> wrote:
I've been noticing packet loss while pinging 4.2.2.1, 4.2.2.2, and 4.2.2.3 in the last few minutes, which are usually my go to IP addresses for testing external server connectivity.
DNS also seems to be intermittent.
-- Dan White _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
-- Dan White BTC Broadband Ph 918.366.0248 (direct) main: (918)366-8000 Fax 918.366.6610 email: dwhite@olp.net http://www.btcbroadband.com _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages

I assume that that 4.2.2.1 is anycast, because my own path is different (going to Chicago) and good. Looks like the problem is in Dallas. nagios:/var/log# mtr -r -c 10 4.2.2.1 HOST: nagios Loss% Snt Last Avg Best Wrst StDev 1. router-core-inside.mtcnet.ne 0.0% 10 0.3 0.2 0.2 0.3 0.0 2. sxct.movl.mtcnet.net 0.0% 10 0.2 0.2 0.2 0.3 0.0 3. siouxcenter.movl.153.netins. 0.0% 10 2.0 2.0 2.0 2.0 0.0 4. ins-kb1-te-13-2-220.kmrr.net 0.0% 10 7.6 8.7 7.6 18.1 3.3 5. ins-kc1-et-9-2.kmrr.netins.n 0.0% 10 7.6 7.6 7.6 7.7 0.0 6. 144.223.35.225 0.0% 10 15.9 15.8 15.8 15.9 0.1 7. sl-st20-chi-0-0-0.sprintlink 0.0% 10 16.0 16.0 15.9 16.2 0.1 8. 144.232.8.114 0.0% 10 49.3 24.3 16.0 64.8 17.7 9. vlan52.ebr2.Chicago2.Level3. 0.0% 10 16.3 16.2 16.2 16.3 0.0 10. ae-5-5.ebr2.Chicago1.Level3. 0.0% 10 16.2 17.3 16.1 22.8 2.1 11. ae-21-52.car1.Chicago1.Level 10.0% 10 16.4 35.6 16.3 102.8 37.7 12. vnsc-pri.sys.gtei.net 0.0% 10 16.5 16.7 16.4 19.2 0.9 nagios:/var/log# Frank -----Original Message----- From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On Behalf Of Geoffrey Mina Sent: Wednesday, October 19, 2011 11:49 AM To: 'Dan White'; 'Josh Luthman' Cc: outages@outages.org Subject: Re: [outages] Intermittent ICMP and DNS for 4.2.2.[123] I am seeing the same. Tracing route to vnsc-pri.sys.gtei.net [4.2.2.1] over a maximum of 30 hops: 1 1 ms <1 ms <1 ms 173-164-56-78-colorado.hfc.comcastbusiness.net [ 173.164.56.78] 2 52 ms 24 ms 29 ms 73.83.164.1 3 10 ms 15 ms 10 ms te-8-1-ur01.boulder.co.denver.comcast.net [68.85 .107.85] 4 8 ms 9 ms 9 ms te-7-4-ur02.boulder.co.denver.comcast.net [68.86 .103.122] 5 12 ms 15 ms 16 ms te-0-8-0-0-ar02.aurora.co.denver.comcast.net [68 .86.179.97] 6 27 ms 11 ms 14 ms pos-3-8-0-0-cr01.denver.co.ibone.comcast.net [68 .86.94.153] 7 52 ms 28 ms 11 ms xe-9-2-0.edge3.Denver1.Level3.net [4.28.20.5] 8 18 ms 11 ms 12 ms vlan51.ebr1.Denver1.Level3.net [4.69.147.94] 9 63 ms 27 ms 29 ms ae-2-2.ebr2.Dallas1.Level3.net [4.69.132.106] 10 34 ms 35 ms 26 ms ae-72-72.csw2.Dallas1.Level3.net [4.69.151.141] 11 79 ms 26 ms 25 ms ae-21-70.car1.Dallas1.Level3.net [4.69.145.67] 12 61 ms * * vnsc-pri.sys.gtei.net [4.2.2.1] 13 65 ms 68 ms * vnsc-pri.sys.gtei.net [4.2.2.1] 14 85 ms * * vnsc-pri.sys.gtei.net [4.2.2.1] 15 75 ms * * vnsc-pri.sys.gtei.net [4.2.2.1] 16 * * * Request timed out. 17 70 ms * 62 ms vnsc-pri.sys.gtei.net [4.2.2.1] Trace complete. Geoff MINA --------------------------------- CTO/Co-Founder Connect First Inc. CCI * VB * 911 P: 720.335.5924 T: 888.410.3071 F: 678.265.1158 E: gmina@connectfirst.com --------------------------------- www.connectfirst.com -----Original Message----- From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On Behalf Of Dan White Sent: Wednesday, October 19, 2011 10:21 AM To: Josh Luthman Cc: outages@outages.org Subject: Re: [outages] Intermittent ICMP and DNS for 4.2.2.[123] Additional information (I'm still seeing the problem) ~$ traceroute 4.2.2.2 traceroute to 4.2.2.2 (4.2.2.2), 30 hops max, 60 byte packets 1 10.0.2.1 (10.0.2.1) 3.813 ms 3.779 ms 4.183 ms 2 olp-67-217-152-5.olp.net (67.217.152.5) 2.547 ms 2.582 ms 2.761 ms 3 router.olp.net (67.217.151.97) 2.520 ms 2.514 ms 2.509 ms 4 olp-67-217-152-34.olp.net (67.217.152.34) 2.678 ms 2.659 ms 2.582 ms 5 ge-6-10-183.car1.dallas1.level3.net (4.71.13.121) 11.008 ms 11.010 ms 11.002 ms 6 * * * 7 * * * 8 * * * 9 * * * 10 vnsc-bak.sys.gtei.net (4.2.2.2) 56.222 ms 61.235 ms 61.239 ms ~$ ping -c 10 4.2.2.2 PING 4.2.2.2 (4.2.2.2) 56(84) bytes of data. 64 bytes from 4.2.2.2: icmp_req=1 ttl=59 time=51.1 ms 64 bytes from 4.2.2.2: icmp_req=6 ttl=59 time=57.0 ms 64 bytes from 4.2.2.2: icmp_req=7 ttl=59 time=35.5 ms 64 bytes from 4.2.2.2: icmp_req=8 ttl=59 time=31.7 ms --- 4.2.2.2 ping statistics --- 10 packets transmitted, 4 received, 60% packet loss, time 9045ms rtt min/avg/max/mdev = 31.754/43.865/57.000/10.493 ms ~$ ping -c 10 4.71.13.121 PING 4.71.13.121 (4.71.13.121) 56(84) bytes of data. 64 bytes from 4.71.13.121: icmp_req=1 ttl=251 time=14.7 ms 64 bytes from 4.71.13.121: icmp_req=2 ttl=251 time=9.15 ms 64 bytes from 4.71.13.121: icmp_req=3 ttl=251 time=9.14 ms 64 bytes from 4.71.13.121: icmp_req=4 ttl=251 time=8.82 ms 64 bytes from 4.71.13.121: icmp_req=5 ttl=251 time=11.3 ms 64 bytes from 4.71.13.121: icmp_req=6 ttl=251 time=9.72 ms 64 bytes from 4.71.13.121: icmp_req=7 ttl=251 time=8.47 ms 64 bytes from 4.71.13.121: icmp_req=8 ttl=251 time=8.47 ms 64 bytes from 4.71.13.121: icmp_req=9 ttl=251 time=9.57 ms 64 bytes from 4.71.13.121: icmp_req=10 ttl=251 time=9.49 ms --- 4.71.13.121 ping statistics --- 10 packets transmitted, 10 received, 0% packet loss, time 9014ms rtt min/avg/max/mdev = 8.471/9.898/14.793/1.809 ms On 19/10/11 12:14 -0400, Josh Luthman wrote: ping -c 10 4.2.2
Rock solid at 26 ms for me
-> traceroute 4.2.2.2 traceroute to 4.2.2.2 (4.2.2.2), 30 hops max, 40 byte packets 1 rrcs-74-218-208-193.central.biz.rr.com (74.218.208.193) 0.531 ms 0.485 ms 0.450 ms 2 rrcs-74-218-88-133.central.biz.rr.com (74.218.88.133) 0.418 ms 0.390 ms 0.589 ms 3 rrcs-74-218-88-129.central.biz.rr.com (74.218.88.129) 1.061 ms 2.046 ms 2.024 ms 4 vln864.dytnoh1-swt101.woh.rr.com (74.218.88.73) 3.000 ms 2.977 ms 2.943 ms 5 gig2-1-801.dytnoh1-rtr102.woh.rr.com (70.60.47.5) 2.910 ms 2.884 ms 2.858 ms 6 gig5-0.dytnoh1-rtr1.woh.rr.com (24.29.160.21) 2.840 ms 2.845 ms 1.922 ms 7 tge5-1-0.tr00.clmkohpe.mwrtn.rr.com (65.25.137.177) 8.852 ms 8.812 ms 8.788 ms 8 ae-4-0.cr0.chi30.tbone.rr.com (66.109.6.68) 18.744 ms 18.953 ms ae-9-0.cr0.chi30.tbone.rr.com (107.14.19.16) 18.692 ms 9 107.14.17.147 (107.14.17.147) 18.899 ms ae-1-0.pr0.chi10.tbone.rr.com (66.109.6.155) 18.874 ms 18.839 ms 10 xe-10-2-0.edge2.Chicago2.Level3.net (4.59.28.109) 25.573 ms xe-10-0-0.edge2.Chicago2.Level3.net (4.59.28.101) 25.549 ms xe-10-2-0.edge2.Chicago2.Level3.net (4.59.28.109) 25.507 ms 11 vlan51.ebr1.Chicago2.Level3.net (4.69.138.158) 25.482 ms 25.458 ms vlan52.ebr2.Chicago2.Level3.net (4.69.138.190) 25.432 ms 12 ae-6-6.ebr1.Chicago1.Level3.net (4.69.140.189) 41.152 ms 34.879 ms 35.936 ms 13 ae-11-51.car1.Chicago1.Level3.net (4.69.138.3) 35.880 ms ae-21-52.car1.Chicago1.Level3.net (4.69.138.35) 35.848 ms 35.822 ms 14 vnsc-bak.sys.gtei.net (4.2.2.2) 36.780 ms 36.755 ms
Josh Luthman Office: 937-552-2340 Direct: 937-552-2343 1100 Wayne St Suite 1337 Troy, OH 45373
On Wed, Oct 19, 2011 at 11:59 AM, Dan White <dwhite@olp.net> wrote:
I've been noticing packet loss while pinging 4.2.2.1, 4.2.2.2, and 4.2.2.3 in the last few minutes, which are usually my go to IP addresses for testing external server connectivity.
DNS also seems to be intermittent.
-- Dan White _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
-- Dan White BTC Broadband Ph 918.366.0248 (direct) main: (918)366-8000 Fax 918.366.6610 email: dwhite@olp.net http://www.btcbroadband.com _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages

You guys know that 4.2.2.[123] are anycast, so you get to different servers depending on where you enter the Level3 network, right? Mentioning just in case some have missed this. -mark SanJose: 2 ae-41-90.car1.SanJose1.Level3.net (4.69.152.195) [AS 3356] 2ms 1ms 1ms 3 vnsc-bak.sys.gtei.net (4.2.2.2) [AS 3356] 2ms 1ms 1ms NewYork: 2 ae-41-90.car1.NewYork1.Level3.net (4.69.155.195) [AS 3356] 0ms 4ms 0ms 3 vnsc-bak.sys.gtei.net (4.2.2.2) [AS 3356] 4ms 0ms 0ms WashDC: 2 ge-5-0-60.hsa1.Washington2.Level3.net (4.69.149.51) [AS 3356] 1ms 1ms 1ms 3 vnsc-bak.sys.gtei.net (4.2.2.2) [AS 3356] 2ms 2ms 1ms

Our edge DNS servers have been alerting for DNS lookups from Denver, CO using 4.2.2.1 and 4.2.2.2 as forwarders. --- Kyle Smith -----Original Message----- From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On Behalf Of Josh Luthman Sent: Wednesday, October 19, 2011 12:15 PM To: Dan White Cc: outages@outages.org Subject: Re: [outages] Intermittent ICMP and DNS for 4.2.2.[123] Rock solid at 26 ms for me -> traceroute 4.2.2.2 traceroute to 4.2.2.2 (4.2.2.2), 30 hops max, 40 byte packets 1 rrcs-74-218-208-193.central.biz.rr.com (74.218.208.193) 0.531 ms 0.485 ms 0.450 ms 2 rrcs-74-218-88-133.central.biz.rr.com (74.218.88.133) 0.418 ms 0.390 ms 0.589 ms 3 rrcs-74-218-88-129.central.biz.rr.com (74.218.88.129) 1.061 ms 2.046 ms 2.024 ms 4 vln864.dytnoh1-swt101.woh.rr.com (74.218.88.73) 3.000 ms 2.977 ms 2.943 ms 5 gig2-1-801.dytnoh1-rtr102.woh.rr.com (70.60.47.5) 2.910 ms 2.884 ms 2.858 ms 6 gig5-0.dytnoh1-rtr1.woh.rr.com (24.29.160.21) 2.840 ms 2.845 ms 1.922 ms 7 tge5-1-0.tr00.clmkohpe.mwrtn.rr.com (65.25.137.177) 8.852 ms 8.812 ms 8.788 ms 8 ae-4-0.cr0.chi30.tbone.rr.com (66.109.6.68) 18.744 ms 18.953 ms ae-9-0.cr0.chi30.tbone.rr.com (107.14.19.16) 18.692 ms 9 107.14.17.147 (107.14.17.147) 18.899 ms ae-1-0.pr0.chi10.tbone.rr.com (66.109.6.155) 18.874 ms 18.839 ms 10 xe-10-2-0.edge2.Chicago2.Level3.net (4.59.28.109) 25.573 ms xe-10-0-0.edge2.Chicago2.Level3.net (4.59.28.101) 25.549 ms xe-10-2-0.edge2.Chicago2.Level3.net (4.59.28.109) 25.507 ms 11 vlan51.ebr1.Chicago2.Level3.net (4.69.138.158) 25.482 ms 25.458 ms vlan52.ebr2.Chicago2.Level3.net (4.69.138.190) 25.432 ms 12 ae-6-6.ebr1.Chicago1.Level3.net (4.69.140.189) 41.152 ms 34.879 ms 35.936 ms 13 ae-11-51.car1.Chicago1.Level3.net (4.69.138.3) 35.880 ms ae-21-52.car1.Chicago1.Level3.net (4.69.138.35) 35.848 ms 35.822 ms 14 vnsc-bak.sys.gtei.net (4.2.2.2) 36.780 ms 36.755 ms Josh Luthman Office: 937-552-2340 Direct: 937-552-2343 1100 Wayne St Suite 1337 Troy, OH 45373 On Wed, Oct 19, 2011 at 11:59 AM, Dan White <dwhite@olp.net> wrote:
I've been noticing packet loss while pinging 4.2.2.1, 4.2.2.2, and 4.2.2.3 in the last few minutes, which are usually my go to IP addresses for testing external server connectivity.
DNS also seems to be intermittent.
-- Dan White _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages The information in this message is for the intended recipient(s) only and may be the proprietary and/or confidential property of Litle & Co., LLC, and thus protected from disclosure. If you are not the intended recipient(s), or an employee or agent responsible for delivering this message to the intended recipient, you are hereby notified that any use, dissemination, distribution or copying of this communication is prohibited. If you have received this communication in error, please notify Litle & Co. immediately by replying to this message and then promptly deleting it and your reply permanently from your computer.

Smokeping for these sites: 4.2.2.1: doesn't show anything http://smokeping.foxygnu.org/?target=DNSopen.vnscprisysgteinet http://home.derekbodner.com/cgi-bin/smokeping.cgi?target=World.Level3 4.2.2.2: shows a little http://smokeping.foxygnu.org/?target=DNSopen.vnscbaksysgteinet 4.2.2.3: doesn't show anything http://smokeping.foxygnu.org/?target=DNSopen.vnsclcsysgteinet Frank -----Original Message----- From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On Behalf Of Dan White Sent: Wednesday, October 19, 2011 10:59 AM To: outages@outages.org Subject: [outages] Intermittent ICMP and DNS for 4.2.2.[123] I've been noticing packet loss while pinging 4.2.2.1, 4.2.2.2, and 4.2.2.3 in the last few minutes, which are usually my go to IP addresses for testing external server connectivity. DNS also seems to be intermittent. -- Dan White _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
participants (7)
-
Dan White
-
Frank Bulk
-
Geoffrey Mina
-
Josh Luthman
-
Mark Kent
-
Smith, Kyle
-
Taner Halicioglu