
Is anyone seeing any issues to 4.2.2.2? At 3:35 EST I starting seeing packet loss (icmp ping) to it - http://imgur.com/ITGL1 I noticed at the same time we had a route change with RR (Time Warner Cable) in Chicago. Josh Luthman Office: 937-552-2340 Direct: 937-552-2343 1100 Wayne St Suite 1337 Troy, OH 45373

I am seeing the same. Thanks! Matthew Anderson Network Operations Manager Enhanced Telecommunications Corporation Office: 812-623-6288 Cell: 812-212-5565 From: Josh Luthman <josh@imaginenetworksllc.com<mailto:josh@imaginenetworksllc.com>> Date: Wednesday, September 12, 2012 4:08 PM To: "outages@outages.org<mailto:outages@outages.org>" <outages@outages.org<mailto:outages@outages.org>> Subject: [outages] Packet loss to 4.2.2.2 Is anyone seeing any issues to 4.2.2.2? At 3:35 EST I starting seeing packet loss (icmp ping) to it - http://imgur.com/ITGL1 I noticed at the same time we had a route change with RR (Time Warner Cable) in Chicago. Josh Luthman Office: 937-552-2340 Direct: 937-552-2343 1100 Wayne St Suite 1337 Troy, OH 45373

Can you share from/to? I was told 4.2.2.2 is anycast (or otherwise geographically diverse), though I have no proof of that. Josh Luthman Office: 937-552-2340 Direct: 937-552-2343 1100 Wayne St Suite 1337 Troy, OH 45373 On Wed, Sep 12, 2012 at 4:10 PM, Matthew Anderson <manderson@etc1.net>wrote:
I am seeing the same.
Thanks!
Matthew Anderson Network Operations Manager Enhanced Telecommunications Corporation Office: 812-623-6288 Cell: 812-212-5565
From: Josh Luthman <josh@imaginenetworksllc.com<mailto: josh@imaginenetworksllc.com>> Date: Wednesday, September 12, 2012 4:08 PM To: "outages@outages.org<mailto:outages@outages.org>" <outages@outages.org <mailto:outages@outages.org>> Subject: [outages] Packet loss to 4.2.2.2
Is anyone seeing any issues to 4.2.2.2?
At 3:35 EST I starting seeing packet loss (icmp ping) to it - http://imgur.com/ITGL1
I noticed at the same time we had a route change with RR (Time Warner Cable) in Chicago.
Josh Luthman Office: 937-552-2340 Direct: 937-552-2343 1100 Wayne St Suite 1337 Troy, OH 45373

$ traceroute 4.2.2.2 traceroute to 4.2.2.2 (4.2.2.2), 64 hops max, 52 byte packets 1 216.8.196.1.etczone.com (216.8.196.1) 3.102 ms 3.102 ms 3.157 ms 2 216.8.254.2.core.etczone.com (216.8.254.2) 5.087 ms 3.327 ms 3.354 ms 3 216.8.252.193.core.etczone.com (216.8.252.193) 3.149 ms 3.210 ms 4.054 ms 4 gige-g3-13.core1.chi1.he.net (216.66.77.217) 13.036 ms 14.810 ms 23.195 ms 5 chi-bb1-link.telia.net (213.248.104.213) 13.075 ms 15.820 ms 12.659 ms 6 chi-bb1-link.telia.net (80.91.246.118) 13.328 ms chi-bb1-link.telia.net (80.91.246.172) 13.506 ms chi-bb1-link.telia.net (80.91.246.168) 12.970 ms 7 xe-9-2-0.edge1.washington4.level3.net (4.68.63.85) 209.779 ms te-8-4.car3.chicago1.level3.net (4.68.127.137) 229.144 ms 72.390 ms 8 ae-11-51.car1.chicago1.level3.net (4.69.138.3) 13.154 ms 13.773 ms 13.425 ms 9 b.resolvers.level3.net (4.2.2.2) 53.985 ms * 33.460 ms Thanks! Matthew Anderson Network Operations Manager Enhanced Telecommunications Corporation Office: 812-623-6288 Cell: 812-212-5565 From: Josh Luthman <josh@imaginenetworksllc.com<mailto:josh@imaginenetworksllc.com>> Date: Wednesday, September 12, 2012 4:12 PM To: MATT <manderson@etc1.net<mailto:manderson@etc1.net>> Cc: "outages@outages.org<mailto:outages@outages.org>" <outages@outages.org<mailto:outages@outages.org>> Subject: Re: [outages] Packet loss to 4.2.2.2 Can you share from/to? I was told 4.2.2.2 is anycast (or otherwise geographically diverse), though I have no proof of that. Josh Luthman Office: 937-552-2340 Direct: 937-552-2343 1100 Wayne St Suite 1337 Troy, OH 45373 On Wed, Sep 12, 2012 at 4:10 PM, Matthew Anderson <manderson@etc1.net<mailto:manderson@etc1.net>> wrote: I am seeing the same. Thanks! Matthew Anderson Network Operations Manager Enhanced Telecommunications Corporation Office: 812-623-6288<tel:812-623-6288> Cell: 812-212-5565<tel:812-212-5565> From: Josh Luthman <josh@imaginenetworksllc.com<mailto:josh@imaginenetworksllc.com><mailto:josh@imaginenetworksllc.com<mailto:josh@imaginenetworksllc.com>>> Date: Wednesday, September 12, 2012 4:08 PM To: "outages@outages.org<mailto:outages@outages.org><mailto:outages@outages.org<mailto:outages@outages.org>>" <outages@outages.org<mailto:outages@outages.org><mailto:outages@outages.org<mailto:outages@outages.org>>> Subject: [outages] Packet loss to 4.2.2.2 Is anyone seeing any issues to 4.2.2.2? At 3:35 EST I starting seeing packet loss (icmp ping) to it - http://imgur.com/ITGL1 I noticed at the same time we had a route change with RR (Time Warner Cable) in Chicago. Josh Luthman Office: 937-552-2340<tel:937-552-2340> Direct: 937-552-2343<tel:937-552-2343> 1100 Wayne St Suite 1337 Troy, OH 45373

aninetops01 Snt: 10 Loss% Last Avg Best Wrst StDev 172.18.1.1 0.0% 0.5 0.5 0.4 0.5 0.0 10.224.8.81 0.0% 0.3 5.0 0.2 47.0 14.8 70.103.75.2 0.0% 0.6 1.0 0.6 1.3 0.2 fe9-8.gw01.lsvg.eli.net 0.0% 1.5 2.1 1.5 2.4 0.3 ge2-10.cr01.lsvlnv23.integra.net 0.0% 9.3 12.2 8.9 28.4 6.4 tg0-3-0-2.sc01.lsancarc.integra.net 0.0% 13.3 12.2 10.8 13.6 1.3 be1.br02.lsancarc.integra.net 0.0% 9.5 9.5 9.2 9.8 0.2 xe-11-1-0.edge1.losangeles6.level3.net 0.0% 9.1 14.2 8.9 40.8 11.1 ae-4-90.edge3.losangeles1.level3.net 0.0% 9.2 9.1 8.5 9.3 0.3 b.resolvers.level3.net 0.0% 9.3 9.1 8.9 9.5 0.2 -----Original Message----- From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On Behalf Of Matthew Anderson Sent: Wednesday, September 12, 2012 3:17 PM To: Josh Luthman Cc: outages@outages.org Subject: Re: [outages] Packet loss to 4.2.2.2 $ traceroute 4.2.2.2 traceroute to 4.2.2.2 (4.2.2.2), 64 hops max, 52 byte packets 1 216.8.196.1.etczone.com (216.8.196.1) 3.102 ms 3.102 ms 3.157 ms 2 216.8.254.2.core.etczone.com (216.8.254.2) 5.087 ms 3.327 ms 3.354 ms 3 216.8.252.193.core.etczone.com (216.8.252.193) 3.149 ms 3.210 ms 4.054 ms 4 gige-g3-13.core1.chi1.he.net (216.66.77.217) 13.036 ms 14.810 ms 23.195 ms 5 chi-bb1-link.telia.net (213.248.104.213) 13.075 ms 15.820 ms 12.659 ms 6 chi-bb1-link.telia.net (80.91.246.118) 13.328 ms chi-bb1-link.telia.net (80.91.246.172) 13.506 ms chi-bb1-link.telia.net (80.91.246.168) 12.970 ms 7 xe-9-2-0.edge1.washington4.level3.net (4.68.63.85) 209.779 ms te-8-4.car3.chicago1.level3.net (4.68.127.137) 229.144 ms 72.390 ms 8 ae-11-51.car1.chicago1.level3.net (4.69.138.3) 13.154 ms 13.773 ms 13.425 ms 9 b.resolvers.level3.net (4.2.2.2) 53.985 ms * 33.460 ms Thanks! Matthew Anderson Network Operations Manager Enhanced Telecommunications Corporation Office: 812-623-6288 Cell: 812-212-5565 From: Josh Luthman <josh@imaginenetworksllc.com<mailto:josh@imaginenetworksllc.com>> Date: Wednesday, September 12, 2012 4:12 PM To: MATT <manderson@etc1.net<mailto:manderson@etc1.net>> Cc: "outages@outages.org<mailto:outages@outages.org>" <outages@outages.org<mailto:outages@outages.org>> Subject: Re: [outages] Packet loss to 4.2.2.2 Can you share from/to? I was told 4.2.2.2 is anycast (or otherwise geographically diverse), though I have no proof of that. Josh Luthman Office: 937-552-2340 Direct: 937-552-2343 1100 Wayne St Suite 1337 Troy, OH 45373 On Wed, Sep 12, 2012 at 4:10 PM, Matthew Anderson <manderson@etc1.net<mailto:manderson@etc1.net>> wrote: I am seeing the same. Thanks! Matthew Anderson Network Operations Manager Enhanced Telecommunications Corporation Office: 812-623-6288<tel:812-623-6288> Cell: 812-212-5565<tel:812-212-5565> From: Josh Luthman <josh@imaginenetworksllc.com<mailto:josh@imaginenetworksllc.com><mailto:josh@imaginenetworksllc.com<mailto:josh@imaginenetworksllc.com>>> Date: Wednesday, September 12, 2012 4:08 PM To: "outages@outages.org<mailto:outages@outages.org><mailto:outages@outages.org<mailto:outages@outages.org>>" <outages@outages.org<mailto:outages@outages.org><mailto:outages@outages.org<mailto:outages@outages.org>>> Subject: [outages] Packet loss to 4.2.2.2 Is anyone seeing any issues to 4.2.2.2? At 3:35 EST I starting seeing packet loss (icmp ping) to it - http://imgur.com/ITGL1 I noticed at the same time we had a route change with RR (Time Warner Cable) in Chicago. Josh Luthman Office: 937-552-2340<tel:937-552-2340> Direct: 937-552-2343<tel:937-552-2343> 1100 Wayne St Suite 1337 Troy, OH 45373 _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages

Hello, We are not seeing any packet loss through Level3 DIA in SFO: 7 33 ms 57 ms 28 ms xe-8-3-0.bar1.SanFrancisco1.Level3.net [4.53.132.37] 8 24 ms 28 ms 38 ms ae-0-11.bar2.SanFrancisco1.Level3.net [4.69.140.146] 9 39 ms 30 ms 40 ms ae-6-6.ebr2.SanJose1.Level3.net [4.69.140.154] 10 47 ms 50 ms 48 ms ae-62-62.csw1.SanJose1.Level3.net [4.69.153.18] 11 * 33 ms 26 ms ae-1-60.edge1.SanJose1.Level3.net [4.69.152.14] 12 29 ms 36 ms 35 ms b.resolvers.Level3.net [4.2.2.2] Trace complete. Joshua D. Breeds E: josh@servedby.net | D: 206.707.9898 ext.100 ServedBy the Net, LLC | www.ServedBy.Net | 1(855)GET-SBN-0 (1-855-438-7260) -----Original Message----- From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On Behalf Of Matthew Anderson Sent: Wednesday, September 12, 2012 1:17 PM To: Josh Luthman Cc: outages@outages.org Subject: Re: [outages] Packet loss to 4.2.2.2 $ traceroute 4.2.2.2 traceroute to 4.2.2.2 (4.2.2.2), 64 hops max, 52 byte packets 1 216.8.196.1.etczone.com (216.8.196.1) 3.102 ms 3.102 ms 3.157 ms 2 216.8.254.2.core.etczone.com (216.8.254.2) 5.087 ms 3.327 ms 3.354 ms 3 216.8.252.193.core.etczone.com (216.8.252.193) 3.149 ms 3.210 ms 4.054 ms 4 gige-g3-13.core1.chi1.he.net (216.66.77.217) 13.036 ms 14.810 ms 23.195 ms 5 chi-bb1-link.telia.net (213.248.104.213) 13.075 ms 15.820 ms 12.659 ms 6 chi-bb1-link.telia.net (80.91.246.118) 13.328 ms chi-bb1-link.telia.net (80.91.246.172) 13.506 ms chi-bb1-link.telia.net (80.91.246.168) 12.970 ms 7 xe-9-2-0.edge1.washington4.level3.net (4.68.63.85) 209.779 ms te-8-4.car3.chicago1.level3.net (4.68.127.137) 229.144 ms 72.390 ms 8 ae-11-51.car1.chicago1.level3.net (4.69.138.3) 13.154 ms 13.773 ms 13.425 ms 9 b.resolvers.level3.net (4.2.2.2) 53.985 ms * 33.460 ms Thanks! Matthew Anderson Network Operations Manager Enhanced Telecommunications Corporation Office: 812-623-6288 Cell: 812-212-5565 From: Josh Luthman <josh@imaginenetworksllc.com<mailto:josh@imaginenetworksllc.com>> Date: Wednesday, September 12, 2012 4:12 PM To: MATT <manderson@etc1.net<mailto:manderson@etc1.net>> Cc: "outages@outages.org<mailto:outages@outages.org>" <outages@outages.org<mailto:outages@outages.org>> Subject: Re: [outages] Packet loss to 4.2.2.2 Can you share from/to? I was told 4.2.2.2 is anycast (or otherwise geographically diverse), though I have no proof of that. Josh Luthman Office: 937-552-2340 Direct: 937-552-2343 1100 Wayne St Suite 1337 Troy, OH 45373 On Wed, Sep 12, 2012 at 4:10 PM, Matthew Anderson <manderson@etc1.net<mailto:manderson@etc1.net>> wrote: I am seeing the same. Thanks! Matthew Anderson Network Operations Manager Enhanced Telecommunications Corporation Office: 812-623-6288<tel:812-623-6288> Cell: 812-212-5565<tel:812-212-5565> From: Josh Luthman <josh@imaginenetworksllc.com<mailto:josh@imaginenetworksllc.com><mailto:josh@imaginenetworksllc.com<mailto:josh@imaginenetworksllc.com>>> Date: Wednesday, September 12, 2012 4:08 PM To: "outages@outages.org<mailto:outages@outages.org><mailto:outages@outages.org<mailto:outages@outages.org>>" <outages@outages.org<mailto:outages@outages.org><mailto:outages@outages.org<mailto:outages@outages.org>>> Subject: [outages] Packet loss to 4.2.2.2 Is anyone seeing any issues to 4.2.2.2? At 3:35 EST I starting seeing packet loss (icmp ping) to it - http://imgur.com/ITGL1 I noticed at the same time we had a route change with RR (Time Warner Cable) in Chicago. Josh Luthman Office: 937-552-2340<tel:937-552-2340> Direct: 937-552-2343<tel:937-552-2343> 1100 Wayne St Suite 1337 Troy, OH 45373 _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages

On Wed, Sep 12, 2012 at 1:12 PM, Josh Luthman <josh@imaginenetworksllc.com> wrote:
Can you share from/to? I was told 4.2.2.2 is anycast (or otherwise geographically diverse), though I have no proof of that.
From a Linode VM in New Jersey:
jof@ewr ~ % traceroute 4.2.2.2 traceroute to 4.2.2.2 (4.2.2.2), 30 hops max, 60 byte packets 1 router1-nac.linode.com (207.99.1.13) 0.618 ms 0.703 ms 0.703 ms 2 207.99.53.41 (207.99.53.41) 0.433 ms 0.502 ms 0.608 ms 3 vlan801.tbr1.mmu.nac.net (209.123.10.9) 0.302 ms 0.416 ms 0.508 ms 4 0.e2-2.tbr1.wbr.nac.net (209.123.11.74) 0.490 ms 0.478 ms 0.532 ms 5 0.e2-2.tbr2.wbr.nac.net (209.123.11.78) 0.499 ms 0.e2-3.tbr2.wbr.nac.net (209.123.11.54) 0.541 ms 0.582 ms 6 ge-8-34.car1.Newark1.Level3.net (4.26.16.189) 1.337 ms 1.365 ms 1.329 ms 7 ae-31-51.ebr1.Newark1.Level3.net (4.69.156.30) 1.215 ms 1.180 ms 1.148 ms 8 ae-2-2.ebr1.NewYork1.Level3.net (4.69.132.97) 1.550 ms 1.760 ms 1.740 ms 9 ae-61-61.csw1.NewYork1.Level3.net (4.69.134.66) 2.953 ms ae-81-81.csw3.NewYork1.Level3.net (4.69.134.74) 1.699 ms ae-91-91.csw4.NewYork1.Level3.net (4.69.134.78) 1.755 ms 10 ae-1-60.edge2.NewYork1.Level3.net (4.69.155.16) 6.131 ms 6.114 ms ae-3-80.edge2.NewYork1.Level3.net (4.69.155.144) 1.728 ms 11 b.resolvers.Level3.net (4.2.2.2) 1.823 ms 1.672 ms 1.759 ms And from an ADSL circuit on Sonic.net in the SF Bay Area: jof@r00ter:~$ traceroute 4.2.2.2 traceroute to 4.2.2.2 (4.2.2.2), 30 hops max, 60 byte packets 1 75-101-x-x.dsl.static.sonic.net (75.101.x.x) 14.832 ms 14.294 ms 13.780 ms 2 gig1-29.cr1.colaca01.sonic.net (70.36.228.109) 13.271 ms 13.327 ms 13.139 ms 3 po3.cr1.lsatca11.sonic.net (75.101.33.166) 14.130 ms 13.493 ms 13.222 ms 4 0.xe-7-1-0.gw.pao1.sonic.net (69.12.211.5) 13.005 ms 13.102 ms 13.143 ms 5 0.xe-6-1-0.gw2.200p-sf.sonic.net (64.142.0.109) 12.674 ms 13.064 ms 12.591 ms 6 as0.gw.200p-sf.sonic.net (208.106.96.249) 14.478 ms 15.139 ms 15.221 ms 7 xe-10-1-0.bar2.SanFrancisco1.Level3.net (4.53.134.9) 14.667 ms 15.620 ms 14.711 ms 8 ae-6-6.ebr2.SanJose1.Level3.net (4.69.140.154) 14.888 ms 17.756 ms 17.242 ms 9 ae-92-92.csw4.SanJose1.Level3.net (4.69.153.30) 18.937 ms 20.211 ms ae-72-72.csw2.SanJose1.Level3.net (4.69.153.22) 15.844 ms 10 ae-2-70.edge1.SanJose1.Level3.net (4.69.152.78) 16.178 ms ae-4-90.edge1.SanJose1.Level3.net (4.69.152.206) 16.217 ms 15.650 ms 11 b.resolvers.Level3.net (4.2.2.2) 15.152 ms 15.528 ms 15.769 ms

I see the loss in Chicago @ Level 3: |------------------------------------------------------------------------------------------| | WinMTR statistics | | Host - % | Sent | Recv | Best | Avrg | Wrst | Last | |------------------------------------------------|------|------|------|------|------|------| | xe-7-3-1.edge4.Chicago2.Level3.net - 0 | 90 | 90 | 38 | 38 | 64 | 49 | | 4.69.158.158 - 0 | 90 | 90 | 38 | 40 | 49 | 38 | | ae-5-5.ebr2.Chicago1.Level3.net - 0 | 90 | 90 | 37 | 41 | 50 | 42 | | ae-21-52.car1.Chicago1.Level3.net - 17 | 55 | 46 | 37 | 48 | 229 | 38 | | b.resolvers.Level3.net - 28 | 43 | 31 | 47 | 58 | 74 | 49 | |________________________________________________|______|______|______|______|______|______| From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On Behalf Of Josh Luthman Sent: Wednesday, September 12, 2012 3:13 PM To: Matthew Anderson Cc: outages@outages.org Subject: Re: [outages] Packet loss to 4.2.2.2 Can you share from/to? I was told 4.2.2.2 is anycast (or otherwise geographically diverse), though I have no proof of that. Josh Luthman Office: 937-552-2340 Direct: 937-552-2343 1100 Wayne St Suite 1337 Troy, OH 45373 On Wed, Sep 12, 2012 at 4:10 PM, Matthew Anderson <manderson@etc1.net<mailto:manderson@etc1.net>> wrote: I am seeing the same. Thanks! Matthew Anderson Network Operations Manager Enhanced Telecommunications Corporation Office: 812-623-6288<tel:812-623-6288> Cell: 812-212-5565<tel:812-212-5565> From: Josh Luthman <josh@imaginenetworksllc.com<mailto:josh@imaginenetworksllc.com><mailto:josh@imaginenetworksllc.com<mailto:josh@imaginenetworksllc.com>>> Date: Wednesday, September 12, 2012 4:08 PM To: "outages@outages.org<mailto:outages@outages.org><mailto:outages@outages.org<mailto:outages@outages.org>>" <outages@outages.org<mailto:outages@outages.org><mailto:outages@outages.org<mailto:outages@outages.org>>> Subject: [outages] Packet loss to 4.2.2.2 Is anyone seeing any issues to 4.2.2.2? At 3:35 EST I starting seeing packet loss (icmp ping) to it - http://imgur.com/ITGL1 I noticed at the same time we had a route change with RR (Time Warner Cable) in Chicago. Josh Luthman Office: 937-552-2340<tel:937-552-2340> Direct: 937-552-2343<tel:937-552-2343> 1100 Wayne St Suite 1337 Troy, OH 45373

Coming from Cogent in Chicago their route looks normal but I do see a high packet loss. traceroute 4.2.2.2 Type escape sequence to abort. Tracing the route to b.resolvers.Level3.net (4.2.2.2) VRF info: (vrf in name/id, vrf out name/id) 1 fa0-10.na01.b000657-0.ord01.atlas.cogentco.com (38.122.113.149) 0 msec 0 msec 4 msec 2 vl3525.mag02.ord01.atlas.cogentco.com (66.250.9.193) 0 msec 4 msec 4 msec 3 te0-5-0-3.mpd21.ord01.atlas.cogentco.com (154.54.45.205) 360 msec te0-5-0-3.mpd22.ord01.atlas.cogentco.com (154.54.45.213) 4 msec te0-5-0-3.mpd21.ord01.atlas.cogentco.com (154.54.45.205) 0 msec 4 te0-5-0-1.ccr21.ord03.atlas.cogentco.com (154.54.44.170) 4 msec te0-1-0-1.ccr21.ord03.atlas.cogentco.com (154.54.5.18) 4 msec te0-1-0-0.ccr22.ord01.atlas.cogentco.com (154.54.6.178) 4 msec 5 te0-1-0-0.ccr22.ord03.atlas.cogentco.com (66.28.4.66) 0 msec 154.54.86.6 8 msec te0-1-0-0.ccr22.ord03.atlas.cogentco.com (66.28.4.66) 4 msec 6 level3.ord03.atlas.cogentco.com (154.54.13.10) 0 msec 4 msec 0 msec 7 ae-11-51.car1.Chicago1.Level3.net (4.69.138.3) 4 msec 4 msec 4 msec 8 b.resolvers.Level3.net (4.2.2.2) 4 msec 4 msec 8 msec ping 4.2.2.2 repeat 100 Type escape sequence to abort. Sending 100, 100-byte ICMP Echos to 4.2.2.2, timeout is 2 seconds: !!!!.!!!!!!!!!!!!!!!!!!!!!..!! !.!!!!!!.!!.!.!!!!!!.!!!.!..!!.!!.!!!.!!!...!.!.!.!..!!.!.!!.!!.!!!!.. Success rate is 72 percent (72/100), round-trip min/avg/max = 1/3/16 ms On Wed, Sep 12, 2012 at 3:26 PM, Dikkema, Michael (Business Technology) < MDikkema@postmedia.com> wrote:
I see the loss in Chicago @ Level 3:****
** **
|------------------------------------------------------------------------------------------| ****
| WinMTR statistics |****
| Host - % | Sent | Recv | Best | Avrg | Wrst | Last |****
|------------------------------------------------|------|------|------|------|------|------| ****
| xe-7-3-1.edge4.Chicago2.Level3.net - 0 | 90 | 90 | 38 | 38 | 64 | 49 |****
| 4.69.158.158 - 0 | 90 | 90 | 38 | 40 | 49 | 38 |****
| ae-5-5.ebr2.Chicago1.Level3.net - 0 | 90 | 90 | 37 | 41 | 50 | 42 |****
| ae-21-52.car1.Chicago1.Level3.net - 17 | 55 | 46 | 37 | 48 | 229 | 38 |****
| b.resolvers.Level3.net - 28 | 43 | 31 | 47 | 58 | 74 | 49 |****
|________________________________________________|______|______|______|______|______|______| ****
** **
*From:* outages-bounces@outages.org [mailto:outages-bounces@outages.org] *On Behalf Of *Josh Luthman *Sent:* Wednesday, September 12, 2012 3:13 PM *To:* Matthew Anderson *Cc:* outages@outages.org *Subject:* Re: [outages] Packet loss to 4.2.2.2****
** **
Can you share from/to? I was told 4.2.2.2 is anycast (or otherwise geographically diverse), though I have no proof of that.****
Josh Luthman Office: 937-552-2340 Direct: 937-552-2343 1100 Wayne St Suite 1337 Troy, OH 45373
****
On Wed, Sep 12, 2012 at 4:10 PM, Matthew Anderson <manderson@etc1.net> wrote:****
I am seeing the same.
Thanks!
Matthew Anderson Network Operations Manager Enhanced Telecommunications Corporation Office: 812-623-6288 Cell: 812-212-5565
From: Josh Luthman <josh@imaginenetworksllc.com<mailto: josh@imaginenetworksllc.com>> Date: Wednesday, September 12, 2012 4:08 PM To: "outages@outages.org<mailto:outages@outages.org>" <outages@outages.org <mailto:outages@outages.org>> Subject: [outages] Packet loss to 4.2.2.2****
Is anyone seeing any issues to 4.2.2.2?
At 3:35 EST I starting seeing packet loss (icmp ping) to it - http://imgur.com/ITGL1
I noticed at the same time we had a route change with RR (Time Warner Cable) in Chicago.
Josh Luthman Office: 937-552-2340 Direct: 937-552-2343 1100 Wayne St Suite 1337 Troy, OH 45373****
** **
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages

----- Original Message -----
From: "Josh Luthman" <josh@imaginenetworksllc.com>
Can you share from/to? I was told 4.2.2.2 is anycast (or otherwise geographically diverse), though I have no proof of that.
4.2.2.x are *authoritatively* multicast; I have that from someone who is involved in running recursive DNS for whatever the hell they're calling themselves this year. :-) In other news: not all 6 of those servers will always respond to ping traffic even if they're resolving; it's random; presumably they don't want anyone to get too settled into using them as a ping beacon to watching the weather on the 'net. Understandably. Cheers, -- jra -- Jay R. Ashworth Baylink jra@baylink.com Designer The Things I Think RFC 2100 Ashworth & Associates http://baylink.pitas.com 2000 Land Rover DII St Petersburg FL USA #natog +1 727 647 1274

----- Original Message -----
From: "Jay Ashworth" <jra@baylink.com>
4.2.2.x are *authoritatively* multicast;
Or, y'know, "anycast". Cheers, -- jr 'tnxka' a -- Jay R. Ashworth Baylink jra@baylink.com Designer The Things I Think RFC 2100 Ashworth & Associates http://baylink.pitas.com 2000 Land Rover DII St Petersburg FL USA #natog +1 727 647 1274

More importantly, unless something has changed in the past year or so Level 3 has always stated that the 4.2.2.X (yes there are others like .1 - .6) DNS severs are NOT production level systems and they will not provide any support for them. A good piece of evidence to this is that they don't charge for their use and Level 3 doesn't give much of anything away for free. If this is wrong and someone from Level3 wants to speak to it, I'd be happy to stand corrected. Sometimes a forum like NANOG is an appropriate place to mention that there is a possible issue (you can find this in the achieves), but I'm pretty sure that either way this would not qualify as an "outage". If you are using 4.2.2.X for DNS resolution for anything in production in any way other than as a last resort, I feel sorry for your customers. We all like to use it for testing because it's easy to remember and generally reliable, but I always do so with caution. -Scott -----Original Message----- From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On Behalf Of Jay Ashworth Sent: Wednesday, September 12, 2012 6:21 PM To: outages@outages.org Subject: Re: [outages] Packet loss to 4.2.2.2 ----- Original Message -----
From: "Josh Luthman" <josh@imaginenetworksllc.com>
Can you share from/to? I was told 4.2.2.2 is anycast (or otherwise geographically diverse), though I have no proof of that.
4.2.2.x are *authoritatively* multicast; I have that from someone who is involved in running recursive DNS for whatever the hell they're calling themselves this year. :-) In other news: not all 6 of those servers will always respond to ping traffic even if they're resolving; it's random; presumably they don't want anyone to get too settled into using them as a ping beacon to watching the weather on the 'net. Understandably. Cheers, -- jra -- Jay R. Ashworth Baylink jra@baylink.com Designer The Things I Think RFC 2100 Ashworth & Associates http://baylink.pitas.com 2000 Land Rover DII St Petersburg FL USA #natog +1 727 647 1274 _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages

----- Original Message -----
From: "Scott Berkman" <scott@sberkman.net>
More importantly, unless something has changed in the past year or so, Level 3 has always stated that the 4.2.2.X (yes there are others like .1 -> .6) DNS severs are NOT production level systems and they will not provide any support for them. A good piece of evidence to this is that they don't charge for their use and Level 3 doesn't give much of anything away for free. If this is wrong and someone from Level3 wants to speak to it, I'd be happy to stand corrected.
My understanding is that they're not supported *for anyone who isn't a direct customer of GTE/BBN/L3*; they *are* the customer resolver servers for their direct end-user customer base. But that's not evidence, just my understanding. Cheers, -- jra -- Jay R. Ashworth Baylink jra@baylink.com Designer The Things I Think RFC 2100 Ashworth & Associates http://baylink.pitas.com 2000 Land Rover DII St Petersburg FL USA #natog +1 727 647 1274

Just for the record, it was resolved around 3AM Eastern (for me) http://imgur.com/7B7Vp Josh Luthman Office: 937-552-2340 Direct: 937-552-2343 1100 Wayne St Suite 1337 Troy, OH 45373 On Thu, Sep 13, 2012 at 10:17 AM, Jay Ashworth <jra@baylink.com> wrote:
----- Original Message -----
From: "Scott Berkman" <scott@sberkman.net>
More importantly, unless something has changed in the past year or so, Level 3 has always stated that the 4.2.2.X (yes there are others like .1 -> .6) DNS severs are NOT production level systems and they will not provide any support for them. A good piece of evidence to this is that they don't charge for their use and Level 3 doesn't give much of anything away for free. If this is wrong and someone from Level3 wants to speak to it, I'd be happy to stand corrected.
My understanding is that they're not supported *for anyone who isn't a direct customer of GTE/BBN/L3*; they *are* the customer resolver servers for their direct end-user customer base. But that's not evidence, just my understanding.
Cheers, -- jra -- Jay R. Ashworth Baylink jra@baylink.com Designer The Things I Think RFC 2100 Ashworth & Associates http://baylink.pitas.com 2000 Land Rover DII St Petersburg FL USA #natog +1 727 647 1274 _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages

I'm not seeing any loss over integra to level3 from las vegas. -----Original Message----- From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On Behalf Of Matthew Anderson Sent: Wednesday, September 12, 2012 3:11 PM To: Josh Luthman; outages@outages.org Subject: Re: [outages] Packet loss to 4.2.2.2 I am seeing the same. Thanks! Matthew Anderson Network Operations Manager Enhanced Telecommunications Corporation Office: 812-623-6288 Cell: 812-212-5565 From: Josh Luthman <josh@imaginenetworksllc.com<mailto:josh@imaginenetworksllc.com>> Date: Wednesday, September 12, 2012 4:08 PM To: "outages@outages.org<mailto:outages@outages.org>" <outages@outages.org<mailto:outages@outages.org>> Subject: [outages] Packet loss to 4.2.2.2 Is anyone seeing any issues to 4.2.2.2? At 3:35 EST I starting seeing packet loss (icmp ping) to it - http://imgur.com/ITGL1 I noticed at the same time we had a route change with RR (Time Warner Cable) in Chicago. Josh Luthman Office: 937-552-2340 Direct: 937-552-2343 1100 Wayne St Suite 1337 Troy, OH 45373 _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages

I can reproduce it over here as well. Maybe Level 3 is finally getting sick of people using it for ping testing after all these years? :) Mike Phipps Media Genesis, Inc. -----Original Message----- From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On Behalf Of Matthew Anderson Sent: Wednesday, September 12, 2012 4:11 PM To: Josh Luthman; outages@outages.org Subject: Re: [outages] Packet loss to 4.2.2.2 I am seeing the same. Thanks! Matthew Anderson Network Operations Manager Enhanced Telecommunications Corporation Office: 812-623-6288 Cell: 812-212-5565 From: Josh Luthman <josh@imaginenetworksllc.com<mailto:josh@imaginenetworksllc.com>> Date: Wednesday, September 12, 2012 4:08 PM To: "outages@outages.org<mailto:outages@outages.org>" <outages@outages.org<mailto:outages@outages.org>> Subject: [outages] Packet loss to 4.2.2.2 Is anyone seeing any issues to 4.2.2.2? At 3:35 EST I starting seeing packet loss (icmp ping) to it - http://imgur.com/ITGL1 I noticed at the same time we had a route change with RR (Time Warner Cable) in Chicago. Josh Luthman Office: 937-552-2340 Direct: 937-552-2343 1100 Wayne St Suite 1337 Troy, OH 45373 _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages

I'm getting 50% loss: Pinging 4.2.2.2 with 32 bytes of data: Reply from 4.2.2.2: bytes=32 time=19ms TTL=57 Request timed out. Request timed out. Reply from 4.2.2.2: bytes=32 time=36ms TTL=57 Ping statistics for 4.2.2.2: Packets: Sent = 4, Received = 2, Lost = 2 (50% loss), Approximate round trip times in milli-seconds: Minimum = 19ms, Maximum = 36ms, Average = 27ms On Wed, Sep 12, 2012 at 4:15 PM, Mike Phipps <mphipps@mediag.com> wrote:
I can reproduce it over here as well. Maybe Level 3 is finally getting sick of people using it for ping testing after all these years? :)
Mike Phipps Media Genesis, Inc.
-----Original Message----- From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On Behalf Of Matthew Anderson Sent: Wednesday, September 12, 2012 4:11 PM To: Josh Luthman; outages@outages.org Subject: Re: [outages] Packet loss to 4.2.2.2
I am seeing the same.
Thanks!
Matthew Anderson Network Operations Manager Enhanced Telecommunications Corporation Office: 812-623-6288 Cell: 812-212-5565
From: Josh Luthman <josh@imaginenetworksllc.com<mailto:josh@imaginenetworksllc.com>> Date: Wednesday, September 12, 2012 4:08 PM To: "outages@outages.org<mailto:outages@outages.org>" <outages@outages.org<mailto:outages@outages.org>> Subject: [outages] Packet loss to 4.2.2.2
Is anyone seeing any issues to 4.2.2.2?
At 3:35 EST I starting seeing packet loss (icmp ping) to it - http://imgur.com/ITGL1
I noticed at the same time we had a route change with RR (Time Warner Cable) in Chicago.
Josh Luthman Office: 937-552-2340 Direct: 937-552-2343 1100 Wayne St Suite 1337 Troy, OH 45373
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages

I do not see this problem, I am in Philly Metro. ------------------------------------- Ping statistics for 4.2.2.2: Packets: Sent = 509, Received = 509, Lost = 0 (0% loss), Approximate round trip times in milli-seconds: Minimum = 4ms, Maximum = 15ms, Average = 4ms Control-C [X:\] [X:\]tr 4.2.2.2 Tracing route to b.resolvers.Level3.net [4.2.2.2] over a maximum of 30 hops: 1 4 ms 3 ms 1 ms acs2-faste2-0-85.asinternet.net [63.175.140.5] 2 71 ms 12 ms 12 ms vlan276.car1.Philadelphia1.Level3.net [4.59.140.97] 3 2 ms 2 ms 3 ms ae-3-3.bar1.Philadelphia1.Level3.net [4.69.153.93] 4 3 ms 3 ms 3 ms ae-11-11.bar2.Philadelphia1.Level3.net [4.69.153.90] 5 4 ms 4 ms 4 ms ae-6-6.ebr1.NewYork1.Level3.net [4.69.133.170] 6 5 ms 5 ms 5 ms ae-71-71.csw2.NewYork1.Level3.net [4.69.134.70] 7 4 ms 4 ms 4 ms ae-2-70.edge2.NewYork1.Level3.net [4.69.155.80] 8 4 ms 4 ms 4 ms b.resolvers.Level3.net [4.2.2.2] Trace complete. Tim Glen Sr. Network Engineer tglen@asicentral.com tel: (215) 953-3692 fax: (215) 953-3759 -----Original Message----- From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On Behalf Of Mike Phipps Sent: Wednesday, September 12, 2012 4:16 PM To: outages@outages.org Subject: Re: [outages] Packet loss to 4.2.2.2 I can reproduce it over here as well. Maybe Level 3 is finally getting sick of people using it for ping testing after all these years? :) Mike Phipps Media Genesis, Inc. -----Original Message----- From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On Behalf Of Matthew Anderson Sent: Wednesday, September 12, 2012 4:11 PM To: Josh Luthman; outages@outages.org Subject: Re: [outages] Packet loss to 4.2.2.2 I am seeing the same. Thanks! Matthew Anderson Network Operations Manager Enhanced Telecommunications Corporation Office: 812-623-6288 Cell: 812-212-5565 From: Josh Luthman <josh@imaginenetworksllc.com<mailto:josh@imaginenetworksllc.com>> Date: Wednesday, September 12, 2012 4:08 PM To: "outages@outages.org<mailto:outages@outages.org>" <outages@outages.org<mailto:outages@outages.org>> Subject: [outages] Packet loss to 4.2.2.2 Is anyone seeing any issues to 4.2.2.2? At 3:35 EST I starting seeing packet loss (icmp ping) to it - http://imgur.com/ITGL1 I noticed at the same time we had a route change with RR (Time Warner Cable) in Chicago. Josh Luthman Office: 937-552-2340 Direct: 937-552-2343 1100 Wayne St Suite 1337 Troy, OH 45373 _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages

On Wed, Sep 12, 2012 at 1:08 PM, Josh Luthman <josh@imaginenetworksllc.com> wrote:
Is anyone seeing any issues to 4.2.2.2?
Nope, my local one seems solid. These are anycasted, so I'm hitting the one in San Jose, California. Maybe try and MTR or some other hop-by-hop analysis and see if it's a bad path to 4.2.2.2 or some other link in-between.
At 3:35 EST I starting seeing packet loss (icmp ping) to it - http://imgur.com/ITGL1
Yup, looks lossy and not normal. Cheers, jof

4.2.2.2 is an anycast address, so any answer to this will likely depend on exactly where you are, and which 4.2.2.2 you are hitting... Scott On Wed, Sep 12, 2012 at 1:08 PM, Josh Luthman <josh@imaginenetworksllc.com>wrote:
Is anyone seeing any issues to 4.2.2.2?
At 3:35 EST I starting seeing packet loss (icmp ping) to it - http://imgur.com/ITGL1
I noticed at the same time we had a route change with RR (Time Warner Cable) in Chicago.
Josh Luthman Office: 937-552-2340 Direct: 937-552-2343 1100 Wayne St Suite 1337 Troy, OH 45373
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages

I am on a TWTC circuit in Milwaukee and am seeing no ping loss or DNS failures to and of the 4.2.2 dns servers. -Grant On Wed, Sep 12, 2012 at 3:16 PM, Scott Howard <scott@doc.net.au> wrote:
4.2.2.2 is an anycast address, so any answer to this will likely depend on exactly where you are, and which 4.2.2.2 you are hitting...
Scott
On Wed, Sep 12, 2012 at 1:08 PM, Josh Luthman <josh@imaginenetworksllc.com
wrote:
Is anyone seeing any issues to 4.2.2.2?
At 3:35 EST I starting seeing packet loss (icmp ping) to it - http://imgur.com/ITGL1
I noticed at the same time we had a route change with RR (Time Warner Cable) in Chicago.
Josh Luthman Office: 937-552-2340 Direct: 937-552-2343 1100 Wayne St Suite 1337 Troy, OH 45373
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages

From their LA pop, not seeing any packet loss.
PING 4.2.2.2 (4.2.2.2): 56 data bytes .. --- 4.2.2.2 ping statistics --- 1000 packets transmitted, 1000 packets received, 0.0% packet loss round-trip min/avg/max/stddev = 0.393/0.502/1.044/0.112 ms On Wed, Sep 12, 2012 at 01:16:49PM -0700, Scott Howard wrote:
4.2.2.2 is an anycast address, so any answer to this will likely depend on exactly where you are, and which 4.2.2.2 you are hitting...
Scott
On Wed, Sep 12, 2012 at 1:08 PM, Josh Luthman <josh-hPvxz4IaMr62QFlZL/IlyMI/UQi/AW5J@public.gmane.org> wrote:
Is anyone seeing any issues to 4.2.2.2? At 3:35 EST I starting seeing packet loss (icmp ping) to it - http://imgur.com/ITGL1 I noticed at the same time we had a route change with RR (Time Warner Cable) in Chicago. Josh Luthman Office: 937-552-2340 Direct: 937-552-2343 1100 Wayne St Suite 1337 Troy, OH 45373 _______________________________________________ Outages mailing list Outages-w8q5NolJmqtAfugRpC6u6w@public.gmane.org https://puck.nether.net/mailman/listinfo/outages

On 09/12/12 16:08 -0400, Josh Luthman wrote:
Is anyone seeing any issues to 4.2.2.2?
At 3:35 EST I starting seeing packet loss (icmp ping) to it - http://imgur.com/ITGL1
I noticed at the same time we had a route change with RR (Time Warner Cable) in Chicago.
I am not seeing loss to the 4.2.2.2 located near Dallas: ~$ traceroute 4.2.2.2 traceroute to 4.2.2.2 (4.2.2.2), 30 hops max, 60 byte packets 1 olp-67-217-151-193.olp.net (67.217.151.193) 0.484 ms 0.563 ms 0.645 ms 2 olp-67-217-152-34.olp.net (67.217.152.34) 0.988 ms 1.063 ms 1.135 ms 3 ae5-694.edge9.Dallas1.Level3.net (4.30.66.17) 7.891 ms 9.835 ms 7.910 ms 4 ae-4-90.edge3.Dallas1.Level3.net (4.69.145.200) 7.904 ms ae-2-70.edge3.Dallas1.Level3.net (4.69.145.72) 7.961 ms ae-1-60.edge3.Dallas1.Level3.net (4.69.145.8) 7.939 ms 5 b.resolvers.Level3.net (4.2.2.2) 36.606 ms 8.030 ms 8.034 ms ~# ping -A -c 100 -q 4.2.2.2 PING 4.2.2.2 (4.2.2.2) 56(84) bytes of data. --- 4.2.2.2 ping statistics --- 100 packets transmitted, 100 received, 0% packet loss, time 890ms rtt min/avg/max/mdev = 7.913/8.796/13.416/1.326 ms, pipe 2, ipg/ewma 8.996/8.421 ms -- Dan White

Yep, about 30% ping requests are timing out for me. I'm in Chicago on a Level3 connection too. -george From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On Behalf Of Josh Luthman Sent: Wednesday, September 12, 2012 3:08 PM To: outages@outages.org Subject: [outages] Packet loss to 4.2.2.2 Is anyone seeing any issues to 4.2.2.2? At 3:35 EST I starting seeing packet loss (icmp ping) to it - http://imgur.com/ITGL1 I noticed at the same time we had a route change with RR (Time Warner Cable) in Chicago. Josh Luthman Office: 937-552-2340 Direct: 937-552-2343 1100 Wayne St Suite 1337 Troy, OH 45373

Not seeing loss from Central Oregon: shadowfax # mtr -r -c 10 4.2.2.2 HOST: shadowfax Loss% Snt Last Avg Best Wrst StDev #snipped 3.|-- qwest-border.bendcable.co 0.0% 10 1.0 1.0 0.9 1.4 0.2 4.|-- vlan186.car1.seattle1.lev 0.0% 10 37.2 66.0 37.2 182.5 60.4 5.|-- ae-31-51.ebr1.seattle1.le 0.0% 10 47.7 46.6 37.6 49.8 4.8 6.|-- ae-7-7.ebr2.sanjose1.leve 0.0% 10 45.1 45.1 45.0 45.7 0.2 7.|-- ae-62-62.csw1.sanjose1.le 0.0% 10 47.3 49.3 47.3 53.2 2.0 8.|-- ae-1-60.edge1.sanjose1.le 0.0% 10 45.2 45.3 45.1 45.4 0.1 9.|-- b.resolvers.level3.net 0.0% 10 45.2 45.3 45.2 45.4 0.1 -Dave On Wed, Sep 12, 2012 at 1:08 PM, Josh Luthman <josh@imaginenetworksllc.com>wrote:
Is anyone seeing any issues to 4.2.2.2?
At 3:35 EST I starting seeing packet loss (icmp ping) to it - http://imgur.com/ITGL1
I noticed at the same time we had a route change with RR (Time Warner Cable) in Chicago.
Josh Luthman Office: 937-552-2340 Direct: 937-552-2343 1100 Wayne St Suite 1337 Troy, OH 45373
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages

So far it looks like the only problematic one is in Chicago. On Wed, Sep 12, 2012 at 3:38 PM, Dave Owens <dave@teamunify.com> wrote:
Not seeing loss from Central Oregon:
shadowfax # mtr -r -c 10 4.2.2.2 HOST: shadowfax Loss% Snt Last Avg Best Wrst StDev #snipped 3.|-- qwest-border.bendcable.co 0.0% 10 1.0 1.0 0.9 1.4 0.2 4.|-- vlan186.car1.seattle1.lev 0.0% 10 37.2 66.0 37.2 182.5 60.4 5.|-- ae-31-51.ebr1.seattle1.le 0.0% 10 47.7 46.6 37.6 49.8 4.8 6.|-- ae-7-7.ebr2.sanjose1.leve 0.0% 10 45.1 45.1 45.0 45.7 0.2 7.|-- ae-62-62.csw1.sanjose1.le 0.0% 10 47.3 49.3 47.3 53.2 2.0 8.|-- ae-1-60.edge1.sanjose1.le 0.0% 10 45.2 45.3 45.1 45.4 0.1 9.|-- b.resolvers.level3.net 0.0% 10 45.2 45.3 45.2 45.4 0.1
-Dave
On Wed, Sep 12, 2012 at 1:08 PM, Josh Luthman <josh@imaginenetworksllc.com
wrote:
Is anyone seeing any issues to 4.2.2.2?
At 3:35 EST I starting seeing packet loss (icmp ping) to it - http://imgur.com/ITGL1
I noticed at the same time we had a route change with RR (Time Warner Cable) in Chicago.
Josh Luthman Office: 937-552-2340 Direct: 937-552-2343 1100 Wayne St Suite 1337 Troy, OH 45373
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages

On 09/12/2012 02:44 PM, Grant Ridder wrote:
So far it looks like the only problematic one is in Chicago.
I get sent to Dallas - and that looks fine: 14:49:26-cls@orion:~$ mtr --report-cycles=25 --report --report-wide 4.2.2.2 HOST: orion.axint.net Loss% Snt Last Avg Best Wrst StDev 3.|-- hlrn-agw1.inet.qwest.net 0.0% 25 40.4 43.2 39.1 69.2 9.2 4.|-- dvr-brdr-02.inet.qwest.net 0.0% 25 40.9 42.0 39.8 73.5 6.6 5.|-- 63.146.26.134 0.0% 25 40.4 40.7 39.8 43.6 0.8 6.|-- vlan51.ebr1.Denver1.Level3.net 0.0% 25 40.6 45.8 40.1 54.5 4.3 7.|-- ae-2-2.ebr2.Dallas1.Level3.net 0.0% 25 60.5 60.9 60.1 63.3 0.8 8.|-- ae-62-62.csw1.Dallas1.Level3.net 0.0% 25 71.1 60.1 58.6 71.1 2.3 9.|-- ae-1-60.edge3.Dallas1.Level3.net 0.0% 25 59.4 61.2 59.0 86.4 5.3 10.|-- b.resolvers.Level3.net 0.0% 25 61.1 61.2 60.2 67.2 1.3 Chris

Good from both coasts for me: -East: lucid Snt: 5 Loss% Last Avg Best Wrst StDev j-vl102-d2.acc.dca3.hopone.net 0.0% 0.3 0.3 0.3 0.4 0.0 ge6-1.core2.dca3.hopone.net 0.0% 0.2 0.2 0.2 0.2 0.0 ge9-2.core1.dca3.hopone.net 0.0% 0.2 0.2 0.2 0.3 0.0 ge6-1.core1.dca2.hopone.net 0.0% 2.4 2.4 2.3 2.5 0.1 ge-7-27.car1.washington1.level3.net 0.0% 3.0 3.0 2.8 3.2 0.2 b.resolvers.level3.net 0.0% 3.2 3.2 3.1 3.4 0.1 -West vps Snt: 5 Loss% Last Avg Best Wrst StDev 10.0.50.5 0.0% 0.0 0.1 0.0 0.1 0.0 ge2-3.core1.sea2.hopone.net 0.0% 0.5 0.4 0.3 0.6 0.1 xe-7-2-3.edge2.seattle1.level3.net 0.0% 1.7 1.8 1.7 1.8 0.1 ae-31-51.ebr1.Seattle1.Level3.net 0.0% 1.9 2.7 1.7 5.0 1.4 ae-7-7.ebr2.SanJose1.Level3.net 0.0% 18.8 18.7 18.6 18.8 0.1 ae-62-62.csw1.SanJose1.Level3.net 0.0% 29.2 23.4 18.9 31.3 6.3 ae-1-60.edge1.SanJose1.Level3.net 0.0% 23.9 20.7 18.6 24.0 2.9 b.resolvers.Level3.net 0.0% 18.6 18.7 18.6 18.8 0.1 On 9/12/2012 4:38 PM, Dave Owens wrote:
Not seeing loss from Central Oregon:
shadowfax # mtr -r -c 10 4.2.2.2 HOST: shadowfax Loss% Snt Last Avg Best Wrst StDev #snipped 3.|-- qwest-border.bendcable.co 0.0% 10 1.0 1.0 0.9 1.4 0.2 4.|-- vlan186.car1.seattle1.lev 0.0% 10 37.2 66.0 37.2 182.5 60.4 5.|-- ae-31-51.ebr1.seattle1.le 0.0% 10 47.7 46.6 37.6 49.8 4.8 6.|-- ae-7-7.ebr2.sanjose1.leve 0.0% 10 45.1 45.1 45.0 45.7 0.2 7.|-- ae-62-62.csw1.sanjose1.le 0.0% 10 47.3 49.3 47.3 53.2 2.0 8.|-- ae-1-60.edge1.sanjose1.le 0.0% 10 45.2 45.3 45.1 45.4 0.1 9.|-- b.resolvers.level3.net 0.0% 10 45.2 45.3 45.2 45.4 0.1
-Dave
On Wed, Sep 12, 2012 at 1:08 PM, Josh Luthman <josh@imaginenetworksllc.com>wrote:
Is anyone seeing any issues to 4.2.2.2?
At 3:35 EST I starting seeing packet loss (icmp ping) to it - http://imgur.com/ITGL1
I noticed at the same time we had a route change with RR (Time Warner Cable) in Chicago.
Josh Luthman Office: 937-552-2340 Direct: 937-552-2343 1100 Wayne St Suite 1337 Troy, OH 45373
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
-- Christopher Singhaus, Systems Administrator / Abuse Dept csinghaus@hopone.net *** 703-564-5400 HopOne Internet Corp. - AS 14361 - www.hopone.net "Yesterday's weirdness is tomorrow's reason why." -Hunter S Thompson

Looks good from PHX localhost.localdomain (0.0.0.0) Wed Sep 12 14:01:29 2012 Resolver error: No error returned but no answers given. of fields quit Packets Pings Host Loss% Snt Last Avg Best Wrst StDev 1. vl161.sw1.hipaa365.net 0.0% 6 0.8 0.7 0.7 0.8 0.0 2. 199.66.112.129 0.0% 6 0.7 0.7 0.7 1.1 0.2 3. 69.160.32.26 0.0% 6 1.2 10.0 0.9 55.3 22.2 4. bb2.cwie.10.2.phx0.cwie.net 0.0% 6 0.8 0.8 0.8 0.8 0.0 5. xe-10-2-1.bar2.phoenix1.level3.net 0.0% 5 0.9 0.9 0.9 1.0 0.0 6. ae-4-4.ebr2.losangeles1.level3.net 0.0% 5 9.7 9.8 9.7 9.9 0.1 7. ae-62-62.csw1.losangeles1.level3.net 0.0% 5 12.2 12.7 9.8 16.9 3.1 8. ae-1-60.edge3.losangeles1.level3.net 0.0% 5 9.8 9.8 9.8 9.8 0.0 9. b.resolvers.level3.net 0.0% 5 14.2 12.6 9.9 15.7 2.3 Raman Sud Director, Network Infrastructure PhoenixNAP LLC 3402 East University Drive Phoenix AZ 85034 Phone: 480.449.8834 Fax: 480.682.3236 http://www.phoenixnap.com PGP Key: http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xBB765DA2 THIS MESSAGE IS INTENDED ONLY FOR THE USE OF THE INDIVIDUAL OR ENTITY TO WHICH IT IS ADDRESSED, AND MAY CONTAIN INFORMATION THAT IS PRIVILEGED, CONFIDENTIAL AND EXEMPT FROM DISCLOSURE UNDER APPLICABLE LAW. IF THE READER OF THIS MESSAGE IS NOT THE INTENDED RECIPIENT, YOU ARE HEREBY NOTIFIED THAT ANY DISSEMINATION, DISTRIBUTION OR COPYING OF THIS COMMUNICATION IS STRICTLY PROHIBITED, AND THAT THIS EMAIL AND ALL COPIES AND ATTACHMENTS SHOULD BE IMMEDIATELY DESTROYED. -----Original Message----- From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On Behalf Of Christopher Singhaus Sent: Wednesday, September 12, 2012 1:48 PM To: outages@outages.org Subject: Re: [outages] Packet loss to 4.2.2.2 Good from both coasts for me: -East: lucid Snt: 5 Loss% Last Avg Best Wrst StDev j-vl102-d2.acc.dca3.hopone.net 0.0% 0.3 0.3 0.3 0.4 0.0 ge6-1.core2.dca3.hopone.net 0.0% 0.2 0.2 0.2 0.2 0.0 ge9-2.core1.dca3.hopone.net 0.0% 0.2 0.2 0.2 0.3 0.0 ge6-1.core1.dca2.hopone.net 0.0% 2.4 2.4 2.3 2.5 0.1 ge-7-27.car1.washington1.level3.net 0.0% 3.0 3.0 2.8 3.2 0.2 b.resolvers.level3.net 0.0% 3.2 3.2 3.1 3.4 0.1 -West vps Snt: 5 Loss% Last Avg Best Wrst StDev 10.0.50.5 0.0% 0.0 0.1 0.0 0.1 0.0 ge2-3.core1.sea2.hopone.net 0.0% 0.5 0.4 0.3 0.6 0.1 xe-7-2-3.edge2.seattle1.level3.net 0.0% 1.7 1.8 1.7 1.8 0.1 ae-31-51.ebr1.Seattle1.Level3.net 0.0% 1.9 2.7 1.7 5.0 1.4 ae-7-7.ebr2.SanJose1.Level3.net 0.0% 18.8 18.7 18.6 18.8 0.1 ae-62-62.csw1.SanJose1.Level3.net 0.0% 29.2 23.4 18.9 31.3 6.3 ae-1-60.edge1.SanJose1.Level3.net 0.0% 23.9 20.7 18.6 24.0 2.9 b.resolvers.Level3.net 0.0% 18.6 18.7 18.6 18.8 0.1 On 9/12/2012 4:38 PM, Dave Owens wrote:
Not seeing loss from Central Oregon:
shadowfax # mtr -r -c 10 4.2.2.2 HOST: shadowfax Loss% Snt Last Avg Best Wrst StDev #snipped 3.|-- qwest-border.bendcable.co 0.0% 10 1.0 1.0 0.9 1.4 0.2 4.|-- vlan186.car1.seattle1.lev 0.0% 10 37.2 66.0 37.2 182.5 60.4 5.|-- ae-31-51.ebr1.seattle1.le 0.0% 10 47.7 46.6 37.6 49.8 4.8 6.|-- ae-7-7.ebr2.sanjose1.leve 0.0% 10 45.1 45.1 45.0 45.7 0.2 7.|-- ae-62-62.csw1.sanjose1.le 0.0% 10 47.3 49.3 47.3 53.2 2.0 8.|-- ae-1-60.edge1.sanjose1.le 0.0% 10 45.2 45.3 45.1 45.4 0.1 9.|-- b.resolvers.level3.net 0.0% 10 45.2 45.3 45.2 45.4 0.1
-Dave
On Wed, Sep 12, 2012 at 1:08 PM, Josh Luthman <josh@imaginenetworksllc.com>wrote:
Is anyone seeing any issues to 4.2.2.2?
At 3:35 EST I starting seeing packet loss (icmp ping) to it - http://imgur.com/ITGL1
I noticed at the same time we had a route change with RR (Time Warner Cable) in Chicago.
Josh Luthman Office: 937-552-2340 Direct: 937-552-2343 1100 Wayne St Suite 1337 Troy, OH 45373
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
-- Christopher Singhaus, Systems Administrator / Abuse Dept csinghaus@hopone.net *** 703-564-5400 HopOne Internet Corp. - AS 14361 - www.hopone.net "Yesterday's weirdness is tomorrow's reason why." -Hunter S Thompson _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages

I show pretty clean to the server itself from a Level(3) connection in Detroit, but I was seeing loss to the router one hop up from it (which seems to be getting better with time). -Bill 4. ge-6-16.car2.Detroit1.Level3.net 0.0% 4.5 26.3 3.3 190.9 47.2 5. ae-11-11.car1.Detroit1.Level3.net 0.0% 9.1 30.6 9.1 198.8 48.1 6. ae-8-8.ebr2.Chicago1.Level3.net 0.0% 14.3 12.7 8.9 22.2 3.8 7. ae-21-52.car1.Chicago1.Level3.net 10.0% 137.9 22.8 9.0 137.9 32.1 8. b.resolvers.Level3.net 0.0% 9.6 10.0 9.0 12.3 0.7 ________________________________ From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On Behalf Of Josh Luthman Sent: Wednesday, September 12, 2012 4:08 PM To: outages@outages.org Subject: [outages] Packet loss to 4.2.2.2 Is anyone seeing any issues to 4.2.2.2? At 3:35 EST I starting seeing packet loss (icmp ping) to it - http://imgur.com/ITGL1 I noticed at the same time we had a route change with RR (Time Warner Cable) in Chicago. Josh Luthman Office: 937-552-2340 Direct: 937-552-2343 1100 Wayne St Suite 1337 Troy, OH 45373

See very little packets being dropped. 4 out of 5000. Pinging from Denver. On Wed, Sep 12, 2012 at 2:40 PM, Bill Wichers <billw@waveform.net> wrote:
** ** ** ** ** **
I show pretty clean to the server itself from a Level(3) connection in *** *Detroit****, but I was seeing loss to the router one hop up from it (which seems to be getting better with time).****
** **
-Bill****
** **
4. ge-6-16.car2.Detroit1.Level3.net 0.0% 4.5 26.3 3.3 190.9 47.2****
5. ae-11-11.car1.Detroit1.Level3.net 0.0% 9.1 30.6 9.1 198.8 48.1****
6. ae-8-8.ebr2.Chicago1.Level3.net 0.0% 14.3 12.7 8.9 22.2 3.8****
7. ae-21-52.car1.Chicago1.Level3.net 10.0% 137.9 22.8 9.0 137.9 32.1****
8. b.resolvers.Level3.net 0.0% 9.6 10.0 9.0 12.3 0.7****
** ** ------------------------------
*From:* outages-bounces@outages.org [mailto:outages-bounces@outages.org] *On Behalf Of *Josh Luthman *Sent:* Wednesday, September 12, 2012 4:08 PM *To:* outages@outages.org
*Subject:* [outages] Packet loss to 4.2.2.2 ****
** **
Is anyone seeing any issues to 4.2.2.2?****
** **
At 3:35 EST I starting seeing packet loss (icmp ping) to it - http://imgur.com/ITGL1 ****
** **
I noticed at the same time we had a route change with RR (Time Warner Cable) in ****Chicago****.****
Josh Luthman Office: 937-552-2340 Direct: 937-552-2343 ****1100 Wayne St**** ****Suite** 1337** ****Troy**, **OH** **45373********
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
-- Shriram Ayyar

----- Original Message -----
From: "Bill Wichers" <billw@waveform.net>
I show pretty clean to the server itself from a Level(3) connection in Detroit, but I was seeing loss to the router one hop up from it (which seems to be getting better with time).
Just a reminder to all: intermediate hops on a traceroute may show high packet drops *without that router actually dropping any real traffic packets*. Many routers are required by their design to force ICMP traffic up into the CPU, where actual packets may remain down on the much more efficient cross-linecard-path, which means that if the router gets too busy doing "Real work", the ICMP will be the first thing to go. Evidence of this is usually a large degree of packet loss which starts and ends with one hop in your trace; if *everything past a certain point* seems to be dropping roughly the same percentage of traffic, *that* is probably the outbound link from the router. Cheers, -- jra -- Jay R. Ashworth Baylink jra@baylink.com Designer The Things I Think RFC 2100 Ashworth & Associates http://baylink.pitas.com 2000 Land Rover DII St Petersburg FL USA #natog +1 727 647 1274

Just a reminder to all:
intermediate hops on a traceroute may show high packet drops *without that router actually dropping any real traffic packets*. Many routers are required by their design to force ICMP traffic up into the CPU, where actual packets may remain down on the much more efficient cross-linecard-path, which means that if the router gets too busy doing "Real work", the ICMP will be the first thing to go.
Evidence of this is usually a large degree of packet loss which starts and ends with one hop in your trace; if *everything past a certain point* seems to be dropping roughly the same percentage of traffic, *that* is probably the outbound link from the router.
Bingo. Traceroute is not a true indicator of packet loss; ping on the other hand would be so long as the endpoint or intermediate endpoints aren't interfering with the flow and don't icmp rate limit. In any case, I would like to point an excellent resource for interpreting traceroute results from NANOG47: http://www.nanog.org/meetings/nanog47/presentations/Sunday/RAS_Traceroute_N4... - Naveen

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On 09/12/2012 01:08 PM, Josh Luthman wrote:
Is anyone seeing any issues to 4.2.2.2?
At 3:35 EST I starting seeing packet loss (icmp ping) to it - http://imgur.com/ITGL1
I noticed at the same time we had a route change with RR (Time Warner Cable) in Chicago.
Josh Luthman Office: 937-552-2340 Direct: 937-552-2343 1100 Wayne St Suite 1337 Troy, OH 45373
trace off socal tw looks good. vrode@skywalker:~$ traceroute 4.2.2.2 traceroute to 4.2.2.2 (4.2.2.2), 30 hops max, 60 byte packets 1 Debian-AirPort-Extreme.local (10.0.1.1) 1.267 ms 1.508 ms 1.723 ms 2 cpe-72-130-64-1.socal.res.rr.com (72.130.64.1) 28.018 ms 28.018 ms 42.050 ms 3 tge7-1.orngcani-cer02.socal.rr.com (76.166.13.161) 16.810 ms * * 4 cpe-72-129-30-42.socal.res.rr.com (72.129.30.42) 25.404 ms 25.574 ms 25.576 ms 5 agg23.tustca1-cr01.socal.rr.com (72.129.29.2) 29.292 ms 29.494 ms 29.713 ms 6 107.14.19.30 (107.14.19.30) 27.475 ms 19.381 ms 36.890 ms 7 107.14.19.67 (107.14.19.67) 68.924 ms 69.445 ms 69.010 ms 8 xe-4-1-0.edge1.LosAngeles6.Level3.net (4.26.0.73) 34.800 ms xe-4-2-0.edge1.LosAngeles6.Level3.net (4.26.0.77) 36.642 ms 36.622 ms 9 ae-1-60.edge3.LosAngeles1.Level3.net (4.69.144.9) 77.887 ms ae-4-90.edge3.LosAngeles1.Level3.net (4.69.144.201) 36.311 ms ae-1-60.edge3.LosAngeles1.Level3.net (4.69.144.9) 78.167 ms 10 b.resolvers.Level3.net (4.2.2.2) 36.260 ms 40.238 ms 40.414 ms mtr off socal tw looks good: vrode@skywalker:~$ mtr -c 10 -r 4.2.2.2 HOST: skywalker Loss% Snt Last Avg Best Wrst StDev 1.|-- Debian-AirPort-Extreme.lo 0.0% 10 2.9 2.0 0.9 5.9 1.6 2.|-- cpe-72-130-64-1.socal.res 0.0% 10 12.2 23.3 11.6 51.1 12.2 3.|-- tge7-1.orngcani-cer02.soc 0.0% 10 39.7 17.9 11.6 39.7 8.9 4.|-- cpe-72-129-30-42.socal.re 0.0% 10 19.6 22.2 14.5 35.3 7.0 5.|-- agg23.tustca1-cr01.socal. 0.0% 10 21.4 18.3 13.6 28.5 4.3 6.|-- 107.14.19.30 0.0% 10 14.0 19.5 14.0 26.6 3.8 7.|-- ae-1-0.pr0.lax00.tbone.rr 0.0% 10 19.0 19.0 15.6 38.8 7.0 8.|-- xe-7-2-0.edge1.LosAngeles 0.0% 10 15.0 24.2 14.9 61.4 15.1 9.|-- ae-3-80.edge3.LosAngeles1 0.0% 10 17.3 18.9 15.8 25.8 3.2 10.|-- b.resolvers.Level3.net 0.0% 10 16.4 20.0 15.0 34.5 5.8 trace off verizon aircard (somewhere on 5 fwy): vrode@skywalker:~$ traceroute 4.2.2.2 traceroute to 4.2.2.2 (4.2.2.2), 30 hops max, 60 byte packets 1 164.sub-66-174-33.myvzw.com (66.174.33.164) 45.254 ms 45.207 ms 45.221 ms 2 49.sub-69-83-183.myvzw.com (69.83.183.49) 47.850 ms 47.795 ms 47.364 ms 3 * * * 4 163.sub-66-174-32.myvzw.com (66.174.32.163) 47.276 ms 47.257 ms 47.241 ms 5 * * * 6 0.xe-0-1-3.XT1.LAX9.ALTER.NET (152.63.114.157) 47.179 ms 39.779 ms 49.732 ms 7 0.xe-10-0-0.BR1.LAX15.ALTER.NET (152.63.115.90) 39.845 ms 49.812 ms 49.793 ms 8 ae6.edge1.LosAngeles9.level3.net (4.68.62.169) 49.780 ms 49.764 ms 49.750 ms 9 ae-1-60.edge3.LosAngeles1.Level3.net (4.69.144.9) 49.736 ms 49.644 ms ae-2-70.edge3.LosAngeles1.Level3.net (4.69.144.73) 49.596 ms 10 b.resolvers.Level3.net (4.2.2.2) 49.580 ms 29.875 ms 39.894 ms mrt off verizon aircard (somewhere on 5 fwy): vrode@skywalker:~$ mtr -c 10 -r 4.2.2.2 HOST: skywalker Loss% Snt Last Avg Best Wrst StDev 1.|-- 164.sub-66-174-33.myvzw.c 0.0% 10 37.9 41.5 37.5 46.3 3.5 2.|-- 49.sub-69-83-183.myvzw.co 0.0% 10 47.8 45.3 39.1 48.8 3.0 3.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0 4.|-- 163.sub-66-174-32.myvzw.c 0.0% 10 47.6 51.9 34.6 118.6 24.1 5.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0 6.|-- 0.xe-0-1-3.XT1.LAX9.ALTER 0.0% 10 47.3 44.7 35.7 49.8 4.7 7.|-- 0.xe-10-0-0.BR1.LAX15.ALT 0.0% 10 47.2 45.5 35.5 54.2 5.5 8.|-- ae6.edge1.LosAngeles9.lev 0.0% 10 47.1 53.2 39.6 109.6 20.2 9.|-- ae-3-80.edge3.LosAngeles1 0.0% 10 36.9 44.1 36.6 55.3 6.2 10.|-- b.resolvers.Level3.net 0.0% 10 46.8 46.9 38.0 55.2 5.4 regards, /virendra -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://www.enigmail.net/ iF4EAREIAAYFAlBQ9sQACgkQ3HuimOHfh+El0AD/X4VKmhM++WaF6S6VPbl2qYeb aNT7U4X9mByp4cPzpQkA/2BdyX2h5cG65QotLuZmBLG6giw6RR+UeK0wcUd/WrHI =alUx -----END PGP SIGNATURE-----

On 12 Sep 2012, at 21:55, virendra rode <virendra.rode@outages.org> wrote:
mtr -c 10 -r 4.2.2.2
uk looks good colins-imac:V2 colinj$ traceroute 4.2.2.2 traceroute to 4.2.2.2 (4.2.2.2), 64 hops max, 52 byte packets 1 api (192.168.1.2) 1.972 ms 1.695 ms 1.538 ms 2 par8.the1.hso.uk.com (217.196.224.63) 16.594 ms 16.603 ms 16.194 ms 3 89.248.31.242 (89.248.31.242) 26.993 ms 21.796 ms 24.260 ms 4 eth-1-2-pcr1.the1.hso.uk.com (89.248.31.62) 16.065 ms 15.380 ms 16.179 ms 5 ge-0-2-0.pgr1.the1.hso.uk.com (89.248.31.94) 16.530 ms 15.977 ms 16.264 ms 6 xe-11-2-0-5.lon10.ip4.tinet.net (213.200.77.81) 16.002 ms 16.242 ms 16.098 ms 7 xe-1-0-0.lon11.ip4.tinet.net (89.149.185.170) 16.932 ms xe-1-1-2.lon11.ip4.tinet.net (89.149.183.58) 15.668 ms 16.708 ms 8 4.68.110.113 (4.68.110.113) 16.510 ms 15.957 ms 16.357 ms 9 b.resolvers.level3.net (4.2.2.2) 16.430 ms 15.950 ms 18.010 ms

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On 09/12/2012 01:08 PM, Josh Luthman wrote:
Is anyone seeing any issues to 4.2.2.2?
At 3:35 EST I starting seeing packet loss (icmp ping) to it - http://imgur.com/ITGL1
I noticed at the same time we had a route change with RR (Time Warner Cable) in Chicago.
Josh Luthman Office: 937-552-2340 Direct: 937-552-2343 1100 Wayne St Suite 1337 Troy, OH 45373
- ----------------------- FYI - Dug little deeper, maybe coincidence but right about the same time saw a brief spike (~200ms) between (chicago) level3 and ntt. It appears back to normal now. I have graphs that shows spike in DNS response time off PUCK :) regards, /virendra -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://www.enigmail.net/ iF4EAREIAAYFAlBRAaIACgkQ3HuimOHfh+GcaAD/U06wqSSL30qwSiTI9BQh/24R jRlSkjGTvsvnzyBZ0xsA/0mRR9fFe7ZsVbsGbNnWQHONNUEQJ94Wj3jJfH676glL =wjMt -----END PGP SIGNATURE-----

My issue still exists. Josh Luthman Office: 937-552-2340 Direct: 937-552-2343 1100 Wayne St Suite 1337 Troy, OH 45373 On Wed, Sep 12, 2012 at 5:41 PM, virendra rode <virendra.rode@outages.org>wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256
On 09/12/2012 01:08 PM, Josh Luthman wrote:
Is anyone seeing any issues to 4.2.2.2?
At 3:35 EST I starting seeing packet loss (icmp ping) to it - http://imgur.com/ITGL1
I noticed at the same time we had a route change with RR (Time Warner Cable) in Chicago.
Josh Luthman Office: 937-552-2340 Direct: 937-552-2343 1100 Wayne St Suite 1337 Troy, OH 45373
- ----------------------- FYI - Dug little deeper, maybe coincidence but right about the same time saw a brief spike (~200ms) between (chicago) level3 and ntt. It appears back to normal now.
I have graphs that shows spike in DNS response time off PUCK :)
regards, /virendra -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://www.enigmail.net/
iF4EAREIAAYFAlBRAaIACgkQ3HuimOHfh+GcaAD/U06wqSSL30qwSiTI9BQh/24R jRlSkjGTvsvnzyBZ0xsA/0mRR9fFe7ZsVbsGbNnWQHONNUEQJ94Wj3jJfH676glL =wjMt -----END PGP SIGNATURE----- _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On 09/12/2012 02:46 PM, Josh Luthman wrote:
My issue still exists.
Josh Luthman Office: 937-552-2340 Direct: 937-552-2343 1100 Wayne St Suite 1337 Troy, OH 45373
yup. Show Level 3 (Chicago, IL) Ping to 4.2.2.2 icmp_seq=0 time=28 ms icmp_seq=4 time=24 ms icmp_seq=5 time=24 ms icmp_seq=6 time=24 ms icmp_seq=7 time=24 ms icmp_seq=8 time=24 ms - ---- 4.2.2.2 statistics ---- 10 packets transmitted, 6 packets received, 40% packet loss rtt min/avg/median/max/mdev/stddev = 24/24.667/24/28/1.054/1.491 ms Show Level 3 (Chicago, IL) Traceroute to 4.2.2.2 1 b.resolvers.Level3.net (4.2.2.2) [AS10753 {Level3(customer_multihomed)}] 4 msec 44 msec * 44 msec kinda high off their own network. However, reachability to others look ok to me. Show Level 3 (Chicago, IL) Ping to puck.nether.net (204.42.254.5) icmp_seq=0 time=1 ms icmp_seq=1 time=1 ms icmp_seq=2 time=1 ms icmp_seq=3 time=1 ms icmp_seq=4 time=1 ms icmp_seq=5 time=1 ms icmp_seq=6 time=1 ms icmp_seq=7 time=1 ms icmp_seq=8 time=1 ms icmp_seq=9 time=1 ms - ---- puck.nether.net statistics ---- 10 packets transmitted, 10 packets received, 0% packet loss rtt min/avg/median/max/mdev/stddev = 1/1/1/1/0/0 ms Show Level 3 (Chicago, IL) Ping to tracker.outages.org (98.143.36.109) icmp_seq=0 time=44 ms icmp_seq=1 time=88 ms icmp_seq=2 time=64 ms icmp_seq=3 time=44 ms icmp_seq=4 time=44 ms icmp_seq=5 time=44 ms icmp_seq=6 time=44 ms icmp_seq=7 time=44 ms icmp_seq=8 time=44 ms icmp_seq=9 time=44 ms - ---- tracker.outages.org statistics ---- 10 packets transmitted, 10 packets received, 0% packet loss rtt min/avg/median/max/mdev/stddev = 44/50.4/44/88/3.2/13.879 ms Show Level 3 (Chicago, IL) Traceroute to puck.nether.net (204.42.254.5) 1 ae-31-51.ebr1.Chicago1.Level3.net (4.69.138.30) 12 msec ae-32-52.ebr2.Chicago1.Level3.net (4.69.138.62) 0 msec ae-31-51.ebr1.Chicago1.Level3.net (4.69.138.30) 0 msec 2 ae-5-5.ebr2.Chicago2.Level3.net (4.69.140.194) 12 msec ae-6-6.ebr1.Chicago2.Level3.net (4.69.140.190) 8 msec ae-5-5.ebr2.Chicago2.Level3.net (4.69.140.194) 4 msec 3 ae-1-51.edge4.Chicago3.Level3.net (4.69.138.134) 0 msec ae-2-52.edge4.Chicago3.Level3.net (4.69.138.166) 0 msec ae-1-51.edge4.Chicago3.Level3.net (4.69.138.134) 0 msec 4 xe-0-0-0-9.r06.chcgil09.us.bb.gin.ntt.net (129.250.8.77) [AS2914 {VERIO}] 0 msec verio-ntt-level3-10ge.Chicago3.Level3.net (4.68.63.198) 0 msec 0 msec 5 ae-8.r05.chcgil09.us.bb.gin.ntt.net (129.250.4.221) [AS2914 {VERIO}] 0 msec 0 msec 0 msec 6 puck.nether.net (204.42.254.5) [AS2914 {VERIO}] 0 msec 4 msec 0 msec regards, /virendra
On Wed, Sep 12, 2012 at 5:41 PM, virendra rode <virendra.rode@outages.org>wrote:
On 09/12/2012 01:08 PM, Josh Luthman wrote:
Is anyone seeing any issues to 4.2.2.2?
At 3:35 EST I starting seeing packet loss (icmp ping) to it - http://imgur.com/ITGL1
I noticed at the same time we had a route change with RR (Time Warner Cable) in Chicago.
Josh Luthman Office: 937-552-2340 Direct: 937-552-2343 1100 Wayne St Suite 1337 Troy, OH 45373
----------------------- FYI - Dug little deeper, maybe coincidence but right about the same time saw a brief spike (~200ms) between (chicago) level3 and ntt. It appears back to normal now.
I have graphs that shows spike in DNS response time off PUCK :)
regards, /virendra
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://www.enigmail.net/ iF4EAREIAAYFAlBRBmQACgkQ3HuimOHfh+F8mgD/RdryOmmmC2XndWKkEI+D1ArJ 2c8IDIpo1lQAiB/sY+ABAITMCTySLaEltNf2p1mFXRVHLyGetjcJ5nvLHDm7ycXs =Ia70 -----END PGP SIGNATURE-----

Two routes (both originating from Grand Rapids, MI - 3 hrs to Chicago): US Signal (okay): 1 38 ms 95 ms 15 ms gi2-3.pe02.grr.ussignalcom.net [70.34.130.46] 2 13 ms 13 ms 13 ms te1-0-0.pe01.dtw.ussignalcom.net [74.204.127.254] 3 14 ms 12 ms 13 ms te0-0-0-0.agg01.dtw.ussigalcom.net [70.34.130.194] 4 105 ms 222 ms 13 ms te-4-4.car2.detroit1.level3.net [4.53.76.81] 5 18 ms 25 ms 18 ms ae-11-11.car1.detroit1.level3.net [4.69.133.245] 6 22 ms 26 ms 22 ms ae-8-8.ebr2.chicago1.level3.net [4.69.133.242] 7 64 ms 29 ms 18 ms ae-21-52.car1.chicago1.level3.net [4.69.138.35] 8 23 ms 28 ms 25 ms b.resolvers.Level3.net [4.2.2.2] AT&T MIS Fiber (not okay): 1 10 ms 6 ms 7 ms cr82.gdrmi.ip.att.net [12.122.152.18] 2 7 ms 7 ms 6 ms cr1.cgcil.ip.att.net [12.122.152.5] 3 6 ms 6 ms 5 ms gar8.cgcil.ip.att.net [12.122.133.161] 4 5 ms 5 ms 5 ms 192.205.37.150 5 15 ms 5 ms 5 ms 4.69.158.158 6 6 ms 5 ms 17 ms ae-5-5.ebr2.chicago1.level3.net [4.69.140.193] 7 6 ms 6 ms * ae-21-52.car1.chicago1.level3.net [4.69.138.35] 8 * 23 ms 35 ms b.resolvers.Level3.net [4.2.2.2] -- B.J. Ash bjash@kentrecords.com<mailto:bjash@kentrecords.com> IT Supervisor | Kent Record Management Inc. 1950 Waldorf NW | Grand Rapids, MI 49544 W:616-459-6681 | www.kentrecords.com<http://www.kentrecords.com/> C:616-240-6819 -----Original Message----- From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On Behalf Of virendra rode Sent: Wednesday, September 12, 2012 18:02 To: Josh Luthman Cc: outages@outages.org Subject: Re: [outages] Packet loss to 4.2.2.2 -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On 09/12/2012 02:46 PM, Josh Luthman wrote:
My issue still exists.
Josh Luthman Office: 937-552-2340 Direct: 937-552-2343 1100 Wayne St
Suite 1337 Troy, OH 45373
- -------------------------- yup. Show Level 3 (Chicago, IL) Ping to 4.2.2.2 icmp_seq=0 time=28 ms icmp_seq=4 time=24 ms icmp_seq=5 time=24 ms icmp_seq=6 time=24 ms icmp_seq=7 time=24 ms icmp_seq=8 time=24 ms - ---- 4.2.2.2 statistics ---- 10 packets transmitted, 6 packets received, 40% packet loss rtt min/avg/median/max/mdev/stddev = 24/24.667/24/28/1.054/1.491 ms Show Level 3 (Chicago, IL) Traceroute to 4.2.2.2 1 b.resolvers.Level3.net (4.2.2.2) [AS10753 {Level3(customer_multihomed)}] 4 msec 44 msec * 44 msec kinda high off their own network. However, reachability to others look ok to me. Show Level 3 (Chicago, IL) Ping to puck.nether.net (204.42.254.5) icmp_seq=0 time=1 ms icmp_seq=1 time=1 ms icmp_seq=2 time=1 ms icmp_seq=3 time=1 ms icmp_seq=4 time=1 ms icmp_seq=5 time=1 ms icmp_seq=6 time=1 ms icmp_seq=7 time=1 ms icmp_seq=8 time=1 ms icmp_seq=9 time=1 ms - ---- puck.nether.net statistics ---- 10 packets transmitted, 10 packets received, 0% packet loss rtt min/avg/median/max/mdev/stddev = 1/1/1/1/0/0 ms Show Level 3 (Chicago, IL) Ping to tracker.outages.org (98.143.36.109) icmp_seq=0 time=44 ms icmp_seq=1 time=88 ms icmp_seq=2 time=64 ms icmp_seq=3 time=44 ms icmp_seq=4 time=44 ms icmp_seq=5 time=44 ms icmp_seq=6 time=44 ms icmp_seq=7 time=44 ms icmp_seq=8 time=44 ms icmp_seq=9 time=44 ms - ---- tracker.outages.org statistics ---- 10 packets transmitted, 10 packets received, 0% packet loss rtt min/avg/median/max/mdev/stddev = 44/50.4/44/88/3.2/13.879 ms Show Level 3 (Chicago, IL) Traceroute to puck.nether.net (204.42.254.5) 1 ae-31-51.ebr1.Chicago1.Level3.net (4.69.138.30) 12 msec ae-32-52.ebr2.Chicago1.Level3.net (4.69.138.62) 0 msec ae-31-51.ebr1.Chicago1.Level3.net (4.69.138.30) 0 msec 2 ae-5-5.ebr2.Chicago2.Level3.net (4.69.140.194) 12 msec ae-6-6.ebr1.Chicago2.Level3.net (4.69.140.190) 8 msec ae-5-5.ebr2.Chicago2.Level3.net (4.69.140.194) 4 msec 3 ae-1-51.edge4.Chicago3.Level3.net (4.69.138.134) 0 msec ae-2-52.edge4.Chicago3.Level3.net (4.69.138.166) 0 msec ae-1-51.edge4.Chicago3.Level3.net (4.69.138.134) 0 msec 4 xe-0-0-0-9.r06.chcgil09.us.bb.gin.ntt.net (129.250.8.77) [AS2914 {VERIO}] 0 msec verio-ntt-level3-10ge.Chicago3.Level3.net (4.68.63.198) 0 msec 0 msec 5 ae-8.r05.chcgil09.us.bb.gin.ntt.net (129.250.4.221) [AS2914 {VERIO}] 0 msec 0 msec 0 msec 6 puck.nether.net (204.42.254.5) [AS2914 {VERIO}] 0 msec 4 msec 0 msec regards, /virendra
On Wed, Sep 12, 2012 at 5:41 PM, virendra rode
<virendra.rode@outages.org<mailto:virendra.rode@outages.org>>wrote:
On 09/12/2012 01:08 PM, Josh Luthman wrote:
Is anyone seeing any issues to 4.2.2.2?
At 3:35 EST I starting seeing packet loss (icmp ping) to it
I noticed at the same time we had a route change with RR (Time
Warner Cable) in Chicago.
Josh Luthman Office: 937-552-2340 Direct: 937-552-2343 1100 Wayne
St Suite 1337 Troy, OH 45373
----------------------- FYI - Dug little deeper, maybe coincidence but
right about the same time saw a brief spike (~200ms) between
(chicago) level3 and ntt. It appears back to normal now.
I have graphs that shows spike in DNS response time off PUCK :)
regards, /virendra
_______________________________________________ Outages mailing list
Outages@outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages
-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://www.enigmail.net/ iF4EAREIAAYFAlBRBmQACgkQ3HuimOHfh+F8mgD/RdryOmmmC2XndWKkEI+D1ArJ 2c8IDIpo1lQAiB/sY+ABAITMCTySLaEltNf2p1mFXRVHLyGetjcJ5nvLHDm7ycXs =Ia70 -----END PGP SIGNATURE----- _______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages

ATT in St. Louis, MO is still having issues to 4.2.2.2. C3560-BGP-STL-MO-001#ping 4.2.2.2 repeat 100 Type escape sequence to abort. Sending 100, 100-byte ICMP Echos to 4.2.2.2, timeout is 2 seconds: !!!!!!!!!!!!!!!!!!!!!!!!!!!!....!..!!!!..!!!.!!..!!!..!.!....!!!.!.!!. !..!!.!.!.!!!.!!.!..!.!.!!!!.. Success rate is 66 percent (66/100), round-trip min/avg/max = 8/9/17 ms C3560-BGP-STL-MO-001#traceroute 4.2.2.2 Type escape sequence to abort. Tracing the route to 4.2.2.2 1 12.249.110.201 0 msec 33 msec 34 msec 2 12.122.152.202 [AS 7018] 8 msec 17 msec 8 msec 3 12.122.2.21 [AS 7018] 17 msec 9 msec 17 msec 4 12.122.132.197 [AS 7018] 8 msec 9 msec 8 msec 5 192.205.37.150 [AS 7018] 8 msec 8 msec 9 msec 6 4.69.158.158 [AS 7018] 8 msec 4.69.158.146 [AS 7018] 9 msec 8 msec 7 4.69.140.193 [AS 7018] 17 msec 16 msec 9 msec 8 4.69.138.3 [AS 7018] 8 msec 210 msec 16 msec 9 4.2.2.2 [AS 7018] 9 msec 9 msec 8 msec From: Josh Luthman <josh@imaginenetworksllc.com<mailto:josh@imaginenetworksllc.com>> Date: Wednesday, September 12, 2012 4:08 PM To: Outages <outages@outages.org<mailto:outages@outages.org>> Subject: [outages] Packet loss to 4.2.2.2 Is anyone seeing any issues to 4.2.2.2? At 3:35 EST I starting seeing packet loss (icmp ping) to it - http://imgur.com/ITGL1 I noticed at the same time we had a route change with RR (Time Warner Cable) in Chicago. Josh Luthman Office: 937-552-2340 Direct: 937-552-2343 1100 Wayne St Suite 1337 Troy, OH 45373
participants (27)
-
Andres Herrera
-
B.J. Ash
-
Bill Wichers
-
Chris Stone
-
Christopher Singhaus
-
Colin Johnston
-
Dan White
-
Dave Owens
-
Dikkema, Michael (Business Technology)
-
Grant Ridder
-
Jay Ashworth
-
Jonathan Lassoff
-
Joseph Jackson
-
Josh Breeds
-
Josh Luthman
-
Matthew Anderson
-
Michael Eberline
-
Mike Phipps
-
Mitch
-
Naveen Nathan
-
Raman Sud
-
Sarlas, George
-
Scott Berkman
-
Scott Howard
-
Shriram Ayyar
-
Tim Glen
-
virendra rode