Packet loss to 8.8.8.8 and other google IPs (Dallas)
We are seeing packet loss to google hosts, hosted in Dallas. Any anyone seeing this? MTR output: Host Loss% Snt Last Avg Best Wrst StDev 1. 10.0.2.1 0.0% 51 2.4 1.9 0.5 5.2 0.7 2. olp-67-217-152-5.olp.net 0.0% 51 0.4 3.7 0.3 27.8 4.9 3. olp-67-217-158-29.olp.net 0.0% 50 0.3 4.5 0.3 22.0 5.3 4. olp-67-217-151-125.olp.net 0.0% 50 0.6 4.9 0.6 33.3 6.8 5. olp-67-217-158-30.olp.net 0.0% 50 0.6 6.4 0.5 64.0 10.5 6. olp-67-217-158-21.olp.net 0.0% 50 11.4 4.4 1.1 13.6 4.1 7. xe-8-3-0.bar2.tulsa2.level3.net 0.0% 50 1.2 2.7 1.2 27.5 5.5 8. ae-0-11.bar1.tulsa2.level3.net 0.0% 50 2.0 4.1 1.9 44.5 7.8 9. ae-3-3.ebr4.dallas1.level3.net 91.8% 50 20.5 20.5 20.5 20.5 0.0 10. ae-8-0.ear3.dallas1.level3.net 93.9% 50 20.6 20.6 20.6 20.7 0.0 11. google-level3-100g.dallas1.leve28.0% 50 20.5 20.7 20.4 29.6 1.5 12. 108.170.240.193 22.0% 50 20.8 20.7 20.7 20.9 0.0 13. 64.233.175.9 26.0% 50 20.7 20.7 20.7 20.8 0.0 14. google-public-dns-a.google.com 24.0% 50 20.5 20.6 20.5 25.9 0.9 The problematic hop appears to be either 10 or 11 (level3-100g). Compare to: Host Loss% Snt Last Avg Best Wrst StDev 1. 10.0.2.1 0.0% 124 0.8 1.6 0.5 4.6 0.5 2. olp-67-217-152-5.olp.net 0.0% 124 3.8 4.3 0.3 29.9 5.6 3. olp-67-217-158-29.olp.net 0.0% 124 0.3 3.5 0.3 26.1 4.9 4. olp-67-217-151-125.olp.net 0.0% 124 6.6 3.3 0.6 20.0 3.7 5. olp-67-217-158-30.olp.net 0.0% 124 0.6 7.1 0.6 44.1 9.8 6. olp-67-217-158-21.olp.net 0.8% 124 11.9 4.8 1.1 14.9 4.3 7. xe-8-3-0.bar2.tulsa2.level3.net 0.0% 123 1.3 4.0 1.1 68.3 11.3 8. ae-0-11.bar1.tulsa2.level3.net 0.8% 123 2.1 3.9 1.9 62.3 8.1 9. ae-3-3.ebr4.dallas1.level3.net 94.3% 123 20.5 20.6 20.5 20.6 0.0 10. ae-2-3512.edge2.dallas1.level3. 0.0% 123 20.4 22.1 20.3 68.0 7.1 11. b.resolvers.level3.net 0.0% 123 20.2 20.3 20.2 22.8 0.3 -- Dan White BTC Broadband Network Admin Lead Ph 918.366.0248 (direct) main: (918)366-8000 Fax 918.366.6610 email: dwhite@olp.net http://www.btcbroadband.com
I'm seeing the same. $ sudo mtr -rwc 20 8.8.4.4 HOST: endor Loss% Snt Last Avg Best Wrst StDev 1. 69.28.84.2 0.0% 20 1.1 22.2 0.4 181.5 47.2 2. 38.104.37.141 0.0% 20 1.3 1.3 1.0 2.5 0.3 3. te0-3-1-1.rcr21.dfw02.atlas.cogentco.com 0.0% 20 0.8 0.9 0.6 2.0 0.4 4. be2277.ccr22.dfw01.atlas.cogentco.com 0.0% 20 0.9 1.0 0.9 1.2 0.1 5. be2764.ccr41.dfw03.atlas.cogentco.com 0.0% 20 0.9 1.1 0.9 1.5 0.2 6. tata.dfw03.atlas.cogentco.com 0.0% 20 0.9 1.0 0.7 2.0 0.3 7. 72.14.219.124 25.0% 20 0.8 1.0 0.8 1.7 0.2 8. 108.170.240.193 20.0% 20 1.0 1.1 0.9 1.2 0.1 9. 64.233.175.9 30.0% 20 1.0 1.1 1.0 1.2 0.1 10. google-public-dns-b.google.com 45.0% 20 1.0 1.0 0.9 1.0 0.0 On 2016-08-29 14:14, Dan White via Outages wrote:
We are seeing packet loss to google hosts, hosted in Dallas. Any anyone seeing this?
MTR output:
Host Loss% Snt Last Avg Best Wrst StDev 1. 10.0.2.1 0.0% 51 2.4 1.9 0.5 5.2 0.7 2. olp-67-217-152-5.olp.net 0.0% 51 0.4 3.7 0.3 27.8 4.9 3. olp-67-217-158-29.olp.net 0.0% 50 0.3 4.5 0.3 22.0 5.3 4. olp-67-217-151-125.olp.net 0.0% 50 0.6 4.9 0.6 33.3 6.8 5. olp-67-217-158-30.olp.net 0.0% 50 0.6 6.4 0.5 64.0 10.5 6. olp-67-217-158-21.olp.net 0.0% 50 11.4 4.4 1.1 13.6 4.1 7. xe-8-3-0.bar2.tulsa2.level3.net 0.0% 50 1.2 2.7 1.2 27.5 5.5 8. ae-0-11.bar1.tulsa2.level3.net 0.0% 50 2.0 4.1 1.9 44.5 7.8 9. ae-3-3.ebr4.dallas1.level3.net 91.8% 50 20.5 20.5 20.5 20.5 0.0 10. ae-8-0.ear3.dallas1.level3.net 93.9% 50 20.6 20.6 20.6 20.7 0.0 11. google-level3-100g.dallas1.leve28.0% 50 20.5 20.7 20.4 29.6 1.5 12. 108.170.240.193 22.0% 50 20.8 20.7 20.7 20.9 0.0 13. 64.233.175.9 26.0% 50 20.7 20.7 20.7 20.8 0.0 14. google-public-dns-a.google.com 24.0% 50 20.5 20.6 20.5 25.9 0.9
The problematic hop appears to be either 10 or 11 (level3-100g). Compare to:
Host Loss% Snt Last Avg Best Wrst StDev 1. 10.0.2.1 0.0% 124 0.8 1.6 0.5 4.6 0.5 2. olp-67-217-152-5.olp.net 0.0% 124 3.8 4.3 0.3 29.9 5.6 3. olp-67-217-158-29.olp.net 0.0% 124 0.3 3.5 0.3 26.1 4.9 4. olp-67-217-151-125.olp.net 0.0% 124 6.6 3.3 0.6 20.0 3.7 5. olp-67-217-158-30.olp.net 0.0% 124 0.6 7.1 0.6 44.1 9.8 6. olp-67-217-158-21.olp.net 0.8% 124 11.9 4.8 1.1 14.9 4.3 7. xe-8-3-0.bar2.tulsa2.level3.net 0.0% 123 1.3 4.0 1.1 68.3 11.3 8. ae-0-11.bar1.tulsa2.level3.net 0.8% 123 2.1 3.9 1.9 62.3 8.1 9. ae-3-3.ebr4.dallas1.level3.net 94.3% 123 20.5 20.6 20.5 20.6 0.0 10. ae-2-3512.edge2.dallas1.level3. 0.0% 123 20.4 22.1 20.3 68.0 7.1 11. b.resolvers.level3.net 0.0% 123 20.2 20.3 20.2 22.8 0.3
-- ____________ Mason Moody
I only have 1 lost out of 50 (from Troy OH through Frontier) # ADDRESS LOSS SENT LAST AVG BEST WORST 1 192.149.254.17 0% 51 0.6ms 0.5 0.2 1.5 2 192.149.254.1 0% 51 1.2ms 1 0.5 1.5 3 45.52.8.205 0% 51 4.5ms 6.5 4.2 36.7 4 74.40.5.58 0% 51 19.1ms 19.9 19 30.8 5 74.42.149.189 0% 51 19.5ms 19.8 19 26.5 6 74.40.4.138 0% 51 19ms 19.6 18.7 28.2 7 72.14.212.143 0% 51 19.6ms 19.8 18.9 32.9 8 209.85.255.26 0% 51 20.2ms 20.1 19.3 22.1 9 209.85.254.238 0% 51 20.2ms 20.2 19.5 22.6 10 216.239.59.201 0% 51 43.1ms 43.1 42.5 45.6 11 216.239.62.13 0% 51 42.6ms 43.2 42.3 59.2 12 100% 50 timeout 13 8.8.8.8 0% 50 42.5ms 42.8 42.3 45.1 Josh Luthman Office: 937-552-2340 Direct: 937-552-2343 1100 Wayne St Suite 1337 Troy, OH 45373 On Mon, Aug 29, 2016 at 2:19 PM, Mason Moody via Outages < outages@outages.org> wrote:
I'm seeing the same.
$ sudo mtr -rwc 20 8.8.4.4 HOST: endor Loss% Snt Last Avg Best Wrst StDev 1. 69.28.84.2 0.0% 20 1.1 22.2 0.4 181.5 47.2 2. 38.104.37.141 0.0% 20 1.3 1.3 1.0 2.5 0.3 3. te0-3-1-1.rcr21.dfw02.atlas.cogentco.com 0.0% 20 0.8 0.9 0.6 2.0 0.4 4. be2277.ccr22.dfw01.atlas.cogentco.com 0.0% 20 0.9 1.0 0.9 1.2 0.1 5. be2764.ccr41.dfw03.atlas.cogentco.com 0.0% 20 0.9 1.1 0.9 1.5 0.2 6. tata.dfw03.atlas.cogentco.com 0.0% 20 0.9 1.0 0.7 2.0 0.3 7. 72.14.219.124 25.0% 20 0.8 1.0 0.8 1.7 0.2 8. 108.170.240.193 20.0% 20 1.0 1.1 0.9 1.2 0.1 9. 64.233.175.9 30.0% 20 1.0 1.1 1.0 1.2 0.1 10. google-public-dns-b.google.com 45.0% 20 1.0 1.0 0.9 1.0 0.0
On 2016-08-29 14:14, Dan White via Outages wrote:
We are seeing packet loss to google hosts, hosted in Dallas. Any anyone seeing this?
MTR output:
Host Loss% Snt Last Avg Best Wrst StDev 1. 10.0.2.1 0.0% 51 2.4 1.9 0.5 5.2 0.7 2. olp-67-217-152-5.olp.net 0.0% 51 0.4 3.7 0.3 27.8 4.9 3. olp-67-217-158-29.olp.net 0.0% 50 0.3 4.5 0.3 22.0 5.3 4. olp-67-217-151-125.olp.net 0.0% 50 0.6 4.9 0.6 33.3 6.8 5. olp-67-217-158-30.olp.net 0.0% 50 0.6 6.4 0.5 64.0 10.5 6. olp-67-217-158-21.olp.net 0.0% 50 11.4 4.4 1.1 13.6 4.1 7. xe-8-3-0.bar2.tulsa2.level3.net 0.0% 50 1.2 2.7 1.2 27.5 5.5 8. ae-0-11.bar1.tulsa2.level3.net 0.0% 50 2.0 4.1 1.9 44.5 7.8 9. ae-3-3.ebr4.dallas1.level3.net 91.8% 50 20.5 20.5 20.5 20.5 0.0 10. ae-8-0.ear3.dallas1.level3.net 93.9% 50 20.6 20.6 20.6 20.7 0.0 11. google-level3-100g.dallas1.leve28.0% 50 20.5 20.7 20.4 29.6 1.5 12. 108.170.240.193 22.0% 50 20.8 20.7 20.7 20.9 0.0 13. 64.233.175.9 26.0% 50 20.7 20.7 20.7 20.8 0.0 14. google-public-dns-a.google.com 24.0% 50 20.5 20.6 20.5 25.9 0.9
The problematic hop appears to be either 10 or 11 (level3-100g). Compare to:
Host Loss% Snt Last Avg Best Wrst StDev 1. 10.0.2.1 0.0% 124 0.8 1.6 0.5 4.6 0.5 2. olp-67-217-152-5.olp.net 0.0% 124 3.8 4.3 0.3 29.9 5.6 3. olp-67-217-158-29.olp.net 0.0% 124 0.3 3.5 0.3 26.1 4.9 4. olp-67-217-151-125.olp.net 0.0% 124 6.6 3.3 0.6 20.0 3.7 5. olp-67-217-158-30.olp.net 0.0% 124 0.6 7.1 0.6 44.1 9.8 6. olp-67-217-158-21.olp.net 0.8% 124 11.9 4.8 1.1 14.9 4.3 7. xe-8-3-0.bar2.tulsa2.level3.net 0.0% 123 1.3 4.0 1.1 68.3 11.3 8. ae-0-11.bar1.tulsa2.level3.net 0.8% 123 2.1 3.9 1.9 62.3 8.1 9. ae-3-3.ebr4.dallas1.level3.net 94.3% 123 20.5 20.6 20.5 20.6 0.0 10. ae-2-3512.edge2.dallas1.level3. 0.0% 123 20.4 22.1 20.3 68.0 7.1 11. b.resolvers.level3.net 0.0% 123 20.2 20.3 20.2 22.8 0.3
--
____________
Mason Moody
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
I am not seeing these issues from LAX via Cogent, Denver via Level3 or Denver via Hurricane Electric. I am seeing some packet loss on certain hops on Level3 and Zayo's backbones, but it is not persistent through the remaining hops, indicating some ICMP rate limiting or something. From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Mason Moody via Outages Sent: Monday, August 29, 2016 12:20 PM To: outages@outages.org Subject: Re: [outages] Packet loss to 8.8.8.8 and other google IPs (Dallas) I'm seeing the same. $ sudo mtr -rwc 20 8.8.4.4 HOST: endor Loss% Snt Last Avg Best Wrst StDev 1. 69.28.84.2 0.0% 20 1.1 22.2 0.4 181.5 47.2 2. 38.104.37.141 0.0% 20 1.3 1.3 1.0 2.5 0.3 3. te0-3-1-1.rcr21.dfw02.atlas.cogentco.com 0.0% 20 0.8 0.9 0.6 2.0 0.4 4. be2277.ccr22.dfw01.atlas.cogentco.com 0.0% 20 0.9 1.0 0.9 1.2 0.1 5. be2764.ccr41.dfw03.atlas.cogentco.com 0.0% 20 0.9 1.1 0.9 1.5 0.2 6. tata.dfw03.atlas.cogentco.com 0.0% 20 0.9 1.0 0.7 2.0 0.3 7. 72.14.219.124 25.0% 20 0.8 1.0 0.8 1.7 0.2 8. 108.170.240.193 20.0% 20 1.0 1.1 0.9 1.2 0.1 9. 64.233.175.9 30.0% 20 1.0 1.1 1.0 1.2 0.1 10. google-public-dns-b.google.com 45.0% 20 1.0 1.0 0.9 1.0 0.0 On 2016-08-29 14:14, Dan White via Outages wrote: We are seeing packet loss to google hosts, hosted in Dallas. Any anyone seeing this? MTR output: Host Loss% Snt Last Avg Best Wrst StDev 1. 10.0.2.1 0.0% 51 2.4 1.9 0.5 5.2 0.7 2. olp-67-217-152-5.olp.net 0.0% 51 0.4 3.7 0.3 27.8 4.9 3. olp-67-217-158-29.olp.net 0.0% 50 0.3 4.5 0.3 22.0 5.3 4. olp-67-217-151-125.olp.net 0.0% 50 0.6 4.9 0.6 33.3 6.8 5. olp-67-217-158-30.olp.net 0.0% 50 0.6 6.4 0.5 64.0 10.5 6. olp-67-217-158-21.olp.net 0.0% 50 11.4 4.4 1.1 13.6 4.1 7. xe-8-3-0.bar2.tulsa2.level3.net 0.0% 50 1.2 2.7 1.2 27.5 5.5 8. ae-0-11.bar1.tulsa2.level3.net 0.0% 50 2.0 4.1 1.9 44.5 7.8 9. ae-3-3.ebr4.dallas1.level3.net 91.8% 50 20.5 20.5 20.5 20.5 0.0 10. ae-8-0.ear3.dallas1.level3.net 93.9% 50 20.6 20.6 20.6 20.7 0.0 11. google-level3-100g.dallas1.leve28.0% 50 20.5 20.7 20.4 29.6 1.5 12. 108.170.240.193 22.0% 50 20.8 20.7 20.7 20.9 0.0 13. 64.233.175.9 26.0% 50 20.7 20.7 20.7 20.8 0.0 14. google-public-dns-a.google.com 24.0% 50 20.5 20.6 20.5 25.9 0.9 The problematic hop appears to be either 10 or 11 (level3-100g). Compare to: Host Loss% Snt Last Avg Best Wrst StDev 1. 10.0.2.1 0.0% 124 0.8 1.6 0.5 4.6 0.5 2. olp-67-217-152-5.olp.net 0.0% 124 3.8 4.3 0.3 29.9 5.6 3. olp-67-217-158-29.olp.net 0.0% 124 0.3 3.5 0.3 26.1 4.9 4. olp-67-217-151-125.olp.net 0.0% 124 6.6 3.3 0.6 20.0 3.7 5. olp-67-217-158-30.olp.net 0.0% 124 0.6 7.1 0.6 44.1 9.8 6. olp-67-217-158-21.olp.net 0.8% 124 11.9 4.8 1.1 14.9 4.3 7. xe-8-3-0.bar2.tulsa2.level3.net 0.0% 123 1.3 4.0 1.1 68.3 11.3 8. ae-0-11.bar1.tulsa2.level3.net 0.8% 123 2.1 3.9 1.9 62.3 8.1 9. ae-3-3.ebr4.dallas1.level3.net 94.3% 123 20.5 20.6 20.5 20.6 0.0 10. ae-2-3512.edge2.dallas1.level3. 0.0% 123 20.4 22.1 20.3 68.0 7.1 11. b.resolvers.level3.net 0.0% 123 20.2 20.3 20.2 22.8 0.3 -- ____________ Mason Moody Disclaimer The information contained in this communication from the sender is confidential. It is intended solely for use by the recipient and others authorized to receive it. If you are not the recipient, you are hereby notified that any disclosure, copying, distribution or taking action in relation of the contents of this information is strictly prohibited and may be unlawful. This email has been scanned for viruses and malware, and may have been automatically archived by Mimecast Ltd, an innovator in Software as a Service (SaaS) for business. Providing a safer and more useful place for your human generated data. Specializing in; Security, archiving and compliance. To find out more visit the Mimecast website.
Not seeing this from one of my boxes in Lenoir, North Carolina kmyers@office:~$ sudo mtr -rwc 20 8.8.4.4 HOST: office Loss% Snt Last Avg Best Wrst StDev 1.|-- 162.248.242.189 0.0% 20 0.3 0.5 0.3 2.5 0.4 2.|-- 199.101.186.9 0.0% 20 0.3 0.3 0.3 0.6 0.0 3.|-- 199.255.159.18 0.0% 20 0.4 2.9 0.2 10.6 3.5 4.|-- dct-ds42-ve33.dacentec.com 0.0% 20 11.0 2.0 0.3 11.0 2.9 5.|-- dct-cr03--v51.dacentec.com 0.0% 20 8.9 8.9 8.9 9.0 0.0 6.|-- aix.pr1.atl.google.com 0.0% 20 89.4 73.0 50.6 111.8 15.9 7.|-- 72.14.233.54 0.0% 20 9.3 9.2 9.1 9.3 0.0 8.|-- 209.85.142.140 0.0% 20 9.7 9.7 9.6 10.1 0.0 9.|-- 209.85.143.201 0.0% 20 9.7 9.7 9.7 10.3 0.0 10.|-- ??? 100.0 20 0.0 0.0 0.0 0.0 0.0 On Mon, Aug 29, 2016 at 2:19 PM, Mason Moody via Outages < outages@outages.org> wrote:
I'm seeing the same.
$ sudo mtr -rwc 20 8.8.4.4 HOST: endor Loss% Snt Last Avg Best Wrst StDev 1. 69.28.84.2 0.0% 20 1.1 22.2 0.4 181.5 47.2 2. 38.104.37.141 0.0% 20 1.3 1.3 1.0 2.5 0.3 3. te0-3-1-1.rcr21.dfw02.atlas.cogentco.com 0.0% 20 0.8 0.9 0.6 2.0 0.4 4. be2277.ccr22.dfw01.atlas.cogentco.com 0.0% 20 0.9 1.0 0.9 1.2 0.1 5. be2764.ccr41.dfw03.atlas.cogentco.com 0.0% 20 0.9 1.1 0.9 1.5 0.2 6. tata.dfw03.atlas.cogentco.com 0.0% 20 0.9 1.0 0.7 2.0 0.3 7. 72.14.219.124 25.0% 20 0.8 1.0 0.8 1.7 0.2 8. 108.170.240.193 20.0% 20 1.0 1.1 0.9 1.2 0.1 9. 64.233.175.9 30.0% 20 1.0 1.1 1.0 1.2 0.1 10. google-public-dns-b.google.com 45.0% 20 1.0 1.0 0.9 1.0 0.0
On 2016-08-29 14:14, Dan White via Outages wrote:
We are seeing packet loss to google hosts, hosted in Dallas. Any anyone seeing this?
MTR output:
Host Loss% Snt Last Avg Best Wrst StDev 1. 10.0.2.1 0.0% 51 2.4 1.9 0.5 5.2 0.7 2. olp-67-217-152-5.olp.net 0.0% 51 0.4 3.7 0.3 27.8 4.9 3. olp-67-217-158-29.olp.net 0.0% 50 0.3 4.5 0.3 22.0 5.3 4. olp-67-217-151-125.olp.net 0.0% 50 0.6 4.9 0.6 33.3 6.8 5. olp-67-217-158-30.olp.net 0.0% 50 0.6 6.4 0.5 64.0 10.5 6. olp-67-217-158-21.olp.net 0.0% 50 11.4 4.4 1.1 13.6 4.1 7. xe-8-3-0.bar2.tulsa2.level3.net 0.0% 50 1.2 2.7 1.2 27.5 5.5 8. ae-0-11.bar1.tulsa2.level3.net 0.0% 50 2.0 4.1 1.9 44.5 7.8 9. ae-3-3.ebr4.dallas1.level3.net 91.8% 50 20.5 20.5 20.5 20.5 0.0 10. ae-8-0.ear3.dallas1.level3.net 93.9% 50 20.6 20.6 20.6 20.7 0.0 11. google-level3-100g.dallas1.leve28.0% 50 20.5 20.7 20.4 29.6 1.5 12. 108.170.240.193 22.0% 50 20.8 20.7 20.7 20.9 0.0 13. 64.233.175.9 26.0% 50 20.7 20.7 20.7 20.8 0.0 14. google-public-dns-a.google.com 24.0% 50 20.5 20.6 20.5 25.9 0.9
The problematic hop appears to be either 10 or 11 (level3-100g). Compare to:
Host Loss% Snt Last Avg Best Wrst StDev 1. 10.0.2.1 0.0% 124 0.8 1.6 0.5 4.6 0.5 2. olp-67-217-152-5.olp.net 0.0% 124 3.8 4.3 0.3 29.9 5.6 3. olp-67-217-158-29.olp.net 0.0% 124 0.3 3.5 0.3 26.1 4.9 4. olp-67-217-151-125.olp.net 0.0% 124 6.6 3.3 0.6 20.0 3.7 5. olp-67-217-158-30.olp.net 0.0% 124 0.6 7.1 0.6 44.1 9.8 6. olp-67-217-158-21.olp.net 0.8% 124 11.9 4.8 1.1 14.9 4.3 7. xe-8-3-0.bar2.tulsa2.level3.net 0.0% 123 1.3 4.0 1.1 68.3 11.3 8. ae-0-11.bar1.tulsa2.level3.net 0.8% 123 2.1 3.9 1.9 62.3 8.1 9. ae-3-3.ebr4.dallas1.level3.net 94.3% 123 20.5 20.6 20.5 20.6 0.0 10. ae-2-3512.edge2.dallas1.level3. 0.0% 123 20.4 22.1 20.3 68.0 7.1 11. b.resolvers.level3.net 0.0% 123 20.2 20.3 20.2 22.8 0.3
--
____________
Mason Moody
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
-- [image: --] Keith I Myers [image: http://]http://KMyers.me +Keith I Myers <http://plus.kmyers.me/> Mobile : (305)-929-3475
Not me. Loss% Snt Last Avg Best Wrst StDev 1. ext-215-74.215.64.208.in-addr.arpa 0.0% 20 0.5 0.5 0.4 1.5 0.2 2. 72.20.177.33 0.0% 20 0.5 2.0 0.5 22.3 4.9 3. 74.125.49.233 0.0% 20 0.7 0.9 0.5 2.0 0.5 4. 108.170.240.1 0.0% 20 0.8 0.8 0.7 1.5 0.2 5. 72.14.234.105 0.0% 20 1.0 1.1 0.9 1.8 0.2 6. google-public-dns-b.google.com<http://google-public-dns-b.google.com> 0.0% 20 0.6 1.1 0.6 2.2 0.5 On Aug 29, 2016, at 1:19 PM, Mason Moody via Outages <outages@outages.org<mailto:outages@outages.org>> wrote: I'm seeing the same. $ sudo mtr -rwc 20 8.8.4.4 HOST: endor Loss% Snt Last Avg Best Wrst StDev 1. 69.28.84.2 0.0% 20 1.1 22.2 0.4 181.5 47.2 2. 38.104.37.141 0.0% 20 1.3 1.3 1.0 2.5 0.3 3. te0-3-1-1.rcr21.dfw02.atlas.cogentco.com<http://rcr21.dfw02.atlas.cogentco.com> 0.0% 20 0.8 0.9 0.6 2.0 0.4 4. be2277.ccr22.dfw01.atlas.cogentco.com<http://ccr22.dfw01.atlas.cogentco.com> 0.0% 20 0.9 1.0 0.9 1.2 0.1 5. be2764.ccr41.dfw03.atlas.cogentco.com<http://ccr41.dfw03.atlas.cogentco.com> 0.0% 20 0.9 1.1 0.9 1.5 0.2 6. tata.dfw03.atlas.cogentco.com<http://tata.dfw03.atlas.cogentco.com> 0.0% 20 0.9 1.0 0.7 2.0 0.3 7. 72.14.219.124 25.0% 20 0.8 1.0 0.8 1.7 0.2 8. 108.170.240.193 20.0% 20 1.0 1.1 0.9 1.2 0.1 9. 64.233.175.9 30.0% 20 1.0 1.1 1.0 1.2 0.1 10. google-public-dns-b.google.com<http://google-public-dns-b.google.com> 45.0% 20 1.0 1.0 0.9 1.0 0.0 On 2016-08-29 14:14, Dan White via Outages wrote: We are seeing packet loss to google hosts, hosted in Dallas. Any anyone seeing this? MTR output: Host Loss% Snt Last Avg Best Wrst StDev 1. 10.0.2.1 0.0% 51 2.4 1.9 0.5 5.2 0.7 2. olp-67-217-152-5.olp.net<http://olp-67-217-152-5.olp.net> 0.0% 51 0.4 3.7 0.3 27.8 4.9 3. olp-67-217-158-29.olp.net<http://olp-67-217-158-29.olp.net> 0.0% 50 0.3 4.5 0.3 22.0 5.3 4. olp-67-217-151-125.olp.net<http://olp-67-217-151-125.olp.net> 0.0% 50 0.6 4.9 0.6 33.3 6.8 5. olp-67-217-158-30.olp.net<http://olp-67-217-158-30.olp.net> 0.0% 50 0.6 6.4 0.5 64.0 10.5 6. olp-67-217-158-21.olp.net<http://olp-67-217-158-21.olp.net> 0.0% 50 11.4 4.4 1.1 13.6 4.1 7. xe-8-3-0.bar2.tulsa2.level3.net<http://xe-8-3-0.bar2.tulsa2.level3.net> 0.0% 50 1.2 2.7 1.2 27.5 5.5 8. ae-0-11.bar1.tulsa2.level3.net<http://ae-0-11.bar1.tulsa2.level3.net> 0.0% 50 2.0 4.1 1.9 44.5 7.8 9. ae-3-3.ebr4.dallas1.level3.net<http://ae-3-3.ebr4.dallas1.level3.net> 91.8% 50 20.5 20.5 20.5 20.5 0.0 10. ae-8-0.ear3.dallas1.level3.net<http://ae-8-0.ear3.dallas1.level3.net> 93.9% 50 20.6 20.6 20.6 20.7 0.0 11. google-level3-100g.dallas1.leve28.0% 50 20.5 20.7 20.4 29.6 1.5 12. 108.170.240.193 22.0% 50 20.8 20.7 20.7 20.9 0.0 13. 64.233.175.9 26.0% 50 20.7 20.7 20.7 20.8 0.0 14. google-public-dns-a.google.com<http://google-public-dns-a.google.com> 24.0% 50 20.5 20.6 20.5 25.9 0.9 The problematic hop appears to be either 10 or 11 (level3-100g). Compare to: Host Loss% Snt Last Avg Best Wrst StDev 1. 10.0.2.1 0.0% 124 0.8 1.6 0.5 4.6 0.5 2. olp-67-217-152-5.olp.net<http://olp-67-217-152-5.olp.net> 0.0% 124 3.8 4.3 0.3 29.9 5.6 3. olp-67-217-158-29.olp.net<http://olp-67-217-158-29.olp.net> 0.0% 124 0.3 3.5 0.3 26.1 4.9 4. olp-67-217-151-125.olp.net<http://olp-67-217-151-125.olp.net> 0.0% 124 6.6 3.3 0.6 20.0 3.7 5. olp-67-217-158-30.olp.net<http://olp-67-217-158-30.olp.net> 0.0% 124 0.6 7.1 0.6 44.1 9.8 6. olp-67-217-158-21.olp.net<http://olp-67-217-158-21.olp.net> 0.8% 124 11.9 4.8 1.1 14.9 4.3 7. xe-8-3-0.bar2.tulsa2.level3.net<http://xe-8-3-0.bar2.tulsa2.level3.net> 0.0% 123 1.3 4.0 1.1 68.3 11.3 8. ae-0-11.bar1.tulsa2.level3.net<http://ae-0-11.bar1.tulsa2.level3.net> 0.8% 123 2.1 3.9 1.9 62.3 8.1 9. ae-3-3.ebr4.dallas1.level3.net<http://ae-3-3.ebr4.dallas1.level3.net> 94.3% 123 20.5 20.6 20.5 20.6 0.0 10. ae-2-3512.edge2.dallas1.level3. 0.0% 123 20.4 22.1 20.3 68.0 7.1 11. b.resolvers.level3.net<http://b.resolvers.level3.net> 0.0% 123 20.2 20.3 20.2 22.8 0.3 -- ____________ Mason Moody _______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages --- Keith Stokes
Not sure if it matters but some of you guys are using 8.8.4.4 and the OP is 8.8.8.8 Josh Luthman Office: 937-552-2340 Direct: 937-552-2343 1100 Wayne St Suite 1337 Troy, OH 45373 On Mon, Aug 29, 2016 at 2:27 PM, Keith Stokes via Outages < outages@outages.org> wrote:
Not me.
Loss% Snt Last Avg Best Wrst StDev 1. ext-215-74.215.64.208.in-addr.arpa 0.0% 20 0.5 0.5 0.4 1.5 0.2 2. 72.20.177.33 0.0% 20 0.5 2.0 0.5 22.3 4.9 3. 74.125.49.233 0.0% 20 0.7 0.9 0.5 2.0 0.5 4. 108.170.240.1 0.0% 20 0.8 0.8 0.7 1.5 0.2 5. 72.14.234.105 0.0% 20 1.0 1.1 0.9 1.8 0.2 6. google-public-dns-b.google.com 0.0% 20 0.6 1.1 0.6 2.2 0.5
On Aug 29, 2016, at 1:19 PM, Mason Moody via Outages <outages@outages.org> wrote:
I'm seeing the same.
$ sudo mtr -rwc 20 8.8.4.4 HOST: endor Loss% Snt Last Avg Best Wrst StDev 1. 69.28.84.2 0.0% 20 1.1 22.2 0.4 181.5 47.2 2. 38.104.37.141 0.0% 20 1.3 1.3 1.0 2.5 0.3 3. te0-3-1-1.rcr21.dfw02.atlas.cogentco.com 0.0% 20 0.8 0.9 0.6 2.0 0.4 4. be2277.ccr22.dfw01.atlas.cogentco.com 0.0% 20 0.9 1.0 0.9 1.2 0.1 5. be2764.ccr41.dfw03.atlas.cogentco.com 0.0% 20 0.9 1.1 0.9 1.5 0.2 6. tata.dfw03.atlas.cogentco.com 0.0% 20 0.9 1.0 0.7 2.0 0.3 7. 72.14.219.124 25.0% 20 0.8 1.0 0.8 1.7 0.2 8. 108.170.240.193 20.0% 20 1.0 1.1 0.9 1.2 0.1 9. 64.233.175.9 30.0% 20 1.0 1.1 1.0 1.2 0.1 10. google-public-dns-b.google.com 45.0% 20 1.0 1.0 0.9 1.0 0.0
On 2016-08-29 14:14, Dan White via Outages wrote:
We are seeing packet loss to google hosts, hosted in Dallas. Any anyone seeing this?
MTR output:
Host Loss% Snt Last Avg Best Wrst StDev 1. 10.0.2.1 0.0% 51 2.4 1.9 0.5 5.2 0.7 2. olp-67-217-152-5.olp.net 0.0% 51 0.4 3.7 0.3 27.8 4.9 3. olp-67-217-158-29.olp.net 0.0% 50 0.3 4.5 0.3 22.0 5.3 4. olp-67-217-151-125.olp.net 0.0% 50 0.6 4.9 0.6 33.3 6.8 5. olp-67-217-158-30.olp.net 0.0% 50 0.6 6.4 0.5 64.0 10.5 6. olp-67-217-158-21.olp.net 0.0% 50 11.4 4.4 1.1 13.6 4.1 7. xe-8-3-0.bar2.tulsa2.level3.net 0.0% 50 1.2 2.7 1.2 27.5 5.5 8. ae-0-11.bar1.tulsa2.level3.net 0.0% 50 2.0 4.1 1.9 44.5 7.8 9. ae-3-3.ebr4.dallas1.level3.net 91.8% 50 20.5 20.5 20.5 20.5 0.0 10. ae-8-0.ear3.dallas1.level3.net 93.9% 50 20.6 20.6 20.6 20.7 0.0 11. google-level3-100g.dallas1.leve28.0% 50 20.5 20.7 20.4 29.6 1.5 12. 108.170.240.193 22.0% 50 20.8 20.7 20.7 20.9 0.0 13. 64.233.175.9 26.0% 50 20.7 20.7 20.7 20.8 0.0 14. google-public-dns-a.google.com 24.0% 50 20.5 20.6 20.5 25.9 0.9
The problematic hop appears to be either 10 or 11 (level3-100g). Compare to:
Host Loss% Snt Last Avg Best Wrst StDev 1. 10.0.2.1 0.0% 124 0.8 1.6 0.5 4.6 0.5 2. olp-67-217-152-5.olp.net 0.0% 124 3.8 4.3 0.3 29.9 5.6 3. olp-67-217-158-29.olp.net 0.0% 124 0.3 3.5 0.3 26.1 4.9 4. olp-67-217-151-125.olp.net 0.0% 124 6.6 3.3 0.6 20.0 3.7 5. olp-67-217-158-30.olp.net 0.0% 124 0.6 7.1 0.6 44.1 9.8 6. olp-67-217-158-21.olp.net 0.8% 124 11.9 4.8 1.1 14.9 4.3 7. xe-8-3-0.bar2.tulsa2.level3.net 0.0% 123 1.3 4.0 1.1 68.3 11.3 8. ae-0-11.bar1.tulsa2.level3.net 0.8% 123 2.1 3.9 1.9 62.3 8.1 9. ae-3-3.ebr4.dallas1.level3.net 94.3% 123 20.5 20.6 20.5 20.6 0.0 10. ae-2-3512.edge2.dallas1.level3. 0.0% 123 20.4 22.1 20.3 68.0 7.1 11. b.resolvers.level3.net 0.0% 123 20.2 20.3 20.2 22.8 0.3
--
____________
Mason Moody
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
---
Keith Stokes
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Loss% Snt Last Avg Best Wrst StDev 1. ext-215-74.215.64.208.in-addr.arpa 0.0% 20 0.5 5.7 0.5 16.1 5.3 2. 72.20.177.33 0.0% 20 0.7 0.6 0.4 1.2 0.2 3. 74.125.49.233 0.0% 20 0.6 1.5 0.5 15.2 3.3 4. 108.170.240.193 0.0% 20 0.8 0.9 0.6 2.0 0.3 5. 72.14.238.47 0.0% 20 1.3 1.1 0.9 2.0 0.3 6. google-public-dns-a.google.com<http://google-public-dns-a.google.com> 0.0% 20 0.7 0.7 0.6 0.9 0.1 I’m not sure where I’m going once I enter Google land but this is from a server located in Dallas with a direct connection into Google from my upstream. On Aug 29, 2016, at 1:38 PM, Josh Luthman <josh@imaginenetworksllc.com<mailto:josh@imaginenetworksllc.com>> wrote: Not sure if it matters but some of you guys are using 8.8.4.4 and the OP is 8.8.8.8 Josh Luthman Office: 937-552-2340 Direct: 937-552-2343 1100 Wayne St Suite 1337 Troy, OH 45373 On Mon, Aug 29, 2016 at 2:27 PM, Keith Stokes via Outages <outages@outages.org<mailto:outages@outages.org>> wrote: Not me. Loss% Snt Last Avg Best Wrst StDev 1. ext-215-74.215.64.208.in-addr.arpa 0.0% 20 0.5 0.5 0.4 1.5 0.2 2. 72.20.177.33 0.0% 20 0.5 2.0 0.5 22.3 4.9 3. 74.125.49.233 0.0% 20 0.7 0.9 0.5 2.0 0.5 4. 108.170.240.1 0.0% 20 0.8 0.8 0.7 1.5 0.2 5. 72.14.234.105 0.0% 20 1.0 1.1 0.9 1.8 0.2 6. google-public-dns-b.google.com<http://google-public-dns-b.google.com/> 0.0% 20 0.6 1.1 0.6 2.2 0.5 On Aug 29, 2016, at 1:19 PM, Mason Moody via Outages <outages@outages.org<mailto:outages@outages.org>> wrote: I'm seeing the same. $ sudo mtr -rwc 20 8.8.4.4 HOST: endor Loss% Snt Last Avg Best Wrst StDev 1. 69.28.84.2 0.0% 20 1.1 22.2 0.4 181.5 47.2 2. 38.104.37.141 0.0% 20 1.3 1.3 1.0 2.5 0.3 3. te0-3-1-1.rcr21.dfw02.atlas.cogentco.com<http://rcr21.dfw02.atlas.cogentco.com/> 0.0% 20 0.8 0.9 0.6 2.0 0.4 4. be2277.ccr22.dfw01.atlas.cogentco.com<http://ccr22.dfw01.atlas.cogentco.com/> 0.0% 20 0.9 1.0 0.9 1.2 0.1 5. be2764.ccr41.dfw03.atlas.cogentco.com<http://ccr41.dfw03.atlas.cogentco.com/> 0.0% 20 0.9 1.1 0.9 1.5 0.2 6. tata.dfw03.atlas.cogentco.com<http://tata.dfw03.atlas.cogentco.com/> 0.0% 20 0.9 1.0 0.7 2.0 0.3 7. 72.14.219.124 25.0% 20 0.8 1.0 0.8 1.7 0.2 8. 108.170.240.193 20.0% 20 1.0 1.1 0.9 1.2 0.1 9. 64.233.175.9 30.0% 20 1.0 1.1 1.0 1.2 0.1 10. google-public-dns-b.google.com<http://google-public-dns-b.google.com/> 45.0% 20 1.0 1.0 0.9 1.0 0.0 On 2016-08-29 14:14, Dan White via Outages wrote: We are seeing packet loss to google hosts, hosted in Dallas. Any anyone seeing this? MTR output: Host Loss% Snt Last Avg Best Wrst StDev 1. 10.0.2.1 0.0% 51 2.4 1.9 0.5 5.2 0.7 2. olp-67-217-152-5.olp.net<http://olp-67-217-152-5.olp.net/> 0.0% 51 0.4 3.7 0.3 27.8 4.9 3. olp-67-217-158-29.olp.net<http://olp-67-217-158-29.olp.net/> 0.0% 50 0.3 4.5 0.3 22.0 5.3 4. olp-67-217-151-125.olp.net<http://olp-67-217-151-125.olp.net/> 0.0% 50 0.6 4.9 0.6 33.3 6.8 5. olp-67-217-158-30.olp.net<http://olp-67-217-158-30.olp.net/> 0.0% 50 0.6 6.4 0.5 64.0 10.5 6. olp-67-217-158-21.olp.net<http://olp-67-217-158-21.olp.net/> 0.0% 50 11.4 4.4 1.1 13.6 4.1 7. xe-8-3-0.bar2.tulsa2.level3.net<http://xe-8-3-0.bar2.tulsa2.level3.net/> 0.0% 50 1.2 2.7 1.2 27.5 5.5 8. ae-0-11.bar1.tulsa2.level3.net<http://ae-0-11.bar1.tulsa2.level3.net/> 0.0% 50 2.0 4.1 1.9 44.5 7.8 9. ae-3-3.ebr4.dallas1.level3.net<http://ae-3-3.ebr4.dallas1.level3.net/> 91.8% 50 20.5 20.5 20.5 20.5 0.0 10. ae-8-0.ear3.dallas1.level3.net<http://ae-8-0.ear3.dallas1.level3.net/> 93.9% 50 20.6 20.6 20.6 20.7 0.0 11. google-level3-100g.dallas1.leve28.0% 50 20.5 20.7 20.4 29.6 1.5 12. 108.170.240.193 22.0% 50 20.8 20.7 20.7 20.9 0.0 13. 64.233.175.9 26.0% 50 20.7 20.7 20.7 20.8 0.0 14. google-public-dns-a.google.com<http://google-public-dns-a.google.com/> 24.0% 50 20.5 20.6 20.5 25.9 0.9 The problematic hop appears to be either 10 or 11 (level3-100g). Compare to: Host Loss% Snt Last Avg Best Wrst StDev 1. 10.0.2.1 0.0% 124 0.8 1.6 0.5 4.6 0.5 2. olp-67-217-152-5.olp.net<http://olp-67-217-152-5.olp.net/> 0.0% 124 3.8 4.3 0.3 29.9 5.6 3. olp-67-217-158-29.olp.net<http://olp-67-217-158-29.olp.net/> 0.0% 124 0.3 3.5 0.3 26.1 4.9 4. olp-67-217-151-125.olp.net<http://olp-67-217-151-125.olp.net/> 0.0% 124 6.6 3.3 0.6 20.0 3.7 5. olp-67-217-158-30.olp.net<http://olp-67-217-158-30.olp.net/> 0.0% 124 0.6 7.1 0.6 44.1 9.8 6. olp-67-217-158-21.olp.net<http://olp-67-217-158-21.olp.net/> 0.8% 124 11.9 4.8 1.1 14.9 4.3 7. xe-8-3-0.bar2.tulsa2.level3.net<http://xe-8-3-0.bar2.tulsa2.level3.net/> 0.0% 123 1.3 4.0 1.1 68.3 11.3 8. ae-0-11.bar1.tulsa2.level3.net<http://ae-0-11.bar1.tulsa2.level3.net/> 0.8% 123 2.1 3.9 1.9 62.3 8.1 9. ae-3-3.ebr4.dallas1.level3.net<http://ae-3-3.ebr4.dallas1.level3.net/> 94.3% 123 20.5 20.6 20.5 20.6 0.0 10. ae-2-3512.edge2.dallas1.level3. 0.0% 123 20.4 22.1 20.3 68.0 7.1 11. b.resolvers.level3.net<http://b.resolvers.level3.net/> 0.0% 123 20.2 20.3 20.2 22.8 0.3 -- ____________ Mason Moody _______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages --- Keith Stokes _______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages --- Keith Stokes
Yes, intermittently since Friday. -----Original Message----- From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Dan White via Outages Sent: Monday, August 29, 2016 1:15 PM To: outages@outages.org Cc: noc@google.com Subject: [outages] Packet loss to 8.8.8.8 and other google IPs (Dallas) We are seeing packet loss to google hosts, hosted in Dallas. Any anyone seeing this? MTR output: Host Loss% Snt Last Avg Best Wrst StDev 1. 10.0.2.1 0.0% 51 2.4 1.9 0.5 5.2 0.7 2. olp-67-217-152-5.olp.net 0.0% 51 0.4 3.7 0.3 27.8 4.9 3. olp-67-217-158-29.olp.net 0.0% 50 0.3 4.5 0.3 22.0 5.3 4. olp-67-217-151-125.olp.net 0.0% 50 0.6 4.9 0.6 33.3 6.8 5. olp-67-217-158-30.olp.net 0.0% 50 0.6 6.4 0.5 64.0 10.5 6. olp-67-217-158-21.olp.net 0.0% 50 11.4 4.4 1.1 13.6 4.1 7. xe-8-3-0.bar2.tulsa2.level3.net 0.0% 50 1.2 2.7 1.2 27.5 5.5 8. ae-0-11.bar1.tulsa2.level3.net 0.0% 50 2.0 4.1 1.9 44.5 7.8 9. ae-3-3.ebr4.dallas1.level3.net 91.8% 50 20.5 20.5 20.5 20.5 0.0 10. ae-8-0.ear3.dallas1.level3.net 93.9% 50 20.6 20.6 20.6 20.7 0.0 11. google-level3-100g.dallas1.leve28.0% 50 20.5 20.7 20.4 29.6 1.5 12. 108.170.240.193 22.0% 50 20.8 20.7 20.7 20.9 0.0 13. 64.233.175.9 26.0% 50 20.7 20.7 20.7 20.8 0.0 14. google-public-dns-a.google.com 24.0% 50 20.5 20.6 20.5 25.9 0.9 The problematic hop appears to be either 10 or 11 (level3-100g). Compare to: Host Loss% Snt Last Avg Best Wrst StDev 1. 10.0.2.1 0.0% 124 0.8 1.6 0.5 4.6 0.5 2. olp-67-217-152-5.olp.net 0.0% 124 3.8 4.3 0.3 29.9 5.6 3. olp-67-217-158-29.olp.net 0.0% 124 0.3 3.5 0.3 26.1 4.9 4. olp-67-217-151-125.olp.net 0.0% 124 6.6 3.3 0.6 20.0 3.7 5. olp-67-217-158-30.olp.net 0.0% 124 0.6 7.1 0.6 44.1 9.8 6. olp-67-217-158-21.olp.net 0.8% 124 11.9 4.8 1.1 14.9 4.3 7. xe-8-3-0.bar2.tulsa2.level3.net 0.0% 123 1.3 4.0 1.1 68.3 11.3 8. ae-0-11.bar1.tulsa2.level3.net 0.8% 123 2.1 3.9 1.9 62.3 8.1 9. ae-3-3.ebr4.dallas1.level3.net 94.3% 123 20.5 20.6 20.5 20.6 0.0 10. ae-2-3512.edge2.dallas1.level3. 0.0% 123 20.4 22.1 20.3 68.0 7.1 11. b.resolvers.level3.net 0.0% 123 20.2 20.3 20.2 22.8 0.3 -- Dan White BTC Broadband Network Admin Lead Ph 918.366.0248 (direct) main: (918)366-8000 Fax 918.366.6610 email: dwhite@olp.net http://www.btcbroadband.com _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Google congestion management Google can generally run peering ports at 100% capacity, with low (<1-2%) packet loss for most services. Peers should abide by the capacity augments guideline above. https://peering.google.com/#/options/peering -- Eduardo Schoedler Em segunda-feira, 29 de agosto de 2016, Dan White via Outages < outages@outages.org> escreveu:
We are seeing packet loss to google hosts, hosted in Dallas. Any anyone seeing this?
MTR output:
Host Loss% Snt Last Avg Best Wrst StDev 1. 10.0.2.1 0.0% 51 2.4 1.9 0.5 5.2 0.7 2. olp-67-217-152-5.olp.net 0.0% 51 0.4 3.7 0.3 27.8 4.9 3. olp-67-217-158-29.olp.net 0.0% 50 0.3 4.5 0.3 22.0 5.3 4. olp-67-217-151-125.olp.net 0.0% 50 0.6 4.9 0.6 33.3 6.8 5. olp-67-217-158-30.olp.net 0.0% 50 0.6 6.4 0.5 64.0 10.5 6. olp-67-217-158-21.olp.net 0.0% 50 11.4 4.4 1.1 13.6 4.1 7. xe-8-3-0.bar2.tulsa2.level3.net 0.0% 50 1.2 2.7 1.2 27.5 5.5 8. ae-0-11.bar1.tulsa2.level3.net 0.0% 50 2.0 4.1 1.9 44.5 7.8 9. ae-3-3.ebr4.dallas1.level3.net 91.8% 50 20.5 20.5 20.5 20.5 0.0 10. ae-8-0.ear3.dallas1.level3.net 93.9% 50 20.6 20.6 20.6 20.7 0.0 11. google-level3-100g.dallas1.leve28.0% 50 20.5 20.7 20.4 29.6 1.5 12. 108.170.240.193 22.0% 50 20.8 20.7 20.7 20.9 0.0 13. 64.233.175.9 26.0% 50 20.7 20.7 20.7 20.8 0.0 14. google-public-dns-a.google.com 24.0% 50 20.5 20.6 20.5 25.9 0.9
The problematic hop appears to be either 10 or 11 (level3-100g). Compare to:
Host Loss% Snt Last Avg Best Wrst StDev 1. 10.0.2.1 0.0% 124 0.8 1.6 0.5 4.6 0.5 2. olp-67-217-152-5.olp.net 0.0% 124 3.8 4.3 0.3 29.9 5.6 3. olp-67-217-158-29.olp.net 0.0% 124 0.3 3.5 0.3 26.1 4.9 4. olp-67-217-151-125.olp.net 0.0% 124 6.6 3.3 0.6 20.0 3.7 5. olp-67-217-158-30.olp.net 0.0% 124 0.6 7.1 0.6 44.1 9.8 6. olp-67-217-158-21.olp.net 0.8% 124 11.9 4.8 1.1 14.9 4.3 7. xe-8-3-0.bar2.tulsa2.level3.net 0.0% 123 1.3 4.0 1.1 68.3 11.3 8. ae-0-11.bar1.tulsa2.level3.net 0.8% 123 2.1 3.9 1.9 62.3 8.1 9. ae-3-3.ebr4.dallas1.level3.net 94.3% 123 20.5 20.6 20.5 20.6 0.0 10. ae-2-3512.edge2.dallas1.level3. 0.0% 123 20.4 22.1 20.3 68.0 7.1 11. b.resolvers.level3.net 0.0% 123 20.2 20.3 20.2 22.8 0.3
-- Dan White BTC Broadband Network Admin Lead Ph 918.366.0248 (direct) main: (918)366-8000 Fax 918.366.6610 email: dwhite@olp.net http://www.btcbroadband.com _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
-- Eduardo Schoedler
This appears to be a problem on the Level3 network in the DFW region, which is why only some of you see it. (Those who don't see it are in other regions, peering with Google directly, or using other transit providers.) For those seeing issues, I suggest escalating to Level3, perhaps after trying a reverse-traceroute from http://lg.level3.net/ (I can't do this test since you didn't specify your source IP.) Damian On Mon, Aug 29, 2016 at 11:14 AM, Dan White <dwhite@olp.net> wrote:
We are seeing packet loss to google hosts, hosted in Dallas. Any anyone seeing this?
MTR output:
Host Loss% Snt Last Avg Best Wrst StDev 1. 10.0.2.1 0.0% 51 2.4 1.9 0.5 5.2 0.7 2. olp-67-217-152-5.olp.net 0.0% 51 0.4 3.7 0.3 27.8 4.9 3. olp-67-217-158-29.olp.net 0.0% 50 0.3 4.5 0.3 22.0 5.3 4. olp-67-217-151-125.olp.net 0.0% 50 0.6 4.9 0.6 33.3 6.8 5. olp-67-217-158-30.olp.net 0.0% 50 0.6 6.4 0.5 64.0 10.5 6. olp-67-217-158-21.olp.net 0.0% 50 11.4 4.4 1.1 13.6 4.1 7. xe-8-3-0.bar2.tulsa2.level3.net 0.0% 50 1.2 2.7 1.2 27.5 5.5 8. ae-0-11.bar1.tulsa2.level3.net 0.0% 50 2.0 4.1 1.9 44.5 7.8 9. ae-3-3.ebr4.dallas1.level3.net 91.8% 50 20.5 20.5 20.5 20.5 0.0 10. ae-8-0.ear3.dallas1.level3.net 93.9% 50 20.6 20.6 20.6 20.7 0.0 11. google-level3-100g.dallas1.leve28.0% 50 20.5 20.7 20.4 29.6 1.5 12. 108.170.240.193 22.0% 50 20.8 20.7 20.7 20.9 0.0 13. 64.233.175.9 26.0% 50 20.7 20.7 20.7 20.8 0.0 14. google-public-dns-a.google.com 24.0% 50 20.5 20.6 20.5 25.9 0.9
The problematic hop appears to be either 10 or 11 (level3-100g). Compare to:
Host Loss% Snt Last Avg Best Wrst StDev 1. 10.0.2.1 0.0% 124 0.8 1.6 0.5 4.6 0.5 2. olp-67-217-152-5.olp.net 0.0% 124 3.8 4.3 0.3 29.9 5.6 3. olp-67-217-158-29.olp.net 0.0% 124 0.3 3.5 0.3 26.1 4.9 4. olp-67-217-151-125.olp.net 0.0% 124 6.6 3.3 0.6 20.0 3.7 5. olp-67-217-158-30.olp.net 0.0% 124 0.6 7.1 0.6 44.1 9.8 6. olp-67-217-158-21.olp.net 0.8% 124 11.9 4.8 1.1 14.9 4.3 7. xe-8-3-0.bar2.tulsa2.level3.net 0.0% 123 1.3 4.0 1.1 68.3 11.3 8. ae-0-11.bar1.tulsa2.level3.net 0.8% 123 2.1 3.9 1.9 62.3 8.1 9. ae-3-3.ebr4.dallas1.level3.net 94.3% 123 20.5 20.6 20.5 20.6 0.0 10. ae-2-3512.edge2.dallas1.level3. 0.0% 123 20.4 22.1 20.3 68.0 7.1 11. b.resolvers.level3.net 0.0% 123 20.2 20.3 20.2 22.8 0.3
-- Dan White BTC Broadband Network Admin Lead Ph 918.366.0248 (direct) main: (918)366-8000 Fax 918.366.6610 email: dwhite@olp.net http://www.btcbroadband.com
-- You received this message because you are subscribed to the Google Groups "external-netops" group. To unsubscribe from this group and stop receiving emails from it, send an email to external-netops+unsubscribe@google.com. To post to this group, send email to external-netops@google.com. Visit this group at https://groups.google.com/a/go ogle.com/group/external-netops/. For more options, visit https://groups.google.com/a/google.com/d/optout.
Google public dns is anycast so trying to hit from a different location will most likely result in a different destination. https://developers.google.com/speed/public-dns/faq Matt Blackstone Network Engineer On Mon, Aug 29, 2016 at 1:54 PM, Damian Menscher via Outages < outages@outages.org> wrote:
This appears to be a problem on the Level3 network in the DFW region, which is why only some of you see it. (Those who don't see it are in other regions, peering with Google directly, or using other transit providers.)
For those seeing issues, I suggest escalating to Level3, perhaps after trying a reverse-traceroute from http://lg.level3.net/ (I can't do this test since you didn't specify your source IP.)
Damian
On Mon, Aug 29, 2016 at 11:14 AM, Dan White <dwhite@olp.net> wrote:
We are seeing packet loss to google hosts, hosted in Dallas. Any anyone seeing this?
MTR output:
Host Loss% Snt Last Avg Best Wrst StDev 1. 10.0.2.1 0.0% 51 2.4 1.9 0.5 5.2 0.7 2. olp-67-217-152-5.olp.net 0.0% 51 0.4 3.7 0.3 27.8 4.9 3. olp-67-217-158-29.olp.net 0.0% 50 0.3 4.5 0.3 22.0 5.3 4. olp-67-217-151-125.olp.net 0.0% 50 0.6 4.9 0.6 33.3 6.8 5. olp-67-217-158-30.olp.net 0.0% 50 0.6 6.4 0.5 64.0 10.5 6. olp-67-217-158-21.olp.net 0.0% 50 11.4 4.4 1.1 13.6 4.1 7. xe-8-3-0.bar2.tulsa2.level3.net 0.0% 50 1.2 2.7 1.2 27.5 5.5 8. ae-0-11.bar1.tulsa2.level3.net 0.0% 50 2.0 4.1 1.9 44.5 7.8 9. ae-3-3.ebr4.dallas1.level3.net 91.8% 50 20.5 20.5 20.5 20.5 0.0 10. ae-8-0.ear3.dallas1.level3.net 93.9% 50 20.6 20.6 20.6 20.7 0.0 11. google-level3-100g.dallas1.leve28.0% 50 20.5 20.7 20.4 29.6 1.5 12. 108.170.240.193 22.0% 50 20.8 20.7 20.7 20.9 0.0 13. 64.233.175.9 26.0% 50 20.7 20.7 20.7 20.8 0.0 14. google-public-dns-a.google.com 24.0% 50 20.5 20.6 20.5 25.9 0.9
The problematic hop appears to be either 10 or 11 (level3-100g). Compare to:
Host Loss% Snt Last Avg Best Wrst StDev 1. 10.0.2.1 0.0% 124 0.8 1.6 0.5 4.6 0.5 2. olp-67-217-152-5.olp.net 0.0% 124 3.8 4.3 0.3 29.9 5.6 3. olp-67-217-158-29.olp.net 0.0% 124 0.3 3.5 0.3 26.1 4.9 4. olp-67-217-151-125.olp.net 0.0% 124 6.6 3.3 0.6 20.0 3.7 5. olp-67-217-158-30.olp.net 0.0% 124 0.6 7.1 0.6 44.1 9.8 6. olp-67-217-158-21.olp.net 0.8% 124 11.9 4.8 1.1 14.9 4.3 7. xe-8-3-0.bar2.tulsa2.level3.net 0.0% 123 1.3 4.0 1.1 68.3 11.3 8. ae-0-11.bar1.tulsa2.level3.net 0.8% 123 2.1 3.9 1.9 62.3 8.1 9. ae-3-3.ebr4.dallas1.level3.net 94.3% 123 20.5 20.6 20.5 20.6 0.0 10. ae-2-3512.edge2.dallas1.level3. 0.0% 123 20.4 22.1 20.3 68.0 7.1 11. b.resolvers.level3.net 0.0% 123 20.2 20.3 20.2 22.8 0.3
-- Dan White BTC Broadband Network Admin Lead Ph 918.366.0248 (direct) main: (918)366-8000 Fax 918.366.6610 email: dwhite@olp.net http://www.btcbroadband.com
-- You received this message because you are subscribed to the Google Groups "external-netops" group. To unsubscribe from this group and stop receiving emails from it, send an email to external-netops+unsubscribe@google.com. To post to this group, send email to external-netops@google.com. Visit this group at https://groups.google.com/a/go ogle.com/group/external-netops/. For more options, visit https://groups.google.com/a/google.com/d/optout.
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
We have been seeing this off and on to many google sites (probably most if not all) but we have tested to www.google.com, www.youtube.com, 8.8.8.8, & 8.8.4.4 and been seeing packet loss 15-40%. It is specific to our Level 3 connection and we do not see it on our HE connection (when using alternate IP space to avoid Level 3). We've seen this off and on for over a month, but noticed it pronounced on Friday and pronounced today. I opened a ticket with Level 3, they said that it's a Google problem and to contact them. Since I am not a direct google IP services customer I can't do much more than e-mail noc@google.com and see what happens (which I already have). Our Level 3 upstream router is in Kansas City and we see the issue after the Dallas Level 3 router: tracert 8.8.8.8 Tracing route to google-public-dns-a.google.com [8.8.8.8] over a maximum of 30 hops: 1 1 ms 2 ms 2 ms customer-ip-8-34-97-1.widerangebroadband.net [8.34.97.1] 2 4 ms 4 ms 2 ms wrb-router-8-33-2-121.widerangebroadband.net [8.33.2.121] 3 60 ms 11 ms 142 ms ge-6-15.car1.kansascity1.level3.net [4.53.32.105] 4 16 ms 17 ms 19 ms google-level3-100g.dallas1.level3.net [4.68.72.138] 5 16 ms 16 ms * 108.170.240.193 6 19 ms 18 ms 18 ms 64.233.175.9 7 17 ms 17 ms * google-public-dns-a.google.com [8.8.8.8] 8 17 ms 17 ms 16 ms google-public-dns-a.google.com [8.8.8.8] Trace complete. Damian: If you can assist me with details contact me off-list and I'd be happy to include any Google information in the current Level 3 ticket and escalate it again. At this point Level 3 has said everything is fine until 4.68.72.138 therefore the issue an internal Google network issue starting at 108-170.240.193. --Andrew Duey On Mon, Aug 29, 2016 at 3:24 PM, Matthew Blackstone via Outages < outages@outages.org> wrote:
Google public dns is anycast so trying to hit from a different location will most likely result in a different destination. https://developers.google.com/speed/public-dns/faq
Matt Blackstone Network Engineer
On Mon, Aug 29, 2016 at 1:54 PM, Damian Menscher via Outages < outages@outages.org> wrote:
This appears to be a problem on the Level3 network in the DFW region, which is why only some of you see it. (Those who don't see it are in other regions, peering with Google directly, or using other transit providers.)
For those seeing issues, I suggest escalating to Level3, perhaps after trying a reverse-traceroute from http://lg.level3.net/ (I can't do this test since you didn't specify your source IP.)
Damian
On Mon, Aug 29, 2016 at 11:14 AM, Dan White <dwhite@olp.net> wrote:
We are seeing packet loss to google hosts, hosted in Dallas. Any anyone seeing this?
MTR output:
Host Loss% Snt Last Avg Best Wrst StDev 1. 10.0.2.1 0.0% 51 2.4 1.9 0.5 5.2 0.7 2. olp-67-217-152-5.olp.net 0.0% 51 0.4 3.7 0.3 27.8 4.9 3. olp-67-217-158-29.olp.net 0.0% 50 0.3 4.5 0.3 22.0 5.3 4. olp-67-217-151-125.olp.net 0.0% 50 0.6 4.9 0.6 33.3 6.8 5. olp-67-217-158-30.olp.net 0.0% 50 0.6 6.4 0.5 64.0 10.5 6. olp-67-217-158-21.olp.net 0.0% 50 11.4 4.4 1.1 13.6 4.1 7. xe-8-3-0.bar2.tulsa2.level3.net 0.0% 50 1.2 2.7 1.2 27.5 5.5 8. ae-0-11.bar1.tulsa2.level3.net 0.0% 50 2.0 4.1 1.9 44.5 7.8 9. ae-3-3.ebr4.dallas1.level3.net 91.8% 50 20.5 20.5 20.5 20.5 0.0 10. ae-8-0.ear3.dallas1.level3.net 93.9% 50 20.6 20.6 20.6 20.7 0.0 11. google-level3-100g.dallas1.leve28.0% 50 20.5 20.7 20.4 29.6 1.5 12. 108.170.240.193 22.0% 50 20.8 20.7 20.7 20.9 0.0 13. 64.233.175.9 26.0% 50 20.7 20.7 20.7 20.8 0.0 14. google-public-dns-a.google.com 24.0% 50 20.5 20.6 20.5 25.9 0.9
The problematic hop appears to be either 10 or 11 (level3-100g). Compare to:
Host Loss% Snt Last Avg Best Wrst StDev 1. 10.0.2.1 0.0% 124 0.8 1.6 0.5 4.6 0.5 2. olp-67-217-152-5.olp.net 0.0% 124 3.8 4.3 0.3 29.9 5.6 3. olp-67-217-158-29.olp.net 0.0% 124 0.3 3.5 0.3 26.1 4.9 4. olp-67-217-151-125.olp.net 0.0% 124 6.6 3.3 0.6 20.0 3.7 5. olp-67-217-158-30.olp.net 0.0% 124 0.6 7.1 0.6 44.1 9.8 6. olp-67-217-158-21.olp.net 0.8% 124 11.9 4.8 1.1 14.9 4.3 7. xe-8-3-0.bar2.tulsa2.level3.net 0.0% 123 1.3 4.0 1.1 68.3 11.3 8. ae-0-11.bar1.tulsa2.level3.net 0.8% 123 2.1 3.9 1.9 62.3 8.1 9. ae-3-3.ebr4.dallas1.level3.net 94.3% 123 20.5 20.6 20.5 20.6 0.0 10. ae-2-3512.edge2.dallas1.level3. 0.0% 123 20.4 22.1 20.3 68.0 7.1 11. b.resolvers.level3.net 0.0% 123 20.2 20.3 20.2 22.8 0.3
-- Dan White BTC Broadband Network Admin Lead Ph 918.366.0248 (direct) main: (918)366-8000 Fax 918.366.6610 email: dwhite@olp.net http://www.btcbroadband.com
-- You received this message because you are subscribed to the Google Groups "external-netops" group. To unsubscribe from this group and stop receiving emails from it, send an email to external-netops+unsubscribe@google.com. To post to this group, send email to external-netops@google.com. Visit this group at https://groups.google.com/a/go ogle.com/group/external-netops/. For more options, visit https://groups.google.com/a/google.com/d/optout.
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
-- Thanks, Andrew Duey Network Engineer, Widerange Broadband Direct: 402-327-1101
I am seeing loss and high latency to Google in Atlanta and Miami over Level3. When I finally spoke with someone within Level3 they mentioned they are moving peerings and is a part of the TWT/L3 merger. We are working with them at a higher level to determine a resolution. On Mon, Aug 29, 2016 at 4:41 PM, Andrew Duey via Outages < outages@outages.org> wrote:
We have been seeing this off and on to many google sites (probably most if not all) but we have tested to www.google.com, www.youtube.com, 8.8.8.8, & 8.8.4.4 and been seeing packet loss 15-40%. It is specific to our Level 3 connection and we do not see it on our HE connection (when using alternate IP space to avoid Level 3).
We've seen this off and on for over a month, but noticed it pronounced on Friday and pronounced today.
I opened a ticket with Level 3, they said that it's a Google problem and to contact them. Since I am not a direct google IP services customer I can't do much more than e-mail noc@google.com and see what happens (which I already have).
Our Level 3 upstream router is in Kansas City and we see the issue after the Dallas Level 3 router:
tracert 8.8.8.8 Tracing route to google-public-dns-a.google.com [8.8.8.8] over a maximum of 30 hops: 1 1 ms 2 ms 2 ms customer-ip-8-34-97-1.widerang ebroadband.net [8.34.97.1] 2 4 ms 4 ms 2 ms wrb-router-8-33-2-121.widerang ebroadband.net [8.33.2.121] 3 60 ms 11 ms 142 ms ge-6-15.car1.kansascity1.level3.net [4.53.32.105] 4 16 ms 17 ms 19 ms google-level3-100g.dallas1.level3.net [4.68.72.138] 5 16 ms 16 ms * 108.170.240.193 6 19 ms 18 ms 18 ms 64.233.175.9 7 17 ms 17 ms * google-public-dns-a.google.com [8.8.8.8] 8 17 ms 17 ms 16 ms google-public-dns-a.google.com [8.8.8.8] Trace complete.
Damian: If you can assist me with details contact me off-list and I'd be happy to include any Google information in the current Level 3 ticket and escalate it again. At this point Level 3 has said everything is fine until 4.68.72.138 therefore the issue an internal Google network issue starting at 108-170.240.193.
--Andrew Duey
On Mon, Aug 29, 2016 at 3:24 PM, Matthew Blackstone via Outages < outages@outages.org> wrote:
Google public dns is anycast so trying to hit from a different location will most likely result in a different destination. https://developers.google.com/speed/public-dns/faq
Matt Blackstone Network Engineer
On Mon, Aug 29, 2016 at 1:54 PM, Damian Menscher via Outages < outages@outages.org> wrote:
This appears to be a problem on the Level3 network in the DFW region, which is why only some of you see it. (Those who don't see it are in other regions, peering with Google directly, or using other transit providers.)
For those seeing issues, I suggest escalating to Level3, perhaps after trying a reverse-traceroute from http://lg.level3.net/ (I can't do this test since you didn't specify your source IP.)
Damian
On Mon, Aug 29, 2016 at 11:14 AM, Dan White <dwhite@olp.net> wrote:
We are seeing packet loss to google hosts, hosted in Dallas. Any anyone seeing this?
MTR output:
Host Loss% Snt Last Avg Best Wrst StDev 1. 10.0.2.1 0.0% 51 2.4 1.9 0.5 5.2 0.7 2. olp-67-217-152-5.olp.net 0.0% 51 0.4 3.7 0.3 27.8 4.9 3. olp-67-217-158-29.olp.net 0.0% 50 0.3 4.5 0.3 22.0 5.3 4. olp-67-217-151-125.olp.net 0.0% 50 0.6 4.9 0.6 33.3 6.8 5. olp-67-217-158-30.olp.net 0.0% 50 0.6 6.4 0.5 64.0 10.5 6. olp-67-217-158-21.olp.net 0.0% 50 11.4 4.4 1.1 13.6 4.1 7. xe-8-3-0.bar2.tulsa2.level3.net 0.0% 50 1.2 2.7 1.2 27.5 5.5 8. ae-0-11.bar1.tulsa2.level3.net 0.0% 50 2.0 4.1 1.9 44.5 7.8 9. ae-3-3.ebr4.dallas1.level3.net 91.8% 50 20.5 20.5 20.5 20.5 0.0 10. ae-8-0.ear3.dallas1.level3.net 93.9% 50 20.6 20.6 20.6 20.7 0.0 11. google-level3-100g.dallas1.leve28.0% 50 20.5 20.7 20.4 29.6 1.5 12. 108.170.240.193 22.0% 50 20.8 20.7 20.7 20.9 0.0 13. 64.233.175.9 26.0% 50 20.7 20.7 20.7 20.8 0.0 14. google-public-dns-a.google.com 24.0% 50 20.5 20.6 20.5 25.9 0.9
The problematic hop appears to be either 10 or 11 (level3-100g). Compare to:
Host Loss% Snt Last Avg Best Wrst StDev 1. 10.0.2.1 0.0% 124 0.8 1.6 0.5 4.6 0.5 2. olp-67-217-152-5.olp.net 0.0% 124 3.8 4.3 0.3 29.9 5.6 3. olp-67-217-158-29.olp.net 0.0% 124 0.3 3.5 0.3 26.1 4.9 4. olp-67-217-151-125.olp.net 0.0% 124 6.6 3.3 0.6 20.0 3.7 5. olp-67-217-158-30.olp.net 0.0% 124 0.6 7.1 0.6 44.1 9.8 6. olp-67-217-158-21.olp.net 0.8% 124 11.9 4.8 1.1 14.9 4.3 7. xe-8-3-0.bar2.tulsa2.level3.net 0.0% 123 1.3 4.0 1.1 68.3 11.3 8. ae-0-11.bar1.tulsa2.level3.net 0.8% 123 2.1 3.9 1.9 62.3 8.1 9. ae-3-3.ebr4.dallas1.level3.net 94.3% 123 20.5 20.6 20.5 20.6 0.0 10. ae-2-3512.edge2.dallas1.level3. 0.0% 123 20.4 22.1 20.3 68.0 7.1 11. b.resolvers.level3.net 0.0% 123 20.2 20.3 20.2 22.8 0.3
-- Dan White BTC Broadband Network Admin Lead Ph 918.366.0248 (direct) main: (918)366-8000 Fax 918.366.6610 email: dwhite@olp.net http://www.btcbroadband.com
-- You received this message because you are subscribed to the Google Groups "external-netops" group. To unsubscribe from this group and stop receiving emails from it, send an email to external-netops+unsubscribe@google.com. To post to this group, send email to external-netops@google.com. Visit this group at https://groups.google.com/a/go ogle.com/group/external-netops/. For more options, visit https://groups.google.com/a/google.com/d/optout .
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
-- Thanks, Andrew Duey Network Engineer, Widerange Broadband Direct: 402-327-1101
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
participants (12)
-
Andrew Duey -
Damian Menscher -
Dan White -
Eduardo Schoedler -
Josh Luthman -
Justin -
Keith I Myers -
Keith Stokes -
Mark Borchers -
Mason Moody -
Matthew Blackstone -
Randall (Randy) Raitz