I am seeing loss and high latency to Google in Atlanta and Miami over Level3. When I finally spoke with someone within Level3 they mentioned they are moving peerings and is a part of the TWT/L3 merger. We are working with them at a higher level to determine a resolution. On Mon, Aug 29, 2016 at 4:41 PM, Andrew Duey via Outages < outages@outages.org> wrote:
We have been seeing this off and on to many google sites (probably most if not all) but we have tested to www.google.com, www.youtube.com, 8.8.8.8, & 8.8.4.4 and been seeing packet loss 15-40%. It is specific to our Level 3 connection and we do not see it on our HE connection (when using alternate IP space to avoid Level 3).
We've seen this off and on for over a month, but noticed it pronounced on Friday and pronounced today.
I opened a ticket with Level 3, they said that it's a Google problem and to contact them. Since I am not a direct google IP services customer I can't do much more than e-mail noc@google.com and see what happens (which I already have).
Our Level 3 upstream router is in Kansas City and we see the issue after the Dallas Level 3 router:
tracert 8.8.8.8 Tracing route to google-public-dns-a.google.com [8.8.8.8] over a maximum of 30 hops: 1 1 ms 2 ms 2 ms customer-ip-8-34-97-1.widerang ebroadband.net [8.34.97.1] 2 4 ms 4 ms 2 ms wrb-router-8-33-2-121.widerang ebroadband.net [8.33.2.121] 3 60 ms 11 ms 142 ms ge-6-15.car1.kansascity1.level3.net [4.53.32.105] 4 16 ms 17 ms 19 ms google-level3-100g.dallas1.level3.net [4.68.72.138] 5 16 ms 16 ms * 108.170.240.193 6 19 ms 18 ms 18 ms 64.233.175.9 7 17 ms 17 ms * google-public-dns-a.google.com [8.8.8.8] 8 17 ms 17 ms 16 ms google-public-dns-a.google.com [8.8.8.8] Trace complete.
Damian: If you can assist me with details contact me off-list and I'd be happy to include any Google information in the current Level 3 ticket and escalate it again. At this point Level 3 has said everything is fine until 4.68.72.138 therefore the issue an internal Google network issue starting at 108-170.240.193.
--Andrew Duey
On Mon, Aug 29, 2016 at 3:24 PM, Matthew Blackstone via Outages < outages@outages.org> wrote:
Google public dns is anycast so trying to hit from a different location will most likely result in a different destination. https://developers.google.com/speed/public-dns/faq
Matt Blackstone Network Engineer
On Mon, Aug 29, 2016 at 1:54 PM, Damian Menscher via Outages < outages@outages.org> wrote:
This appears to be a problem on the Level3 network in the DFW region, which is why only some of you see it. (Those who don't see it are in other regions, peering with Google directly, or using other transit providers.)
For those seeing issues, I suggest escalating to Level3, perhaps after trying a reverse-traceroute from http://lg.level3.net/ (I can't do this test since you didn't specify your source IP.)
Damian
On Mon, Aug 29, 2016 at 11:14 AM, Dan White <dwhite@olp.net> wrote:
We are seeing packet loss to google hosts, hosted in Dallas. Any anyone seeing this?
MTR output:
Host Loss% Snt Last Avg Best Wrst StDev 1. 10.0.2.1 0.0% 51 2.4 1.9 0.5 5.2 0.7 2. olp-67-217-152-5.olp.net 0.0% 51 0.4 3.7 0.3 27.8 4.9 3. olp-67-217-158-29.olp.net 0.0% 50 0.3 4.5 0.3 22.0 5.3 4. olp-67-217-151-125.olp.net 0.0% 50 0.6 4.9 0.6 33.3 6.8 5. olp-67-217-158-30.olp.net 0.0% 50 0.6 6.4 0.5 64.0 10.5 6. olp-67-217-158-21.olp.net 0.0% 50 11.4 4.4 1.1 13.6 4.1 7. xe-8-3-0.bar2.tulsa2.level3.net 0.0% 50 1.2 2.7 1.2 27.5 5.5 8. ae-0-11.bar1.tulsa2.level3.net 0.0% 50 2.0 4.1 1.9 44.5 7.8 9. ae-3-3.ebr4.dallas1.level3.net 91.8% 50 20.5 20.5 20.5 20.5 0.0 10. ae-8-0.ear3.dallas1.level3.net 93.9% 50 20.6 20.6 20.6 20.7 0.0 11. google-level3-100g.dallas1.leve28.0% 50 20.5 20.7 20.4 29.6 1.5 12. 108.170.240.193 22.0% 50 20.8 20.7 20.7 20.9 0.0 13. 64.233.175.9 26.0% 50 20.7 20.7 20.7 20.8 0.0 14. google-public-dns-a.google.com 24.0% 50 20.5 20.6 20.5 25.9 0.9
The problematic hop appears to be either 10 or 11 (level3-100g). Compare to:
Host Loss% Snt Last Avg Best Wrst StDev 1. 10.0.2.1 0.0% 124 0.8 1.6 0.5 4.6 0.5 2. olp-67-217-152-5.olp.net 0.0% 124 3.8 4.3 0.3 29.9 5.6 3. olp-67-217-158-29.olp.net 0.0% 124 0.3 3.5 0.3 26.1 4.9 4. olp-67-217-151-125.olp.net 0.0% 124 6.6 3.3 0.6 20.0 3.7 5. olp-67-217-158-30.olp.net 0.0% 124 0.6 7.1 0.6 44.1 9.8 6. olp-67-217-158-21.olp.net 0.8% 124 11.9 4.8 1.1 14.9 4.3 7. xe-8-3-0.bar2.tulsa2.level3.net 0.0% 123 1.3 4.0 1.1 68.3 11.3 8. ae-0-11.bar1.tulsa2.level3.net 0.8% 123 2.1 3.9 1.9 62.3 8.1 9. ae-3-3.ebr4.dallas1.level3.net 94.3% 123 20.5 20.6 20.5 20.6 0.0 10. ae-2-3512.edge2.dallas1.level3. 0.0% 123 20.4 22.1 20.3 68.0 7.1 11. b.resolvers.level3.net 0.0% 123 20.2 20.3 20.2 22.8 0.3
-- Dan White BTC Broadband Network Admin Lead Ph 918.366.0248 (direct) main: (918)366-8000 Fax 918.366.6610 email: dwhite@olp.net http://www.btcbroadband.com
-- You received this message because you are subscribed to the Google Groups "external-netops" group. To unsubscribe from this group and stop receiving emails from it, send an email to external-netops+unsubscribe@google.com. To post to this group, send email to external-netops@google.com. Visit this group at https://groups.google.com/a/go ogle.com/group/external-netops/. For more options, visit https://groups.google.com/a/google.com/d/optout .
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
-- Thanks, Andrew Duey Network Engineer, Widerange Broadband Direct: 402-327-1101
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages