Re: [outages] Google Connectivity Issues in NYC
This info pertains to accounts.google.com: Errors started around noon MST 12-Nov. We didn't start actively troubleshooting until the morning of the 14th. accounts.google.com was resolving to 216.58.219.237 from our DC and our corporate office. We were seeing constant curl timeouts (well over 50%) and mtr showed tons of packet loss at the last two hops (anywhere from 4%-60%). An offsite user found that they were resolving accounts.google.com to 216.58.219.205 and they were not seeing the same curl timeouts. Setup some host file hacks on a few hosts, and didn't see any more errors from those hosts. The 216.58.219.237 endpoint is still problematic today. ---------- Tim Korn Network Ninja On Tue, Nov 15, 2016 at 4:50 PM, Laurie Denness <ldenness@etsy.com> wrote:
Note this affecting Gmail and adwords heavily too, if you're in the NYC metro, see if you can reproduce:
[box:~] $ time curl -4 -vv http://adwords.google.com * About to connect() to adwords.google.com port 80 (#0) * Trying 74.125.29.100... <hang for 1 minute>
tcpdump: 23:47:02.402713 IP (tos 0x0, ttl 64, id 6873, offset 0, flags [DF], proto TCP (6), length 60) box.dc.etsy.com.35746 > qg-in-f113.1e100.net.http: Flags [S], cksum 0x738d (incorrect -> 0x4660), seq 570037774, win 29200, options [mss 1460,sackOK,TS val 1067194520 ecr 0,nop,wscale 9], length 0 23:47:06.410714 IP (tos 0x0, ttl 64, id 6874, offset 0, flags [DF], proto TCP (6), length 60) box.dc.etsy.com.35746 > qg-in-f113.1e100.net.http: Flags [S], cksum 0x738d (incorrect -> 0x36b8), seq 570037774, win 29200, options [mss 1460,sackOK,TS val 1067198528 ecr 0,nop,wscale 9], length 0 23:47:14.426726 IP (tos 0x0, ttl 64, id 6875, offset 0, flags [DF], proto TCP (6), length 60) box.dc.etsy.com.35746 > qg-in-f113.1e100.net.http: Flags [S], cksum 0x738d (incorrect -> 0x1768), seq 570037774, win 29200, options [mss 1460,sackOK,TS val 1067206544 ecr 0,nop,wscale 9], length 0
I'd think we were going nuts, except we're seeing from many independent networks and looks like there's a flurry of NY based people reporting on http://downdetector.com/status/google/map/
Definitely seems NYC based only, although I cannot replicate on Verizon FiOS in Manhattan.
On 15 November 2016 at 18:38, Tim Korn via Outages <outages@outages.org> wrote:
We've been having issues reaching google since last Friday. We're trying from at least four different ISPs (Zayo, Cogent, TimeWarner, Verizon FiOS) and all show the same issues intermittently.
We're having issues with the google apps suite, accounts.google.com, and www.google.com.
From our corporate office: [bam01:~] $ time curl -4 -vv http://www.google.com * Rebuilt URL to: http://www.google.com/ * Trying 74.125.21.103... * connect to 74.125.21.103 port 80 failed: Connection refused * Trying 74.125.21.99... * Connected to www.google.com (74.125.21.99) port 80 (#0)
GET / HTTP/1.1 Host: www.google.com User-Agent: curl/7.43.0 Accept: */*
< HTTP/1.1 200 OK <snip> *real 1m6.523s (1 minute connect time)*
From Datacenter: [dingy01.ny2:~] $ time curl -4 -vv http://www.google.com * About to connect() to www.google.com port 80 (#0) * Trying 216.58.193.132... * Connected to www.google.com (216.58.193.132) port 80 (#0)
GET / HTTP/1.1 User-Agent: curl/7.29.0 Host: www.google.com Accept: */*
< HTTP/1.1 200 OK <snip> *real 1m3.189s (took 1 minute to connect)*
[ding02.ny5:~] $ curl -4 -k https://accounts.google.com *curl: (7) couldn't connect to host* [ding02.ny5:~] $ curl -4 -k https://accounts.google.com *curl: (7) couldn't connect to host* [ding02.ny5:~] $ curl -4 -k https://accounts.google.com <HTML> <HEAD> <TITLE>Moved Temporarily</TITLE> </HEAD> <BODY BGCOLOR="#FFFFFF" TEXT="#000000"> <H1>Moved Temporarily</H1> The document has moved <A HREF="https://accounts.google.com/ManageAccount ">here</A>. </BODY> </HTML>
---------- Tim Korn
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
participants (1)
-
Tim Korn