This morning appears everything looks fine, and still crossing gblx, but not the same hop that was failing. Thanks for the replies :) 3 15 ms 10 ms 10 ms te-8-3-ur02.greeley.co.denver.comcast.net [68.85.220.133] 4 14 ms 14 ms 16 ms te-0-8-0-5-ar02.aurora.co.denver.comcast.net [68.86.103.82] 5 15 ms 15 ms 13 ms pos-3-7-0-0-cr01.denver.co.ibone.comcast.net [68.86.91.101] 6 32 ms 28 ms 33 ms pos-2-15-0-0-cr01.dallas.tx.ibone.comcast.net [68.86.85.174] 7 30 ms 31 ms 27 ms tengigabitethernet6-2.ar3.dal2.gblx.net [67.17.157.221] 8 482 ms 306 ms 286 ms iomart.tengigabitethernet7-1.ar4.lon3.gblx.net [64.211.1.190] 9 134 ms 130 ms 144 ms 78.129.226.1 Trace complete. Blake Pfankuch -----Original Message----- From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On Behalf Of Blake Pfankuch Sent: Tuesday, September 20, 2011 10:58 PM To: Jeremy Chadwick Cc: outages@outages.org Subject: Re: [outages] comcast to Global Crossing issues Correct. Getting to the other side has taken a bit of doing, however traceroutes coming back to my house don't come across GBLX, destination in 75.151.89.0/24. Anything going from local comcast network to this remote location crossing gblx seems to fail. From another server in denver over tinet, looks good (no gblx touching) and same from phoenix. I'm not sure where the issue lies, just appears to me to from comcast destined to gblx, hense asking if anyone else sees something weird. 2 62-233-127-181.as20860.net (62.233.127.181) 1.259 ms 1.260 ms 1.252 ms 3 202.core1.thn.as20860.net (62.233.127.11) 20.223 ms 20.466 ms 20.460 ms 4 xe-10-2-0.londra32.lon.seabone.net (89.221.43.105) 1.214 ms 1.208 ms 1.20 1 ms 5 te4-4.newyork51.new.seabone.net (195.22.216.211) 70.649 ms te4-3.newyork51.new.seabone.net (195.22.195.3) 72.886 ms te4-4.newyork51.new.seabone.net (195.2 2.216.211) 70.647 ms 6 be-10-204-pe01.111eighthave.ny.ibone.comcast.net (75.149.230.77) 71.120 ms 71.365 ms 71.364 ms 7 pos-1-9-0-0-cr01.newyork.ny.ibone.comcast.net (68.86.86.41) 70.609 ms 70.6 03 ms 70.589 ms 8 pos-0-6-0-0-cr01.350ecermak.il.ibone.comcast.net (68.86.88.154) 91.822 ms 91.816 ms 91.803 ms 9 pos-1-7-0-0-cr01.chicago.il.ibone.comcast.net (68.86.88.45) 92.044 ms 91.6 53 ms 92.559 ms 10 * * * 11 * * * You are correct something is always broken, however OOB is not something that's justifiable in this case :) -----Original Message----- From: Jeremy Chadwick [mailto:outages@jdc.parodius.com] Sent: Tuesday, September 20, 2011 10:33 PM To: Blake Pfankuch Cc: outages@outages.org Subject: Re: [outages] comcast to Global Crossing issues Are you absolutely certain this is the fault of GBLX? Your traceroute below is only one direction -- to get a clear picture you need traceroutes from *both* directions; return packets may be taking an entirely different route and that could be what's broken (and would render as such in your traceroute). Most of the time on the Internet routing IS NOT symmetrical. You're a network engineer though so I assume you know this. 78.129.136.1 reachable from Comcast in the California Bay Area. This isn't shown in the below mtr because of what appears to be ACLs somewhere near/around the border of the network shown below blocking ICMP time exceeded. These egress packets obviously don't use GBLX; cannot check ingress, obviously. ICMP echo/echo reply to the endpoint does work. Host Loss% Snt Rcv Last Avg Best Wrst 1. gw.home.lan 0.0% 14 14 0.3 0.4 0.2 0.6 2. c-67-180-84-1.hsd1.ca.comcast.net 0.0% 14 14 25.0 24.5 9.4 40.6 3. te-7-3-ur04.santaclara.ca.sfba.comcast.net 0.0% 13 13 9.7 15.3 8.9 44.8 4. te-1-10-0-7-ar01.oakland.ca.sfba.comcast.ne 0.0% 13 13 12.4 13.7 10.9 24.5 5. pos-2-0-0-0-cr01.sacramento.ca.ibone.comcas 0.0% 13 13 13.3 15.2 13.0 30.3 6. pos-0-2-0-0-cr01.sanjose.ca.ibone.comcast.n 0.0% 13 13 16.8 17.5 15.3 21.3 7. pos-0-4-0-0-pe01.529bryant.ca.ibone.comcast 0.0% 13 13 17.2 18.8 16.6 32.2 8. as6762-pe01.350ecermak.il.ibone.comcast.net 0.0% 13 13 16.7 17.5 15.8 21.7 9. xe-8-0-0.londra32.lon.seabone.net 0.0% 13 13 159.1 160.8 157.9 175.5 10. iomart.londra32.lon.seabone.net 0.0% 13 13 161.0 160.1 158.5 161.1 11. 62-233-127-174.as20860.net 0.0% 13 13 160.8 160.9 159.7 162.5 12. 87.117.211.33 0.0% 13 13 162.0 165.0 158.9 190.9 13. ??? $ ping 78.129.136.1 PING 78.129.136.1 (78.129.136.1): 56 data bytes 64 bytes from 78.129.136.1: icmp_seq=0 ttl=240 time=212.176 ms 64 bytes from 78.129.136.1: icmp_seq=1 ttl=240 time=200.820 ms 64 bytes from 78.129.136.1: icmp_seq=2 ttl=240 time=182.815 ms 64 bytes from 78.129.136.1: icmp_seq=3 ttl=240 time=171.383 ms 64 bytes from 78.129.136.1: icmp_seq=4 ttl=240 time=164.347 ms 64 bytes from 78.129.136.1: icmp_seq=5 ttl=240 time=165.554 ms 64 bytes from 78.129.136.1: icmp_seq=6 ttl=240 time=163.315 ms 64 bytes from 78.129.136.1: icmp_seq=7 ttl=240 time=163.348 ms 64 bytes from 78.129.136.1: icmp_seq=8 ttl=240 time=201.289 ms ^C --- 78.129.136.1 ping statistics --- 9 packets transmitted, 9 packets received, 0.0% packet loss round-trip min/avg/max/stddev = 163.315/180.561/212.176/18.290 ms
From my co-location provider (in Santa Clara, CA) to said IP. Again, no use of GBLX on the egress path, but this time ACLs permit ICMP time exceeded.
Host Loss% Snt Rcv Last Avg Best Wrst 1. 72.20.98.65 0.0% 15 15 0.4 0.5 0.3 1.3 2. er0sc2.bayarea.net.bayarea.net 0.0% 15 15 0.3 0.3 0.2 0.5 3. ge-11-0-1.er2.sjc2.above.net 0.0% 15 15 0.5 0.5 0.4 0.6 4. xe-1-1-0.mpr4.sjc7.us.above.net 0.0% 14 14 14.6 6.3 1.0 53.7 5. xe-0-1-0-6.sjc10.ip4.tinet.net 0.0% 14 14 1.3 5.0 1.0 53.9 6. xe-11-0-0.lon25.ip4.tinet.net 0.0% 14 14 142.5 144.8 142.5 173.2 7. iomart-hosting-gw.ip4.tinet.net 0.0% 14 14 143.6 182.0 143.5 280.8 8. 78.129.136.1 0.0% 14 14 148.0 147.7 145.0 174.1 Reminder to readers: the Internet is broken 24x7x365, and I'm quite serious. For mission-critical services that need maintenance, OOB (out-of-band) is required. If that means dialing internationally into a modem in London to do some device config changes then that's what it means. Rock the POTS lines, don't be afraid. -- | Jeremy Chadwick jdc at parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, US | | Making life hard for others since 1977. PGP 4BD6C0CB | On Wed, Sep 21, 2011 at 03:58:02AM +0000, Blake Pfankuch wrote:
Anyone else seeing any issues going Comcast to Global Crossing this evening? I was smack in the middle of some work on an EU server when it disconnected my ssh session. Thought I broke it, however I cannot load the datacenter website from home. From the office everything shows fine. From home traceroute shows fail past 208.178.62.141 going to 78.129.136.0/24 or 79.129.226.0/24, and I cannot load http://ipstats.globalcrossing.net either. Down for me says it's just me, but I cannot get to any of the affected services from one of my customer networks on Comcast either.
4 11 ms 9 ms 11 ms te-8-4-ur02.greeley.co.denver.comcast.net [68.86.128.22] 5 14 ms 11 ms 16 ms te-0-8-0-5-ar02.aurora.co.denver.comcast.net [68.86.103.82] 6 11 ms 16 ms 16 ms pos-3-7-0-0-cr01.denver.co.ibone.comcast.net [68.86.91.101] 7 33 ms 39 ms 29 ms pos-2-10-0-0-cr01.dallas.tx.ibone.comcast.net [68.86.87.186] 8 36 ms 27 ms 31 ms 208.178.62.141 9 * * * Request timed out. 10 * * * Request timed out.
Blake Pfankuch bpfankuch@cpgreeley.com<mailto:bpfankuch@cpgreeley.com>
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages