Are you absolutely certain this is the fault of GBLX? Your traceroute below is only one direction -- to get a clear picture you need traceroutes from *both* directions; return packets may be taking an entirely different route and that could be what's broken (and would render as such in your traceroute). Most of the time on the Internet routing IS NOT symmetrical. You're a network engineer though so I assume you know this. 78.129.136.1 reachable from Comcast in the California Bay Area. This isn't shown in the below mtr because of what appears to be ACLs somewhere near/around the border of the network shown below blocking ICMP time exceeded. These egress packets obviously don't use GBLX; cannot check ingress, obviously. ICMP echo/echo reply to the endpoint does work. Host Loss% Snt Rcv Last Avg Best Wrst 1. gw.home.lan 0.0% 14 14 0.3 0.4 0.2 0.6 2. c-67-180-84-1.hsd1.ca.comcast.net 0.0% 14 14 25.0 24.5 9.4 40.6 3. te-7-3-ur04.santaclara.ca.sfba.comcast.net 0.0% 13 13 9.7 15.3 8.9 44.8 4. te-1-10-0-7-ar01.oakland.ca.sfba.comcast.ne 0.0% 13 13 12.4 13.7 10.9 24.5 5. pos-2-0-0-0-cr01.sacramento.ca.ibone.comcas 0.0% 13 13 13.3 15.2 13.0 30.3 6. pos-0-2-0-0-cr01.sanjose.ca.ibone.comcast.n 0.0% 13 13 16.8 17.5 15.3 21.3 7. pos-0-4-0-0-pe01.529bryant.ca.ibone.comcast 0.0% 13 13 17.2 18.8 16.6 32.2 8. as6762-pe01.350ecermak.il.ibone.comcast.net 0.0% 13 13 16.7 17.5 15.8 21.7 9. xe-8-0-0.londra32.lon.seabone.net 0.0% 13 13 159.1 160.8 157.9 175.5 10. iomart.londra32.lon.seabone.net 0.0% 13 13 161.0 160.1 158.5 161.1 11. 62-233-127-174.as20860.net 0.0% 13 13 160.8 160.9 159.7 162.5 12. 87.117.211.33 0.0% 13 13 162.0 165.0 158.9 190.9 13. ??? $ ping 78.129.136.1 PING 78.129.136.1 (78.129.136.1): 56 data bytes 64 bytes from 78.129.136.1: icmp_seq=0 ttl=240 time=212.176 ms 64 bytes from 78.129.136.1: icmp_seq=1 ttl=240 time=200.820 ms 64 bytes from 78.129.136.1: icmp_seq=2 ttl=240 time=182.815 ms 64 bytes from 78.129.136.1: icmp_seq=3 ttl=240 time=171.383 ms 64 bytes from 78.129.136.1: icmp_seq=4 ttl=240 time=164.347 ms 64 bytes from 78.129.136.1: icmp_seq=5 ttl=240 time=165.554 ms 64 bytes from 78.129.136.1: icmp_seq=6 ttl=240 time=163.315 ms 64 bytes from 78.129.136.1: icmp_seq=7 ttl=240 time=163.348 ms 64 bytes from 78.129.136.1: icmp_seq=8 ttl=240 time=201.289 ms ^C --- 78.129.136.1 ping statistics --- 9 packets transmitted, 9 packets received, 0.0% packet loss round-trip min/avg/max/stddev = 163.315/180.561/212.176/18.290 ms
From my co-location provider (in Santa Clara, CA) to said IP. Again, no use of GBLX on the egress path, but this time ACLs permit ICMP time exceeded.
Host Loss% Snt Rcv Last Avg Best Wrst 1. 72.20.98.65 0.0% 15 15 0.4 0.5 0.3 1.3 2. er0sc2.bayarea.net.bayarea.net 0.0% 15 15 0.3 0.3 0.2 0.5 3. ge-11-0-1.er2.sjc2.above.net 0.0% 15 15 0.5 0.5 0.4 0.6 4. xe-1-1-0.mpr4.sjc7.us.above.net 0.0% 14 14 14.6 6.3 1.0 53.7 5. xe-0-1-0-6.sjc10.ip4.tinet.net 0.0% 14 14 1.3 5.0 1.0 53.9 6. xe-11-0-0.lon25.ip4.tinet.net 0.0% 14 14 142.5 144.8 142.5 173.2 7. iomart-hosting-gw.ip4.tinet.net 0.0% 14 14 143.6 182.0 143.5 280.8 8. 78.129.136.1 0.0% 14 14 148.0 147.7 145.0 174.1 Reminder to readers: the Internet is broken 24x7x365, and I'm quite serious. For mission-critical services that need maintenance, OOB (out-of-band) is required. If that means dialing internationally into a modem in London to do some device config changes then that's what it means. Rock the POTS lines, don't be afraid. -- | Jeremy Chadwick jdc at parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, US | | Making life hard for others since 1977. PGP 4BD6C0CB | On Wed, Sep 21, 2011 at 03:58:02AM +0000, Blake Pfankuch wrote:
Anyone else seeing any issues going Comcast to Global Crossing this evening? I was smack in the middle of some work on an EU server when it disconnected my ssh session. Thought I broke it, however I cannot load the datacenter website from home. From the office everything shows fine. From home traceroute shows fail past 208.178.62.141 going to 78.129.136.0/24 or 79.129.226.0/24, and I cannot load http://ipstats.globalcrossing.net either. Down for me says it's just me, but I cannot get to any of the affected services from one of my customer networks on Comcast either.
4 11 ms 9 ms 11 ms te-8-4-ur02.greeley.co.denver.comcast.net [68.86.128.22] 5 14 ms 11 ms 16 ms te-0-8-0-5-ar02.aurora.co.denver.comcast.net [68.86.103.82] 6 11 ms 16 ms 16 ms pos-3-7-0-0-cr01.denver.co.ibone.comcast.net [68.86.91.101] 7 33 ms 39 ms 29 ms pos-2-10-0-0-cr01.dallas.tx.ibone.comcast.net [68.86.87.186] 8 36 ms 27 ms 31 ms 208.178.62.141 9 * * * Request timed out. 10 * * * Request timed out.
Blake Pfankuch bpfankuch@cpgreeley.com<mailto:bpfankuch@cpgreeley.com>
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages