Issues happening at Layer42 or PAIX?
Hey folks, is anyone aware of any problems happening right now with Layer42, or within the PAIX? We're seeing issues reaching both www.google.com (going through the PAIX) as well as www.facebook.com (going through the PAIX). Packets not going through the PAIX appear to be working fine. Below are traceroutes showing the problem. And yes, it's still ongoing as of right now (20:43 PDT). |------------------------------------------------------------------------------------------| | WinMTR statistics | | Host - % | Sent | Recv | Best | Avrg | Wrst | Last | |------------------------------------------------|------|------|------|------|------|------| | fw01.noc.XXXXXXXXXXXXXX - 0 | 161 | 161 | 0 | 0 | 7 | 0 | | gw01.nocport.cns.XXXXXXXXXXXXXX - 0 | 161 | 161 | 1 | 1 | 25 | 1 | | gw02.edgecross2.cns.XXXXXXXXXXXXXX - 0 | 161 | 161 | 1 | 1 | 24 | 1 | | gw-XXXXXXXXX.svcolo.com - 0 | 161 | 161 | 2 | 3 | 42 | 2 | | xe2-3.core1.pao.layer42.net - 0 | 161 | 161 | 2 | 10 | 184 | 3 | | paix.br01.pao1.tfbnw.net - 17 | 97 | 81 | 0 | 9 | 63 | 4 | | ae8.bb02.pao1.tfbnw.net - 14 | 105 | 91 | 3 | 7 | 43 | 14 | | ae10.bb02.prn1.tfnw.net - 17 | 97 | 81 | 4 | 15 | 76 | 4 | | ae1.dr01.prn1.tfbnw.net - 17 | 97 | 81 | 0 | 25 | 53 | 28 | | po1016.csw01b.prn1.tfbnw.net - 17 | 97 | 81 | 0 | 25 | 49 | 24 | | www-15-01-prn1.facebook.com - 17 | 97 | 81 | 22 | 25 | 37 | 25 | |________________________________________________|______|______|______|______|______|______| |------------------------------------------------------------------------------------------| | WinMTR statistics | | Host - % | Sent | Recv | Best | Avrg | Wrst | Last | |------------------------------------------------|------|------|------|------|------|------| | fw01.noc.XXXXXXXXXXXXXX - 0 | 218 | 218 | 0 | 0 | 8 | 0 | | gw01.nocport.cns.XXXXXXXXXXXXXX - 0 | 219 | 219 | 1 | 1 | 31 | 1 | | gw02.edgecross1.cns.XXXXXXXXXXXXXX - 0 | 219 | 219 | 1 | 1 | 19 | 1 | | gw-XXXXXXXXX.svcolo.com - 0 | 219 | 219 | 2 | 3 | 89 | 2 | | xe2-3.core1.pao.layer42.net - 0 | 219 | 219 | 2 | 14 | 111 | 32 | | core2-1-1-0.pao.net.google.com - 18 | 126 | 104 | 0 | 3 | 43 | 2 | | 216.239.49.250 - 2 | 203 | 199 | 3 | 10 | 88 | 17 | | 64.233.174.119 - 2 | 207 | 204 | 3 | 3 | 8 | 4 | | 74.125.224.145 - 8 | 167 | 154 | 3 | 3 | 12 | 3 | |________________________________________________|______|______|______|______|______|______| -- | Jeremy Chadwick jdc at parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, US | | Making life hard for others since 1977. PGP 4BD6C0CB |
On 17 Jun 2011, at 20:43 GMT-0700, Jeremy Chadwick wrote:
Hey folks, is anyone aware of any problems happening right now with Layer42, or within the PAIX?
We're seeing issues reaching both www.google.com (going through the PAIX) as well as www.facebook.com (going through the PAIX). Packets not going through the PAIX appear to be working fine.
FWIW, my connectivity via Layer42's network (at their 1555 Plymouth (MV) facility) appears to be fine, including to www.'('facebook'|'google')'.com. Same appears true via IPv4 and IPv6. True as of 21:20 GMT-7. I didn't get your email until just now, so perhaps there was a problem that has since cleared up. -G
On Fri, Jun 17, 2011 at 09:23:08PM -0700, Graham Freeman wrote:
On 17 Jun 2011, at 20:43 GMT-0700, Jeremy Chadwick wrote:
Hey folks, is anyone aware of any problems happening right now with Layer42, or within the PAIX?
We're seeing issues reaching both www.google.com (going through the PAIX) as well as www.facebook.com (going through the PAIX). Packets not going through the PAIX appear to be working fine.
FWIW, my connectivity via Layer42's network (at their 1555 Plymouth (MV) facility) appears to be fine, including to www.'('facebook'|'google')'.com. Same appears true via IPv4 and IPv6.
True as of 21:20 GMT-7. I didn't get your email until just now, so perhaps there was a problem that has since cleared up.
The problem hasn't cleared up. Instead, it appears in both cases someone has induced reconvergence/route priority to route *around* the PAIX. It looks like San Jose is now exclusively in use. Here's what mtrs look like to both Google and Facebook; compare them to the previous. |------------------------------------------------------------------------------------------| | WinMTR statistics | | Host - % | Sent | Recv | Best | Avrg | Wrst | Last | |------------------------------------------------|------|------|------|------|------|------| | fw01.noc.sv2.tellme.com - 0 | 71 | 71 | 0 | 0 | 3 | 0 | | gw01.nocport.cns.sv2.tellme.com - 0 | 71 | 71 | 1 | 2 | 12 | 1 | | gw02.edgecross2.cns.sv2.tellme.com - 0 | 71 | 71 | 1 | 2 | 19 | 1 | | gw-tellme.sv.svcolo.com - 0 | 71 | 71 | 2 | 3 | 12 | 2 | | xe1-5-901.core1.sv1.layer42.net - 0 | 71 | 71 | 2 | 6 | 62 | 4 | | sv1.br01.sjc1.tfbnw.net - 0 | 71 | 71 | 3 | 6 | 83 | 3 | | ae1.bb02.sjc1.tfbnw.net - 0 | 71 | 71 | 4 | 5 | 23 | 5 | | ae2.dr01.snc5.tfbnw.net - 0 | 71 | 71 | 4 | 7 | 71 | 4 | | po510.csw02a.snc5.tfbnw.net - 0 | 71 | 71 | 4 | 5 | 12 | 4 | | www-13-02-snc5.facebook.com - 0 | 71 | 71 | 4 | 4 | 13 | 4 | |________________________________________________|______|______|______|______|______|______| |------------------------------------------------------------------------------------------| | WinMTR statistics | | Host - % | Sent | Recv | Best | Avrg | Wrst | Last | |------------------------------------------------|------|------|------|------|------|------| | fw01.noc.sv2.tellme.com - 0 | 21 | 21 | 0 | 0 | 2 | 1 | | gw01.nocport.cns.sv2.tellme.com - 0 | 21 | 21 | 1 | 1 | 12 | 4 | | gw02.edgecross1.cns.sv2.tellme.com - 0 | 21 | 21 | 1 | 3 | 11 | 1 | | gw-tellme.sv.svcolo.com - 0 | 21 | 21 | 3 | 4 | 12 | 7 | | xe1-5-901.core1.sv1.layer42.net - 0 | 21 | 21 | 3 | 5 | 12 | 8 | | eqixsj-google-gige.google.com - 0 | 21 | 21 | 3 | 7 | 56 | 10 | | 216.239.49.168 - 0 | 21 | 21 | 3 | 6 | 14 | 9 | | 64.233.174.119 - 0 | 21 | 21 | 3 | 5 | 12 | 10 | | 74.125.224.144 - 0 | 21 | 21 | 3 | 4 | 12 | 8 | |________________________________________________|______|______|______|______|______|______| So, anyone know what's going on in the PAIX? :-) -- | Jeremy Chadwick jdc at parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, US | | Making life hard for others since 1977. PGP 4BD6C0CB |
The problem hasn't cleared up. Instead, it appears in both cases someone has induced reconvergence/route priority to route *around* the PAIX. It looks like San Jose is now exclusively in use.
What you saw was probably related to this, as we saw a few peers flapping during that time period. Dear Equinix Customer, IBX(s): SV8 PRODUCTS: Equinix Internet Exchange SR#: 1755835 Date and Time of Occurrence: 06/17/2011 20:28:14 Site Local Time INCIDENT SUMMARY: Equinix Exchange INCIDENT DESCRIPTION: You may have experienced a brief interruption with your PAIX/Equinix Exchange services. This issue is now resolved and service returned to normal as of 22:00 Site Local Time. Sincerely, Equinix US NOC Phone: 888-892-0607 or 408-451-5200 -- Steve Rubin ser@layer42.net Layer42 Networks http://www.layer42.net/ Expect More from A Web Solutions Provider (408)450-5742
participants (3)
-
Graham Freeman -
Jeremy Chadwick -
Steve Rubin