Hi list, I'm seeing what appears to be a broken Layer 3 or 4 port channel (load balancing) somewhere in Twitter land. I have no idea if this is affecting anyone other than us as searching hasn't yielded anything, and Twitter's status page shows no issues. At the end of my message I've put some example of what I'm seeing. Behaviour changes based on source/dest IP, based on many customer and infrastructure addresses in different subnets, and a variety of the A records returned for twitter.com. We are multihomed so that might obscure the problem, I've included a sample out our 2nd provider. The problem appears to be somewhere past ae60.pao1-cr1.twttr.com (199.16.159.85) If anyone from Twitter is out there please drop me a line... we are AS 23252 If anyone else is experiencing this I'd love to know also Thanks --- Ross Halliday Network Operations WTC Communications Office: 613-547-6939 x203 Helpdesk: 866-547-6939 option 2 http://www.wtccommunications.ca edge-wprt-c6509#traceroute 199.59.150.39 source 66.102.70.237 Type escape sequence to abort. Tracing the route to r-199-59-150-39.twttr.com (199.59.150.39) 1 ge-6-1-117.car1.Toronto2.Level3.net (4.59.180.17) [AS 3356] 4 msec 4 msec 4 msec 2 4.69.153.77 [AS 3356] [MPLS: Label 493184 Exp 0] 4 msec 4 msec 4 msec 3 ae-0-11.bar2.Toronto1.Level3.net (4.69.151.242) [AS 3356] [MPLS: Label 492736 Exp 0] 4 msec 8 msec 4 msec 4 ae-7-1.ebr2.NewYork1.Level3.net (4.69.151.114) [AS 3356] [MPLS: Label 1494 Exp 0] 24 msec 24 msec 20 msec 5 ae-6-6.ebr2.NewYork2.Level3.net (4.69.141.22) [AS 3356] [MPLS: Label 1343 Exp 0] 16 msec 20 msec 16 msec 6 ae-1-100.ebr1.NewYork2.Level3.net (4.69.135.253) [AS 3356] [MPLS: Label 1278 Exp 0] 20 msec 24 msec 24 msec 7 ae-3-3.ebr2.Washington1.Level3.net (4.69.132.89) [AS 3356] [MPLS: Label 1355 Exp 0] 24 msec 24 msec 20 msec 8 ae-82-82.csw3.Washington1.Level3.net (4.69.134.154) [AS 3356] [MPLS: Label 1452 Exp 0] 24 msec ae-72-72.csw2.Washington1.Level3.net (4.69.134.150) [AS 3356] [MPLS: Label 1390 Exp 0] 24 msec ae-62-62.csw1.Washington1.Level3.net (4.69.134.146) [AS 3356] [MPLS: Label 1388 Exp 0] 24 msec 9 ae-2-70.edge2.Washington4.Level3.net (4.69.149.80) [AS 3356] 24 msec 24 msec ae-1-60.edge2.Washington4.Level3.net (4.69.149.16) [AS 3356] 36 msec 10 TWITTER-INC.edge2.Washington4.Level3.net (4.53.114.102) [AS 3356] 20 msec 20 msec 20 msec 11 ae60.pao1-cr1.twttr.com (199.16.159.85) [AS 13414] 96 msec 92 msec 92 msec 12 * * * 13 r-199-59-150-39.twttr.com (199.59.150.39) [AS 13414] 92 msec 100 msec 100 msec edge-wprt-c6509#traceroute 199.59.150.39 source 66.102.70.233 Type escape sequence to abort. Tracing the route to r-199-59-150-39.twttr.com (199.59.150.39) 1 ge-6-1-117.car1.Toronto2.Level3.net (4.59.180.17) [AS 3356] 4 msec 4 msec 4 msec 2 4.69.153.77 [AS 3356] [MPLS: Label 493184 Exp 0] 28 msec 32 msec 4 msec 3 ae-0-11.bar2.Toronto1.Level3.net (4.69.151.242) [AS 3356] [MPLS: Label 492736 Exp 0] 16 msec 36 msec 4 msec 4 ae-7-1.ebr2.NewYork1.Level3.net (4.69.151.114) [AS 3356] [MPLS: Label 1494 Exp 0] 24 msec 24 msec 24 msec 5 ae-6-6.ebr2.NewYork2.Level3.net (4.69.141.22) [AS 3356] [MPLS: Label 1343 Exp 0] 20 msec 16 msec 20 msec 6 ae-1-100.ebr1.NewYork2.Level3.net (4.69.135.253) [AS 3356] [MPLS: Label 1278 Exp 0] 16 msec 20 msec 16 msec 7 ae-3-3.ebr2.Washington1.Level3.net (4.69.132.89) [AS 3356] [MPLS: Label 1355 Exp 0] 20 msec 24 msec 24 msec 8 ae-92-92.csw4.Washington1.Level3.net (4.69.134.158) [AS 3356] [MPLS: Label 1396 Exp 0] 28 msec ae-72-72.csw2.Washington1.Level3.net (4.69.134.150) [AS 3356] [MPLS: Label 1390 Exp 0] 28 msec 24 msec 9 ae-4-90.edge2.Washington4.Level3.net (4.69.149.208) [AS 3356] 24 msec ae-2-70.edge2.Washington4.Level3.net (4.69.149.80) [AS 3356] 92 msec ae-3-80.edge2.Washington4.Level3.net (4.69.149.144) [AS 3356] 24 msec 10 TWITTER-INC.edge2.Washington4.Level3.net (4.53.114.102) [AS 3356] 20 msec 20 msec 24 msec 11 ae60.pao1-cr1.twttr.com (199.16.159.85) [AS 13414] 92 msec 92 msec 92 msec 12 * * * 13 * * * 14 * * * 15 * <snip> edge-wprt-c6509#traceroute 199.59.150.39 Type escape sequence to abort. Tracing the route to r-199-59-150-39.twttr.com (199.59.150.39) 1 ge5-1-5.core1.toronto1.nexicom.net (76.75.127.145) [AS 11666] 0 msec 8 msec 4 msec 2 xe2-2-0-1.core1.toronto1.nexicom.net (98.124.49.233) [AS 32881] 4 msec 4 msec 4 msec 3 206.126.236.97 [AS 8218] 20 msec 20 msec 20 msec 4 ae60.pao1-cr1.twttr.com (199.16.159.85) [AS 13414] 80 msec 80 msec 88 msec 5 * * * 6 * * * 7 * * * 8 * * * 9 * * * 10 * * * 11 * * * 12 * * * 13 * * * 14 * * *