August 2012 - Outages - Outages Lists

Telia problems in London
by Patrick W. Gilmore 26 Aug '12

26 Aug '12

Telia has routing problems in London. Some of their routers are leaving BGP sessions up & dropping all packets. Check your 1299 circuits / paths. -- TTFN, patrick

1 0

AUTO: Nora Kennedy/The Hub Group Inc. is out of the office. (returning 08/28/2012)
by Nora Kennedy 25 Aug '12

25 Aug '12

I am out of the office until 08/28/2012. Please contact Paula Kordell (pkordell(a)hubgroup.com) if your message requires attention before my return, including any issues with Blackberries or other wireless devices Note: This is an automated response to your message "Outages Digest, Vol 51, Issue 23" sent on 8/25/2012 11:00:04 AM. This is the only notification you will receive while this person is away.

1 0

FW: Sprint Outage - Chicago
by Frank Bulk 24 Aug '12

24 Aug '12

FYI, confirmation for what Colleen Velo reported this afternoon. -----Original Message----- From: John Schneider [mailto:str8steelerfan@gmail.com] Sent: Friday, August 24, 2012 3:27 PM To: nanog(a)nanog.org Subject: Sprint Outage - Chicago Sprint appears to be having major network issues in the Chicago area. Confirmed by sprint as CRS down in Chicago affecting multiple customers nationwide.

1 0

cogent to Cablevision issues in NYC?
by Wallace Keith 22 Aug '12

22 Aug '12

Have lost connectivity to a bunch of users in CT just before noon, and again at 12:39. Going up and down. Can't reach cablevision by phone (fast busy) Tracing route to ool-4b7fe759.static.optonline.net [75.127.231.89] over a maximum of 30 hops: 4 3 ms 3 ms 3 ms 63.251.130.157 5 3 ms 3 ms 3 ms 63.251.128.19 6 3 ms 3 ms 3 ms xe-0-1-0.bos11.ip4.tinet.net [77.67.77.53] 7 50 ms 9 ms 9 ms xe-8-3-0.nyc30.ip4.tinet.net [89.149.183.42] 8 9 ms 9 ms 9 ms te0-2-0-7.ccr21.jfk07.atlas.cogentco.com [154.54.10.141] 9 9 ms 9 ms 9 ms te0-5-0-5.ccr21.jfk02.atlas.cogentco.com [154.54.80.173] 10 10 ms 9 ms 9 ms te8-1.ccr01.jfk04.atlas.cogentco.com [154.54.47.18] 11 * * * Request timed out. 12 * * * Request timed out. 13

4 4

AUTO: Nora Kennedy/The Hub Group Inc. is out of the office. (returning 08/21/2012)
by Nora Kennedy 18 Aug '12

18 Aug '12

I am out of the office until 08/21/2012. Please contact Paula Kordell (pkordell(a)hubgroup.com) if your message requires attention before my return, including any issues with Blackberries or other wireless devices Note: This is an automated response to your message "Outages Digest, Vol 51, Issue 19" sent on 8/18/2012 11:00:06 AM. This is the only notification you will receive while this person is away.

1 0

Anyone seeing Sprint problems?
by Jeremy Charles 17 Aug '12

17 Aug '12

Has anyone noticed any problems with Sprint circuits this afternoon? We have four T1s that run from our site in Verona, WI to the Claranet POP in New York. The alarm lights have been on here in Verona for all four T1s for most of the last couple hours. (The last mile is a SONET ring provided by our ILEC, TDS Telecom. I understand it's entirely possible that the problem could be at our ILEC or the cross connect between Sprint and the ILEC, but I wouldn't expect many on this list to see the problem if it is within our ILEC. ;-) ) === Jeremy Charles, jcharles(a)epic.com Epic's Computer and Technology Services Division Phone: 608-271-9000, Fax 608-271-7237

2 2

Re: [outages] AT&T COMMON BACKBONE IS DOWN - AFFECTS ALL THEIR DNS
by Nick Olsen 16 Aug '12

16 Aug '12

I can confirm this is also causing MMS issues. My ATT Galaxy S3 is unable to send or receive MMS messages. Network APN contains an MMS proxy and MMS address of mmsc.mobile.att.net and proxy.mobile.att.net Both fail to resolve 90% of the time. Confirmed on every ATT based mobile I have access to. Nick Olsen Network Operations (855) FLSPEED x106 ---------------------------------------- From: "virendra rode" <virendra.rode(a)outages.org> Sent: Wednesday, August 15, 2012 12:33 PM To: outages(a)outages.org Subject: Re: [outages] AT&T COMMON BACKBONE IS DOWN - AFFECTS ALL THEIR DNS -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 yup, tracker.outages.org is seeing streams of hashtags in regards to ATT dns issues. Don't have much insight but hopefully someone will pipe up and tracker. Not sure if folks saw this, http://www.circleid.com/posts/20120814_tracking_outages/ regards, /virendra On 08/15/2012 09:06 AM, Peter Tarnowski wrote: > > > Peter Tarnowski Vice President - IT Infrastructure Fortress > Investment Group 1345 Avenue of the Americas 26th Floor New York, > NY 10105 > > 212-479-7047 > > This email and any attached documents (the "Information") are > confidential. The Information is solely intended for the person(s) > to whom it is being provided, and may not be reproduced, > redistributed or used for any other purpose. By accessing the > Information, you acknowledge and agree that you will not reproduce, > redistribute or make it available to any other person, in whole or > in part, except to your own professional advisors for the sole > purpose of having them assist you in evaluating the Information. > > > > _______________________________________________ Outages mailing > list Outages(a)outages.org > https://puck.nether.net/mailman/listinfo/outages > -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iF4EAREIAAYFAlArzkcACgkQ3HuimOHfh+GSAwD/TtDEGCMTUrCPmb6kg/niBOV3 VN55nZqU+6hW3ra06lMA/0545tA3I5jS3fXD8cvTJuVUwppA5g46kMMAIRLlAipe =7b2q -----END PGP SIGNATURE----- _______________________________________________ Outages mailing list Outages(a)outages.org https://puck.nether.net/mailman/listinfo/outages

6 10

Overnight Issues?
by sjk 16 Aug '12

16 Aug '12

I have reports from our overnight team of access issues, and am trying to track it down - as our monitoring reported no problems. I believe it is related to our XO transit link, and was wondering if anyone else experienced upstream XO issues around the Chicago area last night? Thanks -- Steve -- sjk(a)cupacoffee.net fingerprint: 1024D/89420B8E 2001-09-16 No one can understand the truth until he drinks of coffee's frothy goodness. ~Sheik Abd-al-Kadir

1 0

Level 3 Problems
by Dan Stralka 15 Aug '12

15 Aug '12

Seeing high latency on Level 3 out of northern New Jersey. We saw very slow website loads across many sites and a few page timeouts. We also see high latency between our Abovenet circuit and Level 3 where they peer. Level 3 tech support confirms latency problems and did say a lot of customers are calling about this. See traceroutes below: Level 3 Router: Type escape sequence to abort. Tracing the route to france.com (96.126.117.104) VRF info: (vrf in name/id, vrf out name/id) 1 159.63.23.213 1 msec 1 msec 1 msec 2 64.215.195.230 [AS 3549] 100 msec 103 msec 114 msec 3 ae-4-90.edge3.NewYork1.Level3.net (4.69.155.209) [AS 3356] 49 msec 38 msec ae-3-80.edge3.NewYork1.Level3.net (4.69.155.145) [AS 3356] 35 msec 4 te2-5.bbr01.tl01.nyc01.networklayer.com (4.26.34.22) [AS 3356] 99 msec 97 msec 98 msec 5 ae1.bbr01.eq01.chi01.networklayer.com (173.192.18.132) [AS 36351] [MPLS: Label 348745 Exp 0] 119 msec * 119 msec 6 ae20.bbr01.eq01.dal03.networklayer.com (173.192.18.136) [AS 36351] 77 msec 76 msec 128 msec 7 po31.dsr01.dllstx3.networklayer.com (173.192.18.225) [AS 36351] 76 msec 79 msec 80 msec 8 te2-4.dsr01.dllstx2.networklayer.com (70.87.255.110) [AS 21844] [MPLS: Label 991 Exp 0] 73 msec 69 msec 72 msec 9 po1.car01.dllstx2.networklayer.com (70.87.254.74) [AS 21844] 76 msec 76 msec 82 msec 10 router1-dal.linode.com (67.18.7.90) [AS 21844] 137 msec 142 msec * NJASR1#trace 4.69.151.49 Type escape sequence to abort. Tracing the route to ge-7-15-19.car1.Atlanta1.Level3.net (4.69.151.49) VRF info: (vrf in name/id, vrf out name/id) 1 159.63.23.213 1 msec 1 msec 1 msec 2 64.215.195.230 [AS 3549] 94 msec * * 3 vlan60.csw1.NewYork1.Level3.net (4.69.155.62) [AS 3356] 34 msec * 36 msec 4 ae-71-71.ebr1.NewYork1.Level3.net (4.69.134.69) [AS 3356] 30 msec ae-61-61.ebr1.NewYork1.Level3.net (4.69.134.65) [AS 3356] 100 msec ae-71-71.ebr1.NewYork1.Level3.net (4.69.134.69) [AS 3356] 34 msec 5 * ae-10-10.ebr2.Washington12.Level3.net (4.69.148.50) [AS 3356] 36 msec 46 msec 6 * ae-1-100.ebr1.Washington12.Level3.net (4.69.143.213) [AS 3356] 122 msec 108 msec 7 ae-6-6.ebr1.Atlanta2.Level3.net (4.69.148.105) [AS 3356] 132 msec * * 8 ae-11-51.car1.Atlanta1.Level3.net (4.69.150.3) [AS 3356] 23 msec * 23 msec Abovenet (for comparison): Type escape sequence to abort. Tracing the route to france.com (96.126.117.104) 1 209.66.88.145 1 msec 1 msec 1 msec 2 xe-4-1-0.cr1.lga5.us.above.net (64.125.31.170) [AS 6461] [MPLS: Label 677655 Exp 0] 1 msec 1 msec 1 msec 3 xe-1-0-1.er3.lga5.us.above.net (64.125.24.165) [AS 6461] 1 msec 1 msec 1 msec 4 above-ntt-2.lga5.us.above.net (64.125.13.182) [AS 6461] 1 msec 1 msec 1 msec 5 ae-3.r05.nycmny01.us.bb.gin.ntt.net (129.250.4.204) [AS 2914] 2 msec 1 msec 2 msec 6 xe-1-0-0.bbr01.tl01.nyc01.networklayer.com (165.254.16.2) [AS 2914] 1 msec 1 msec 1 msec 7 ae7.bbr02.tl01.nyc01.networklayer.com (173.192.18.177) [AS 36351] [MPLS: Label 400085 Exp 0] 1 msec 2 msec 1 msec 8 ae1.bbr01.eq01.chi01.networklayer.com (173.192.18.132) [AS 36351] [MPLS: Label 347833 Exp 0] 20 msec 40 msec 34 msec 9 ae20.bbr01.eq01.dal03.networklayer.com (173.192.18.136) [AS 36351] 44 msec 42 msec 45 msec 10 po31.dsr02.dllstx3.networklayer.com (173.192.18.227) [AS 36351] 42 msec po31.dsr01.dllstx3.networklayer.com (173.192.18.225) [AS 36351] 40 msec 41 msec 11 te2-4.dsr02.dllstx2.networklayer.com (70.87.255.126) [AS 21844] [MPLS: Label 504 Exp 0] 45 msec te2-4.dsr01.dllstx2.networklayer.com (70.87.255.110) [AS 21844] [MPLS: Label 991 Exp 0] 44 msec 40 msec 12 po2.car01.dllstx2.networklayer.com (70.87.254.78) [AS 21844] 40 msec po1.car01.dllstx2.networklayer.com (70.87.254.74) [AS 21844] 44 msec po2.car01.dllstx2.networklayer.com (70.87.254.78) [AS 21844] 43 msec 13 router1-dal.linode.com (67.18.7.90) [AS 21844] 42 msec 41 msec 44 msec Tracing the route to ge-7-15-19.car1.Atlanta1.Level3.net (4.69.151.49) 1 66.6.235.11 2 msec 2 msec 2 msec 2 209.66.88.149 [AS 6461] 2 msec 1 msec 2 msec 3 xe-3-3-0.cr2.lga5.us.above.net (64.125.28.218) [AS 6461] [MPLS: Label 769773 Exp 0] 2 msec 2 msec 1 msec 4 xe-2-2-0.cr2.dca2.us.above.net (64.125.26.105) [AS 6461] [MPLS: Label 590170 Exp 0] 47 msec 6 msec 6 msec 5 xe-1-0-1.er2.iad10.us.above.net (64.125.26.242) [AS 6461] 6 msec 7 msec 7 msec 6 above-level3.iad10.us.above.net (64.125.13.234) [AS 6461] 33 msec 32 msec 32 msec 7 vlan60.csw1.Washington1.Level3.net (4.69.149.62) [AS 3356] 105 msec vlan70.csw2.Washington1.Level3.net (4.69.149.126) [AS 3356] 45 msec * 8 ae-61-61.ebr1.Washington1.Level3.net (4.69.134.129) [AS 3356] 31 msec ae-81-81.ebr1.Washington1.Level3.net (4.69.134.137) [AS 3356] 20 msec ae-71-71.ebr1.Washington1.Level3.net (4.69.134.133) [AS 3356] 26 msec 9 ae-2-2.ebr3.Atlanta2.Level3.net (4.69.132.85) [AS 3356] 51 msec 51 msec 50 msec 10 4.69.148.242 [AS 3356] 125 msec ae-73-73.ebr2.Atlanta2.Level3.net (4.69.148.254) [AS 3356] 53 msec 48 msec 11 ae-11-51.car1.Atlanta1.Level3.net (4.69.150.3) [AS 3356] 61 msec * 44 msec Looking glass trace from Level 3: 1 ge-6-17-17.car1.Atlanta1.Level3.net (4.69.151.45) 12 msec ge-6-13-18.car2.Atlanta1.Level3.net (4.69.151.73) 16 msec ge-6-17-17.car1.Atlanta1.Level3.net (4.69.151.45) 12 msec 2 ae-2-52.edge5.Atlanta2.Level3.net (4.69.150.78) 28 msec * * 3 64.208.27.209 [AS3549 {GBLX}] 12 msec 12 msec 16 msec 4 67.16.131.134 [AS3549 {GBLX}] 60 msec 64 msec pos3-0-0-2488M.asr1.JFK1.gblx.net (67.16.131.138) [AS3549 {GBLX}] 128 msec 5 159.63.23.214 [AS3549 {GBLX}] 64 msec 60 msec 128 msec 6 * * 66.6.224.47 [AS40073 {NBA-ASN-USA}] !filtered -- Webmail

3 3

AT&T COMMON BACKBONE IS DOWN - AFFECTS ALL THEIR DNS
by Peter Tarnowski 15 Aug '12

15 Aug '12

Peter Tarnowski Vice President - IT Infrastructure Fortress Investment Group 1345 Avenue of the Americas 26th Floor New York, NY 10105 212-479-7047 This email and any attached documents (the "Information") are confidential. The Information is solely intended for the person(s) to whom it is being provided, and may not be reproduced, redistributed or used for any other purpose. By accessing the Information, you acknowledge and agree that you will not reproduce, redistribute or make it available to any other person, in whole or in part, except to your own professional advisors for the sole purpose of having them assist you in evaluating the Information.

2 1