We witnessed something similar, but the time frame does not match up with what you provided. At 19:11 PST we witnessed packet loss to a Verizon router, followed by the physif on two of our routers going down (SONET links, OC12) at 19:17 PST. Things recovered by 19:19 PST. The physifs are directly tied to AT&T MPLS but for Verizon IP traffic. Our routers are located (physically) in an AT&T datacenter in Arizona. So in our case AT&T would be the LEC, not Verizon. Routers in question, with SONET pathing data (indicating router names and IP addresses of those AT&T routers on AT&T's side): router01: Last flapped : 2011-12-10 03:17:19 UTC (01:56:12 ago) Host name : phmaz31ck Interface : POS4/1 IP Address : 12.113.38.5 router02: Last flapped : 2011-12-10 03:17:19 UTC (01:57:32 ago) Host name : phmaz31ck Interface : POS4/2 IP Address : 12.113.38.9 I don't know if this is what you saw or not. I will point out, however, that AT&T has a very long and well-established history of "doing whatever, whenever", no matter how large the scale. A few years ago we witnessed them doing router OS upgrades across the entire United States over the course of two evenings, with absolutely no BGP preferencing announcements done prior; just straight up "bam, router down, too bad". I am not implying this is what AT&T was doing, but rather that they may have administratively done something to part of their mesh without bothering to tell anyone, thinking "nobody would notice". -- | Jeremy Chadwick jdc at parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, US | | Making life hard for others since 1977. PGP 4BD6C0CB | On Fri, Dec 09, 2011 at 10:38:52PM -0600, Stefan wrote:
Lost two sites, simultaneously, over 2.5 hrs ago, in PA area. MPLS services from AT&T, supposedly running with Verizon as LEC. Anybody else?!!
***Stefan Mititelu http://twitter.com/netfortius http://www.linkedin.com/in/netfortius _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages