August 2012 - Outages - Outages Lists

Linode outage in Newark, NJ starting at 3:30 pm
by Smith, Kyle 31 Aug '12

31 Aug '12

FYI, http://status.linode.com/ We started getting alerts at roughly ~3:30 - Kyle The information in this message is for the intended recipient(s) only and may be the proprietary and/or confidential property of Litle & Co., LLC, and thus protected from disclosure. If you are not the intended recipient(s), or an employee or agent responsible for delivering this message to the intended recipient, you are hereby notified that any use, dissemination, distribution or copying of this communication is prohibited. If you have received this communication in error, please notify Litle & Co. immediately by replying to this message and then promptly deleting it and your reply permanently from your computer.

11 12

Re: [outages] Linode outage in Newark, NJ starting at 3:30 pm
by Alex Rubenstein 31 Aug '12

31 Aug '12

The fault was related to an automatically operated circuit breaker on one of several generators. It would not close automatically, and had to be persuaded, which took some time. Further details will be made available to customers as soon as the post-event investigation is complete. ----- Original Message ----- From: outages-bounces(a)outages.org <outages-bounces(a)outages.org> To: reza a <reza(a)lethalnetworks.com>; outages(a)outages.org <outages(a)outages.org> Sent: Fri Aug 31 18:52:35 2012 Subject: Re: [outages] Linode outage in Newark, NJ starting at 3:30 pm It is common to have not only strings of batteries but an ATS, or automatic transfer switch. In this case, it sounds like the fault was on the ATS side of the plant based on the IR below. - LB -----Original Message----- From: outages-bounces(a)outages.org [mailto:outages-bounces@outages.org] On Behalf Of reza a Sent: Friday, August 31, 2012 3:29 PM To: outages(a)outages.org Subject: Re: [outages] Linode outage in Newark, NJ starting at 3:30 pm Is it common practice to have an array of batteries in between the colo and the generators to give you time to start the generators? ----- Original Message ----- From: "Sadiq Saif" <sadiq(a)asininetech.com> To: "Jack Carrozzo" <jack(a)crepinc.com> Cc: outages(a)outages.org Sent: Friday, August 31, 2012 2:59:16 PM Subject: Re: [outages] Linode outage in Newark, NJ starting at 3:30 pm This is the incident report from NAC: Subsequent to the utility power failure at our Cedar Knolls facility the generator that powers systems A and D did not start automatically. Manual intervention was required to get generator power running. Unfortunately before the generator was manually engaged customers on these systems experienced a complete power loss. We are still investigating why the generator did not engage automatically.Utility power has been restored, and transfer back to utility was successful. On Fri, Aug 31, 2012 at 3:59 PM, Jack Carrozzo <jack(a)crepinc.com> wrote: > 3:52pm (EDT): NAC has informed us of a power issue affecting at least > some portion of the datacenter. As soon as power is restored we are > poised to execute our recovery procedures to all affected systems. > > On Fri, Aug 31, 2012 at 3:46 PM, Smith, Kyle <ksmith(a)litle.com> wrote: >> FYI, >> >> >> >> http://status.linode.com/ >> >> >> >> We started getting alerts at roughly ~3:30 >> >> >> >> - Kyle >> >> >> >> The information in this message is for the intended recipient(s) only >> and may be the proprietary and/or confidential property of Litle & >> Co., LLC, and thus protected from disclosure. If you are not the >> intended recipient(s), or an employee or agent responsible for >> delivering this message to the intended recipient, you are hereby >> notified that any use, dissemination, distribution or copying of this >> communication is prohibited. If you have received this communication >> in error, please notify Litle & Co. immediately by replying to this >> message and then promptly deleting it and your reply permanently from your computer. >> >> _______________________________________________ >> Outages mailing list >> Outages(a)outages.org >> https://puck.nether.net/mailman/listinfo/outages >> > _______________________________________________ > Outages mailing list > Outages(a)outages.org > https://puck.nether.net/mailman/listinfo/outages -- Sadiq S O< ascii ribbon campaign - stop html mail - www.asciiribbon.org _______________________________________________ Outages mailing list Outages(a)outages.org https://puck.nether.net/mailman/listinfo/outages _______________________________________________ Outages mailing list Outages(a)outages.org https://puck.nether.net/mailman/listinfo/outages _______________________________________________ Outages mailing list Outages(a)outages.org https://puck.nether.net/mailman/listinfo/outages

2 1

Linode NJ Down
by Paul Norton 31 Aug '12

31 Aug '12

HI Folks - Just an FYI, My VPS on Linode NJ is down, confirmed facility wide issues on Linode status blog - http://status.linode.com/. According to that blog, they're doing an initial investigation and no ETA yet. -- Paul Norton Systems Administrator Neoverve - http://www.neoverve.com

4 4

Fwd: [Outages-discussion] Fibercut near Wallingford, CT
by virendra rode 31 Aug '12

31 Aug '12

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 - -------- Original Message -------- Subject: [Outages-discussion] Fibercut near Wallingford, CT Date: Fri, 31 Aug 2012 12:45:49 -0400 From: Tim Donahue <tim.donahue(a)gmail.com> To: outages-discussion(a)outages.org <outages-discussion(a)outages.org> Hi all, Sorry for the noise. Has anyone heard anything about a fibercut in or near Wallingford, CT? Tim -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://www.enigmail.net/ iF4EAREIAAYFAlBA638ACgkQ3HuimOHfh+ExPAD+JamBoIvFqeS2vUHY4/hXMAjD t/Rd1ZCpM/dbPLdqUfYA/jMnx6fU7k/yBi/Bf43jrF1VMchBtbBjtM790dLwrfYK =F5iT -----END PGP SIGNATURE-----

1 0

TWC outages
by Adam Greene 30 Aug '12

30 Aug '12

Time Warner seems to be having an issue affecting all accounts in the North East, as per their support line. From 14:45 - 15:00 EST we saw packet loss and high latency on our fiber circuit to them. Additionally, phone service provided over coax experienced a total outage. As of 15:00 the packet loss has ceased and dial tone is back, however, latency is very high (100ms - 800ms) to locations outside the TWC network. FYI.

3 3

Fwd: Re: [Outages-discussion] Fiber Cut in Southern NJ
by virendra rode 30 Aug '12

30 Aug '12

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 - -------- Original Message -------- Subject: Re: [Outages-discussion] Fiber Cut in Southern NJ Date: Thu, 30 Aug 2012 17:55:35 +0000 From: George Bonser <gbonser(a)seven.com> To: Jon Radel <jradel(a)vantage.com>, "outages-discussion(a)outages.org" <outages-discussion(a)outages.org> Seeing fiber get shot is normally something seen in the countryside during hunting season. Generally a few cases a year of that happening. Rare for the city. > -----Original Message----- From: > outages-discussion-bounces(a)outages.org [mailto:outages- > discussion-bounces(a)outages.org] On Behalf Of Jon Radel Sent: > Thursday, August 30, 2012 10:53 AM To: > outages-discussion(a)outages.org; tim.donahue(a)gmail.com Subject: Re: > [Outages-discussion] Fiber Cut in Southern NJ > > Ah, now we get some details: ------ > > We have further information regarding the cause of the fiber cut. > The fiber was apparently shot by a gun. Fibercut has been isolated > to 5700 feet from 401 N. Broad Street. Windstream crews will have > to resplice 72 of aerial fiber. They are currently still pulling > slack and will begin splicing soon, ETR is 2 hours. > > ------ > > I honestly didn't realize it was that type of neighborhood..... > > --Jon Radel jradel(a)vantage.com 267-756-1014 > > On 8/30/12 1:11 PM, Jon Radel wrote: >> Yes. We lost part of our XO MPLS-based network in the >> Philadelphia region. We've been told that there was a fiber cut >> involving at > least >> 72 strands. We've been told, as of 11am ET, that some repair >> has > been >> achieved but there's no ETR available for all splicing to be > completed. >> >> Anything interesting, such as exact location of the cut, other >> carriers involved, what happened, etc., have been carefully >> filtered out of what I'm being told. ;-) >> >> --Jon Radel jradel(a)vantage.com 267-756-1014 >> >> On 8/30/12 1:02 PM, Tim Donahue wrote: >>> Has anyone heard anything about a fibercut near the Southern >>> NJ/Philadelphia Area? We have been getting random error >>> messages on phone calls today and we heard a rumor that there >>> might be a outage in that area. >>> >>> Tim _______________________________________________ >>> Outages-discussion mailing list Outages-discussion(a)outages.org >>> https://puck.nether.net/mailman/listinfo/outages-discussion >> >> >> >> _______________________________________________ >> Outages-discussion mailing list Outages-discussion(a)outages.org >> https://puck.nether.net/mailman/listinfo/outages-discussion > > > > _______________________________________________ Outages-discussion > mailing list Outages-discussion(a)outages.org > https://puck.nether.net/mailman/listinfo/outages-discussion _______________________________________________ Outages-discussion mailing list Outages-discussion(a)outages.org https://puck.nether.net/mailman/listinfo/outages-discussion -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iF4EAREIAAYFAlA/qf0ACgkQ3HuimOHfh+H3QgD/Vg9usDsmH5plbTj6c0NuuUWV cjYhRr60y2+WZZ1nlT8A/jjf5IU5h4HZBKwV55lRmf/2Pnn/H2R34tLo3kySJs1e =d09T -----END PGP SIGNATURE-----

1 0

Verizon Business (SIP) One-way Audio
by Eric Wieling 29 Aug '12

29 Aug '12

Is anyone experiencing one-way audio on Verizon Business (SIP) service? We have multiple customers experiencing this. The SDP looks fine and we see two-way audio flowing, but one side can't hear anything. We are confident it is not an issue local to us, but it would be great to get confirmation that it is a larger outage. If we re-route to Level 3 the issue does not happen.

3 5

Level3 - Fiber Cut near Houston, TX
by Bill.Ingrum＠t-systems.com 29 Aug '12

29 Aug '12

Does anyone have information on a major fiber cut near Houston, TX? We've received generic updates from Level3 and Alpheus. Bill Ingrum

1 0

Re: [outages] Sprint Outage - Chicago
by Jay Ashworth 27 Aug '12

27 Aug '12

----- Original Message ----- > From: "Randy Bush" <randy(a)psg.com> > Subject: Re: Sprint Outage - Chicago > > You mean outages@... > > chris, this is not productive. outages are a very apt subject for > nanog. I'm pretty sure Chris was implying that "people were notifies by outages@, and not by the carrier's actual notification", Randy. Cheers, -- jr 'PINCH' a -- Jay R. Ashworth Baylink jra(a)baylink.com Designer The Things I Think RFC 2100 Ashworth & Associates http://baylink.pitas.com 2000 Land Rover DII St Petersburg FL USA #natog +1 727 647 1274

1 1

Re: [outages] Telia problems in London
by Patrick W. Gilmore 26 Aug '12

26 Aug '12

On Aug 26, 2012, at 10:06 , <paul.haering(a)thomsonreuters.com> wrote: > Is there any way to identify if and when Telia may have this issue corrected? When packet loss stops? :) More seriously, they claim to have already fixed it. -- TTFN, patrick > -----Original Message----- > From: outages-bounces(a)outages.org [mailto:outages-bounces@outages.org] On Behalf Of Patrick W. Gilmore > Sent: Sunday, August 26, 2012 8:51 AM > To: outages(a)outages.org > Subject: [outages] Telia problems in London > > Telia has routing problems in London. Some of their routers are leaving BGP sessions up & dropping all packets. > > Check your 1299 circuits / paths. > > -- > TTFN, > patrick > > > _______________________________________________ > Outages mailing list > Outages(a)outages.org > https://puck.nether.net/mailman/listinfo/outages >

1 0