February 2017 - Outages - Outages Lists

Amazon S3 down
by Carlos Alvarez 03 Mar '17

03 Mar '17

We are unable to reach it from Cox cable, Highwinds, or Cogent from the Phoenix area. Also several of the outage testing sites say it's down. -- Carlos Alvarez 602-368-6403

33 37

CenturyLink(old Qwest) issues in Seattle
by Matthew Blackstone 27 Feb '17

27 Feb '17

Getting reports from CenturyLink that an outage affecting our site is due to the loss of a core router in Seattle. Circuit down since Sunday. If anyone else is having issues in the area this may explain it. Matt Blackstone

1 0

Ziggo outage in Amsterdam, NL
by Simon Lockhart 26 Feb '17

26 Feb '17

We currently have multiple Ziggo circuits in Amsterdam showing major packet loss. They have been completely uncontactable by phone for the last 4 hours. Nothing obvious on their own twitter feeds, but others are reporting similar issues: https://twitter.com/ITTDesk/status/835541853072158720 Simon

2 2

Re: [outages] Charter issues this morning?
by Jeremy Charles 24 Feb '17

24 Feb '17

It seems like the Charter issue this morning was "regional" (perhaps a rather small "region"). It sounds like it primarily affected the Madison, WI area, which we hang off of (we're in Verona, WI). We re-enabled our BGP advertisements via Charter about 15 minutes ago and everything seems to be operating normally. Yes, we like to hold off for a while after the up-stream gives the all-clear, just in case there are some lingering issues that still need to shake out. Thanks to those who replied with observations and offers of assistance, both on and off-list. From: Jeremy Charles Sent: Friday, February 24, 2017 10:01 AM To: 'outages(a)outages.org' <outages(a)outages.org> Subject: RE: Charter issues this morning? Update received from Charter just a moment ago.... I think we'll give it a bit for things to settle down before we put traffic back on them. STATUS: The SE NOC has been advised that the bad fiber has been re-spliced at this time. Following the completion of these actions, both fibers have been verified and the original links have restored. If you have any questions or concerns please call 866-603-3199 and reference your ticket number. From: Jeremy Charles Sent: Friday, February 24, 2017 8:45 AM To: 'outages(a)outages.org' <outages(a)outages.org<mailto:outages@outages.org>> Subject: Charter issues this morning? Anyone else using Charter Internet have issues with connectivity this morning? We (AS 10359) have site-to-site VPN tunnels to about 150 customers across the US. We started seeing loss of connectivity to about 20% of them at about 06:25 CST. When we filtered out all advertisements sent/received via Charter, we regained connectivity to everyone. (We have two ISPs in addition to Charter.) Additionally, our total aggregate Internet consumption jumped from about 200 Mbps to 500 Mbps as soon as Charter was taken out of the mix. It looks like a lot of traffic was preferring to come to us via Charter (which is normal) but just wasn't making it here. One of our engineers waited on hold for Charter for about 20 minutes and then gave up to go get coffee. :-) [https://mrtg1.epic.com/mrtg/internet/inbound-borders.png] == Jeremy Charles Epic's Computer and Technology Services Division jcharles(a)epic.com<mailto:jcharles@epic.com> 608-271-9000

1 0

Re: [outages] Charter issues this morning?
by Thom Cleary 24 Feb '17

24 Feb '17

AS4150 here. Charter is better now. According to down detector: Andrew J Traxler <https://disqus.com/by/andrewjtraxler/>•14 minutes ago <http://downdetector.com/status/charter-communications/map/#comment-31728508…> Someone that works at Charter said there was system maintenance last night that went bad and they had to take everything down. No ETA. She said it was State Wide All of our peering stuff in Minneapolis is coming up.

2 1

Re: [outages] Charter issues this morning?
by Jeremy Charles 24 Feb '17

24 Feb '17

Update received from Charter just a moment ago.... I think we'll give it a bit for things to settle down before we put traffic back on them. STATUS: The SE NOC has been advised that the bad fiber has been re-spliced at this time. Following the completion of these actions, both fibers have been verified and the original links have restored. If you have any questions or concerns please call 866-603-3199 and reference your ticket number. From: Jeremy Charles Sent: Friday, February 24, 2017 8:45 AM To: 'outages(a)outages.org' <outages(a)outages.org> Subject: Charter issues this morning? Anyone else using Charter Internet have issues with connectivity this morning? We (AS 10359) have site-to-site VPN tunnels to about 150 customers across the US. We started seeing loss of connectivity to about 20% of them at about 06:25 CST. When we filtered out all advertisements sent/received via Charter, we regained connectivity to everyone. (We have two ISPs in addition to Charter.) Additionally, our total aggregate Internet consumption jumped from about 200 Mbps to 500 Mbps as soon as Charter was taken out of the mix. It looks like a lot of traffic was preferring to come to us via Charter (which is normal) but just wasn't making it here. One of our engineers waited on hold for Charter for about 20 minutes and then gave up to go get coffee. :-) [https://mrtg1.epic.com/mrtg/internet/inbound-borders.png] == Jeremy Charles Epic's Computer and Technology Services Division jcharles(a)epic.com<mailto:jcharles@epic.com> 608-271-9000

1 0

Charter issues this morning?
by Jeremy Charles 24 Feb '17

24 Feb '17

Anyone else using Charter Internet have issues with connectivity this morning? We (AS 10359) have site-to-site VPN tunnels to about 150 customers across the US. We started seeing loss of connectivity to about 20% of them at about 06:25 CST. When we filtered out all advertisements sent/received via Charter, we regained connectivity to everyone. (We have two ISPs in addition to Charter.) Additionally, our total aggregate Internet consumption jumped from about 200 Mbps to 500 Mbps as soon as Charter was taken out of the mix. It looks like a lot of traffic was preferring to come to us via Charter (which is normal) but just wasn't making it here. One of our engineers waited on hold for Charter for about 20 minutes and then gave up to go get coffee. :-) [https://mrtg1.epic.com/mrtg/internet/inbound-borders.png] == Jeremy Charles Epic's Computer and Technology Services Division jcharles(a)epic.com 608-271-9000

2 1

Re: [outages] Level 3 issues?
by Bart Boryczko 23 Feb '17

23 Feb '17

Updates here: https://www.reddit.com/r/sysadmin/comments/5vr0uf On Feb 23, 2017 1:30 PM, "Aaron C. de Bruyn via Outages" < outages(a)outages.org> wrote: Most (all?) of the traffic from the ISP that serves my house (AS21577) goes out Level 3 in Seattle. I've been running into packet loss and latency all morning long. Web meetings were miserable, various Google services are sluggish, SSH connections intermittently don't keep up with my typing, etc... Seems to come and go in bursts every 30-90 seconds and the lag lasts for ~10 seconds. -A On Thu, Feb 23, 2017 at 1:07 PM, Chris Bromley via Outages < outages(a)outages.org> wrote: > http://internetpulse.keynote.com/ > > Shows some issues between Level3 and XO/Verizon.. not sure how much stock > to put in that, depending on your pathway, though. > > On 2/23/2017 3:40 PM, outages--- via Outages wrote: > > We have started seeing some slowness on Level3 in Milwaukee, WI. > > On Thu, Feb 23, 2017 at 9:36 AM, Michael Snyder via Outages < > outages(a)outages.org> wrote: > >> I have sites in New York City, Austin and Omaha, no down, but extremely >> slow. Anyone else seeing issues? >> >> -- >> >> Michael Snyder >> >> Technical Lead | Infrastructure >> >> XO Group Inc. (NYSE: XOXO) >> >> >> >> [image: https://www.facebook.com/XOGroupInc] >> <https://www.facebook.com/XOGroupInc>[image: >> https://twitter.com/xogroupinc] <https://twitter.com/xogroupinc>[image: >> https://instagram.com/xogroupinc/] <https://instagram.com/xogroupinc/> >> _______________________________________________ Outages mailing list >> Outages(a)outages.org https://puck.nether.net/mailman/listinfo/outages > > _______________________________________________ > Outages mailing listOutages@outages.orghttps://puck.nether.net/mailman/listinfo/outages > > -- > Chris Bromley > Cloud Services Systems Administrator > TMW Systems Inc. | A TRIMBLE COMPANYwww.tmwsystems.com > > > This email and any files transmitted with it are confidential and intended > solely for the use of the addressee. If you are not the intended > addressee, then you have received this email in error and any use, > dissemination, forwarding, printing, or copying of this email is strictly > prohibited. > > _______________________________________________ > Outages mailing list > Outages(a)outages.org > https://puck.nether.net/mailman/listinfo/outages > > _______________________________________________ Outages mailing list Outages(a)outages.org https://puck.nether.net/mailman/listinfo/outages

1 0

Re: [outages] Level 3 issues?
by outages＠outages.org 23 Feb '17

23 Feb '17

3 2

Level 3 issues?
by outages＠outages.org 23 Feb '17

23 Feb '17

1 0