September 2021 - Outages

And, for your further amusement…
by Bill Woodcock 04 Oct '21

04 Oct '21

…apparently Slack still don’t know why they’re down. Probably because they try to use Slack to discuss it. https://status.slack.com/2021-09/06c1e17de93e7dc2 -Bill

15 15

Fwd: [dns-operations] slack.com bogus
by Bill Woodcock 30 Sep '21

30 Sep '21

Forwarding from elsewhere, for those who may be wondering why Slack is offline today. -Bill Begin forwarded message: > From: Peter van Dijk <peter.van.dijk(a)powerdns.com> > Date: September 30, 2021 at 8:16:26 PM GMT+2 > To: dns-operations(a)lists.dns-oarc.net > Subject: [dns-operations] slack.com bogus > > Hello, > > normally I would not send an email to dns-operations for a domain that > has gone bogus, but slack.com did something quite interesting today, > that quite looks like a failed attempt to sign, plus a panic withdraw > of that. > > Judging from dnsviz, a DS was present in the .com zone for slack.com > around 15:25 UTC today, and records inside slack.com were correctly > signed with the related KSK/ZSK set. > > Judging from the DS as I see it coming out of some resolvers, the DS is > about 15 hours old at this point (so, introduced around 03:15 UTC I > think). Those cached DSes still have 10 hours to go. > > However, the DS is gone from the .com zone, and the DNSKEYs and RRSIGs > are also gone. Hence, slack.com is Bogus for a big part of the Internet > viewer population. > > The text at https://status.slack.com/2021-09/06c1e17de93e7dc2 suggests > to me that Slack might not actually know what's going on. I do not have > a contact with them; perhaps somebody in here does? > > Kind regards, > -- > Peter van Dijk > PowerDNS.COM BV - https://www.powerdns.com/ > > _______________________________________________ > dns-operations mailing list > dns-operations(a)lists.dns-oarc.net > https://lists.dns-oarc.net/mailman/listinfo/dns-operations

1 0

Looks like BANDWIDTH is at it again.
by Michael R. Self 29 Sep '21

29 Sep '21

Getting alerts from our early warning systems. ————————— Best regards, Michael Self This email composed on my Google iPhone Essential Pixel S15+ on 7G technology while driving with my knees. Please excuse brevity and typos due to Siri’s hearing problems, accidents, cops or other mishaps.

3 5

Second Bandwidth Outage Sunday
by Peter Beckman 29 Sep '21

29 Sep '21

Again today, on the heels of the 2h 47m DDoS-related outage yesterday, right at 11:01am US EDT (15:01:28 UTC), our monitoring alerted us to another incident at Bandwidth. The impact to us was much reduced this time, as we saw some flapping of their SIP Inbound SBCs, but around 15:20:12 UTC it was responsive enough to fall below our thresholds. The API started acting up at 14:56:00 UTC, but recovered, and then has been flapping between OK and degraded since then. As of 21:52 UTC, the API and Portal/Dashboard are either non-responsive, slow, or degraded. Inbound Voice Traffic has remaind at expected historical levels throughout. I have not checked on Termination, but it was working earlier today without issue. Latest from BW https://status.bandwidth.com/ Update - Bandwidth teams have observed expected call behavior for 911 calls. 911 calls are processing as designed and are fully restored. Teams continue to work on standard voice traffic. Sep 26, 17:20 EDT I question the phrase "processing as designed" which implies it is not working as the end user expects... odd. Beckman --------------------------------------------------------------------------- Peter Beckman Internet Guy beckman(a)angryox.com https://www.angryox.com/ ---------------------------------------------------------------------------

7 6

Frontier Sarasota Florida EVPL outage
by Nick Olsen 27 Sep '21

27 Sep '21

Seeing high loss and latency (And in one case hard down) on multiple Frontier EVPL circuits in Sarasota Florida. Frontier states "We believe it's a fiber cut".

2 1

AWS problem with EBS in us-east-1
by Paul M - outages 27 Sep '21

27 Sep '21

https://status.aws.amazon.com/ is reporting EBS performance problems in us-east-1 or North Virginia. Not strictly an outage but it could be causing problems which may manifest downstream as an outage with dependent services. e.g. we had problems with some EC2 instances failing to start normally.

2 1

Fwd: Bandwidth Incident - Voice and Messaging Incident - 25 September 2021
by Bruce Freshwater 25 Sep '21

25 Sep '21

Things appear fully stable on our side also. -bruce ________________________________ From: Network-update(a)bandwidth.com <Network-update(a)bandwidth.com> Sent: Saturday, September 25, 2021 7:30:16 PM To: telephony(a)sierraexperts.com <telephony(a)sierraexperts.com> Subject: Bandwidth Incident - Voice and Messaging Incident - 25 September 2021 Monitoring - 36 components affected - Bandwidth teams have seen overall network... ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ Bandwidth Voice and Messaging Incident Incident status: Monitoring Bandwidth teams have seen overall network stability with expected traffic behavior. We will continue to monitor the network. Time posted Sep 25, 19:29 EDT Components affected [Operational] Inbound Calling Services - Inbound Ca... [Operational] Inbound Calling Services - Inbound Ca... [Operational] Inbound Calling Services - Toll-Free ... [Operational] Inbound Calling Services - REST APIs ... [Operational] Inbound Calling Services - REST APIs ... ...and 31 more components. View 7 previous incident updates<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fu12100069.ct.sendgrid.n…> You received this email because you are subscribed to Bandwidth's service status notifications. Manage your subscription<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fu12100069.ct.sendgrid.n…> Powered by Atlassian<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fu12100069.ct.sendgrid.n…> Statuspage<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fu12100069.ct.sendgrid.n…>

1 0

Fwd: Your Bandwidth support ticket is updated (Connection issue on trunk)
by Bruce Freshwater 25 Sep '21

25 Sep '21

>From the source -bruce ________________________________ From: Joe DaSilva (Bandwidth Support) <ticketing(a)bandwidth.com> Sent: Saturday, September 25, 2021 6:06:53 PM To: Ben Beige <bbeige(a)sierraexperts.com> Cc: Telephony <telephony(a)sierraexperts.com> Subject: Your Bandwidth support ticket is updated (Connection issue on trunk) [https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fgo.bandwidth.com%2frs%2…] Request #4174578<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fsupport.bandwidth.com%2…> has been updated. You can add a comment by replying to this email or by following this link: https://support.bandwidth.com/hc/requests/4174578<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fsupport.bandwidth.com%2…>. [Your current ticket status.] [https://mybwc.zendesk.com/system/photos/1318695047/joedasilva.png] Joe DaSilva (Bandwidth Support) Sep 25, 2021, 6:06 PM EDT Good afternoon, Bandwidth is currently experiencing a DDoS attack which is intermittently impacting our network. Our network operations and engineering teams are actively engaged in critical efforts to protect our network and fully restore all services as rapidly as possible. We will be sure to post key information and updates to status.bandwidth.com<https://linkprotect.cudasvc.com/url?a=http%3a%2f%2fstatus.bandwidth.com%2f&…> as our efforts progress and we have additional information to share. Thank you, Joe DaSilva TAC Manager [https://mybwc.zendesk.com/system/photos/1318695047/joedasilva.png] Joe DaSilva (Bandwidth Support) Sep 25, 2021, 4:34 PM EDT Good afternoon Ben, Thank you for contacting Bandwidth. Currently we are experiencing an incident impacting voice and messaging traffic on the Bandwidth network. We have updated our status page<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fstatus.bandwidth.com%2f…> to reflect this incent. We will provide additional updates via this ticket and on our status page as soon as possible. Best Regards Joseph DaSilva TAC Manager ________________________________ http://status.bandwidth.com/<https://linkprotect.cudasvc.com/url?a=http%3a%2f%2fstatus.bandwidth.com%2f&…> Bandwidth Status Welcome to Bandwidth's home for real-time and historical data on system performance. [https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fsecure.gravatar.com%2fa…] Ben Beige Sep 25, 2021, 4:32 PM EDT We're having issues connection to both 911 peers provided(67.231.12.71 & 67.231.1.137), and 216.82.224.36. 67.231.12.81 is still up though This email is a service from Bandwidth Support. [https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fgo.bandwidth.com%2frs%2…] THE POWER TO COMMUNICATE Privacy Notice<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.bandwidth.com%2fpri…> [OW2KZY-5ZEZ]

2 1

Bandwidth.com issues
by Steinbacher, Caleb 25 Sep '21

25 Sep '21

Seeing some inbound calling issues with our Bandwidth.com clients. Looking like they're having some issues, including 911 calls. https://status.bandwidth.com/ Let's hope this isn't the next VoIP.ms... -- Caleb Steinbacher Brand Ambassador NEPA Geeks p: 570-235-1946 e: caleb.steinbacher(a)nepageeks.com a: 237 Old River Road, Wilkes-Barre, PA 18702

2 1

Reddit very slow but not dead
by Paul M - outages 22 Sep '21

22 Sep '21

Slow page and app loading: https://www.redditstatus.com/ As per the list rules, web sites normally don't matter but a big service like reddit might raise complaints to customer services.

3 3