Second Bandwidth Outage Sunday
Again today, on the heels of the 2h 47m DDoS-related outage yesterday, right at 11:01am US EDT (15:01:28 UTC), our monitoring alerted us to another incident at Bandwidth. The impact to us was much reduced this time, as we saw some flapping of their SIP Inbound SBCs, but around 15:20:12 UTC it was responsive enough to fall below our thresholds. The API started acting up at 14:56:00 UTC, but recovered, and then has been flapping between OK and degraded since then. As of 21:52 UTC, the API and Portal/Dashboard are either non-responsive, slow, or degraded. Inbound Voice Traffic has remaind at expected historical levels throughout. I have not checked on Termination, but it was working earlier today without issue. Latest from BW https://status.bandwidth.com/ Update - Bandwidth teams have observed expected call behavior for 911 calls. 911 calls are processing as designed and are fully restored. Teams continue to work on standard voice traffic. Sep 26, 17:20 EDT I question the phrase "processing as designed" which implies it is not working as the end user expects... odd. Beckman --------------------------------------------------------------------------- Peter Beckman Internet Guy beckman@angryox.com https://www.angryox.com/ ---------------------------------------------------------------------------
Processes 10,000 calls per second as designed, currently processing 10,000 calls exactly per second according to its internal statistics. What's the issue!?!?!! lol
On Sep 26, 2021, at 6:00 PM, Peter Beckman via Outages <outages@outages.org> wrote:
I question the phrase "processing as designed" which implies it is not working as the end user expects... odd.
Seems to be acting up again… https://status.bandwidth.com On Sun, Sep 26, 2021 at 18:12 Peter Beckman via Outages <outages@outages.org> wrote:
Again today, on the heels of the 2h 47m DDoS-related outage yesterday, right at 11:01am US EDT (15:01:28 UTC), our monitoring alerted us to another incident at Bandwidth. The impact to us was much reduced this time, as we saw some flapping of their SIP Inbound SBCs, but around 15:20:12 UTC it was responsive enough to fall below our thresholds.
The API started acting up at 14:56:00 UTC, but recovered, and then has been flapping between OK and degraded since then. As of 21:52 UTC, the API and Portal/Dashboard are either non-responsive, slow, or degraded.
Inbound Voice Traffic has remaind at expected historical levels throughout. I have not checked on Termination, but it was working earlier today without issue.
Latest from BW https://status.bandwidth.com/
Update - Bandwidth teams have observed expected call behavior for 911 calls. 911 calls are processing as designed and are fully restored. Teams continue to work on standard voice traffic.
Sep 26, 17:20 EDT
I question the phrase "processing as designed" which implies it is not working as the end user expects... odd.
Beckman --------------------------------------------------------------------------- Peter Beckman Internet Guy beckman@angryox.com https://www.angryox.com/ --------------------------------------------------------------------------- _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
i believe they are having issues again, not sure if its ddos related. but they are experiencing some major issues. https://status.bandwidth.com/ Update - Bandwidth teams are also investigating delays with Toll-Free Messaging at this time along with V2 Voice, Standard Voice, 911 traffic, Portal & API with intermittent service disruptions. Sep 27, 13:01 EDT Update - We continue to investigate this incident. Sep 27, 11:57 EDT Update - Bandwidth teams are also investigating delayed ports at this time along with V2 Voice, Standard Voice, 911 traffic, Portal & API with intermittent service disruptions. Sep 27, 11:07 EDT Update - We continue to investigate this incidnet. Sep 27, 11:02 EDT Update - Bandwidth teams are investigating an incident impacting Voice, V2 Voice, 911 traffic, Portal & API with intermittent service disruption. Sep 27, 10:00 EDT Update - Bandwidth teams are investigating an incident impacting Voice, V2 Voice, and 911 traffic with intermittent service disruption. Sep 27, 09:42 EDT Investigating - Bandwidth teams are investigating an incident impacting our V2 Voice services. Customers may notice intermittent service disruptions. Sep 27, 09:12 EDT On Sun, Sep 26, 2021 at 3:13 PM Peter Beckman via Outages < outages@outages.org> wrote:
Again today, on the heels of the 2h 47m DDoS-related outage yesterday, right at 11:01am US EDT (15:01:28 UTC), our monitoring alerted us to another incident at Bandwidth. The impact to us was much reduced this time, as we saw some flapping of their SIP Inbound SBCs, but around 15:20:12 UTC it was responsive enough to fall below our thresholds.
The API started acting up at 14:56:00 UTC, but recovered, and then has been flapping between OK and degraded since then. As of 21:52 UTC, the API and Portal/Dashboard are either non-responsive, slow, or degraded.
Inbound Voice Traffic has remaind at expected historical levels throughout. I have not checked on Termination, but it was working earlier today without issue.
Latest from BW https://status.bandwidth.com/
Update - Bandwidth teams have observed expected call behavior for 911 calls. 911 calls are processing as designed and are fully restored. Teams continue to work on standard voice traffic.
Sep 26, 17:20 EDT
I question the phrase "processing as designed" which implies it is not working as the end user expects... odd.
Beckman --------------------------------------------------------------------------- Peter Beckman Internet Guy beckman@angryox.com https://www.angryox.com/ --------------------------------------------------------------------------- _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
I’d agree. We’re intermittently losing e911, CNAM, and LD trunking with them to our Bandwidth.com Dallas and Atlanta peers. We’re out of New York state. Other symptoms seem to be DTMF frequently not working as well. [cid:thumbnail_Capture_2f362413-48ac-4bb3-b2ec-6157617ffdb8.png] Bryon Adams Business Unlimited Installation Technician + T 315 624 2023 + badams@northland.net + www.northland.net<http://www.northland.net> DATA + VOICE + EQUIPMENT SOLUTIONS [cid:fb_c6cc9b1e-0c26-4fb5-8303-05b4511df3a5.png]<http://www.facebook.com/NorthlandCommunications> [cid:in_30ee67ab-2a3f-4c62-9ee5-e2fc9c943c81.png] <https://www.linkedin.com/company/northland-communications-company/> [cid:tw_bcea77d6-ef61-47da-bbdf-93d704743c68.png] <https://twitter.com/NorthlandCom> <https://www.northland.net/lp/referral/> From: Outages <outages-bounces@outages.org> On Behalf Of Drew Matthews via Outages Sent: Monday, September 27, 2021 13:54 To: outages@outages.org Subject: Re: [outages] Second Bandwidth Outage Sunday i believe they are having issues again, not sure if its ddos related. but they are experiencing some major issues. https://status.bandwidth.com/ Update - Bandwidth teams are also investigating delays with Toll-Free Messaging at this time along with V2 Voice, Standard Voice, 911 traffic, Portal & API with intermittent service disruptions. Sep 27, 13:01 EDT Update - We continue to investigate this incident. Sep 27, 11:57 EDT Update - Bandwidth teams are also investigating delayed ports at this time along with V2 Voice, Standard Voice, 911 traffic, Portal & API with intermittent service disruptions. Sep 27, 11:07 EDT Update - We continue to investigate this incidnet. Sep 27, 11:02 EDT Update - Bandwidth teams are investigating an incident impacting Voice, V2 Voice, 911 traffic, Portal & API with intermittent service disruption. Sep 27, 10:00 EDT Update - Bandwidth teams are investigating an incident impacting Voice, V2 Voice, and 911 traffic with intermittent service disruption. Sep 27, 09:42 EDT Investigating - Bandwidth teams are investigating an incident impacting our V2 Voice services. Customers may notice intermittent service disruptions. Sep 27, 09:12 EDT On Sun, Sep 26, 2021 at 3:13 PM Peter Beckman via Outages <outages@outages.org<mailto:outages@outages.org>> wrote: Again today, on the heels of the 2h 47m DDoS-related outage yesterday, right at 11:01am US EDT (15:01:28 UTC), our monitoring alerted us to another incident at Bandwidth. The impact to us was much reduced this time, as we saw some flapping of their SIP Inbound SBCs, but around 15:20:12 UTC it was responsive enough to fall below our thresholds. The API started acting up at 14:56:00 UTC, but recovered, and then has been flapping between OK and degraded since then. As of 21:52 UTC, the API and Portal/Dashboard are either non-responsive, slow, or degraded. Inbound Voice Traffic has remaind at expected historical levels throughout. I have not checked on Termination, but it was working earlier today without issue. Latest from BW https://status.bandwidth.com/ Update - Bandwidth teams have observed expected call behavior for 911 calls. 911 calls are processing as designed and are fully restored. Teams continue to work on standard voice traffic. Sep 26, 17:20 EDT I question the phrase "processing as designed" which implies it is not working as the end user expects... odd. Beckman --------------------------------------------------------------------------- Peter Beckman Internet Guy beckman@angryox.com<mailto:beckman@angryox.com> https://www.angryox.com/ --------------------------------------------------------------------------- _______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages
Appears to be another outage again today! From: Outages <outages-bounces@outages.org> On Behalf Of Bryon Adams via Outages Sent: Monday, September 27, 2021 1:13 PM To: outages@outages.org Subject: Re: [outages] Second Bandwidth Outage Sunday External Message I’d agree. We’re intermittently losing e911, CNAM, and LD trunking with them to our Bandwidth.com Dallas and Atlanta peers. We’re out of New York state. Other symptoms seem to be DTMF frequently not working as well. [cid:thumbnail_Capture_2f362413-48ac-4bb3-b2ec-6157617ffdb8.png] Bryon Adams Business Unlimited Installation Technician + T 315 624 2023 + badams@northland.net<mailto:badams@northland.net> + www.northland.net<http://www.northland.net> DATA + VOICE + EQUIPMENT SOLUTIONS [cid:fb_c6cc9b1e-0c26-4fb5-8303-05b4511df3a5.png]<http://www.facebook.com/NorthlandCommunications> [cid:in_30ee67ab-2a3f-4c62-9ee5-e2fc9c943c81.png] <https://www.linkedin.com/company/northland-communications-company/> [cid:tw_bcea77d6-ef61-47da-bbdf-93d704743c68.png] <https://twitter.com/NorthlandCom> From: Outages <outages-bounces@outages.org<mailto:outages-bounces@outages.org>> On Behalf Of Drew Matthews via Outages Sent: Monday, September 27, 2021 13:54 To: outages@outages.org<mailto:outages@outages.org> Subject: Re: [outages] Second Bandwidth Outage Sunday i believe they are having issues again, not sure if its ddos related. but they are experiencing some major issues. https://status.bandwidth.com/ Update - Bandwidth teams are also investigating delays with Toll-Free Messaging at this time along with V2 Voice, Standard Voice, 911 traffic, Portal & API with intermittent service disruptions. Sep 27, 13:01 EDT Update - We continue to investigate this incident. Sep 27, 11:57 EDT Update - Bandwidth teams are also investigating delayed ports at this time along with V2 Voice, Standard Voice, 911 traffic, Portal & API with intermittent service disruptions. Sep 27, 11:07 EDT Update - We continue to investigate this incidnet. Sep 27, 11:02 EDT Update - Bandwidth teams are investigating an incident impacting Voice, V2 Voice, 911 traffic, Portal & API with intermittent service disruption. Sep 27, 10:00 EDT Update - Bandwidth teams are investigating an incident impacting Voice, V2 Voice, and 911 traffic with intermittent service disruption. Sep 27, 09:42 EDT Investigating - Bandwidth teams are investigating an incident impacting our V2 Voice services. Customers may notice intermittent service disruptions. Sep 27, 09:12 EDT On Sun, Sep 26, 2021 at 3:13 PM Peter Beckman via Outages <outages@outages.org<mailto:outages@outages.org>> wrote: Again today, on the heels of the 2h 47m DDoS-related outage yesterday, right at 11:01am US EDT (15:01:28 UTC), our monitoring alerted us to another incident at Bandwidth. The impact to us was much reduced this time, as we saw some flapping of their SIP Inbound SBCs, but around 15:20:12 UTC it was responsive enough to fall below our thresholds. The API started acting up at 14:56:00 UTC, but recovered, and then has been flapping between OK and degraded since then. As of 21:52 UTC, the API and Portal/Dashboard are either non-responsive, slow, or degraded. Inbound Voice Traffic has remaind at expected historical levels throughout. I have not checked on Termination, but it was working earlier today without issue. Latest from BW https://status.bandwidth.com/ Update - Bandwidth teams have observed expected call behavior for 911 calls. 911 calls are processing as designed and are fully restored. Teams continue to work on standard voice traffic. Sep 26, 17:20 EDT I question the phrase "processing as designed" which implies it is not working as the end user expects... odd. Beckman --------------------------------------------------------------------------- Peter Beckman Internet Guy beckman@angryox.com<mailto:beckman@angryox.com> https://www.angryox.com/ --------------------------------------------------------------------------- _______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages
It's a DDoS attack, sounds like it's still ongoing. https://www.zdnet.com/article/bandwidth-ceo-confirms-outages-caused-by-ddos-... [https://www.zdnet.com/a/hub/i/r/2021/02/22/a7b40545-4115-4c96-acc9-d292c6871c95/thumbnail/770x578/4eecfca6373858783753e71675d0a8be/cyber-ddos-globe-white.png]<https://www.zdnet.com/article/bandwidth-ceo-confirms-outages-caused-by-ddos-attack/> Bandwidth CEO confirms outages caused by DDoS attack | ZDNet<https://www.zdnet.com/article/bandwidth-ceo-confirms-outages-caused-by-ddos-attack/> Bleeping Computer was the first to report on Monday evening that Bandwidth.com was facing issues because of a distributed denial of service attack, which are routinely targeted at VoIP providers ... www.zdnet.com ________________________________ From: Outages <outages-bounces@outages.org> on behalf of Sean Salvadalena via Outages <outages@outages.org> Sent: Tuesday, September 28, 2021 10:07 AM To: outages@outages.org <outages@outages.org> Subject: [outages] FW: Second Bandwidth Outage Sunday Appears to be another outage again today! From: Outages <outages-bounces@outages.org> On Behalf Of Bryon Adams via Outages Sent: Monday, September 27, 2021 1:13 PM To: outages@outages.org Subject: Re: [outages] Second Bandwidth Outage Sunday External Message I’d agree. We’re intermittently losing e911, CNAM, and LD trunking with them to our Bandwidth.com Dallas and Atlanta peers. We’re out of New York state. Other symptoms seem to be DTMF frequently not working as well. [cid:thumbnail_Capture_2f362413-48ac-4bb3-b2ec-6157617ffdb8.png] Bryon Adams Business Unlimited Installation Technician + T 315 624 2023 + badams@northland.net<mailto:badams@northland.net> + www.northland.net<http://www.northland.net> DATA + VOICE + EQUIPMENT SOLUTIONS [cid:fb_c6cc9b1e-0c26-4fb5-8303-05b4511df3a5.png]<http://www.facebook.com/NorthlandCommunications> [cid:in_30ee67ab-2a3f-4c62-9ee5-e2fc9c943c81.png] <https://www.linkedin.com/company/northland-communications-company/> [cid:tw_bcea77d6-ef61-47da-bbdf-93d704743c68.png] <https://twitter.com/NorthlandCom> From: Outages <outages-bounces@outages.org<mailto:outages-bounces@outages.org>> On Behalf Of Drew Matthews via Outages Sent: Monday, September 27, 2021 13:54 To: outages@outages.org<mailto:outages@outages.org> Subject: Re: [outages] Second Bandwidth Outage Sunday i believe they are having issues again, not sure if its ddos related. but they are experiencing some major issues. https://status.bandwidth.com/ Update - Bandwidth teams are also investigating delays with Toll-Free Messaging at this time along with V2 Voice, Standard Voice, 911 traffic, Portal & API with intermittent service disruptions. Sep 27, 13:01 EDT Update - We continue to investigate this incident. Sep 27, 11:57 EDT Update - Bandwidth teams are also investigating delayed ports at this time along with V2 Voice, Standard Voice, 911 traffic, Portal & API with intermittent service disruptions. Sep 27, 11:07 EDT Update - We continue to investigate this incidnet. Sep 27, 11:02 EDT Update - Bandwidth teams are investigating an incident impacting Voice, V2 Voice, 911 traffic, Portal & API with intermittent service disruption. Sep 27, 10:00 EDT Update - Bandwidth teams are investigating an incident impacting Voice, V2 Voice, and 911 traffic with intermittent service disruption. Sep 27, 09:42 EDT Investigating - Bandwidth teams are investigating an incident impacting our V2 Voice services. Customers may notice intermittent service disruptions. Sep 27, 09:12 EDT On Sun, Sep 26, 2021 at 3:13 PM Peter Beckman via Outages <outages@outages.org<mailto:outages@outages.org>> wrote: Again today, on the heels of the 2h 47m DDoS-related outage yesterday, right at 11:01am US EDT (15:01:28 UTC), our monitoring alerted us to another incident at Bandwidth. The impact to us was much reduced this time, as we saw some flapping of their SIP Inbound SBCs, but around 15:20:12 UTC it was responsive enough to fall below our thresholds. The API started acting up at 14:56:00 UTC, but recovered, and then has been flapping between OK and degraded since then. As of 21:52 UTC, the API and Portal/Dashboard are either non-responsive, slow, or degraded. Inbound Voice Traffic has remaind at expected historical levels throughout. I have not checked on Termination, but it was working earlier today without issue. Latest from BW https://status.bandwidth.com/ Update - Bandwidth teams have observed expected call behavior for 911 calls. 911 calls are processing as designed and are fully restored. Teams continue to work on standard voice traffic. Sep 26, 17:20 EDT I question the phrase "processing as designed" which implies it is not working as the end user expects... odd. Beckman --------------------------------------------------------------------------- Peter Beckman Internet Guy beckman@angryox.com<mailto:beckman@angryox.com> https://www.angryox.com/ --------------------------------------------------------------------------- _______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages
participants (7)
-
Bret Clark -
Bryon Adams -
Dovid Bender -
Drew Matthews -
Paul Timmins -
Peter Beckman -
Sean Salvadalena