Does anyone actually moderate outages-discuss? I’ve tried to join multiple times over the years…. Mike Castoro | VP of IT Adrianna Papell Group | 646-723-3885 Chat with me in Teams<https://teams.microsoft.com/l/chat/0/0?users=mcastoro@adriannapapell.com> From: Outages <outages-bounces@outages.org> On Behalf Of Jason Kuehl via Outages Sent: Wednesday, June 10, 2020 10:01 AM To: Andy Ringsmuth <andy@andyring.com> Cc: Jason Kuehl via Outages <outages@outages.org> Subject: Re: [outages] IBM Cloud / NetworkLayer Outage External Email! Didnt know what was a list. On Wed, Jun 10, 2020 at 9:45 AM Andy Ringsmuth via Outages <outages@outages.org<mailto:outages@outages.org>> wrote: Could we move further discussion to outages-discuss please? ---- Andy Ringsmuth 5609 Harding Drive Lincoln, NE 68521-5831 (402) 304-0083 andy@andyring.com<mailto:andy@andyring.com> “Better even die free, than to live slaves.” - Frederick Douglas, 1863
On Jun 10, 2020, at 8:22 AM, Jason Kuehl via Outages <outages@outages.org<mailto:outages@outages.org>> wrote:
Someone didn't have proper protections in place it looks like.
On Wed, Jun 10, 2020 at 9:20 AM Mike Bolitho via Outages <outages@outages.org<mailto:outages@outages.org>> wrote: So another BGP highjacking...
- Mike Bolitho
On Wed, Jun 10, 2020, 5:40 AM Greg Dickinson via Outages <outages@outages.org<mailto:outages@outages.org>> wrote: Found this on Reddit as an initial RFO, so keep in mind the source (although r/sysadmin is usually pretty good):
“A 3rd party network provider was advertising routes which resulted in our WorldWide traffic becoming severely impeded. This led to IBM Cloud clients being unable to log-in to their accounts, greatly limited internet/DC connectivity and other significant network route related impacts. Network Specialists have made adjustments to route policies to restore network access, and alleviate the impacts. The overall incident lasted from 5:55pm - 9:30pm ET. We will be providing a fully detailed Customer Incident Report/Root Cause Analysis as soon as possible”
Greg Dickinson
Network Engineer
234 Goodwin Crest Drive, Suite 500
Birmingham, AL 35209
Office: 205.917.2407 | Mobile: 205.234.6427
Fax: 205.945.0515 |
greg.dickinson@bryantbank.com<mailto:greg.dickinson@bryantbank.com> | BryantBank.com
Unbeatable Service. Legendary Results.
Facebook | Twitter
From: Outages <outages-bounces@outages.org<mailto:outages-bounces@outages.org>> On Behalf Of Jason Kuehl via Outages Sent: Tuesday, June 9, 2020 8:38 PM To: James Brown <jbrown@easypost.com<mailto:jbrown@easypost.com>> Cc: outages@outages.org<mailto:outages@outages.org> Subject: Re: [outages] IBM Cloud / NetworkLayer Outage
This Message originates from outside Bryant Bank. Please use caution when opening this correspondence, attachments or hyperlinks (URLs). If you have questions, please contact IT Support. Thank you.
I can't wait for this post mortem.
On Tue, Jun 9, 2020 at 9:21 PM James Brown via Outages <outages@outages.org<mailto:outages@outages.org>> wrote:
As of a few minutes ago, all US SoftLayer (IBM Cloud) locations look normal to me.
On Tue, Jun 9, 2020 at 5:27 PM James Brown <jbrown@easypost.com<mailto:jbrown@easypost.com>> wrote:
Just wanted to share updates:
I got through to someone at IBM who confirmed the level of chaos over there. It looks like they've started deploying some mitigations because I'm seeing packet loss drop off but latency is now awful for all US locations. It looks like IBM has failed over all of their peering to go through their European facilities because every return packet from every POP is coming from lon03.networklayer.com<http://lon03.networklayer.com> or ams02.networklayer.com<http://ams02.networklayer.com> (via Telia).
On Tue, Jun 9, 2020 at 4:12 PM James Brown <jbrown@easypost.com<mailto:jbrown@easypost.com>> wrote:
It looks like IBM Cloud / SoftLayer / NetworkLayer is having a global outage since about 15:04 Pacific time affecting routing to AWS, Level 3 / CenturyLink, and a bunch of other providers. Has anyone been able to get more details?
--
James Brown
Network Engineer
--
James Brown
Engineer
--
James Brown
Engineer
_______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages
--
Sincerely,
Jason W Kuehl Cell 920-419-8983 jason.w.kuehl@gmail.com<mailto:jason.w.kuehl@gmail.com>
NOTICE: This electronic mail message and any files transmitted with it are intended exclusively for the individual or entity to which it is addressed. The message, together with any attachment, may contain confidential and/or privileged information. Any unauthorized review, use, print, save, copy, disclosure or distribution is strictly prohibited. If you have received this message in error, please immediately advise the sender by reply email and delete copies. Thank you. _______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages _______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages
-- Sincerely,
Jason W Kuehl Cell 920-419-8983 jason.w.kuehl@gmail.com<mailto:jason.w.kuehl@gmail.com> _______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages -- Sincerely, Jason W Kuehl Cell 920-419-8983 jason.w.kuehl@gmail.com<mailto:jason.w.kuehl@gmail.com>