It looks like IBM Cloud / SoftLayer / NetworkLayer is having a global outage since about 15:04 Pacific time affecting routing to AWS, Level 3 / CenturyLink, and a bunch of other providers. Has anyone been able to get more details? -- James Brown Network Engineer
https://cloud.ibm.com/status/ ================================= MAINTENANCE: IBM Cloud Monitoring will be unavailable during maintenance *Component: *IBM Cloud Monitoring with Sysdig *Location: *London *Start time: *9 Jun 2020 3:00 PM *End time: *9 Jun 2020 6:00 PM *Update time: *1 Jun 2020 12:19 PM ------------------------------ *Type: *Maintenance *ID: *CHG0683971 *What are we changing?* We are introducing changes to the IBM Cloud Monitoring with Sysdig service which require database maintenance that will require up to 30 minutes of downtime for the service. *Why are we making this change?* We are making changes to our service infrastructure to improve the service stability and to improve our ability to release non-disruptive changes in the future. These changes to our data storage infrastructure require an outage to ensure that customer data is not impacted by these changes. *How will it impact the environment?* During the maintenance window there will be up to 30 minutes of downtime while we perform the database maintenance. During this time the service will not be available. There will be gaps in the metric data once the service is restored for the outage period during the maintenance window. Alerting will be unavailable during the downtime and there may be some alerts triggered after the downtime due to the missing metric data depending on your alert configurations. *Maintenance Duration*: 180 minutes ================================================= JeffP jeffp@jeffp.us <jeffp@jeffp.us> *Please consider the environment before printing this e-mail.* NOTICE: This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. Use or dissemination by persons other than the intended recipient(s) is strictly prohibited. If you have received this message in error, please notify the sender immediately by reply e-mail to correct our records. Please then delete the original message (including any attachments) in its entirety, including any backup processes that your system may perform. Please note that any views or opinions presented in this email are solely those of the author. Email transmission cannot be guaranteed to be secure or error-free as information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or contain viruses. The sender, therefore, does not accept liability for any errors or omissions in the contents of this message, which arise as a result of e-mail transmission. On Tue, Jun 9, 2020 at 4:15 PM James Brown via Outages <outages@outages.org> wrote:
It looks like IBM Cloud / SoftLayer / NetworkLayer is having a global outage since about 15:04 Pacific time affecting routing to AWS, Level 3 / CenturyLink, and a bunch of other providers. Has anyone been able to get more details?
-- James Brown Network Engineer _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
I think that’s not what people are seeing, probably more of this sort of thing: PING softlayer.ip (softlayer.ip): 56 data bytes Request timeout for icmp_seq 0 Request timeout for icmp_seq 1 64 bytes from softlayer.ip: icmp_seq=2 ttl=46 time=361.999 ms Request timeout for icmp_seq 3 64 bytes from softlayer.ip: icmp_seq=4 ttl=46 time=353.756 ms 64 bytes from softlayer.ip: icmp_seq=5 ttl=46 time=355.315 ms Request timeout for icmp_seq 6 64 bytes from softlayer.ip: icmp_seq=7 ttl=46 time=355.523 ms Request timeout for icmp_seq 8 Request timeout for icmp_seq 9 Request timeout for icmp_seq 10 Request timeout for icmp_seq 11 Request timeout for icmp_seq 12 Request timeout for icmp_seq 13 64 bytes from softlayer.ip: icmp_seq=14 ttl=46 time=350.946 ms Request timeout for icmp_seq 15 ^C --- softlayer.ip ping statistics --- 17 packets transmitted, 5 packets received, 70.6% packet loss round-trip min/avg/max/stddev = 350.946/355.508/361.999/3.635 ms
On Jun 9, 2020, at 7:39 PM, Jeff P via Outages <outages@outages.org> wrote:
https://cloud.ibm.com/status/ <https://cloud.ibm.com/status/> ================================= MAINTENANCE: IBM Cloud Monitoring will be unavailable during maintenance Component: IBM Cloud Monitoring with Sysdig Location: London Start time: 9 Jun 2020 3:00 PM End time: 9 Jun 2020 6:00 PM Update time: 1 Jun 2020 12:19 PM Type: Maintenance ID: CHG0683971 What are we changing?
We are introducing changes to the IBM Cloud Monitoring with Sysdig service which require database maintenance that will require up to 30 minutes of downtime for the service.
Why are we making this change?
We are making changes to our service infrastructure to improve the service stability and to improve our ability to release non-disruptive changes in the future. These changes to our data storage infrastructure require an outage to ensure that customer data is not impacted by these changes.
How will it impact the environment?
During the maintenance window there will be up to 30 minutes of downtime while we perform the database maintenance. During this time the service will not be available. There will be gaps in the metric data once the service is restored for the outage period during the maintenance window. Alerting will be unavailable during the downtime and there may be some alerts triggered after the downtime due to the missing metric data depending on your alert configurations.
Maintenance Duration: 180 minutes =================================================
JeffP jeffp@jeffp.us <mailto:jeffp@jeffp.us>
Please consider the environment before printing this e-mail.
NOTICE: This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. Use or dissemination by persons other than the intended recipient(s) is strictly prohibited. If you have received this message in error, please notify the sender immediately by reply e-mail to correct our records. Please then delete the original message (including any attachments) in its entirety, including any backup processes that your system may perform. Please note that any views or opinions presented in this email are solely those of the author. Email transmission cannot be guaranteed to be secure or error-free as information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or contain viruses. The sender, therefore, does not accept liability for any errors or omissions in the contents of this message, which arise as a result of e-mail transmission.
On Tue, Jun 9, 2020 at 4:15 PM James Brown via Outages <outages@outages.org <mailto:outages@outages.org>> wrote: It looks like IBM Cloud / SoftLayer / NetworkLayer is having a global outage since about 15:04 Pacific time affecting routing to AWS, Level 3 / CenturyLink, and a bunch of other providers. Has anyone been able to get more details?
-- James Brown Network Engineer _______________________________________________ Outages mailing list Outages@outages.org <mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages <https://puck.nether.net/mailman/listinfo/outages> _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Looks like it's just cleared up for me. On 6/9/2020 6:54 PM, Charles Sprickman via Outages wrote:
I think that’s not what people are seeing, probably more of this sort of thing:
PING softlayer.ip (softlayer.ip): 56 data bytes Request timeout for icmp_seq 0 Request timeout for icmp_seq 1 64 bytes from softlayer.ip: icmp_seq=2 ttl=46 time=361.999 ms Request timeout for icmp_seq 3 64 bytes from softlayer.ip: icmp_seq=4 ttl=46 time=353.756 ms 64 bytes from softlayer.ip: icmp_seq=5 ttl=46 time=355.315 ms Request timeout for icmp_seq 6 64 bytes from softlayer.ip: icmp_seq=7 ttl=46 time=355.523 ms Request timeout for icmp_seq 8 Request timeout for icmp_seq 9 Request timeout for icmp_seq 10 Request timeout for icmp_seq 11 Request timeout for icmp_seq 12 Request timeout for icmp_seq 13 64 bytes from softlayer.ip: icmp_seq=14 ttl=46 time=350.946 ms Request timeout for icmp_seq 15 ^C --- softlayer.ip ping statistics --- 17 packets transmitted, 5 packets received, 70.6% packet loss round-trip min/avg/max/stddev = 350.946/355.508/361.999/3.635 ms
On Jun 9, 2020, at 7:39 PM, Jeff P via Outages <outages@outages.org <mailto:outages@outages.org>> wrote:
https://cloud.ibm.com/status/ ================================= MAINTENANCE: IBM Cloud Monitoring will be unavailable during maintenance *Component: *IBM Cloud Monitoring with Sysdig *Location: *London *Start time: *9 Jun 2020 3:00 PM *End time: *9 Jun 2020 6:00 PM *Update time: *1 Jun 2020 12:19 PM ------------------------------------------------------------------------ *Type: *Maintenance *ID: *CHG0683971
*What are we changing?*
We are introducing changes to the IBM Cloud Monitoring with Sysdig service which require database maintenance that will require up to 30 minutes of downtime for the service.
*Why are we making this change?*
We are making changes to our service infrastructure to improve the service stability and to improve our ability to release non-disruptive changes in the future. These changes to our data storage infrastructure require an outage to ensure that customer data is not impacted by these changes.
*How will it impact the environment?*
During the maintenance window there will be up to 30 minutes of downtime while we perform the database maintenance. During this time the service will not be available. There will be gaps in the metric data once the service is restored for the outage period during the maintenance window. Alerting will be unavailable during the downtime and there may be some alerts triggered after the downtime due to the missing metric data depending on your alert configurations.
*Maintenance Duration*: 180 minutes =================================================
JeffP jeffp@jeffp.us <mailto:jeffp@jeffp.us>
* Please consider the environment before printing this e-mail.*
NOTICE: This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. Use or dissemination by persons other than the intended recipient(s) is strictly prohibited. If you have received this message in error, please notify the sender immediately by reply e-mail to correct our records. Please then delete the original message (including any attachments) in its entirety, including any backup processes that your system may perform. Please note that any views or opinions presented in this email are solely those of the author. Email transmission cannot be guaranteed to be secure or error-free as information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or contain viruses. The sender, therefore, does not accept liability for any errors or omissions in the contents of this message, which arise as a result of e-mail transmission.
On Tue, Jun 9, 2020 at 4:15 PM James Brown via Outages <outages@outages.org <mailto:outages@outages.org>> wrote:
It looks like IBM Cloud / SoftLayer / NetworkLayer is having a global outage since about 15:04 Pacific time affecting routing to AWS, Level 3 / CenturyLink, and a bunch of other providers. Has anyone been able to get more details?
-- James Brown Network Engineer _______________________________________________ Outages mailing list Outages@outages.org <mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org <mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Just wanted to share updates: I got through to someone at IBM who confirmed the level of chaos over there. It looks like they've started deploying some mitigations because I'm seeing packet loss drop off but latency is now awful for all US locations. It looks like IBM has failed over all of their peering to go through their European facilities because every return packet from every POP is coming from lon03.networklayer.com or ams02.networklayer.com (via Telia). On Tue, Jun 9, 2020 at 4:12 PM James Brown <jbrown@easypost.com> wrote:
It looks like IBM Cloud / SoftLayer / NetworkLayer is having a global outage since about 15:04 Pacific time affecting routing to AWS, Level 3 / CenturyLink, and a bunch of other providers. Has anyone been able to get more details?
-- James Brown Network Engineer
-- James Brown Engineer
As of a few minutes ago, all US SoftLayer (IBM Cloud) locations look normal to me. On Tue, Jun 9, 2020 at 5:27 PM James Brown <jbrown@easypost.com> wrote:
Just wanted to share updates:
I got through to someone at IBM who confirmed the level of chaos over there. It looks like they've started deploying some mitigations because I'm seeing packet loss drop off but latency is now awful for all US locations. It looks like IBM has failed over all of their peering to go through their European facilities because every return packet from every POP is coming from lon03.networklayer.com or ams02.networklayer.com (via Telia).
On Tue, Jun 9, 2020 at 4:12 PM James Brown <jbrown@easypost.com> wrote:
It looks like IBM Cloud / SoftLayer / NetworkLayer is having a global outage since about 15:04 Pacific time affecting routing to AWS, Level 3 / CenturyLink, and a bunch of other providers. Has anyone been able to get more details?
-- James Brown Network Engineer
-- James Brown Engineer
-- James Brown Engineer
I can't wait for this post mortem. On Tue, Jun 9, 2020 at 9:21 PM James Brown via Outages <outages@outages.org> wrote:
As of a few minutes ago, all US SoftLayer (IBM Cloud) locations look normal to me.
On Tue, Jun 9, 2020 at 5:27 PM James Brown <jbrown@easypost.com> wrote:
Just wanted to share updates:
I got through to someone at IBM who confirmed the level of chaos over there. It looks like they've started deploying some mitigations because I'm seeing packet loss drop off but latency is now awful for all US locations. It looks like IBM has failed over all of their peering to go through their European facilities because every return packet from every POP is coming from lon03.networklayer.com or ams02.networklayer.com (via Telia).
On Tue, Jun 9, 2020 at 4:12 PM James Brown <jbrown@easypost.com> wrote:
It looks like IBM Cloud / SoftLayer / NetworkLayer is having a global outage since about 15:04 Pacific time affecting routing to AWS, Level 3 / CenturyLink, and a bunch of other providers. Has anyone been able to get more details?
-- James Brown Network Engineer
-- James Brown Engineer
-- James Brown Engineer _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
-- Sincerely, Jason W Kuehl Cell 920-419-8983 jason.w.kuehl@gmail.com
Found this on Reddit as an initial RFO, so keep in mind the source (although r/sysadmin is usually pretty good): “A 3rd party network provider was advertising routes which resulted in our WorldWide traffic becoming severely impeded. This led to IBM Cloud clients being unable to log-in to their accounts, greatly limited internet/DC connectivity and other significant network route related impacts. Network Specialists have made adjustments to route policies to restore network access, and alleviate the impacts. The overall incident lasted from 5:55pm - 9:30pm ET. We will be providing a fully detailed Customer Incident Report/Root Cause Analysis as soon as possible” Greg Dickinson Network Engineer [cid:image001.png@01D2DDE3.06E76B70] 234 Goodwin Crest Drive, Suite 500 Birmingham, AL 35209 Office: 205.917.2407 | Mobile: 205.234.6427 Fax: 205.945.0515 | greg.dickinson@bryantbank.com<mailto:greg.dickinson@bryantbank.com> | BryantBank.com<http://bryantbank.com/> Unbeatable Service. Legendary Results. Facebook<https://facebook.com/bryantbank/> | Twitter<https://twitter.com/bryantbank/> From: Outages <outages-bounces@outages.org> On Behalf Of Jason Kuehl via Outages Sent: Tuesday, June 9, 2020 8:38 PM To: James Brown <jbrown@easypost.com> Cc: outages@outages.org Subject: Re: [outages] IBM Cloud / NetworkLayer Outage This Message originates from outside Bryant Bank. Please use caution when opening this correspondence, attachments or hyperlinks (URLs). If you have questions, please contact IT Support. Thank you. I can't wait for this post mortem. On Tue, Jun 9, 2020 at 9:21 PM James Brown via Outages <outages@outages.org<mailto:outages@outages.org>> wrote: As of a few minutes ago, all US SoftLayer (IBM Cloud) locations look normal to me. On Tue, Jun 9, 2020 at 5:27 PM James Brown <jbrown@easypost.com<mailto:jbrown@easypost.com>> wrote: Just wanted to share updates: I got through to someone at IBM who confirmed the level of chaos over there. It looks like they've started deploying some mitigations because I'm seeing packet loss drop off but latency is now awful for all US locations. It looks like IBM has failed over all of their peering to go through their European facilities because every return packet from every POP is coming from lon03.networklayer.com<http://secure-web.cisco.com/1ZCENVaw45JMEdCoXCwG0rvb1UUCeCpWh4Gtfftz2u_gZYgdQhoKwrBfewM4QrECWqHUjB3ntgfnOudlN7Txq8mYVsGWJWeXWg-v2cvwS_INwBf66XwGN6PMKDQHpa9-nztcpzjJKAL45c9ehKsPd3RmZsKBti3c78NtK4_5orSmMjjftj2O5ihZtM3DI39J1jzcHEJkQ4qn1RaZRZvtck6dmc6mHvWEK07J6n5lfEiaFtNTzbRMqYg5PRkS5mXxpZdjs13BLwOYthPbwpG2y9sxQhA4vTT6kJ0WTImFe8A_gFn1O_YWB9bnGM_p6egB-/http%3A%2F%2Flon03.networklayer.com> or ams02.networklayer.com<http://secure-web.cisco.com/11XtT4bESggqtqUrb-8NP8jSqmiZcXpSKONGPf8fLiEIvSGY8GXfJOQvPgNH2fin3Mf1oX1qtIrUEhzncE_p6bafeLZfwQNedalFA2ZzWGHl31bfPAPv2AZR6vyTjFMCcWz92HYy6YIl1zhiQebTrb_76Ea-AEzmUrontPsu-oMbxR1e7nrBlRL1dTA7t3blZO8P3p58WDvMLWgLkm-fwMGANaDJ16Pw8BEmKXJJ-fHjl_4kXPUHTptOJJ2tBu8JYCQDVILZauSUz7kHx_JyJDNQEv4x9MojMPj2lnajAHeRvPjmy7vFPJnybP1MU8vGw/http%3A%2F%2Fams02.networklayer.com> (via Telia). On Tue, Jun 9, 2020 at 4:12 PM James Brown <jbrown@easypost.com<mailto:jbrown@easypost.com>> wrote: It looks like IBM Cloud / SoftLayer / NetworkLayer is having a global outage since about 15:04 Pacific time affecting routing to AWS, Level 3 / CenturyLink, and a bunch of other providers. Has anyone been able to get more details? -- James Brown Network Engineer -- James Brown Engineer -- James Brown Engineer _______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages<https://secure-web.cisco.com/1ALbUsNtj5hDIBcrx-EkowvDfnZpZsLIXlTU9xpfnU2i2nr2PgCQpVywQsS39qJHljpuu-05y4c4OqVp6vnUXfkwMjdFzdWAx6ufZY6amRzi-W8ZDQuOeeScc9jnHNYi4_XXaHMDw62k5a0GGAPi2e7kdcA7oL1IKNx_qZPZumgIs16f_vwL28g93YV-Kco95rDiKdE0bmSRoXy19iByqDxxST8yHpluFq-50ihMwmJnR5cX7NXCjKltHkLQJmNokhORvV20qvLxA6jaejf_FP1PGqUQYp8tZbyW7-wXmtTDlX2L3Pi07ilf9uVHRvOBhR9NhsuDyhRSpbieu3tNNcQ/https%3A%2F%2Fpuck.nether.net%2Fmailman%2Flistinfo%2Foutages> -- Sincerely, Jason W Kuehl Cell 920-419-8983 jason.w.kuehl@gmail.com<mailto:jason.w.kuehl@gmail.com> NOTICE: This electronic mail message and any files transmitted with it are intended exclusively for the individual or entity to which it is addressed. The message, together with any attachment, may contain confidential and/or privileged information. Any unauthorized review, use, print, save, copy, disclosure or distribution is strictly prohibited. If you have received this message in error, please immediately advise the sender by reply email and delete copies. Thank you.
FWIW: https [:] //www [.] computing.co.uk/news/4016300/ibm-cloud-suffers-global-outage-bringing-customer-websites Includes Twitter shots and links. IBM Cloud suffered a widespread outage on Tuesday that brought down multiple services hosted on the platform, as well as IBM's own cloud status page. The problem likely started around 2:30pm Pacific Time (PT) and escalated at around 5:30pm PT. Quickly, it started to impact services such as Cloud Object Storage, App Connect, Kubernetes Service, Continuous Delivery, Identity and Access Management (IAM), VPN for VPC, and Watson AI cloud services. IBM's own status page also failed to load during the outage and returned an internal server error message. DownDetector.com, a website that monitors website outages, showed that the outage was impacting IBM Cloud services in different parts of the US, South America, Japan and Australia. IBM's cloud-based streaming and file transfer services, which come under the Aspera brand, were also mostly out. IBM Aspera, an IBM unit that operates under hybrid cloud business, said that following the first outage, it was alerted to a service disruption affecting all regions. Aspera further revealed that AoC Managed Storage was suffering "major outage" in Dallas, Amsterdam, Frankfurt, Melbourne, and Toronto. "Our engineers are currently investigating the incident and we will provide updates when more information is available," an Aspera advisory stated. IBM Cloud Transfer Clusters in Frankfurt, Washington DC, Hong Kong, Chennai, London, Montreal, Mexico, Milan, Oslo, Seoul, San Jose, Sao Paulo, Tokyo, Paris, and Singapore were also listed as suffering major outages. It also emerged that IBM Cloud Monitoring service Sysdig had a three-hour maintenance window earlier in the day at its London facilities. IBM had informed customers that Sysdig maintenance work would be completed between 3pm to 6pm, and that it would cause only a 30-minute disruption to Cloud Monitoring service. IBM Cloud eventually posted an update on Twitter, stating that it was aware of the issues and that the issue was being investigated. At present, the issue appears to have been fully resolved. IBM also said in a tweet at around 7:00pm PT that all IBM Cloud services are now working normally. - andy From: Outages <outages-bounces@outages.org> on behalf of Greg Dickinson via Outages <outages@outages.org> Reply-To: Greg Dickinson <Greg.Dickinson@bryantbank.com> Date: Wednesday, June 10, 2020 at 8:44 AM To: "'outages@outages.org'" <outages@outages.org> Subject: Re: [outages] IBM Cloud / NetworkLayer Outage Found this on Reddit as an initial RFO, so keep in mind the source (although r/sysadmin is usually pretty good): “A 3rd party network provider was advertising routes which resulted in our WorldWide traffic becoming severely impeded. This led to IBM Cloud clients being unable to log-in to their accounts, greatly limited internet/DC connectivity and other significant network route related impacts. Network Specialists have made adjustments to route policies to restore network access, and alleviate the impacts. The overall incident lasted from 5:55pm - 9:30pm ET. We will be providing a fully detailed Customer Incident Report/Root Cause Analysis as soon as possible” Greg Dickinson Network Engineer [cid:image001.png@01D63F03.CAA2D890] 234 Goodwin Crest Drive, Suite 500 Birmingham, AL 35209 Office: 205.917.2407 | Mobile: 205.234.6427 Fax: 205.945.0515 | greg.dickinson@bryantbank.com<mailto:greg.dickinson@bryantbank.com> | BryantBank.com<http://bryantbank.com/> Unbeatable Service. Legendary Results. Facebook<https://facebook.com/bryantbank/> | Twitter<https://twitter.com/bryantbank/> From: Outages <outages-bounces@outages.org> On Behalf Of Jason Kuehl via Outages Sent: Tuesday, June 9, 2020 8:38 PM To: James Brown <jbrown@easypost.com> Cc: outages@outages.org Subject: Re: [outages] IBM Cloud / NetworkLayer Outage This Message originates from outside Bryant Bank. Please use caution when opening this correspondence, attachments or hyperlinks (URLs). If you have questions, please contact IT Support. Thank you. I can't wait for this post mortem. On Tue, Jun 9, 2020 at 9:21 PM James Brown via Outages <outages@outages.org<mailto:outages@outages.org>> wrote: As of a few minutes ago, all US SoftLayer (IBM Cloud) locations look normal to me. On Tue, Jun 9, 2020 at 5:27 PM James Brown <jbrown@easypost.com<mailto:jbrown@easypost.com>> wrote: Just wanted to share updates: I got through to someone at IBM who confirmed the level of chaos over there. It looks like they've started deploying some mitigations because I'm seeing packet loss drop off but latency is now awful for all US locations. It looks like IBM has failed over all of their peering to go through their European facilities because every return packet from every POP is coming from lon03.networklayer.com<http://secure-web.cisco.com/1ZCENVaw45JMEdCoXCwG0rvb1UUCeCpWh4Gtfftz2u_gZYgdQhoKwrBfewM4QrECWqHUjB3ntgfnOudlN7Txq8mYVsGWJWeXWg-v2cvwS_INwBf66XwGN6PMKDQHpa9-nztcpzjJKAL45c9ehKsPd3RmZsKBti3c78NtK4_5orSmMjjftj2O5ihZtM3DI39J1jzcHEJkQ4qn1RaZRZvtck6dmc6mHvWEK07J6n5lfEiaFtNTzbRMqYg5PRkS5mXxpZdjs13BLwOYthPbwpG2y9sxQhA4vTT6kJ0WTImFe8A_gFn1O_YWB9bnGM_p6egB-/http%3A%2F%2Flon03.networklayer.com> or ams02.networklayer.com<http://secure-web.cisco.com/11XtT4bESggqtqUrb-8NP8jSqmiZcXpSKONGPf8fLiEIvSGY8GXfJOQvPgNH2fin3Mf1oX1qtIrUEhzncE_p6bafeLZfwQNedalFA2ZzWGHl31bfPAPv2AZR6vyTjFMCcWz92HYy6YIl1zhiQebTrb_76Ea-AEzmUrontPsu-oMbxR1e7nrBlRL1dTA7t3blZO8P3p58WDvMLWgLkm-fwMGANaDJ16Pw8BEmKXJJ-fHjl_4kXPUHTptOJJ2tBu8JYCQDVILZauSUz7kHx_JyJDNQEv4x9MojMPj2lnajAHeRvPjmy7vFPJnybP1MU8vGw/http%3A%2F%2Fams02.networklayer.com> (via Telia). On Tue, Jun 9, 2020 at 4:12 PM James Brown <jbrown@easypost.com<mailto:jbrown@easypost.com>> wrote: It looks like IBM Cloud / SoftLayer / NetworkLayer is having a global outage since about 15:04 Pacific time affecting routing to AWS, Level 3 / CenturyLink, and a bunch of other providers. Has anyone been able to get more details? -- James Brown Network Engineer -- James Brown Engineer -- James Brown Engineer _______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages<https://secure-web.cisco.com/1ALbUsNtj5hDIBcrx-EkowvDfnZpZsLIXlTU9xpfnU2i2nr2PgCQpVywQsS39qJHljpuu-05y4c4OqVp6vnUXfkwMjdFzdWAx6ufZY6amRzi-W8ZDQuOeeScc9jnHNYi4_XXaHMDw62k5a0GGAPi2e7kdcA7oL1IKNx_qZPZumgIs16f_vwL28g93YV-Kco95rDiKdE0bmSRoXy19iByqDxxST8yHpluFq-50ihMwmJnR5cX7NXCjKltHkLQJmNokhORvV20qvLxA6jaejf_FP1PGqUQYp8tZbyW7-wXmtTDlX2L3Pi07ilf9uVHRvOBhR9NhsuDyhRSpbieu3tNNcQ/https%3A%2F%2Fpuck.nether.net%2Fmailman%2Flistinfo%2Foutages> -- Sincerely, Jason W Kuehl Cell 920-419-8983 jason.w.kuehl@gmail.com<mailto:jason.w.kuehl@gmail.com> NOTICE: This electronic mail message and any files transmitted with it are intended exclusively for the individual or entity to which it is addressed. The message, together with any attachment, may contain confidential and/or privileged information. Any unauthorized review, use, print, save, copy, disclosure or distribution is strictly prohibited. If you have received this message in error, please immediately advise the sender by reply email and delete copies. Thank you.
So another BGP highjacking... - Mike Bolitho On Wed, Jun 10, 2020, 5:40 AM Greg Dickinson via Outages < outages@outages.org> wrote:
Found this on Reddit as an initial RFO, so keep in mind the source (although r/sysadmin is usually pretty good):
“A 3rd party network provider was advertising routes which resulted in our WorldWide traffic becoming severely impeded. This led to IBM Cloud clients being unable to log-in to their accounts, greatly limited internet/DC connectivity and other significant network route related impacts. Network Specialists have made adjustments to route policies to restore network access, and alleviate the impacts. The overall incident lasted from 5:55pm - 9:30pm ET. We will be providing a fully detailed Customer Incident Report/Root Cause Analysis as soon as possible”
Greg Dickinson
Network Engineer
[image: cid:image001.png@01D2DDE3.06E76B70]
234 Goodwin Crest Drive, Suite 500
Birmingham, AL 35209
Office: 205.917.2407 | Mobile: 205.234.6427
Fax: 205.945.0515 |
greg.dickinson@bryantbank.com | BryantBank.com <http://bryantbank.com/>
Unbeatable Service. Legendary Results.
Facebook <https://facebook.com/bryantbank/> | Twitter <https://twitter.com/bryantbank/>
*From:* Outages <outages-bounces@outages.org> *On Behalf Of *Jason Kuehl via Outages *Sent:* Tuesday, June 9, 2020 8:38 PM *To:* James Brown <jbrown@easypost.com> *Cc:* outages@outages.org *Subject:* Re: [outages] IBM Cloud / NetworkLayer Outage
This Message originates from* outside* Bryant Bank. Please use *caution* when opening this correspondence, attachments or hyperlinks (URLs). If you have questions, please contact IT Support. Thank you.
I can't wait for this post mortem.
On Tue, Jun 9, 2020 at 9:21 PM James Brown via Outages < outages@outages.org> wrote:
As of a few minutes ago, all US SoftLayer (IBM Cloud) locations look normal to me.
On Tue, Jun 9, 2020 at 5:27 PM James Brown <jbrown@easypost.com> wrote:
Just wanted to share updates:
I got through to someone at IBM who confirmed the level of chaos over there. It looks like they've started deploying some mitigations because I'm seeing packet loss drop off but latency is now awful for all US locations. It looks like IBM has failed over all of their peering to go through their European facilities because every return packet from every POP is coming from lon03.networklayer.com <http://secure-web.cisco.com/1ZCENVaw45JMEdCoXCwG0rvb1UUCeCpWh4Gtfftz2u_gZYgdQhoKwrBfewM4QrECWqHUjB3ntgfnOudlN7Txq8mYVsGWJWeXWg-v2cvwS_INwBf66XwGN6PMKDQHpa9-nztcpzjJKAL45c9ehKsPd3RmZsKBti3c78NtK4_5orSmMjjftj2O5ihZtM3DI39J1jzcHEJkQ4qn1RaZRZvtck6dmc6mHvWEK07J6n5lfEiaFtNTzbRMqYg5PRkS5mXxpZdjs13BLwOYthPbwpG2y9sxQhA4vTT6kJ0WTImFe8A_gFn1O_YWB9bnGM_p6egB-/http%3A%2F%2Flon03.networklayer.com> or ams02.networklayer.com <http://secure-web.cisco.com/11XtT4bESggqtqUrb-8NP8jSqmiZcXpSKONGPf8fLiEIvSGY8GXfJOQvPgNH2fin3Mf1oX1qtIrUEhzncE_p6bafeLZfwQNedalFA2ZzWGHl31bfPAPv2AZR6vyTjFMCcWz92HYy6YIl1zhiQebTrb_76Ea-AEzmUrontPsu-oMbxR1e7nrBlRL1dTA7t3blZO8P3p58WDvMLWgLkm-fwMGANaDJ16Pw8BEmKXJJ-fHjl_4kXPUHTptOJJ2tBu8JYCQDVILZauSUz7kHx_JyJDNQEv4x9MojMPj2lnajAHeRvPjmy7vFPJnybP1MU8vGw/http%3A%2F%2Fams02.networklayer.com> (via Telia).
On Tue, Jun 9, 2020 at 4:12 PM James Brown <jbrown@easypost.com> wrote:
It looks like IBM Cloud / SoftLayer / NetworkLayer is having a global outage since about 15:04 Pacific time affecting routing to AWS, Level 3 / CenturyLink, and a bunch of other providers. Has anyone been able to get more details?
--
James Brown
Network Engineer
--
James Brown
Engineer
--
James Brown
Engineer
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages <https://secure-web.cisco.com/1ALbUsNtj5hDIBcrx-EkowvDfnZpZsLIXlTU9xpfnU2i2nr2PgCQpVywQsS39qJHljpuu-05y4c4OqVp6vnUXfkwMjdFzdWAx6ufZY6amRzi-W8ZDQuOeeScc9jnHNYi4_XXaHMDw62k5a0GGAPi2e7kdcA7oL1IKNx_qZPZumgIs16f_vwL28g93YV-Kco95rDiKdE0bmSRoXy19iByqDxxST8yHpluFq-50ihMwmJnR5cX7NXCjKltHkLQJmNokhORvV20qvLxA6jaejf_FP1PGqUQYp8tZbyW7-wXmtTDlX2L3Pi07ilf9uVHRvOBhR9NhsuDyhRSpbieu3tNNcQ/https%3A%2F%2Fpuck.nether.net%2Fmailman%2Flistinfo%2Foutages>
--
Sincerely,
Jason W Kuehl Cell 920-419-8983 jason.w.kuehl@gmail.com
NOTICE: This electronic mail message and any files transmitted with it are intended exclusively for the individual or entity to which it is addressed. The message, together with any attachment, may contain confidential and/or privileged information. Any unauthorized review, use, print, save, copy, disclosure or distribution is strictly prohibited. If you have received this message in error, please immediately advise the sender by reply email and delete copies. Thank you. _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Someone didn't have proper protections in place it looks like. On Wed, Jun 10, 2020 at 9:20 AM Mike Bolitho via Outages < outages@outages.org> wrote:
So another BGP highjacking...
- Mike Bolitho
On Wed, Jun 10, 2020, 5:40 AM Greg Dickinson via Outages < outages@outages.org> wrote:
Found this on Reddit as an initial RFO, so keep in mind the source (although r/sysadmin is usually pretty good):
“A 3rd party network provider was advertising routes which resulted in our WorldWide traffic becoming severely impeded. This led to IBM Cloud clients being unable to log-in to their accounts, greatly limited internet/DC connectivity and other significant network route related impacts. Network Specialists have made adjustments to route policies to restore network access, and alleviate the impacts. The overall incident lasted from 5:55pm - 9:30pm ET. We will be providing a fully detailed Customer Incident Report/Root Cause Analysis as soon as possible”
Greg Dickinson
Network Engineer
[image: cid:image001.png@01D2DDE3.06E76B70]
234 Goodwin Crest Drive, Suite 500
Birmingham, AL 35209
Office: 205.917.2407 | Mobile: 205.234.6427
Fax: 205.945.0515 |
greg.dickinson@bryantbank.com | BryantBank.com <http://bryantbank.com/>
Unbeatable Service. Legendary Results.
Facebook <https://facebook.com/bryantbank/> | Twitter <https://twitter.com/bryantbank/>
*From:* Outages <outages-bounces@outages.org> *On Behalf Of *Jason Kuehl via Outages *Sent:* Tuesday, June 9, 2020 8:38 PM *To:* James Brown <jbrown@easypost.com> *Cc:* outages@outages.org *Subject:* Re: [outages] IBM Cloud / NetworkLayer Outage
This Message originates from* outside* Bryant Bank. Please use *caution* when opening this correspondence, attachments or hyperlinks (URLs). If you have questions, please contact IT Support. Thank you.
I can't wait for this post mortem.
On Tue, Jun 9, 2020 at 9:21 PM James Brown via Outages < outages@outages.org> wrote:
As of a few minutes ago, all US SoftLayer (IBM Cloud) locations look normal to me.
On Tue, Jun 9, 2020 at 5:27 PM James Brown <jbrown@easypost.com> wrote:
Just wanted to share updates:
I got through to someone at IBM who confirmed the level of chaos over there. It looks like they've started deploying some mitigations because I'm seeing packet loss drop off but latency is now awful for all US locations. It looks like IBM has failed over all of their peering to go through their European facilities because every return packet from every POP is coming from lon03.networklayer.com <http://secure-web.cisco.com/1ZCENVaw45JMEdCoXCwG0rvb1UUCeCpWh4Gtfftz2u_gZYgdQhoKwrBfewM4QrECWqHUjB3ntgfnOudlN7Txq8mYVsGWJWeXWg-v2cvwS_INwBf66XwGN6PMKDQHpa9-nztcpzjJKAL45c9ehKsPd3RmZsKBti3c78NtK4_5orSmMjjftj2O5ihZtM3DI39J1jzcHEJkQ4qn1RaZRZvtck6dmc6mHvWEK07J6n5lfEiaFtNTzbRMqYg5PRkS5mXxpZdjs13BLwOYthPbwpG2y9sxQhA4vTT6kJ0WTImFe8A_gFn1O_YWB9bnGM_p6egB-/http%3A%2F%2Flon03.networklayer.com> or ams02.networklayer.com <http://secure-web.cisco.com/11XtT4bESggqtqUrb-8NP8jSqmiZcXpSKONGPf8fLiEIvSGY8GXfJOQvPgNH2fin3Mf1oX1qtIrUEhzncE_p6bafeLZfwQNedalFA2ZzWGHl31bfPAPv2AZR6vyTjFMCcWz92HYy6YIl1zhiQebTrb_76Ea-AEzmUrontPsu-oMbxR1e7nrBlRL1dTA7t3blZO8P3p58WDvMLWgLkm-fwMGANaDJ16Pw8BEmKXJJ-fHjl_4kXPUHTptOJJ2tBu8JYCQDVILZauSUz7kHx_JyJDNQEv4x9MojMPj2lnajAHeRvPjmy7vFPJnybP1MU8vGw/http%3A%2F%2Fams02.networklayer.com> (via Telia).
On Tue, Jun 9, 2020 at 4:12 PM James Brown <jbrown@easypost.com> wrote:
It looks like IBM Cloud / SoftLayer / NetworkLayer is having a global outage since about 15:04 Pacific time affecting routing to AWS, Level 3 / CenturyLink, and a bunch of other providers. Has anyone been able to get more details?
--
James Brown
Network Engineer
--
James Brown
Engineer
--
James Brown
Engineer
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages <https://secure-web.cisco.com/1ALbUsNtj5hDIBcrx-EkowvDfnZpZsLIXlTU9xpfnU2i2nr2PgCQpVywQsS39qJHljpuu-05y4c4OqVp6vnUXfkwMjdFzdWAx6ufZY6amRzi-W8ZDQuOeeScc9jnHNYi4_XXaHMDw62k5a0GGAPi2e7kdcA7oL1IKNx_qZPZumgIs16f_vwL28g93YV-Kco95rDiKdE0bmSRoXy19iByqDxxST8yHpluFq-50ihMwmJnR5cX7NXCjKltHkLQJmNokhORvV20qvLxA6jaejf_FP1PGqUQYp8tZbyW7-wXmtTDlX2L3Pi07ilf9uVHRvOBhR9NhsuDyhRSpbieu3tNNcQ/https%3A%2F%2Fpuck.nether.net%2Fmailman%2Flistinfo%2Foutages>
--
Sincerely,
Jason W Kuehl Cell 920-419-8983 jason.w.kuehl@gmail.com
NOTICE: This electronic mail message and any files transmitted with it are intended exclusively for the individual or entity to which it is addressed. The message, together with any attachment, may contain confidential and/or privileged information. Any unauthorized review, use, print, save, copy, disclosure or distribution is strictly prohibited. If you have received this message in error, please immediately advise the sender by reply email and delete copies. Thank you. _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
-- Sincerely, Jason W Kuehl Cell 920-419-8983 jason.w.kuehl@gmail.com
Could we move further discussion to outages-discuss please? ---- Andy Ringsmuth 5609 Harding Drive Lincoln, NE 68521-5831 (402) 304-0083 andy@andyring.com “Better even die free, than to live slaves.” - Frederick Douglas, 1863
On Jun 10, 2020, at 8:22 AM, Jason Kuehl via Outages <outages@outages.org> wrote:
Someone didn't have proper protections in place it looks like.
On Wed, Jun 10, 2020 at 9:20 AM Mike Bolitho via Outages <outages@outages.org> wrote: So another BGP highjacking...
- Mike Bolitho
On Wed, Jun 10, 2020, 5:40 AM Greg Dickinson via Outages <outages@outages.org> wrote: Found this on Reddit as an initial RFO, so keep in mind the source (although r/sysadmin is usually pretty good):
“A 3rd party network provider was advertising routes which resulted in our WorldWide traffic becoming severely impeded. This led to IBM Cloud clients being unable to log-in to their accounts, greatly limited internet/DC connectivity and other significant network route related impacts. Network Specialists have made adjustments to route policies to restore network access, and alleviate the impacts. The overall incident lasted from 5:55pm - 9:30pm ET. We will be providing a fully detailed Customer Incident Report/Root Cause Analysis as soon as possible”
Greg Dickinson
Network Engineer
234 Goodwin Crest Drive, Suite 500
Birmingham, AL 35209
Office: 205.917.2407 | Mobile: 205.234.6427
Fax: 205.945.0515 |
greg.dickinson@bryantbank.com | BryantBank.com
Unbeatable Service. Legendary Results.
Facebook | Twitter
From: Outages <outages-bounces@outages.org> On Behalf Of Jason Kuehl via Outages Sent: Tuesday, June 9, 2020 8:38 PM To: James Brown <jbrown@easypost.com> Cc: outages@outages.org Subject: Re: [outages] IBM Cloud / NetworkLayer Outage
This Message originates from outside Bryant Bank. Please use caution when opening this correspondence, attachments or hyperlinks (URLs). If you have questions, please contact IT Support. Thank you.
I can't wait for this post mortem.
On Tue, Jun 9, 2020 at 9:21 PM James Brown via Outages <outages@outages.org> wrote:
As of a few minutes ago, all US SoftLayer (IBM Cloud) locations look normal to me.
On Tue, Jun 9, 2020 at 5:27 PM James Brown <jbrown@easypost.com> wrote:
Just wanted to share updates:
I got through to someone at IBM who confirmed the level of chaos over there. It looks like they've started deploying some mitigations because I'm seeing packet loss drop off but latency is now awful for all US locations. It looks like IBM has failed over all of their peering to go through their European facilities because every return packet from every POP is coming from lon03.networklayer.com or ams02.networklayer.com (via Telia).
On Tue, Jun 9, 2020 at 4:12 PM James Brown <jbrown@easypost.com> wrote:
It looks like IBM Cloud / SoftLayer / NetworkLayer is having a global outage since about 15:04 Pacific time affecting routing to AWS, Level 3 / CenturyLink, and a bunch of other providers. Has anyone been able to get more details?
--
James Brown
Network Engineer
--
James Brown
Engineer
--
James Brown
Engineer
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
--
Sincerely,
Jason W Kuehl Cell 920-419-8983 jason.w.kuehl@gmail.com
NOTICE: This electronic mail message and any files transmitted with it are intended exclusively for the individual or entity to which it is addressed. The message, together with any attachment, may contain confidential and/or privileged information. Any unauthorized review, use, print, save, copy, disclosure or distribution is strictly prohibited. If you have received this message in error, please immediately advise the sender by reply email and delete copies. Thank you. _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
-- Sincerely,
Jason W Kuehl Cell 920-419-8983 jason.w.kuehl@gmail.com _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Didnt know what was a list. On Wed, Jun 10, 2020 at 9:45 AM Andy Ringsmuth via Outages < outages@outages.org> wrote:
Could we move further discussion to outages-discuss please?
---- Andy Ringsmuth 5609 Harding Drive Lincoln, NE 68521-5831 (402) 304-0083 andy@andyring.com
“Better even die free, than to live slaves.” - Frederick Douglas, 1863
On Jun 10, 2020, at 8:22 AM, Jason Kuehl via Outages < outages@outages.org> wrote:
Someone didn't have proper protections in place it looks like.
On Wed, Jun 10, 2020 at 9:20 AM Mike Bolitho via Outages < outages@outages.org> wrote: So another BGP highjacking...
- Mike Bolitho
On Wed, Jun 10, 2020, 5:40 AM Greg Dickinson via Outages < outages@outages.org> wrote: Found this on Reddit as an initial RFO, so keep in mind the source (although r/sysadmin is usually pretty good):
“A 3rd party network provider was advertising routes which resulted in our WorldWide traffic becoming severely impeded. This led to IBM Cloud clients being unable to log-in to their accounts, greatly limited internet/DC connectivity and other significant network route related impacts. Network Specialists have made adjustments to route policies to restore network access, and alleviate the impacts. The overall incident lasted from 5:55pm - 9:30pm ET. We will be providing a fully detailed Customer Incident Report/Root Cause Analysis as soon as possible”
Greg Dickinson
Network Engineer
234 Goodwin Crest Drive, Suite 500
Birmingham, AL 35209
Office: 205.917.2407 | Mobile: 205.234.6427
Fax: 205.945.0515 |
greg.dickinson@bryantbank.com | BryantBank.com
Unbeatable Service. Legendary Results.
Facebook | Twitter
From: Outages <outages-bounces@outages.org> On Behalf Of Jason Kuehl via Outages Sent: Tuesday, June 9, 2020 8:38 PM To: James Brown <jbrown@easypost.com> Cc: outages@outages.org Subject: Re: [outages] IBM Cloud / NetworkLayer Outage
This Message originates from outside Bryant Bank. Please use caution when opening this correspondence, attachments or hyperlinks (URLs). If you have questions, please contact IT Support. Thank you.
I can't wait for this post mortem.
On Tue, Jun 9, 2020 at 9:21 PM James Brown via Outages < outages@outages.org> wrote:
As of a few minutes ago, all US SoftLayer (IBM Cloud) locations look normal to me.
On Tue, Jun 9, 2020 at 5:27 PM James Brown <jbrown@easypost.com> wrote:
Just wanted to share updates:
I got through to someone at IBM who confirmed the level of chaos over there. It looks like they've started deploying some mitigations because I'm seeing packet loss drop off but latency is now awful for all US locations. It looks like IBM has failed over all of their peering to go through their European facilities because every return packet from every POP is coming from lon03.networklayer.com or ams02.networklayer.com (via Telia).
On Tue, Jun 9, 2020 at 4:12 PM James Brown <jbrown@easypost.com> wrote:
It looks like IBM Cloud / SoftLayer / NetworkLayer is having a global outage since about 15:04 Pacific time affecting routing to AWS, Level 3 / CenturyLink, and a bunch of other providers. Has anyone been able to get more details?
--
James Brown
Network Engineer
--
James Brown
Engineer
--
James Brown
Engineer
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
--
Sincerely,
Jason W Kuehl Cell 920-419-8983 jason.w.kuehl@gmail.com
NOTICE: This electronic mail message and any files transmitted with it are intended exclusively for the individual or entity to which it is addressed. The message, together with any attachment, may contain confidential and/or privileged information. Any unauthorized review, use, print, save, copy, disclosure or distribution is strictly prohibited. If you have received this message in error, please immediately advise the sender by reply email and delete copies. Thank you. _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
-- Sincerely,
Jason W Kuehl Cell 920-419-8983 jason.w.kuehl@gmail.com _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
-- Sincerely, Jason W Kuehl Cell 920-419-8983 jason.w.kuehl@gmail.com
Does anyone actually moderate outages-discuss? I’ve tried to join multiple times over the years…. Mike Castoro | VP of IT Adrianna Papell Group | 646-723-3885 Chat with me in Teams<https://teams.microsoft.com/l/chat/0/0?users=mcastoro@adriannapapell.com> From: Outages <outages-bounces@outages.org> On Behalf Of Jason Kuehl via Outages Sent: Wednesday, June 10, 2020 10:01 AM To: Andy Ringsmuth <andy@andyring.com> Cc: Jason Kuehl via Outages <outages@outages.org> Subject: Re: [outages] IBM Cloud / NetworkLayer Outage External Email! Didnt know what was a list. On Wed, Jun 10, 2020 at 9:45 AM Andy Ringsmuth via Outages <outages@outages.org<mailto:outages@outages.org>> wrote: Could we move further discussion to outages-discuss please? ---- Andy Ringsmuth 5609 Harding Drive Lincoln, NE 68521-5831 (402) 304-0083 andy@andyring.com<mailto:andy@andyring.com> “Better even die free, than to live slaves.” - Frederick Douglas, 1863
On Jun 10, 2020, at 8:22 AM, Jason Kuehl via Outages <outages@outages.org<mailto:outages@outages.org>> wrote:
Someone didn't have proper protections in place it looks like.
On Wed, Jun 10, 2020 at 9:20 AM Mike Bolitho via Outages <outages@outages.org<mailto:outages@outages.org>> wrote: So another BGP highjacking...
- Mike Bolitho
On Wed, Jun 10, 2020, 5:40 AM Greg Dickinson via Outages <outages@outages.org<mailto:outages@outages.org>> wrote: Found this on Reddit as an initial RFO, so keep in mind the source (although r/sysadmin is usually pretty good):
“A 3rd party network provider was advertising routes which resulted in our WorldWide traffic becoming severely impeded. This led to IBM Cloud clients being unable to log-in to their accounts, greatly limited internet/DC connectivity and other significant network route related impacts. Network Specialists have made adjustments to route policies to restore network access, and alleviate the impacts. The overall incident lasted from 5:55pm - 9:30pm ET. We will be providing a fully detailed Customer Incident Report/Root Cause Analysis as soon as possible”
Greg Dickinson
Network Engineer
234 Goodwin Crest Drive, Suite 500
Birmingham, AL 35209
Office: 205.917.2407 | Mobile: 205.234.6427
Fax: 205.945.0515 |
greg.dickinson@bryantbank.com<mailto:greg.dickinson@bryantbank.com> | BryantBank.com
Unbeatable Service. Legendary Results.
Facebook | Twitter
From: Outages <outages-bounces@outages.org<mailto:outages-bounces@outages.org>> On Behalf Of Jason Kuehl via Outages Sent: Tuesday, June 9, 2020 8:38 PM To: James Brown <jbrown@easypost.com<mailto:jbrown@easypost.com>> Cc: outages@outages.org<mailto:outages@outages.org> Subject: Re: [outages] IBM Cloud / NetworkLayer Outage
This Message originates from outside Bryant Bank. Please use caution when opening this correspondence, attachments or hyperlinks (URLs). If you have questions, please contact IT Support. Thank you.
I can't wait for this post mortem.
On Tue, Jun 9, 2020 at 9:21 PM James Brown via Outages <outages@outages.org<mailto:outages@outages.org>> wrote:
As of a few minutes ago, all US SoftLayer (IBM Cloud) locations look normal to me.
On Tue, Jun 9, 2020 at 5:27 PM James Brown <jbrown@easypost.com<mailto:jbrown@easypost.com>> wrote:
Just wanted to share updates:
I got through to someone at IBM who confirmed the level of chaos over there. It looks like they've started deploying some mitigations because I'm seeing packet loss drop off but latency is now awful for all US locations. It looks like IBM has failed over all of their peering to go through their European facilities because every return packet from every POP is coming from lon03.networklayer.com<http://lon03.networklayer.com> or ams02.networklayer.com<http://ams02.networklayer.com> (via Telia).
On Tue, Jun 9, 2020 at 4:12 PM James Brown <jbrown@easypost.com<mailto:jbrown@easypost.com>> wrote:
It looks like IBM Cloud / SoftLayer / NetworkLayer is having a global outage since about 15:04 Pacific time affecting routing to AWS, Level 3 / CenturyLink, and a bunch of other providers. Has anyone been able to get more details?
--
James Brown
Network Engineer
--
James Brown
Engineer
--
James Brown
Engineer
_______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages
--
Sincerely,
Jason W Kuehl Cell 920-419-8983 jason.w.kuehl@gmail.com<mailto:jason.w.kuehl@gmail.com>
NOTICE: This electronic mail message and any files transmitted with it are intended exclusively for the individual or entity to which it is addressed. The message, together with any attachment, may contain confidential and/or privileged information. Any unauthorized review, use, print, save, copy, disclosure or distribution is strictly prohibited. If you have received this message in error, please immediately advise the sender by reply email and delete copies. Thank you. _______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages _______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages
-- Sincerely,
Jason W Kuehl Cell 920-419-8983 jason.w.kuehl@gmail.com<mailto:jason.w.kuehl@gmail.com> _______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages -- Sincerely, Jason W Kuehl Cell 920-419-8983 jason.w.kuehl@gmail.com<mailto:jason.w.kuehl@gmail.com>
https://puck.nether.net/mailman/listinfo/outages-discussion I’ve received messages as recently as June 7 on -discussion. ---- Andy Ringsmuth 5609 Harding Drive Lincoln, NE 68521-5831 (402) 304-0083 andy@andyring.com “Better even die free, than to live slaves.” - Frederick Douglas, 1863
On Jun 10, 2020, at 10:19 AM, Mike Castoro <mcastoro@AdriannaPapell.com> wrote:
Does anyone actually moderate outages-discuss? I’ve tried to join multiple times over the years….
Mike Castoro | VP of IT Adrianna Papell Group | 646-723-3885 Chat with me in Teams
From: Outages <outages-bounces@outages.org> On Behalf Of Jason Kuehl via Outages Sent: Wednesday, June 10, 2020 10:01 AM To: Andy Ringsmuth <andy@andyring.com> Cc: Jason Kuehl via Outages <outages@outages.org> Subject: Re: [outages] IBM Cloud / NetworkLayer Outage
External Email!
Didnt know what was a list.
On Wed, Jun 10, 2020 at 9:45 AM Andy Ringsmuth via Outages <outages@outages.org> wrote: Could we move further discussion to outages-discuss please?
---- Andy Ringsmuth 5609 Harding Drive Lincoln, NE 68521-5831 (402) 304-0083 andy@andyring.com
“Better even die free, than to live slaves.” - Frederick Douglas, 1863
On Jun 10, 2020, at 8:22 AM, Jason Kuehl via Outages <outages@outages.org> wrote:
Someone didn't have proper protections in place it looks like.
On Wed, Jun 10, 2020 at 9:20 AM Mike Bolitho via Outages <outages@outages.org> wrote: So another BGP highjacking...
- Mike Bolitho
On Wed, Jun 10, 2020, 5:40 AM Greg Dickinson via Outages <outages@outages.org> wrote: Found this on Reddit as an initial RFO, so keep in mind the source (although r/sysadmin is usually pretty good):
“A 3rd party network provider was advertising routes which resulted in our WorldWide traffic becoming severely impeded. This led to IBM Cloud clients being unable to log-in to their accounts, greatly limited internet/DC connectivity and other significant network route related impacts. Network Specialists have made adjustments to route policies to restore network access, and alleviate the impacts. The overall incident lasted from 5:55pm - 9:30pm ET. We will be providing a fully detailed Customer Incident Report/Root Cause Analysis as soon as possible”
Greg Dickinson
Network Engineer
234 Goodwin Crest Drive, Suite 500
Birmingham, AL 35209
Office: 205.917.2407 | Mobile: 205.234.6427
Fax: 205.945.0515 |
greg.dickinson@bryantbank.com | BryantBank.com
Unbeatable Service. Legendary Results.
Facebook | Twitter
From: Outages <outages-bounces@outages.org> On Behalf Of Jason Kuehl via Outages Sent: Tuesday, June 9, 2020 8:38 PM To: James Brown <jbrown@easypost.com> Cc: outages@outages.org Subject: Re: [outages] IBM Cloud / NetworkLayer Outage
This Message originates from outside Bryant Bank. Please use caution when opening this correspondence, attachments or hyperlinks (URLs). If you have questions, please contact IT Support. Thank you.
I can't wait for this post mortem.
On Tue, Jun 9, 2020 at 9:21 PM James Brown via Outages <outages@outages.org> wrote:
As of a few minutes ago, all US SoftLayer (IBM Cloud) locations look normal to me.
On Tue, Jun 9, 2020 at 5:27 PM James Brown <jbrown@easypost.com> wrote:
Just wanted to share updates:
I got through to someone at IBM who confirmed the level of chaos over there. It looks like they've started deploying some mitigations because I'm seeing packet loss drop off but latency is now awful for all US locations. It looks like IBM has failed over all of their peering to go through their European facilities because every return packet from every POP is coming from lon03.networklayer.com or ams02.networklayer.com (via Telia).
On Tue, Jun 9, 2020 at 4:12 PM James Brown <jbrown@easypost.com> wrote:
It looks like IBM Cloud / SoftLayer / NetworkLayer is having a global outage since about 15:04 Pacific time affecting routing to AWS, Level 3 / CenturyLink, and a bunch of other providers. Has anyone been able to get more details?
--
James Brown
Network Engineer
--
James Brown
Engineer
--
James Brown
Engineer
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
--
Sincerely,
Jason W Kuehl Cell 920-419-8983 jason.w.kuehl@gmail.com
NOTICE: This electronic mail message and any files transmitted with it are intended exclusively for the individual or entity to which it is addressed. The message, together with any attachment, may contain confidential and/or privileged information. Any unauthorized review, use, print, save, copy, disclosure or distribution is strictly prohibited. If you have received this message in error, please immediately advise the sender by reply email and delete copies. Thank you. _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
-- Sincerely,
Jason W Kuehl Cell 920-419-8983 jason.w.kuehl@gmail.com _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
-- Sincerely,
Jason W Kuehl Cell 920-419-8983 jason.w.kuehl@gmail.com
participants (10)
-
Andy Jabbour -
Andy Ringsmuth -
Charles Sprickman -
Greg Dickinson -
James Brown -
Jason Kuehl -
Jeff P -
Mike Bolitho -
Mike Castoro -
Nate Carlson