US West Coast Massive Tier 1 Backbone Issues
Hey All, There's a few threads on this already, but they all call out specific ISPs when this trouble isn't limited to any one last-mile provider. Nearly all tier 1 transit providers are experiencing loss on the US West Coast today. Some examples: Comcast/Level3 peering point in Seattle: [image: image.png] Telia/AT&T in Seattle: [image: image.png] Hurricane Electric/Cox in San Jose: [image: image.png] And this isn't the half of it. Very few tier 1 transit providers are unaffected. So much for carrier redundancy! - Cary
I was hoping somebody was going to climb up to 30,000 feet on this, because it sort of felt to me like it was pervasive as well. I encourage further analysis on this point, and apologize to anybody who would prefer that it be on the discussion list - I think it actually belongs here. </mod> On April 21, 2020 11:58:30 AM EDT, Cary Wiedemann via Outages <outages@outages.org> wrote:
Hey All,
There's a few threads on this already, but they all call out specific ISPs when this trouble isn't limited to any one last-mile provider. Nearly all tier 1 transit providers are experiencing loss on the US West Coast today. Some examples:
Comcast/Level3 peering point in Seattle: [image: image.png]
Telia/AT&T in Seattle: [image: image.png]
Hurricane Electric/Cox in San Jose: [image: image.png]
And this isn't the half of it. Very few tier 1 transit providers are unaffected.
So much for carrier redundancy!
- Cary
-- Sent from my Android device with K-9 Mail. Please excuse my brevity.
Try the 60,000 foot view; the midwest is having a bad day as well, though it sounds like a fiber cut is to blame. https://www.reddit.com/r/sysadmin/comments/g5gyms/midwest_internet_outage/ On Tue, Apr 21, 2020 at 12:38 PM Jay Ashworth via Outages < outages@outages.org> wrote:
I was hoping somebody was going to climb up to 30,000 feet on this, because it sort of felt to me like it was pervasive as well. I encourage further analysis on this point, and apologize to anybody who would prefer that it be on the discussion list - I think it actually belongs here. </mod>
On April 21, 2020 11:58:30 AM EDT, Cary Wiedemann via Outages < outages@outages.org> wrote:
Hey All,
There's a few threads on this already, but they all call out specific ISPs when this trouble isn't limited to any one last-mile provider. Nearly all tier 1 transit providers are experiencing loss on the US West Coast today. Some examples:
Comcast/Level3 peering point in Seattle: [image: image.png]
Telia/AT&T in Seattle: [image: image.png]
Hurricane Electric/Cox in San Jose: [image: image.png]
And this isn't the half of it. Very few tier 1 transit providers are unaffected.
So much for carrier redundancy!
- Cary
-- Sent from my Android device with K-9 Mail. Please excuse my brevity. _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Yeah, the CenturyLink ticket is running in another thread on the list, and the fiber cut comments hit the list about 4 minutes before your message. The internet as a whole has had substantially less Reserve capacity, as the power generation people like to put it, for about the last month, and while I hear tell that excess capacity is being turned up to release some of the pressure, I'm not in a position to know how fast that's happening. On April 21, 2020 12:44:48 PM EDT, Brett Dikeman <brett.dikeman@gmail.com> wrote:
Try the 60,000 foot view; the midwest is having a bad day as well, though it sounds like a fiber cut is to blame.
https://www.reddit.com/r/sysadmin/comments/g5gyms/midwest_internet_outage/
On Tue, Apr 21, 2020 at 12:38 PM Jay Ashworth via Outages < outages@outages.org> wrote:
I was hoping somebody was going to climb up to 30,000 feet on this, because it sort of felt to me like it was pervasive as well. I encourage further analysis on this point, and apologize to anybody who would prefer that it be on the discussion list - I think it actually belongs here. </mod>
On April 21, 2020 11:58:30 AM EDT, Cary Wiedemann via Outages < outages@outages.org> wrote:
Hey All,
There's a few threads on this already, but they all call out
specific
ISPs when this trouble isn't limited to any one last-mile provider. Nearly all tier 1 transit providers are experiencing loss on the US West Coast today. Some examples:
Comcast/Level3 peering point in Seattle: [image: image.png]
Telia/AT&T in Seattle: [image: image.png]
Hurricane Electric/Cox in San Jose: [image: image.png]
And this isn't the half of it. Very few tier 1 transit providers are unaffected.
So much for carrier redundancy!
- Cary
-- Sent from my Android device with K-9 Mail. Please excuse my brevity. _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
-- Sent from my Android device with K-9 Mail. Please excuse my brevity.
Great read on carrier diversity: https://transition.fcc.gov/bureaus/pshs/docs/clearinghouse/ATIS_NDAI_Final_R... From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Jay Ashworth via Outages Sent: Tuesday, April 21, 2020 9:50 AM To: Brett Dikeman <brett.dikeman@gmail.com> Cc: Cary Wiedemann via Outages <outages@outages.org> Subject: Re: [outages] US West Coast Massive Tier 1 Backbone Issues Yeah, the CenturyLink ticket is running in another thread on the list, and the fiber cut comments hit the list about 4 minutes before your message. The internet as a whole has had substantially less Reserve capacity, as the power generation people like to put it, for about the last month, and while I hear tell that excess capacity is being turned up to release some of the pressure, I'm not in a position to know how fast that's happening. On April 21, 2020 12:44:48 PM EDT, Brett Dikeman <brett.dikeman@gmail.com<mailto:brett.dikeman@gmail.com>> wrote: Try the 60,000 foot view; the midwest is having a bad day as well, though it sounds like a fiber cut is to blame. https://www.reddit.com/r/sysadmin/comments/g5gyms/midwest_internet_outage/<https://urldefense.proofpoint.com/v2/url?u=https-3A__www.reddit.com_r_sysadmin_comments_g5gyms_midwest-5Finternet-5Foutage_&d=DwMFaQ&c=N13-TaG7c-EYAiUNohBk74oLRjUiBTwVm-KSnr4bPSc&r=-GzOCp0ppLaBQPFaZ7lZ4bUUBQxpFBukitRP75oaRdQ&m=icB-gA7RcG0Wjt9BVrR_NoAKFCXQZQ8DUg68_4AG1Z8&s=ulznjaY0roT-mCOZ_0PosVaKatrqWHLvAef5-2sAaYI&e=> ---------------------------------------------------------------------- This message and any attachments are intended only for the use of the addressee and may contain information that is privileged and confidential. If the reader of the message is not the intended recipient or an authorized representative of the intended recipient, you are hereby notified that any dissemination of this communication is strictly prohibited. If you have received this communication in error, notify the sender immediately by return email and delete the message and any attachments from your system.
FWIW, I’m not seeing the same sort of loss and extra latency to DC on HE out of Seattle, that I am on CTL out of Spokane. --Eric From: Outages <outages-bounces@outages.org> On Behalf Of Cary Wiedemann via Outages Sent: Tuesday, April 21, 2020 8:59 AM To: outages <outages@outages.org> Subject: ADV: [outages] US West Coast Massive Tier 1 Backbone Issues Hey All, There's a few threads on this already, but they all call out specific ISPs when this trouble isn't limited to any one last-mile provider. Nearly all tier 1 transit providers are experiencing loss on the US West Coast today. Some examples: Comcast/Level3 peering point in Seattle: [cid:image001.png@01D617BE.F3CBE7A0] Telia/AT&T in Seattle: [cid:image002.png@01D617BE.F3CBE7A0] Hurricane Electric/Cox in San Jose: [cid:image003.png@01D617BE.F3CBE7A0] And this isn't the half of it. Very few tier 1 transit providers are unaffected. So much for carrier redundancy! - Cary
participants (5)
-
Brett Dikeman -
Cary Wiedemann -
Eric Flanery -
Hiers, David -
Jay Ashworth