TCP errors observed in AWS US-West-2
I'm noting a significant increase in TCP errors across multiple accounts in AWS us-west-2. Curious if anyone else with accounts are noting this too, or if it's localized to some hardware we happen to be on.
I had some users complain about weirdly slow queries in RDS and connection issues so... maybe? On Thu, Aug 18, 2022 at 5:10 PM Cody Wilson via Outages <outages@outages.org> wrote:
I'm noting a significant increase in TCP errors across multiple accounts in AWS us-west-2. Curious if anyone else with accounts are noting this too, or if it's localized to some hardware we happen to be on. _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
AWS Health Dashboard looks clean. Is there a common carrier between your hosts and AWS? https://health.aws.amazon.com/health/status Chris Wright | Network Data Operations Engineer Network Operations and Integrations Commnet Broadband E: chris.wright@commnetbroadband.com M: 423-584-9578 [cid:image001.jpg@01D8B2FB.F9C19B20]<https://www.commnetbroadband.com/> From: Outages <outages-bounces@outages.org> On Behalf Of Cody Wilson via Outages Sent: Thursday, August 18, 2022 12:10 PM To: outages.org <outages@outages.org> Subject: [outages] TCP errors observed in AWS US-West-2 I'm noting a significant increase in TCP errors across multiple accounts in AWS us-west-2. Curious if anyone else with accounts are noting this too, or if it's localized to some hardware we happen to be on.
While we can see and replicate these issues externally, we're also seeing it intra-AWS networks, so it's certainly something within us- west-2 that's amiss here. Moreso looking for any corroboration with anyone else's issues at this point. Thanks Chris! On August 18, 2022, Chris Wright <chris.wright@commnetbroadband.com> wrote:
AWS Health Dashboard looks clean. Is there a common carrier between your hosts and AWS?
<https://health.aws.amazon.com/health/status>
Chris Wright | Network Data Operations Engineer
Network Operations and Integrations
Commnet Broadband
E: chris.wright@commnetbroadband.com M: 423-584-9578
<https://www.commnetbroadband.com/>
From: Outages <outages-bounces@outages.org> On Behalf Of Cody Wilson via Outages Sent: Thursday, August 18, 2022 12:10 PM To: outages.org <outages@outages.org> Subject: [outages] TCP errors observed in AWS US-West-2
I'm noting a significant increase in TCP errors across multiple accounts in AWS us-west-2. Curious if anyone else with accounts are noting this too, or if it's localized to some hardware we happen to be on.
Hey Cody, can you please throw what you have in a support case and share its id with me? If you don’t have a support plan then some details direct to my Amazon email (bonfigg @ amazon com) would be appreciated too but I might not be able to respond/share findings directly (privacy data security etc etc). Thanks! Giorgio On 18 Aug 2022, at 17:25, Cody Wilson via Outages <outages@outages.org <mailto:outages@outages.org> > wrote: While we can see and replicate these issues externally, we're also seeing it intra-AWS networks, so it's certainly something within us-west-2 that's amiss here. Moreso looking for any corroboration with anyone else's issues at this point. Thanks Chris! On August 18, 2022, Chris Wright <chris.wright@commnetbroadband.com <mailto:chris.wright@commnetbroadband.com> > wrote: AWS Health Dashboard looks clean. Is there a common carrier between your hosts and AWS? https://health.aws.amazon.com/health/status <https://health.aws.amazon.com/health/status> Chris Wright | Network Data Operations Engineer Network Operations and Integrations Commnet Broadband E: chris.wright@commnetbroadband.com <mailto:chris.wright@commnetbroadband.com> M: 423-584-9578 <https://www.commnetbroadband.com/> <image001.jpg> From: Outages <outages-bounces@outages.org <mailto:outages-bounces@outages.org> > On Behalf Of Cody Wilson via Outages Sent: Thursday, August 18, 2022 12:10 PM To: outages.org <http://outages.org> <outages@outages.org <mailto:outages@outages.org> > Subject: [outages] TCP errors observed in AWS US-West-2 I'm noting a significant increase in TCP errors across multiple accounts in AWS us-west-2. Curious if anyone else with accounts are noting this too, or if it's localized to some hardware we happen to be on. _______________________________________________ Outages mailing list Outages@outages.org <mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages <https://puck.nether.net/mailman/listinfo/outages>
Thanks Giorgio! Sending case ID directly to you via your Amazon address. On August 18, 2022, Giorgio Bonfiglio <me@grg.pw> wrote:
Hey Cody, can you please throw what you have in a support case and share its id with me? If you don’t have a support plan then some details direct to my Amazon email (bonfigg @ amazon com) would be appreciated too but I might not be able to respond/share findings directly (privacy data security etc etc).
Thanks! Giorgio
On 18 Aug 2022, at 17:25, Cody Wilson via Outages <outages@outages.org <mailto:outages@outages.org>> wrote:
While we can see and replicate these issues externally, we're also seeing it intra-AWS networks, so it's certainly something within us- west-2 that's amiss here. Moreso looking for any corroboration with anyone else's issues at this point. Thanks Chris!
On August 18, 2022, Chris Wright <chris.wright@commnetbroadband.com <mailto:chris.wright@commnetbroadband.com>> wrote:
AWS Health Dashboard looks clean. Is there a common carrier between your hosts and AWS?
<https://health.aws.amazon.com/health/status>
Chris Wright | Network Data Operations Engineer Network Operations and Integrations Commnet Broadband E: chris.wright@commnetbroadband.com <mailto:chris.wright@commnetbroadband.com> M: 423-584-9578 <https://www.commnetbroadband.com/> <image001.jpg>
From: Outages <outages-bounces@outages.org <mailto:outages- bounces@outages.org>> On Behalf Of Cody Wilson via Outages Sent: Thursday, August 18, 2022 12:10 PM To: outages.org <http://outages.org> <outages@outages.org <mailto:outages@outages.org>> Subject: [outages] TCP errors observed in AWS US-West-2
I'm noting a significant increase in TCP errors across multiple accounts in AWS us-west-2. Curious if anyone else with accounts are noting this too, or if it's localized to some hardware we happen to be on.
_______________________________________________ Outages mailing list Outages@outages.org <mailto:Outages@outages.org> <https://puck.nether.net/mailman/listinfo/outages>
I believe this was on the right track, as we are still seeing issues. I believe that it's specifically Frontier (at least in FL) that is having issues contacting AWS resources. We use a loyalty program called Astro that is on AWS and our point of sale (non AWS) is still timing out when the API call is being made. Giorgio/Cody, did you all come up with a resolution? Would love a solution, as our retail stores are doing most of their processes manually right now to get around this. -Jeff On Thu, Aug 18, 2022 at 2:38 PM Cody Wilson via Outages <outages@outages.org> wrote:
Thanks Giorgio! Sending case ID directly to you via your Amazon address.
On August 18, 2022, Giorgio Bonfiglio <me@grg.pw> wrote:
Hey Cody, can you please throw what you have in a support case and share its id with me? If you don’t have a support plan then some details direct to my Amazon email (bonfigg @ amazon com) would be appreciated too but I might not be able to respond/share findings directly (privacy data security etc etc).
Thanks! Giorgio
On 18 Aug 2022, at 17:25, Cody Wilson via Outages <outages@outages.org> wrote:
While we can see and replicate these issues externally, we're also seeing it intra-AWS networks, so it's certainly something within us-west-2 that's amiss here. Moreso looking for any corroboration with anyone else's issues at this point. Thanks Chris!
On August 18, 2022, Chris Wright <chris.wright@commnetbroadband.com> wrote:
AWS Health Dashboard looks clean. Is there a common carrier between your hosts and AWS?
https://health.aws.amazon.com/health/status
*Chris Wright* | Network Data Operations Engineer Network Operations and Integrations Commnet Broadband *E:* chris.wright@commnetbroadband.com *M: *423-584-9578 <https://www.commnetbroadband.com/> <image001.jpg>
*From:* Outages <outages-bounces@outages.org> *On Behalf Of *Cody Wilson via Outages *Sent:* Thursday, August 18, 2022 12:10 PM *To:* outages.org <outages@outages.org> *Subject:* [outages] TCP errors observed in AWS US-West-2
I'm noting a significant increase in TCP errors across multiple accounts in AWS us-west-2. Curious if anyone else with accounts are noting this too, or if it's localized to some hardware we happen to be on.
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Not really yet, two more people have reached out but symptoms and timing are completely off so I’m wondering whether we are looking at a single event or multiple customer specific details. Can you share with me some of your source IPs please and the endpoints (better if they are already resolved to IPs) timing out? I’ll try to narrow down to this specific access network and see if something stands out. Thanks Giorgio On 20 Aug 2022, at 22:26, Jeff Belcher <jeffreybelcher@gmail.com> wrote: I believe this was on the right track, as we are still seeing issues. I believe that it's specifically Frontier (at least in FL) that is having issues contacting AWS resources. We use a loyalty program called Astro that is on AWS and our point of sale (non AWS) is still timing out when the API call is being made. Giorgio/Cody, did you all come up with a resolution? Would love a solution, as our retail stores are doing most of their processes manually right now to get around this. -Jeff On Thu, Aug 18, 2022 at 2:38 PM Cody Wilson via Outages <outages@outages.org <mailto:outages@outages.org> > wrote: Thanks Giorgio! Sending case ID directly to you via your Amazon address. On August 18, 2022, Giorgio Bonfiglio <me@grg.pw <mailto:me@grg.pw> > wrote: Hey Cody, can you please throw what you have in a support case and share its id with me? If you don’t have a support plan then some details direct to my Amazon email (bonfigg @ amazon com) would be appreciated too but I might not be able to respond/share findings directly (privacy data security etc etc). Thanks! Giorgio On 18 Aug 2022, at 17:25, Cody Wilson via Outages <outages@outages.org <mailto:outages@outages.org> > wrote: While we can see and replicate these issues externally, we're also seeing it intra-AWS networks, so it's certainly something within us-west-2 that's amiss here. Moreso looking for any corroboration with anyone else's issues at this point. Thanks Chris! On August 18, 2022, Chris Wright <chris.wright@commnetbroadband.com <mailto:chris.wright@commnetbroadband.com> > wrote: AWS Health Dashboard looks clean. Is there a common carrier between your hosts and AWS? https://health.aws.amazon.com/health/status <https://health.aws.amazon.com/health/status> Chris Wright | Network Data Operations Engineer Network Operations and Integrations Commnet Broadband E: chris.wright@commnetbroadband.com <mailto:chris.wright@commnetbroadband.com> M: 423-584-9578 <https://www.commnetbroadband.com/> <image001.jpg> From: Outages <outages-bounces@outages.org <mailto:outages-bounces@outages.org> > On Behalf Of Cody Wilson via Outages Sent: Thursday, August 18, 2022 12:10 PM To: outages.org <http://outages.org> <outages@outages.org <mailto:outages@outages.org> > Subject: [outages] TCP errors observed in AWS US-West-2 I'm noting a significant increase in TCP errors across multiple accounts in AWS us-west-2. Curious if anyone else with accounts are noting this too, or if it's localized to some hardware we happen to be on. _______________________________________________ Outages mailing list Outages@outages.org <mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages <https://puck.nether.net/mailman/listinfo/outages> _______________________________________________ Outages mailing list Outages@outages.org <mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages <https://puck.nether.net/mailman/listinfo/outages>
Absolutely. I can also do live testing, if that would help at all. On Sun, Aug 21, 2022, 11:01 AM Giorgio Bonfiglio <me@grg.pw> wrote:
Not really yet, two more people have reached out but symptoms and timing are completely off so I’m wondering whether we are looking at a single event or multiple customer specific details.
Can you share with me some of your source IPs please and the endpoints (better if they are already resolved to IPs) timing out? I’ll try to narrow down to this specific access network and see if something stands out.
Thanks Giorgio
On 20 Aug 2022, at 22:26, Jeff Belcher <jeffreybelcher@gmail.com> wrote:
I believe this was on the right track, as we are still seeing issues. I believe that it's specifically Frontier (at least in FL) that is having issues contacting AWS resources. We use a loyalty program called Astro that is on AWS and our point of sale (non AWS) is still timing out when the API call is being made.
Giorgio/Cody, did you all come up with a resolution? Would love a solution, as our retail stores are doing most of their processes manually right now to get around this.
-Jeff
On Thu, Aug 18, 2022 at 2:38 PM Cody Wilson via Outages < outages@outages.org> wrote:
Thanks Giorgio! Sending case ID directly to you via your Amazon address.
On August 18, 2022, Giorgio Bonfiglio <me@grg.pw> wrote:
Hey Cody, can you please throw what you have in a support case and share its id with me? If you don’t have a support plan then some details direct to my Amazon email (bonfigg @ amazon com) would be appreciated too but I might not be able to respond/share findings directly (privacy data security etc etc).
Thanks! Giorgio
On 18 Aug 2022, at 17:25, Cody Wilson via Outages <outages@outages.org> wrote:
While we can see and replicate these issues externally, we're also seeing it intra-AWS networks, so it's certainly something within us-west-2 that's amiss here. Moreso looking for any corroboration with anyone else's issues at this point. Thanks Chris!
On August 18, 2022, Chris Wright <chris.wright@commnetbroadband.com> wrote:
AWS Health Dashboard looks clean. Is there a common carrier between your hosts and AWS?
https://health.aws.amazon.com/health/status
*Chris Wright* | Network Data Operations Engineer Network Operations and Integrations Commnet Broadband *E:* chris.wright@commnetbroadband.com *M: *423-584-9578 <https://www.commnetbroadband.com/> <image001.jpg>
*From:* Outages <outages-bounces@outages.org> *On Behalf Of *Cody Wilson via Outages *Sent:* Thursday, August 18, 2022 12:10 PM *To:* outages.org <outages@outages.org> *Subject:* [outages] TCP errors observed in AWS US-West-2
I'm noting a significant increase in TCP errors across multiple accounts in AWS us-west-2. Curious if anyone else with accounts are noting this too, or if it's localized to some hardware we happen to be on.
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
We’re also seeing issues in AWS US-West-2. Any thoughts or things we should try? On Mon, Aug 22, 2022 at 01:16 Jeff Belcher via Outages <outages@outages.org> wrote:
Absolutely. I can also do live testing, if that would help at all.
On Sun, Aug 21, 2022, 11:01 AM Giorgio Bonfiglio <me@grg.pw> wrote:
Not really yet, two more people have reached out but symptoms and timing are completely off so I’m wondering whether we are looking at a single event or multiple customer specific details.
Can you share with me some of your source IPs please and the endpoints (better if they are already resolved to IPs) timing out? I’ll try to narrow down to this specific access network and see if something stands out.
Thanks Giorgio
On 20 Aug 2022, at 22:26, Jeff Belcher <jeffreybelcher@gmail.com> wrote:
I believe this was on the right track, as we are still seeing issues. I believe that it's specifically Frontier (at least in FL) that is having issues contacting AWS resources. We use a loyalty program called Astro that is on AWS and our point of sale (non AWS) is still timing out when the API call is being made.
Giorgio/Cody, did you all come up with a resolution? Would love a solution, as our retail stores are doing most of their processes manually right now to get around this.
-Jeff
On Thu, Aug 18, 2022 at 2:38 PM Cody Wilson via Outages < outages@outages.org> wrote:
Thanks Giorgio! Sending case ID directly to you via your Amazon address.
On August 18, 2022, Giorgio Bonfiglio <me@grg.pw> wrote:
Hey Cody, can you please throw what you have in a support case and share its id with me? If you don’t have a support plan then some details direct to my Amazon email (bonfigg @ amazon com) would be appreciated too but I might not be able to respond/share findings directly (privacy data security etc etc).
Thanks! Giorgio
On 18 Aug 2022, at 17:25, Cody Wilson via Outages <outages@outages.org> wrote:
While we can see and replicate these issues externally, we're also seeing it intra-AWS networks, so it's certainly something within us-west-2 that's amiss here. Moreso looking for any corroboration with anyone else's issues at this point. Thanks Chris!
On August 18, 2022, Chris Wright <chris.wright@commnetbroadband.com> wrote:
AWS Health Dashboard looks clean. Is there a common carrier between your hosts and AWS?
https://health.aws.amazon.com/health/status
*Chris Wright* | Network Data Operations Engineer Network Operations and Integrations Commnet Broadband *E:* chris.wright@commnetbroadband.com *M: *423-584-9578 <https://www.commnetbroadband.com/> <image001.jpg>
*From:* Outages <outages-bounces@outages.org> *On Behalf Of *Cody Wilson via Outages *Sent:* Thursday, August 18, 2022 12:10 PM *To:* outages.org <outages@outages.org> *Subject:* [outages] TCP errors observed in AWS US-West-2
I'm noting a significant increase in TCP errors across multiple accounts in AWS us-west-2. Curious if anyone else with accounts are noting this too, or if it's localized to some hardware we happen to be on.
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
-- Sent from Gmail Mobile
Can you please share some src/dst IPs, timestamps and description of the issues? On 21 Aug 2022, at 17:30, Michael B. Williams <Michael.Williams@glexia.com> wrote: We’re also seeing issues in AWS US-West-2. Any thoughts or things we should try? On Mon, Aug 22, 2022 at 01:16 Jeff Belcher via Outages <outages@outages.org <mailto:outages@outages.org> > wrote: Absolutely. I can also do live testing, if that would help at all. On Sun, Aug 21, 2022, 11:01 AM Giorgio Bonfiglio <me@grg.pw <mailto:me@grg.pw> > wrote: Not really yet, two more people have reached out but symptoms and timing are completely off so I’m wondering whether we are looking at a single event or multiple customer specific details. Can you share with me some of your source IPs please and the endpoints (better if they are already resolved to IPs) timing out? I’ll try to narrow down to this specific access network and see if something stands out. Thanks Giorgio On 20 Aug 2022, at 22:26, Jeff Belcher <jeffreybelcher@gmail.com <mailto:jeffreybelcher@gmail.com> > wrote: I believe this was on the right track, as we are still seeing issues. I believe that it's specifically Frontier (at least in FL) that is having issues contacting AWS resources. We use a loyalty program called Astro that is on AWS and our point of sale (non AWS) is still timing out when the API call is being made. Giorgio/Cody, did you all come up with a resolution? Would love a solution, as our retail stores are doing most of their processes manually right now to get around this. -Jeff On Thu, Aug 18, 2022 at 2:38 PM Cody Wilson via Outages <outages@outages.org <mailto:outages@outages.org> > wrote: Thanks Giorgio! Sending case ID directly to you via your Amazon address. On August 18, 2022, Giorgio Bonfiglio <me@grg.pw <mailto:me@grg.pw> > wrote: Hey Cody, can you please throw what you have in a support case and share its id with me? If you don’t have a support plan then some details direct to my Amazon email (bonfigg @ amazon com) would be appreciated too but I might not be able to respond/share findings directly (privacy data security etc etc). Thanks! Giorgio On 18 Aug 2022, at 17:25, Cody Wilson via Outages <outages@outages.org <mailto:outages@outages.org> > wrote: While we can see and replicate these issues externally, we're also seeing it intra-AWS networks, so it's certainly something within us-west-2 that's amiss here. Moreso looking for any corroboration with anyone else's issues at this point. Thanks Chris! On August 18, 2022, Chris Wright <chris.wright@commnetbroadband.com <mailto:chris.wright@commnetbroadband.com> > wrote: AWS Health Dashboard looks clean. Is there a common carrier between your hosts and AWS? https://health.aws.amazon.com/health/status <https://health.aws.amazon.com/health/status> Chris Wright | Network Data Operations Engineer Network Operations and Integrations Commnet Broadband E: chris.wright@commnetbroadband.com <mailto:chris.wright@commnetbroadband.com> M: 423-584-9578 <https://www.commnetbroadband.com/> <image001.jpg> From: Outages <outages-bounces@outages.org <mailto:outages-bounces@outages.org> > On Behalf Of Cody Wilson via Outages Sent: Thursday, August 18, 2022 12:10 PM To: outages.org <http://outages.org> <outages@outages.org <mailto:outages@outages.org> > Subject: [outages] TCP errors observed in AWS US-West-2 I'm noting a significant increase in TCP errors across multiple accounts in AWS us-west-2. Curious if anyone else with accounts are noting this too, or if it's localized to some hardware we happen to be on. _______________________________________________ Outages mailing list Outages@outages.org <mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages <https://puck.nether.net/mailman/listinfo/outages> _______________________________________________ Outages mailing list Outages@outages.org <mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages <https://puck.nether.net/mailman/listinfo/outages> _______________________________________________ Outages mailing list Outages@outages.org <mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages <https://puck.nether.net/mailman/listinfo/outages> -- Sent from Gmail Mobile
participants (6)
-
Chris Wright -
Cody Wilson -
Giorgio Bonfiglio -
Jeff Belcher -
Michael B. Williams -
Tamás Kádár (KTamas)