START: 02-09-2020 20:31:39 UTC (~3:31pm EST) END: 02-09-2020 23:40:47 UTC (~6:41pm EST) Thanks Mel -- we're waiting on an RFO from Vitelity/Voyant. Services recovered approximately 3 hours and 10 minutes after we initially detected the network issues. Round Trip Times dropped from 50% packet loss and 375ms RTT back down to 68ms RTT with 0% packet loss. I'll be curious if the CenturyLink issue is related, though our route is via Zayo or Above.net, and I don't know how they might be related. Cheers, fellow Beckman :-) On Mon, 10 Feb 2020, Mel Beckman wrote:
This may not be related, but on CenturyLink our gigabit fiber BPG peering point router is experiencing random high CPU surges that we’ve isolated to traffic arriving on this link. We haven’t identified the specific traffic yet because the surges are intermittent, lasting just a minute or tow. This began at 7:30AM PT and we’ve had about a dozen events in the last nine hours.
-mel beckman
On Feb 9, 2020, at 1:50 PM, Peter Beckman via Outages <outages@outages.org> wrote:
START: 02-09-2020 20:31:39 UTC END: ongoing
Vitelity is experiencing some sort of network issue. While the Voyant NOC says they are aware, their portal is unstable due to the 40-50% packet loss and jump from 40-50ms to 375ms round trip times.
This is affecting all services AFAICT: API endpoints, Portal, incoming and outgoing calls.
We have moved most of our traffic away from Vitelity, except for phone numbers they host, as there is no ETA and 375ms plus 40-50% packet loss means pretty terrible call quality.
I'll update as I learn more.
--------------------------------------------------------------------------- Peter Beckman Internet Guy beckman@angryox.com http://www.angryox.com/ ---------------------------------------------------------------------------