DC Region Verizon FiOS Outage Event 29-April 18:30
There was just a massive DC region Verizon FiOS backbone routing failure starting at 6:27pm EST and running for ~11 minutes. All paths were impacted but some were more loss prone than others. Here's a loss/latency graph to Google's local anycasted 8.8.8.8 resolver: [image: image.png] vs a graph to Cogent: [image: image.png] vs one to Azure: [image: image.png] MTR graphs show major loss down every path that touches one of Verizon's " ALTER.NET" named routers. Really just sending this one in because the footprint was massive. I'm used to smaller DC-region path failures, but this impacted nearly every path for nearly every FiOS user in DC and the close-in Virginia and Maryland suburbs. This was far worse than their usual routing hiccups. - Cary
We noticed it as well (Northern VA). Latency spiked from normally 8ms between Cox and FIOS to over 150ms, though not too much packet loss. There was an initial loss of connections and packets on SSH, and my son reported losing access to his online XBOX game. My monitoring detected 100% packet loss for about 1 minute, 22:32:32 to 22:33:26 UTC, high RTT from 22:33:26 to 22:37:30, and recovery by 22:37:30 UTC. From my POV it lasted only about 5-6 minutes, though that was from Cox into Verizon FIOS monitoring. Tests were done either every 30 seconds or 1 minute, +/- a few seconds. It's too bad we likely won't hear the root cause or recovery story, I'm sure it'd be fascinating. Beckman Cary wrote:
There was just a massive DC region Verizon FiOS backbone routing failure starting at 6:27pm EST and running for ~11 minutes. All paths were impacted but some were more loss prone than others.
--------------------------------------------------------------------------- Peter Beckman Internet Guy beckman@angryox.com https://www.angryox.com/ ---------------------------------------------------------------------------
participants (2)
-
Cary Wiedemann -
Peter Beckman