Level3 or Centurylink transport issue?
Hey has anyone noticed significant packet loss from level3 or centurylink? We are having an issue with packet loss to BW media IPs 67.231.13.87 and 88. We rerouted all media traffic to out other DCs, and the poor MOS scores have disappeared. Rob Whitlatch Director of DevOps White Label Communications 215-987-4500
Yes, I am currently experiencing an outage on Level3 on a wave between Ashburn and Raleigh. Sounds as if their protect path is oversubscribed.
On Nov 15, 2019, at 10:26, Rob Whitlatch via Outages <outages@outages.org> wrote:
Hey has anyone noticed significant packet loss from level3 or centurylink? We are having an issue with packet loss to BW media IPs 67.231.13.87 and 88. We rerouted all media traffic to out other DCs, and the poor MOS scores have disappeared.
Rob Whitlatch Director of DevOps White Label Communications 215-987-4500
_______________________________________________ Outages mailing list Outages@outages.org <mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages <https://puck.nether.net/mailman/listinfo/outages>
There's a possible fibercut between Ashburn and McLean VA on Centurylink. That may have something to do with it. On 11/15/19 10:33 AM, Kate Gerry via Outages wrote:
Yes, I am currently experiencing an outage on Level3 on a wave between Ashburn and Raleigh.
Sounds as if their protect path is oversubscribed.
On Nov 15, 2019, at 10:26, Rob Whitlatch via Outages <outages@outages.org <mailto:outages@outages.org>> wrote:
Hey has anyone noticed significant packet loss from level3 or centurylink? We are having an issue with packet loss to BW media IPs 67.231.13.87 and 88. We rerouted all media traffic to out other DCs, and the poor MOS scores have disappeared.
Rob Whitlatch Director of DevOps White Label Communications 215-987-4500
_______________________________________________ Outages mailing list Outages@outages.org <mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Had a 20-second outage on a Level3 circuit in Arlington, VA 2 hours ago, but no connectivity/loss issues since. -- Christopher Conley Systems Administrator | Fors Marsh Group 1010 N. Glebe Rd, Suite 510 Arlington, VA 22201 From: Outages <outages-bounces@outages.org> On Behalf Of Kate Gerry via Outages Sent: Friday, November 15, 2019 1:33 PM To: Rob Whitlatch <rwhitlatch@whitelabelcomm.com> Cc: outages@outages.org Subject: Re: [outages] Level3 or Centurylink transport issue? Yes, I am currently experiencing an outage on Level3 on a wave between Ashburn and Raleigh. Sounds as if their protect path is oversubscribed. On Nov 15, 2019, at 10:26, Rob Whitlatch via Outages <outages@outages.org<mailto:outages@outages.org>> wrote: Hey has anyone noticed significant packet loss from level3 or centurylink? We are having an issue with packet loss to BW media IPs 67.231.13.87 and 88. We rerouted all media traffic to out other DCs, and the poor MOS scores have disappeared. Rob Whitlatch Director of DevOps White Label Communications 215-987-4500 _______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages
I lost my transit too. On 15/11/2019 19:33, Kate Gerry via Outages wrote:
Yes, I am currently experiencing an outage on Level3 on a wave between Ashburn and Raleigh.
Sounds as if their protect path is oversubscribed.
On Nov 15, 2019, at 10:26, Rob Whitlatch via Outages <outages@outages.org <mailto:outages@outages.org>> wrote:
Hey has anyone noticed significant packet loss from level3 or centurylink? We are having an issue with packet loss to BW media IPs 67.231.13.87 and 88. We rerouted all media traffic to out other DCs, and the poor MOS scores have disappeared.
Rob Whitlatch Director of DevOps White Label Communications 215-987-4500
_______________________________________________ Outages mailing list Outages@outages.org <mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
-- Alarig Le Lay Grifon
864 count fiber got cut by a vertical boring operation.
CenturyLink
This is to confirm that an update has been recorded on CenturyLink ticket x. *** CASCADED EXTERNAL NOTES 15-Nov-2019 20:21:17 GMT From CASE: x - Event Field Operations and the Outside Plant Engineer (OSP) have located the exact point of failure. A vertical boring operation has struck and cut an 864-strand fiber. The construction crew is onsite fully assessing the damage and are formulating a repair plan at this time. The fiber splicers are en route with an estimated time of arrival (ETA) of 21:00 GMT. The replacement fiber is also en route; however, the ETA for its arrival is currently unavailable. On 11/15/19, 3:21 PM, "Outages on behalf of Alarig Le Lay via Outages" <outages-bounces@outages.org on behalf of outages@outages.org> wrote: I lost my transit too. On 15/11/2019 19:33, Kate Gerry via Outages wrote: > Yes, I am currently experiencing an outage on Level3 on a wave between > Ashburn and Raleigh. > > Sounds as if their protect path is oversubscribed. > >> On Nov 15, 2019, at 10:26, Rob Whitlatch via Outages >> <outages@outages.org <mailto:outages@outages.org>> wrote: >> >> Hey has anyone noticed significant packet loss from level3 or centurylink? >> We are having an issue with packet loss to BW media IPs 67.231.13.87 >> and 88. We rerouted all media traffic to out other DCs, and the poor >> MOS scores have disappeared. >> >> >> >> Rob Whitlatch >> Director of DevOps >> White Label Communications >> 215-987-4500 >> >> >> >> _______________________________________________ >> Outages mailing list >> Outages@outages.org <mailto:Outages@outages.org> >> https://puck.nether.net/mailman/listinfo/outages > > > _______________________________________________ > Outages mailing list > Outages@outages.org > https://puck.nether.net/mailman/listinfo/outages > -- Alarig Le Lay Grifon _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Update from CenturyLink: The Outside Plant team on site has advised that the auger has been removed and the cable has been cut away from it. Efforts to manually hand dig down to the damage are currently ongoing. It has also been advised that cabling as well as some of the duct will need to be replaced between two handholes. The full extent of the damage to the duct and fiber will not be able to be confirmed until the excavation process is completed. On 11/15/19, 3:41 PM, "Outages on behalf of Wright, Chris via Outages" <outages-bounces@outages.org on behalf of outages@outages.org> wrote: 864 count fiber got cut by a vertical boring operation. >>>CenturyLink This is to confirm that an update has been recorded on CenturyLink ticket x. *** CASCADED EXTERNAL NOTES 15-Nov-2019 20:21:17 GMT From CASE: x - Event Field Operations and the Outside Plant Engineer (OSP) have located the exact point of failure. A vertical boring operation has struck and cut an 864-strand fiber. The construction crew is onsite fully assessing the damage and are formulating a repair plan at this time. The fiber splicers are en route with an estimated time of arrival (ETA) of 21:00 GMT. The replacement fiber is also en route; however, the ETA for its arrival is currently unavailable. On 11/15/19, 3:21 PM, "Outages on behalf of Alarig Le Lay via Outages" <outages-bounces@outages.org on behalf of outages@outages.org> wrote: I lost my transit too. On 15/11/2019 19:33, Kate Gerry via Outages wrote: > Yes, I am currently experiencing an outage on Level3 on a wave between > Ashburn and Raleigh. > > Sounds as if their protect path is oversubscribed. > >> On Nov 15, 2019, at 10:26, Rob Whitlatch via Outages >> <outages@outages.org <mailto:outages@outages.org>> wrote: >> >> Hey has anyone noticed significant packet loss from level3 or centurylink? >> We are having an issue with packet loss to BW media IPs 67.231.13.87 >> and 88. We rerouted all media traffic to out other DCs, and the poor >> MOS scores have disappeared. >> >> >> >> Rob Whitlatch >> Director of DevOps >> White Label Communications >> 215-987-4500 >> >> >> >> _______________________________________________ >> Outages mailing list >> Outages@outages.org <mailto:Outages@outages.org> >> https://puck.nether.net/mailman/listinfo/outages > > > _______________________________________________ > Outages mailing list > Outages@outages.org > https://puck.nether.net/mailman/listinfo/outages > -- Alarig Le Lay Grifon _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
participants (6)
-
Alarig Le Lay -
Christopher Conley -
Kate Gerry -
Rob Whitlatch -
Tim Choe -
Wright, Chris