Re: [outages] Possible Issues with Level3 - Confirmed Chicago Area, possibly others
We're now seeing 20% packet loss and 200+ms latency. From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Bryan Inks Sent: Tuesday, December 31, 2013 8:25 AM To: outages@outages.org Subject: [outages] Possible Issues with Level3 - Confirmed Chicago Area, possibly others Just received a call from a client here in CA. They are seeing bi-directional issues with Level3 in the Chicago area. Approximately 7% packet loss and 70+ms latency.
Using level3 for SIP trunking. Latency spikes up to 3,000 ms since yesterday 11 PM CST. Andres Sent from my iPhone On Dec 31, 2013, at 10:52 AM, "Bryan Inks" <Binks@keyinfo.com> wrote:
We’re now seeing 20% packet loss and 200+ms latency.
From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Bryan Inks Sent: Tuesday, December 31, 2013 8:25 AM To: outages@outages.org Subject: [outages] Possible Issues with Level3 - Confirmed Chicago Area, possibly others
Just received a call from a client here in CA. They are seeing bi-directional issues with Level3 in the Chicago area.
Approximately 7% packet loss and 70+ms latency. _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Running a trace from WA to Northern CA I'm seeing packet loss after Dallas: 2 pin-206-126-16-161.static.pocketinet.com (206.126.16.161) 2.616 ms 2.587 ms 2.524 ms 3 core03-sea-eth0-6.net.pocketinet.com (64.185.97.122) 9.413 ms 9.308 ms 9.323 ms 4 core01-pdx.net.pocketinet.com (206.126.21.148) 13.481 ms 13.396 ms 13.419 ms 5 core02-la-eth2-942.net.pocketinet.com (64.185.97.68) 45.096 ms 45.067 ms 44.954 ms 6 ge-0-7-0-23.r04.lsanca03.us.bb.gin.ntt.net (129.250.200.29) 42.364 ms 42.704 ms 45.097 ms 7 xe-2.level3.lsanca03.us.bb.gin.ntt.net (129.250.9.226) 41.696 ms xe-3.level3.lsanca03.us.bb.gin.ntt.net (129.250.9.230) 39.691 ms xe-1.level3.lsanca03.us.bb.gin.ntt.net (129.250.9.202) 39.236 ms 8 vlan80.csw3.LosAngeles1.Level3.net (4.69.144.190) 75.114 ms vlan90.csw4.LosAngeles1.Level3.net (4.69.144.254) 73.287 ms vlan70.csw2.LosAngeles1.Level3.net (4.69.144.126) 78.397 ms 9 ae-92-92.ebr2.LosAngeles1.Level3.net (4.69.137.29) 73.749 ms ae-62-62.ebr2.LosAngeles1.Level3.net (4.69.137.17) 74.027 ms ae-92-92.ebr2.LosAngeles1.Level3.net (4.69.137.29) 73.149 ms 10 ae-3-3.ebr3.Dallas1.Level3.net (4.69.132.78) 71.008 ms 72.101 ms 71.152 ms 11 ae-83-83.csw3.Dallas1.Level3.net (4.69.151.157) 72.575 ms ae-63-63.csw1.Dallas1.Level3.net (4.69.151.133) 71.824 ms ae-83-83.csw3.Dallas1.Level3.net (4.69.151.157) 72.172 ms 12 ae-4-90.edge9.Dallas1.Level3.net (4.69.145.208) 72.157 ms ae-1-60.edge9.Dallas1.Level3.net (4.69.145.16) 75.127 ms 74.670 ms 13 HUGHES-NETW.edge9.Dallas1.Level3.net (4.30.69.114) 93.496 ms 93.923 ms 94.188 ms 14 * * * 15 * * * 16 * * * 17 * * * 18 * * * 19 * * * 20 * * * 21 * * * 22 * * * 23 * * * 24 * * * 25 * * * 26 * * * 27 * * * 28 * * * 29 * * * 30 * * * On 12/31/2013 09:32 AM, Andrés Herrera wrote:
Using level3 for SIP trunking. Latency spikes up to 3,000 ms since yesterday 11 PM CST.
Andres
Sent from my iPhone
On Dec 31, 2013, at 10:52 AM, "Bryan Inks" <Binks@keyinfo.com <mailto:Binks@keyinfo.com>> wrote:
We're now seeing 20% packet loss and 200+ms latency.
*From:*Outages [mailto:outages-bounces@outages.org] *On Behalf Of *Bryan Inks *Sent:* Tuesday, December 31, 2013 8:25 AM *To:* outages@outages.org <mailto:outages@outages.org> *Subject:* [outages] Possible Issues with Level3 - Confirmed Chicago Area, possibly others
Just received a call from a client here in CA. They are seeing bi-directional issues with Level3 in the Chicago area.
Approximately 7% packet loss and 70+ms latency.
_______________________________________________ Outages mailing list Outages@outages.org <mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
-- () ascii ribbon campaign - against html e-mail /\ www.asciiribbon.org - against proprietary attachments
We’re seeing some stuff down from Level 3 and Zayo in the Herndon/Ashburn area. No confirmation from anyone about where the physical problem is yet.
Seeing this too. Physical problem seems to be a cut. Last update was from Zayo was that they are shooting fibers from M St. in DC and getting splicers en-route. ETA 2 hours as of 11:30am EST. From: Outages [mailto:outages-bounces@outages.org] On Behalf Of George Carey Sent: Thursday, January 02, 2014 12:00 PM To: outages@outages.org Subject: [outages] Northern VA fiber outage We're seeing some stuff down from Level 3 and Zayo in the Herndon/Ashburn area. No confirmation from anyone about where the physical problem is yet. ________________________________ IMPORTANT: The information contained in this email and/or its attachments is confidential. If you are not the intended recipient, please notify the sender immediately by reply and immediately delete this message and all its attachments. Any review, use, reproduction, disclosure or dissemination of this message or any attachment by an unintended recipient is strictly prohibited. Neither this message nor any attachment is intended as or should be construed as an offer, solicitation or recommendation to buy or sell any security or other financial instrument. Neither the sender, his or her employer nor any of their respective affiliates makes any warranties as to the completeness or accuracy of any of the information contained herein or that this message or any of its attachments is free of viruses.
Level 3 is showing nothing on their trouble board On Thu, Jan 2, 2014 at 9:23 AM, Hillyard, Chad <Chad.Hillyard@sig.com>wrote:
Seeing this too. Physical problem seems to be a cut. Last update was from Zayo was that they are shooting fibers from M St. in DC and getting splicers en-route. ETA 2 hours as of 11:30am EST.
*From:* Outages [mailto:outages-bounces@outages.org] *On Behalf Of *George Carey *Sent:* Thursday, January 02, 2014 12:00 PM *To:* outages@outages.org *Subject:* [outages] Northern VA fiber outage
We’re seeing some stuff down from Level 3 and Zayo in the Herndon/Ashburn area.
No confirmation from anyone about where the physical problem is yet.
------------------------------
IMPORTANT: The information contained in this email and/or its attachments is confidential. If you are not the intended recipient, please notify the sender immediately by reply and immediately delete this message and all its attachments. Any review, use, reproduction, disclosure or dissemination of this message or any attachment by an unintended recipient is strictly prohibited. Neither this message nor any attachment is intended as or should be construed as an offer, solicitation or recommendation to buy or sell any security or other financial instrument. Neither the sender, his or her employer nor any of their respective affiliates makes any warranties as to the completeness or accuracy of any of the information contained herein or that this message or any of its attachments is free of viruses.
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Zayo says they have an OC-192 down which happens to be affecting a couple of our fiber paths. Tech should be on site by now but no official ETR yet. On Thu, Jan 2, 2014 at 12:28 PM, Luke Rockwell <luke.rockwell@alumni.berkeley.edu> wrote:
Level 3 is showing nothing on their trouble board
On Thu, Jan 2, 2014 at 9:23 AM, Hillyard, Chad <Chad.Hillyard@sig.com> wrote:
Seeing this too. Physical problem seems to be a cut. Last update was from Zayo was that they are shooting fibers from M St. in DC and getting splicers en-route. ETA 2 hours as of 11:30am EST.
From: Outages [mailto:outages-bounces@outages.org] On Behalf Of George Carey Sent: Thursday, January 02, 2014 12:00 PM To: outages@outages.org Subject: [outages] Northern VA fiber outage
We’re seeing some stuff down from Level 3 and Zayo in the Herndon/Ashburn area.
No confirmation from anyone about where the physical problem is yet.
________________________________
IMPORTANT: The information contained in this email and/or its attachments is confidential. If you are not the intended recipient, please notify the sender immediately by reply and immediately delete this message and all its attachments. Any review, use, reproduction, disclosure or dissemination of this message or any attachment by an unintended recipient is strictly prohibited. Neither this message nor any attachment is intended as or should be construed as an offer, solicitation or recommendation to buy or sell any security or other financial instrument. Neither the sender, his or her employer nor any of their respective affiliates makes any warranties as to the completeness or accuracy of any of the information contained herein or that this message or any of its attachments is free of viruses.
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
-- - Dennis
Our Abovenet circuit in Reston, VA is down but I'm told it is a cut near Loudon County Parkway and the WO & D trail. That would be near the Equinix complex in Ashburn However we're seeing higher latency on Cogent as well. My question is, are there two outages one in DC on M St and one in Ashburn? On Thu, Jan 2, 2014 at 11:46 AM, Dennis Carrasquillo <dennissc@gmail.com>wrote:
Zayo says they have an OC-192 down which happens to be affecting a couple of our fiber paths. Tech should be on site by now but no official ETR yet.
On Thu, Jan 2, 2014 at 12:28 PM, Luke Rockwell <luke.rockwell@alumni.berkeley.edu> wrote:
Level 3 is showing nothing on their trouble board
On Thu, Jan 2, 2014 at 9:23 AM, Hillyard, Chad <Chad.Hillyard@sig.com> wrote:
Seeing this too. Physical problem seems to be a cut. Last update was
from
Zayo was that they are shooting fibers from M St. in DC and getting splicers en-route. ETA 2 hours as of 11:30am EST.
From: Outages [mailto:outages-bounces@outages.org] On Behalf Of George Carey Sent: Thursday, January 02, 2014 12:00 PM To: outages@outages.org Subject: [outages] Northern VA fiber outage
We’re seeing some stuff down from Level 3 and Zayo in the Herndon/Ashburn area.
No confirmation from anyone about where the physical problem is yet.
________________________________
IMPORTANT: The information contained in this email and/or its attachments is confidential. If you are not the intended recipient, please notify the sender immediately by reply and immediately delete this message and all its attachments. Any review, use, reproduction, disclosure or dissemination of this message or any attachment by an unintended recipient is strictly prohibited. Neither this message nor any attachment is intended as or should be construed as an offer, solicitation or recommendation to buy or sell any security or other financial instrument. Neither the sender, his or her employer nor any of their respective affiliates makes any warranties as to the completeness or accuracy of any of the information contained herein or that this message or any of its attachments is free of viruses.
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
-- - Dennis
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Looks like drilling damage to some fiber. "Third Party Construction Bi-Directional drilling impacted our cable. Construction crews are 30 minutes out. Zayo OSP PM are on site at this time." Nothing to do but wait for the repairs. -Dennis On Thu, Jan 2, 2014 at 2:46 PM, Dennis Carrasquillo <dennissc@gmail.com> wrote:
Zayo says they have an OC-192 down which happens to be affecting a couple of our fiber paths. Tech should be on site by now but no official ETR yet.
On Thu, Jan 2, 2014 at 12:28 PM, Luke Rockwell <luke.rockwell@alumni.berkeley.edu> wrote:
Level 3 is showing nothing on their trouble board
On Thu, Jan 2, 2014 at 9:23 AM, Hillyard, Chad <Chad.Hillyard@sig.com> wrote:
Seeing this too. Physical problem seems to be a cut. Last update was from Zayo was that they are shooting fibers from M St. in DC and getting splicers en-route. ETA 2 hours as of 11:30am EST.
From: Outages [mailto:outages-bounces@outages.org] On Behalf Of George Carey Sent: Thursday, January 02, 2014 12:00 PM To: outages@outages.org Subject: [outages] Northern VA fiber outage
We’re seeing some stuff down from Level 3 and Zayo in the Herndon/Ashburn area.
No confirmation from anyone about where the physical problem is yet.
________________________________
IMPORTANT: The information contained in this email and/or its attachments is confidential. If you are not the intended recipient, please notify the sender immediately by reply and immediately delete this message and all its attachments. Any review, use, reproduction, disclosure or dissemination of this message or any attachment by an unintended recipient is strictly prohibited. Neither this message nor any attachment is intended as or should be construed as an offer, solicitation or recommendation to buy or sell any security or other financial instrument. Neither the sender, his or her employer nor any of their respective affiliates makes any warranties as to the completeness or accuracy of any of the information contained herein or that this message or any of its attachments is free of viruses.
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
-- - Dennis
-- - Dennis
We're still OOS. Seems they are having issues exposing enough fiber to begin the splicing operation. George
Just got an update that they have re-spliced the last of the 36 cables and are going to begin closing things up out there. Took them until about 2am to get the cable exposed as apparently it was under the parkway and had been damaged pretty heavily by the drill. The whole section needed to be replaced. We've been looking good here for a few hours. "Dear Valued Zayo Customer, There are 36 ribbons on this fiber.Ribbons 34,35,36 are complete (note Ribbon 34 had light on a fiber pair so it was restored as well). All ribbons with customer traffic per the VA-B27 Impact report have been completed. The ETR is within the hour. We apologize for the inconvenience and thank you for your patience." From: Outages [mailto:outages-bounces@outages.org] On Behalf Of George Carey Sent: Thursday, January 02, 2014 10:11 PM To: Dennis Carrasquillo Cc: outages@outages.org Subject: Re: [outages] Northern VA fiber outage We're still OOS. Seems they are having issues exposing enough fiber to begin the splicing operation. George ________________________________ IMPORTANT: The information contained in this email and/or its attachments is confidential. If you are not the intended recipient, please notify the sender immediately by reply and immediately delete this message and all its attachments. Any review, use, reproduction, disclosure or dissemination of this message or any attachment by an unintended recipient is strictly prohibited. Neither this message nor any attachment is intended as or should be construed as an offer, solicitation or recommendation to buy or sell any security or other financial instrument. Neither the sender, his or her employer nor any of their respective affiliates makes any warranties as to the completeness or accuracy of any of the information contained herein or that this message or any of its attachments is free of viruses.
participants (8)
-
Andrés Herrera -
Bryan Inks -
Dennis Carrasquillo -
George Carey -
Hillyard, Chad -
Jared Geiger -
Luke Rockwell -
Scott Noel-Hemming