Anyone else having issues on Level3? We’re seeing packet loss with anything that traverses Chicago. Thanks, Matt This message and attachment(s), if any, is intended for the sole use of the individual and/or entity for which it is addressed, and may contain information that is privileged, confidential and prohibited from disclosure under applicable law. If you are not the addressee, or authorized to receive this on behalf of the addressee, you are hereby notified that you may not use, copy, disclose or distribute to anyone this message or any part thereof. If you have received this in error, please immediately advise the sender by e-mail and delete this information and all attachments from your computer and network. Thank you.
Yes, have noticed some packet loss as well. Out of midwest, east, west, NYC as well. On Wed, Aug 26, 2015 at 3:16 PM, Matt Elliott via Outages < outages@outages.org> wrote:
Anyone else having issues on Level3? We’re seeing packet loss with anything that traverses Chicago.
Thanks, Matt This message and attachment(s), if any, is intended for the sole use of the individual and/or entity for which it is addressed, and may contain information that is privileged, confidential and prohibited from disclosure under applicable law. If you are not the addressee, or authorized to receive this on behalf of the addressee, you are hereby notified that you may not use, copy, disclose or distribute to anyone this message or any part thereof. If you have received this in error, please immediately advise the sender by e-mail and delete this information and all attachments from your computer and network. Thank you.
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Can confirm. Seeing packet loss in the DFW area. Thanks, Bobin From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Rafael Possamai via Outages Sent: Wednesday, August 26, 2015 1:21 PM To: outages@outages.org Subject: Re: [outages] Level3/TW Telecom issues? Yes, have noticed some packet loss as well. Out of midwest, east, west, NYC as well. On Wed, Aug 26, 2015 at 3:16 PM, Matt Elliott via Outages <outages@outages.org<mailto:outages@outages.org>> wrote: Anyone else having issues on Level3? We’re seeing packet loss with anything that traverses Chicago. Thanks, Matt This message and attachment(s), if any, is intended for the sole use of the individual and/or entity for which it is addressed, and may contain information that is privileged, confidential and prohibited from disclosure under applicable law. If you are not the addressee, or authorized to receive this on behalf of the addressee, you are hereby notified that you may not use, copy, disclose or distribute to anyone this message or any part thereof. If you have received this in error, please immediately advise the sender by e-mail and delete this information and all attachments from your computer and network. Thank you. _______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages
Can confirm here as well.. I'm in Nashville, TN, and work remotely, home base is in Milwaukee, and I'm seeing 35% loss Signature To ensure prompt service, please use "Reply All" when responding Robert Colon | TCSA Tier 3 | EthoStream LLC | P: (877) 282-2519 x2114 | F: (414) 258-8307 | 20800 Swenson Drive, Suite 175 Waukesha, WI 53186 | www.telkonet.com <http://www.telkonet.com/> | www.ethostream.com <http://www.ethostream.com/> | @Telkonet <https://twitter.com/telkonet> On 8/26/2015 3:24 PM, Bobin Joseph via Outages wrote:
Can confirm. Seeing packet loss in the DFW area.
Thanks,
Bobin
*From:*Outages [mailto:outages-bounces@outages.org] *On Behalf Of *Rafael Possamai via Outages *Sent:* Wednesday, August 26, 2015 1:21 PM *To:* outages@outages.org *Subject:* Re: [outages] Level3/TW Telecom issues?
Yes, have noticed some packet loss as well. Out of midwest, east, west, NYC as well.
On Wed, Aug 26, 2015 at 3:16 PM, Matt Elliott via Outages <outages@outages.org <mailto:outages@outages.org>> wrote:
Anyone else having issues on Level3? We’re seeing packet loss with anything that traverses Chicago.
Thanks,
Matt
This message and attachment(s), if any, is intended for the sole use of the individual and/or entity for which it is addressed, and may contain information that is privileged, confidential and prohibited from disclosure under applicable law. If you are not the addressee, or authorized to receive this on behalf of the addressee, you are hereby notified that you may not use, copy, disclose or distribute to anyone this message or any part thereof. If you have received this in error, please immediately advise the sender by e-mail and delete this information and all attachments from your computer and network. Thank you.
_______________________________________________ Outages mailing list Outages@outages.org <mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
-- ------------------------------ <http://www.telkonet.com> <http://www.ethostream.com> This e-mail and any files transmitted with it are private and confidential and are solely for the use of the addressee. It may contain material which is legally privileged. If you are not the addressee or the person responsible for delivering to the addressee, be advised that you have received this e-mail in error and that any use of it is strictly prohibited. If you have received this e-mail in error please contact us at (414) 223-0473.
Seeing similar impact on a circuit in the Lexington, KY area. Also hearing unconfirmed rumor that Level 3 back bone engineering has been engaged. Bob From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Bob Colon via Outages Sent: Wednesday, August 26, 2015 3:29 PM To: outages@outages.org Subject: Re: [outages] Level3/TW Telecom issues? Can confirm here as well.. I'm in Nashville, TN, and work remotely, home base is in Milwaukee, and I'm seeing 35% loss To ensure prompt service, please use "Reply All" when responding Robert Colon | TCSA Tier 3 | EthoStream LLC | P: (877) 282-2519 x2114 | F: (414) 258-8307 | 20800 Swenson Drive, Suite 175 Waukesha, WI 53186 | www.telkonet.com<http://cp.mcafee.com/d/2DRPoA86Qm6bK9Td7by9KVJ6WqbVEVsjvKrhKCy-qen6mnzqdQQnPhOUC-yrhKCVtVYs-Or84ZxqJyEalGHY9JGPY_R-ndGRl-4SRp-vW_bCXWyDW3X_nV54sYyUzRXBQXzHYCMZtZO_bnhIyyHssNOEuvkzaT0QSCrudEZtazp2BEO8urKr01jYYHk6YKrGTfifQbAaJMJZ0llKuAvFM04SyeousdwLQzh0qmWJPQzZ3iWq80HVDPSs182Cy04-vWVEwBizvQ96SjhOrlqKc> | www.ethostream.com<http://cp.mcafee.com/d/5fHCN0p43qb35T4XCzBN4TsSztd5YQsK9LTdETjhvd7bzbbNJ6WqbVEVsjvhdETjsKY-evpdA2uMJmNk5aRl-4SRp-vW_bCRqG_2rqI_fZvBPtZhjZ1Z_HYyyeuhshWZOWtNR-jouK-VvBHEShhlKeoVkffGhBrwqrpdL6QuKBhIxiQp4fdTdw0LCjHisvRnBYdRrDF7W5O5mUm-waGTfifQU02rh7cfe6MnWhEwdbtmVWh-xFtd40lYPVXe0A1jh02vfZsQgiFhLW4zr9EVdWa8W> | @Telkonet<http://cp.mcafee.com/d/avndxMO86Qm6bK9Td7by9KVJ6WqbVEVsjvKrhKCy-qen6mnzqdQQnPhOUC-yrhKCVtVYs-Or84ZxqJyEalGHY9JGPY_R-ndGRl-4SRp-vW_bCXWyDW3X_nV54sYyUzRXBQXzHYCMZtZO_bnhIyyHssNOEuvkzaT0QSMrudEZtazp2BEO8urKr9PCJhbcCoBevNVv9-ulG3jtmVWh-xsxlK5LE2GJPQzZe00CQhP3PxI5-Aq83iTlKuAvEqnjh05vc-uPw90kQg0DP_nd44Gkr-x8SOqejqJM6nsyJ9IWT> On 8/26/2015 3:24 PM, Bobin Joseph via Outages wrote: Can confirm. Seeing packet loss in the DFW area. Thanks, Bobin From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Rafael Possamai via Outages Sent: Wednesday, August 26, 2015 1:21 PM To: outages@outages.org<mailto:outages@outages.org> Subject: Re: [outages] Level3/TW Telecom issues? Yes, have noticed some packet loss as well. Out of midwest, east, west, NYC as well. On Wed, Aug 26, 2015 at 3:16 PM, Matt Elliott via Outages <outages@outages.org<mailto:outages@outages.org>> wrote: Anyone else having issues on Level3? We're seeing packet loss with anything that traverses Chicago. Thanks, Matt This message and attachment(s), if any, is intended for the sole use of the individual and/or entity for which it is addressed, and may contain information that is privileged, confidential and prohibited from disclosure under applicable law. If you are not the addressee, or authorized to receive this on behalf of the addressee, you are hereby notified that you may not use, copy, disclose or distribute to anyone this message or any part thereof. If you have received this in error, please immediately advise the sender by e-mail and delete this information and all attachments from your computer and network. Thank you. _______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages<http://cp.mcafee.com/d/avndxMO76Qm6bK9Td7by9KVJ6WqbVEVsjvKrhKCy-qen6mnzqdQQnPhOUC-yrhKCVtVYs-Or84ZxqJyEalGHY9JGPY_R-ndGRl-4SRp-vW_bCXWyDW3X_nV54sYyUzRXBQXzHYCMZtZO_bnhIyyHssNOEuvkzaT0QS-rudEZtazp2BEO8urKr9PCJhbcmrIlU6A_zMdMjlS67OFek7qVlKuAvErGTfifQbAaJMJZ0llKuAvFM04SyeousdwLQzh0qmWJPQzZ3iWq80HVDPSs182Cy04-vWVEwBizvQ96SjhOrhLBCZJyY> _______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages<http://cp.mcafee.com/d/k-Kr3x8p3zqb35T4XCzBN4TsSztd5YQsK9LTdETjhvd7bzbbNJ6WqbVEVsjvhdETjsKY-evpdA2uMJmNk5aRl-4SRp-vW_bCRqG_2rqI_fZvBPtZhjZ1Z_HYyyeuhshWZOWtNR-jouK-VvBHEShhlKeoVkffGhBrwqrsdL6QuKBhIxiQp4fdTdAVPmEBCbdSaY3ivNU6U9GX33VkDa3JsGTfifQdRrDF7W5O5mUm-waGTfifQU02rh7cfe6MnWhEwdbtmVWh-xFtd40lYPVXe0A1jh02vfZsQgiFhLW4zr9EVdSMvRkaYQ> ________________________________ [http://www.telkonet.com/images/signature/telkonetlogo.gif][http://www.telkonet.com/images/signature/telkosig.gif]<http://cp.mcafee.com/d/avndxNJ5xyXytPhOUyrKrhKCy-qen4TXCQrFELCzBNBBUSztd5YQsK9LECQrFKnuv7fICO1fomHoG2BqG_2rqI_fZvBPqJlvxdJmvD-LOVK-EF-w-_R-hh7f8K8ZuVteUW_9IfnvsLORQr8EGT7csG7DR8OJMddzzrNJ7HFkr8kJ6h3PtPo0avDBqwTBPtmVWh-xsxlK5LE2GJPQzZe00CQhP3PxI5-Aq83iTlKuAvEqnjh05vc-uPw90kQg0DP_nd44Gkr-x8SOqejoW-S5yzCSd5N>[http://www.ethostream.com/images/signature/ethosig.gif]<http://cp.mcafee.com/d/avndzgA73gOrhooKUDsQsK8CXCQrFELCzBNd-VJ6WqbVEVsppudETjhvd7byrW9J6WrBTDNPX9IwjS5GSawFmGLMCSHfP_nVsSHlnUjrlDV_HYKrLGavEfLZvAkhPObyfnKnjKeLOr3RTTbYJt6OaaJNP7axVZicHs3jrNJUSzRQGdAamz8xVKVI05YOtqjz-GYKrGTfifQbAaJMJZ0llKuAvFM04SyeousdwLQzh0qmWJPQzZ3iWq80HVDPSs182Cy04-vWVEwBizvQ96SjhOrLkf6> This e-mail and any files transmitted with it are private and confidential and are solely for the use of the addressee. It may contain material which is legally privileged. If you are not the addressee or the person responsible for delivering to the addressee, be advised that you have received this e-mail in error and that any use of it is strictly prohibited. If you have received this e-mail in error please contact us at (414) 223-0473. The information contained in this E-mail message is privileged, confidential, and may be protected from disclosure; please be aware that any unauthorized use, printing, copying, disclosure or dissemination of this communication may be subject to legal restriction or sanction. If you think that you have received this E-mail message in error, please reply to the sender. This E-mail message and any attachments have been scanned for viruses and are believed to be free of any virus or other defect that might affect any computer system into which it is received and opened. It is the responsibility, however, of the recipient to ensure that it is virus free.
Don't know if this is the cause of all of the issues on this thread but Level 3 is experiencing issues related to one or more fiber cuts in NYC. Event Ticket ID: 9631341 Tim From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Bob Hancock via Outages Sent: Wednesday, August 26, 2015 4:35 PM To: outages@outages.org Subject: Re: [outages] Level3/TW Telecom issues? Seeing similar impact on a circuit in the Lexington, KY area. Also hearing unconfirmed rumor that Level 3 back bone engineering has been engaged. Bob From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Bob Colon via Outages Sent: Wednesday, August 26, 2015 3:29 PM To: outages@outages.org Subject: Re: [outages] Level3/TW Telecom issues? Can confirm here as well.. I'm in Nashville, TN, and work remotely, home base is in Milwaukee, and I'm seeing 35% loss To ensure prompt service, please use "Reply All" when responding Robert Colon | TCSA Tier 3 | EthoStream LLC | P: (877) 282-2519 x2114 | F: (414) 258-8307 | 20800 Swenson Drive, Suite 175 Waukesha, WI 53186 | www.telkonet.com<http://cp.mcafee.com/d/2DRPoA86Qm6bK9Td7by9KVJ6WqbVEVsjvKrhKCy-qen6mnzqdQQnPhOUC-yrhKCVtVYs-Or84ZxqJyEalGHY9JGPY_R-ndGRl-4SRp-vW_bCXWyDW3X_nV54sYyUzRXBQXzHYCMZtZO_bnhIyyHssNOEuvkzaT0QSCrudEZtazp2BEO8urKr01jYYHk6YKrGTfifQbAaJMJZ0llKuAvFM04SyeousdwLQzh0qmWJPQzZ3iWq80HVDPSs182Cy04-vWVEwBizvQ96SjhOrlqKc> | www.ethostream.com<http://cp.mcafee.com/d/5fHCN0p43qb35T4XCzBN4TsSztd5YQsK9LTdETjhvd7bzbbNJ6WqbVEVsjvhdETjsKY-evpdA2uMJmNk5aRl-4SRp-vW_bCRqG_2rqI_fZvBPtZhjZ1Z_HYyyeuhshWZOWtNR-jouK-VvBHEShhlKeoVkffGhBrwqrpdL6QuKBhIxiQp4fdTdw0LCjHisvRnBYdRrDF7W5O5mUm-waGTfifQU02rh7cfe6MnWhEwdbtmVWh-xFtd40lYPVXe0A1jh02vfZsQgiFhLW4zr9EVdWa8W> | @Telkonet<http://cp.mcafee.com/d/avndxMO86Qm6bK9Td7by9KVJ6WqbVEVsjvKrhKCy-qen6mnzqdQQnPhOUC-yrhKCVtVYs-Or84ZxqJyEalGHY9JGPY_R-ndGRl-4SRp-vW_bCXWyDW3X_nV54sYyUzRXBQXzHYCMZtZO_bnhIyyHssNOEuvkzaT0QSMrudEZtazp2BEO8urKr9PCJhbcCoBevNVv9-ulG3jtmVWh-xsxlK5LE2GJPQzZe00CQhP3PxI5-Aq83iTlKuAvEqnjh05vc-uPw90kQg0DP_nd44Gkr-x8SOqejqJM6nsyJ9IWT> On 8/26/2015 3:24 PM, Bobin Joseph via Outages wrote: Can confirm. Seeing packet loss in the DFW area. Thanks, Bobin From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Rafael Possamai via Outages Sent: Wednesday, August 26, 2015 1:21 PM To: outages@outages.org<mailto:outages@outages.org> Subject: Re: [outages] Level3/TW Telecom issues? Yes, have noticed some packet loss as well. Out of midwest, east, west, NYC as well. On Wed, Aug 26, 2015 at 3:16 PM, Matt Elliott via Outages <outages@outages.org<mailto:outages@outages.org>> wrote: Anyone else having issues on Level3? We're seeing packet loss with anything that traverses Chicago. Thanks, Matt This message and attachment(s), if any, is intended for the sole use of the individual and/or entity for which it is addressed, and may contain information that is privileged, confidential and prohibited from disclosure under applicable law. If you are not the addressee, or authorized to receive this on behalf of the addressee, you are hereby notified that you may not use, copy, disclose or distribute to anyone this message or any part thereof. If you have received this in error, please immediately advise the sender by e-mail and delete this information and all attachments from your computer and network. Thank you. _______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages<http://cp.mcafee.com/d/avndxMO76Qm6bK9Td7by9KVJ6WqbVEVsjvKrhKCy-qen6mnzqdQQnPhOUC-yrhKCVtVYs-Or84ZxqJyEalGHY9JGPY_R-ndGRl-4SRp-vW_bCXWyDW3X_nV54sYyUzRXBQXzHYCMZtZO_bnhIyyHssNOEuvkzaT0QS-rudEZtazp2BEO8urKr9PCJhbcmrIlU6A_zMdMjlS67OFek7qVlKuAvErGTfifQbAaJMJZ0llKuAvFM04SyeousdwLQzh0qmWJPQzZ3iWq80HVDPSs182Cy04-vWVEwBizvQ96SjhOrhLBCZJyY> _______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages<http://cp.mcafee.com/d/k-Kr3x8p3zqb35T4XCzBN4TsSztd5YQsK9LTdETjhvd7bzbbNJ6WqbVEVsjvhdETjsKY-evpdA2uMJmNk5aRl-4SRp-vW_bCRqG_2rqI_fZvBPtZhjZ1Z_HYyyeuhshWZOWtNR-jouK-VvBHEShhlKeoVkffGhBrwqrsdL6QuKBhIxiQp4fdTdAVPmEBCbdSaY3ivNU6U9GX33VkDa3JsGTfifQdRrDF7W5O5mUm-waGTfifQU02rh7cfe6MnWhEwdbtmVWh-xFtd40lYPVXe0A1jh02vfZsQgiFhLW4zr9EVdSMvRkaYQ> ________________________________ [cid:~WRD000.jpg][cid:~WRD000.jpg]<http://cp.mcafee.com/d/avndxNJ5xyXytPhOUyrKrhKCy-qen4TXCQrFELCzBNBBUSztd5YQsK9LECQrFKnuv7fICO1fomHoG2BqG_2rqI_fZvBPqJlvxdJmvD-LOVK-EF-w-_R-hh7f8K8ZuVteUW_9IfnvsLORQr8EGT7csG7DR8OJMddzzrNJ7HFkr8kJ6h3PtPo0avDBqwTBPtmVWh-xsxlK5LE2GJPQzZe00CQhP3PxI5-Aq83iTlKuAvEqnjh05vc-uPw90kQg0DP_nd44Gkr-x8SOqejoW-S5yzCSd5N>[cid:~WRD000.jpg]<http://cp.mcafee.com/d/avndzgA73gOrhooKUDsQsK8CXCQrFELCzBNd-VJ6WqbVEVsppudETjhvd7byrW9J6WrBTDNPX9IwjS5GSawFmGLMCSHfP_nVsSHlnUjrlDV_HYKrLGavEfLZvAkhPObyfnKnjKeLOr3RTTbYJt6OaaJNP7axVZicHs3jrNJUSzRQGdAamz8xVKVI05YOtqjz-GYKrGTfifQbAaJMJZ0llKuAvFM04SyeousdwLQzh0qmWJPQzZ3iWq80HVDPSs182Cy04-vWVEwBizvQ96SjhOrLkf6> This e-mail and any files transmitted with it are private and confidential and are solely for the use of the addressee. It may contain material which is legally privileged. If you are not the addressee or the person responsible for delivering to the addressee, be advised that you have received this e-mail in error and that any use of it is strictly prohibited. If you have received this e-mail in error please contact us at (414) 223-0473. The information contained in this E-mail message is privileged, confidential, and may be protected from disclosure; please be aware that any unauthorized use, printing, copying, disclosure or dissemination of this communication may be subject to legal restriction or sanction. If you think that you have received this E-mail message in error, please reply to the sender. This E-mail message and any attachments have been scanned for viruses and are believed to be free of any virus or other defect that might affect any computer system into which it is received and opened. It is the responsibility, however, of the recipient to ensure that it is virus free.
Seeing some TW issues in Albuquerque at this hop: abq1-ar3-xe-2-0-0-0.us.twtelecom.net From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Bobin Joseph via Outages Sent: Wednesday, August 26, 2015 1:25 PM To: outages@outages.org<mailto:outages@outages.org> Subject: Re: [outages] Level3/TW Telecom issues? Can confirm. Seeing packet loss in the DFW area. Thanks, Bobin From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Rafael Possamai via Outages Sent: Wednesday, August 26, 2015 1:21 PM To: outages@outages.org<mailto:outages@outages.org> Subject: Re: [outages] Level3/TW Telecom issues? Yes, have noticed some packet loss as well. Out of midwest, east, west, NYC as well. On Wed, Aug 26, 2015 at 3:16 PM, Matt Elliott via Outages <outages@outages.org<mailto:outages@outages.org>> wrote: Anyone else having issues on Level3? We’re seeing packet loss with anything that traverses Chicago. Thanks, Matt This message and attachment(s), if any, is intended for the sole use of the individual and/or entity for which it is addressed, and may contain information that is privileged, confidential and prohibited from disclosure under applicable law. If you are not the addressee, or authorized to receive this on behalf of the addressee, you are hereby notified that you may not use, copy, disclose or distribute to anyone this message or any part thereof. If you have received this in error, please immediately advise the sender by e-mail and delete this information and all attachments from your computer and network. Thank you. _______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages Attention: Information contained in this message and or attachments is intended only for the recipient(s) named above and may contain confidential and or privileged material that is protected under State or Federal law. If you are not the intended recipient, any disclosure, copying, distribution or action taken on it is prohibited. If you believe you have received this email in error, please contact the sender, delete this email and destroy all copies.
Here's a graph showing some packet loss: http://imgur.com/KVAzetA On Wed, Aug 26, 2015 at 3:16 PM, Matt Elliott via Outages < outages@outages.org> wrote:
Anyone else having issues on Level3? We’re seeing packet loss with anything that traverses Chicago.
Thanks, Matt This message and attachment(s), if any, is intended for the sole use of the individual and/or entity for which it is addressed, and may contain information that is privileged, confidential and prohibited from disclosure under applicable law. If you are not the addressee, or authorized to receive this on behalf of the addressee, you are hereby notified that you may not use, copy, disclose or distribute to anyone this message or any part thereof. If you have received this in error, please immediately advise the sender by e-mail and delete this information and all attachments from your computer and network. Thank you.
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Was anyone able to get some useful information out of Level3 regarding this yesterday? Also is anyone else seeing similar issues? On Aug 26, 2015 2:40 PM, "Rafael Possamai via Outages" <outages@outages.org> wrote:
Here's a graph showing some packet loss: http://imgur.com/KVAzetA
On Wed, Aug 26, 2015 at 3:16 PM, Matt Elliott via Outages < outages@outages.org> wrote:
Anyone else having issues on Level3? We’re seeing packet loss with anything that traverses Chicago.
Thanks, Matt This message and attachment(s), if any, is intended for the sole use of the individual and/or entity for which it is addressed, and may contain information that is privileged, confidential and prohibited from disclosure under applicable law. If you are not the addressee, or authorized to receive this on behalf of the addressee, you are hereby notified that you may not use, copy, disclose or distribute to anyone this message or any part thereof. If you have received this in error, please immediately advise the sender by e-mail and delete this information and all attachments from your computer and network. Thank you.
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
We got this from Level3 Hello, The network outage has been resolved. *Event Conclusion Summary* Start: August 26, 2015 19:15 GMT Stop: August 27, 2015 00:14 GMT Root Cause: An unknown issue in Chicago, IL was impacting IP services. Fix Action: Adjustments were made to resolve the issue and restore services. Summary: The IP NOC responded to an unknown issue in Chicago, IL that was impacting IP services. Tier III Technical Support were engaged to assist with investigations, and adjustments were made to resolve the issue and stabilize traffic. Monitoring continues to ensure stability. Should further issues be experienced, please contact the Level 3 Technical Service Center. Thanks, Bobin From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Ryan Gelobter via Outages Sent: Thursday, August 27, 2015 9:44 AM To: Rafael Possamai Cc: outages@outages.org Subject: Re: [outages] Level3/TW Telecom issues? Was anyone able to get some useful information out of Level3 regarding this yesterday? Also is anyone else seeing similar issues? On Aug 26, 2015 2:40 PM, "Rafael Possamai via Outages" <outages@outages.org<mailto:outages@outages.org>> wrote: Here's a graph showing some packet loss: http://imgur.com/KVAzetA On Wed, Aug 26, 2015 at 3:16 PM, Matt Elliott via Outages <outages@outages.org<mailto:outages@outages.org>> wrote: Anyone else having issues on Level3? We’re seeing packet loss with anything that traverses Chicago. Thanks, Matt This message and attachment(s), if any, is intended for the sole use of the individual and/or entity for which it is addressed, and may contain information that is privileged, confidential and prohibited from disclosure under applicable law. If you are not the addressee, or authorized to receive this on behalf of the addressee, you are hereby notified that you may not use, copy, disclose or distribute to anyone this message or any part thereof. If you have received this in error, please immediately advise the sender by e-mail and delete this information and all attachments from your computer and network. Thank you. _______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages _______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages
we are still seeing significant packet loss to 3 of our sites in the Midwest. On Thu, Aug 27, 2015 at 12:47 PM, Bobin Joseph via Outages < outages@outages.org> wrote:
We got this from Level3
Hello,
The network outage has been resolved.
*Event Conclusion Summary*
Start: August 26, 2015 19:15 GMT Stop: August 27, 2015 00:14 GMT
Root Cause: An unknown issue in Chicago, IL was impacting IP services. Fix Action: Adjustments were made to resolve the issue and restore services.
Summary: The IP NOC responded to an unknown issue in Chicago, IL that was impacting IP services. Tier III Technical Support were engaged to assist with investigations, and adjustments were made to resolve the issue and stabilize traffic. Monitoring continues to ensure stability. Should further issues be experienced, please contact the Level 3 Technical Service Center.
Thanks,
Bobin
*From:* Outages [mailto:outages-bounces@outages.org] *On Behalf Of *Ryan Gelobter via Outages *Sent:* Thursday, August 27, 2015 9:44 AM *To:* Rafael Possamai *Cc:* outages@outages.org *Subject:* Re: [outages] Level3/TW Telecom issues?
Was anyone able to get some useful information out of Level3 regarding this yesterday? Also is anyone else seeing similar issues?
On Aug 26, 2015 2:40 PM, "Rafael Possamai via Outages" < outages@outages.org> wrote:
Here's a graph showing some packet loss: http://imgur.com/KVAzetA
On Wed, Aug 26, 2015 at 3:16 PM, Matt Elliott via Outages < outages@outages.org> wrote:
Anyone else having issues on Level3? We’re seeing packet loss with anything that traverses Chicago.
Thanks,
Matt
This message and attachment(s), if any, is intended for the sole use of the individual and/or entity for which it is addressed, and may contain information that is privileged, confidential and prohibited from disclosure under applicable law. If you are not the addressee, or authorized to receive this on behalf of the addressee, you are hereby notified that you may not use, copy, disclose or distribute to anyone this message or any part thereof. If you have received this in error, please immediately advise the sender by e-mail and delete this information and all attachments from your computer and network. Thank you.
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
-- To him who is able to keep you from falling and to present you before his glorious presence without fault and with great joy
We were seeing substantial loss through Chicago this morning but it seems to have subsided at this time. Bob Hancock On 08/27/2015 12:19 PM, Steven Fischer via Outages wrote:
we are still seeing significant packet loss to 3 of our sites in the Midwest.
On Thu, Aug 27, 2015 at 12:47 PM, Bobin Joseph via Outages <outages@outages.org <mailto:outages@outages.org>> wrote:
We got this from Level3____
__ __
Hello,
The network outage has been resolved.
*Event Conclusion Summary*
Start: August 26, 2015 19:15 GMT Stop: August 27, 2015 00:14 GMT
Root Cause: An unknown issue in Chicago, IL was impacting IP services. Fix Action: Adjustments were made to resolve the issue and restore services.
Summary: The IP NOC responded to an unknown issue in Chicago, IL that was impacting IP services. Tier III Technical Support were engaged to assist with investigations, and adjustments were made to resolve the issue and stabilize traffic. Monitoring continues to ensure stability. Should further issues be experienced, please contact the Level 3 Technical Service Center.
____
__ __
Thanks,____
Bobin____
__ __
*From:*Outages [mailto:outages-bounces@outages.org <mailto:outages-bounces@outages.org>] *On Behalf Of *Ryan Gelobter via Outages *Sent:* Thursday, August 27, 2015 9:44 AM *To:* Rafael Possamai *Cc:* outages@outages.org <mailto:outages@outages.org> *Subject:* Re: [outages] Level3/TW Telecom issues?____
__ __
Was anyone able to get some useful information out of Level3 regarding this yesterday? Also is anyone else seeing similar issues?____
On Aug 26, 2015 2:40 PM, "Rafael Possamai via Outages" <outages@outages.org <mailto:outages@outages.org>> wrote:____
Here's a graph showing some packet loss: http://imgur.com/KVAzetA <http://cp.mcafee.com/d/FZsSd38w83hJ5xyXxEVsKCMrKrhKCy-qen4TXCQrFELCzBNBBUSztd5YQsK9LECQrFKnuv7fICO1fomHoG2BqG_2rqI_fZvBPqJlvxdJmvD-LOVJV1NxR1Z_HYYMM-YeoWZOWarxEVV5MsMqemrKth5dqWqJTT-l3PWApmU6CQjr1J7HFkr8kJ6h3PtPo0eiF5KfbU1gp41jVy1KHsZ8_gKgGT2TQ1lmVWh-D00jqb2pEVsdwLQzh0qmWJPQzZ3iWq80HVDPSs182Cy04-vWVEwBizvQ96SjhOrehG_98MC>____
__ __
__ __
On Wed, Aug 26, 2015 at 3:16 PM, Matt Elliott via Outages <outages@outages.org <mailto:outages@outages.org>> wrote:____
Anyone else having issues on Level3? We’re seeing packet loss with anything that traverses Chicago.____
__ __
Thanks,____
Matt____
This message and attachment(s), if any, is intended for the sole use of the individual and/or entity for which it is addressed, and may contain information that is privileged, confidential and prohibited from disclosure under applicable law. If you are not the addressee, or authorized to receive this on behalf of the addressee, you are hereby notified that you may not use, copy, disclose or distribute to anyone this message or any part thereof. If you have received this in error, please immediately advise the sender by e-mail and delete this information and all attachments from your computer and network. Thank you. ____
_______________________________________________ Outages mailing list Outages@outages.org <mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages <http://cp.mcafee.com/d/avndy0QrhooKUqenbFI6XCQrFELCzBNd-VJ6WqbVEVsppudETjhvd7byrW9J6WrBTDNPX9IwjS5GSawFmGLMCSHfP_nVsSHlnUjrlDV_HYKrugsotgvvW_fccfL3CeLsKyCUqeuhs7c6zBCXDkhjmKCHtZ_BgY-F6lK1FJcSMrhWWl6O5bhAgYTsSjDdqymoIToHMd9_7wrwCHIcfBisEeROHsZ8_gTlKuAvEn8lrxrW0GHsZ8_jw09J5xcQsK6MnWhEwdbtmVWh-xFtd40lYPVXe0A1jh02vfZsQgiFhLW4zr9EVdBPdH>____
__ __
_______________________________________________ Outages mailing list Outages@outages.org <mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages <http://cp.mcafee.com/d/2DRPoQ938ArhooKUqenbFI6XCQrFELCzBNd-VJ6WqbVEVsppudETjhvd7byrW9J6WrBTDNPX9IwjS5GSawFmGLMCSHfP_nVsSHlnUjrlDV_HYKrugsotgvvW_fccfL3CeLsKyCUqeuhs7c6zBCXDkhjmKCHtZ_BgY-F6lK1FJASMrhWWl6O5bhAgYTsSjDdqymoIToHMd9_7wrwCHIcfBisEeROHsZ8_gTlKuAvEn8lrxrW0GHsZ8_jw09J5xcQsK6MnWhEwdbtmVWh-xFtd40lYPVXe0A1jh02vfZsQgiFhLW4zr9EVdWoUq>____
_______________________________________________ Outages mailing list Outages@outages.org <mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages <http://cp.mcafee.com/d/k-Kr41ESyMNtMQsKnjodTdETjhvd7byrZPqdQQnPhOUOOYrhKCy-qen4TQjqdQTbLfzDSjp0DIblIl1iJlvxdJmvD-LOVJmGLMCSHfP_nVsSYwUMWw-_R-uoovu7ctuVt5dMQsYyUeod7bdTeEyCJtdmXX_axVZicHs3jr1JwSzRQGdAamz8xVKVIDeqR4INpKNnwqj-f0T1dnoovaAVgtHBmVWh-xKHsZ8_gKgGT2TQ1lmVWh-D00jqb2pEVsdwLQzh0qmWJPQzZ3iWq80HVDPSs182Cy04-vWVEwBizvQ96SjhOrVbh4>
-- To him who is able to keep you from falling and to present you before his glorious presence without fault and with great joy
_______________________________________________ Outages mailing list Outages@outages.org http://cp.mcafee.com/d/avndzhJ5xyXxEVsKC-rKrhKCy-qen4TXCQrFELCzBNBBUSztd5YQs...
The information contained in this E-mail message is privileged, confidential, and may be protected from disclosure; please be aware that any unauthorized use, printing, copying, disclosure or dissemination of this communication may be subject to legal restriction or sanction. If you think that you have received this E-mail message in error, please reply to the sender. This E-mail message and any attachments have been scanned for viruses and are believed to be free of any virus or other defect that might affect any computer system into which it is received and opened. It is the responsibility, however, of the recipient to ensure that it is virus free.
We are seeing the same things that happened yesterday on our circuits in Milwaukee and Chicago areas. Lots of packet loss and large latency through Chicago exchanges. On Thu, Aug 27, 2015 at 12:33 PM, Bob Hancock via Outages < outages@outages.org> wrote:
We were seeing substantial loss through Chicago this morning but it seems to have subsided at this time.
Bob Hancock
On 08/27/2015 12:19 PM, Steven Fischer via Outages wrote:
we are still seeing significant packet loss to 3 of our sites in the Midwest.
On Thu, Aug 27, 2015 at 12:47 PM, Bobin Joseph via Outages <outages@outages.org <mailto:outages@outages.org>> wrote:
We got this from Level3____
__ __
Hello,
The network outage has been resolved.
*Event Conclusion Summary*
Start: August 26, 2015 19:15 GMT Stop: August 27, 2015 00:14 GMT
Root Cause: An unknown issue in Chicago, IL was impacting IP services. Fix Action: Adjustments were made to resolve the issue and restore services.
Summary: The IP NOC responded to an unknown issue in Chicago, IL that was impacting IP services. Tier III Technical Support were engaged to assist with investigations, and adjustments were made to resolve the issue and stabilize traffic. Monitoring continues to ensure stability. Should further issues be experienced, please contact the Level 3 Technical Service Center.
____
__ __
Thanks,____
Bobin____
__ __
*From:*Outages [mailto:outages-bounces@outages.org <mailto:outages-bounces@outages.org>] *On Behalf Of *Ryan Gelobter via Outages *Sent:* Thursday, August 27, 2015 9:44 AM *To:* Rafael Possamai *Cc:* outages@outages.org <mailto:outages@outages.org> *Subject:* Re: [outages] Level3/TW Telecom issues?____
__ __
Was anyone able to get some useful information out of Level3 regarding this yesterday? Also is anyone else seeing similar issues?____
On Aug 26, 2015 2:40 PM, "Rafael Possamai via Outages" <outages@outages.org <mailto:outages@outages.org>> wrote:____
Here's a graph showing some packet loss: http://imgur.com/KVAzetA < http://cp.mcafee.com/d/FZsSd38w83hJ5xyXxEVsKCMrKrhKCy-qen4TXCQrFELCzBNBBUSzt... ____
__ __
__ __
On Wed, Aug 26, 2015 at 3:16 PM, Matt Elliott via Outages <outages@outages.org <mailto:outages@outages.org>> wrote:____
Anyone else having issues on Level3? We’re seeing packet loss with anything that traverses Chicago.____
__ __
Thanks,____
Matt____
This message and attachment(s), if any, is intended for the sole use of the individual and/or entity for which it is addressed, and may contain information that is privileged, confidential and prohibited from disclosure under applicable law. If you are not the addressee, or authorized to receive this on behalf of the addressee, you are hereby notified that you may not use, copy, disclose or distribute to anyone this message or any part thereof. If you have received this in error, please immediately advise the sender by e-mail and delete this information and all attachments from your computer and network. Thank you. ____
_______________________________________________ Outages mailing list Outages@outages.org <mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages < http://cp.mcafee.com/d/avndy0QrhooKUqenbFI6XCQrFELCzBNd-VJ6WqbVEVsppudETjhvd... ____
__ __
_______________________________________________ Outages mailing list Outages@outages.org <mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages < http://cp.mcafee.com/d/2DRPoQ938ArhooKUqenbFI6XCQrFELCzBNd-VJ6WqbVEVsppudETj... ____
_______________________________________________ Outages mailing list Outages@outages.org <mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages < http://cp.mcafee.com/d/k-Kr41ESyMNtMQsKnjodTdETjhvd7byrZPqdQQnPhOUOOYrhKCy-q...
-- To him who is able to keep you from falling and to present you before his glorious presence without fault and with great joy
_______________________________________________ Outages mailing list Outages@outages.org
http://cp.mcafee.com/d/avndzhJ5xyXxEVsKC-rKrhKCy-qen4TXCQrFELCzBNBBUSztd5YQs...
The information contained in this E-mail message is privileged, confidential, and may be protected from disclosure; please be aware that any unauthorized use, printing, copying, disclosure or dissemination of this communication may be subject to legal restriction or sanction. If you think that you have received this E-mail message in error, please reply to the sender. This E-mail message and any attachments have been scanned for viruses and are believed to be free of any virus or other defect that might affect any computer system into which it is received and opened. It is the responsibility, however, of the recipient to ensure that it is virus free. _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
We just got another update saying they still have issues. IP services are affected across multiple markets. The IP NOC and IP and Transport Tier III are actively troubleshooting the issue. Operations Engineering is engaged and Field Services is on site in Chicago, IL investigating the issue. tw telecom has identified a network failure affecting your services. tw telecom tier-2 group is investigating the issue at this time. Thanks, Bobin From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Aaron Gross via Outages Sent: Thursday, August 27, 2015 10:55 AM To: Bob Hancock Cc: outages@outages.org Subject: Re: [outages] Level3/TW Telecom issues? We are seeing the same things that happened yesterday on our circuits in Milwaukee and Chicago areas. Lots of packet loss and large latency through Chicago exchanges. On Thu, Aug 27, 2015 at 12:33 PM, Bob Hancock via Outages <outages@outages.org<mailto:outages@outages.org>> wrote: We were seeing substantial loss through Chicago this morning but it seems to have subsided at this time. Bob Hancock On 08/27/2015 12:19 PM, Steven Fischer via Outages wrote:
we are still seeing significant packet loss to 3 of our sites in the Midwest.
On Thu, Aug 27, 2015 at 12:47 PM, Bobin Joseph via Outages <outages@outages.org<mailto:outages@outages.org> <mailto:outages@outages.org<mailto:outages@outages.org>>> wrote:
We got this from Level3____
__ __
Hello,
The network outage has been resolved.
*Event Conclusion Summary*
Start: August 26, 2015 19:15 GMT Stop: August 27, 2015 00:14 GMT
Root Cause: An unknown issue in Chicago, IL was impacting IP services. Fix Action: Adjustments were made to resolve the issue and restore services.
Summary: The IP NOC responded to an unknown issue in Chicago, IL that was impacting IP services. Tier III Technical Support were engaged to assist with investigations, and adjustments were made to resolve the issue and stabilize traffic. Monitoring continues to ensure stability. Should further issues be experienced, please contact the Level 3 Technical Service Center.
____
__ __
Thanks,____
Bobin____
__ __
*From:*Outages [mailto:outages-bounces@outages.org<mailto:outages-bounces@outages.org> <mailto:outages-bounces@outages.org<mailto:outages-bounces@outages.org>>] *On Behalf Of *Ryan Gelobter via Outages *Sent:* Thursday, August 27, 2015 9:44 AM *To:* Rafael Possamai *Cc:* outages@outages.org<mailto:outages@outages.org> <mailto:outages@outages.org<mailto:outages@outages.org>> *Subject:* Re: [outages] Level3/TW Telecom issues?____
__ __
Was anyone able to get some useful information out of Level3 regarding this yesterday? Also is anyone else seeing similar issues?____
On Aug 26, 2015 2:40 PM, "Rafael Possamai via Outages" <outages@outages.org<mailto:outages@outages.org> <mailto:outages@outages.org<mailto:outages@outages.org>>> wrote:____
Here's a graph showing some packet loss: http://imgur.com/KVAzetA <http://cp.mcafee.com/d/FZsSd38w83hJ5xyXxEVsKCMrKrhKCy-qen4TXCQrFELCzBNBBUSztd5YQsK9LECQrFKnuv7fICO1fomHoG2BqG_2rqI_fZvBPqJlvxdJmvD-LOVJV1NxR1Z_HYYMM-YeoWZOWarxEVV5MsMqemrKth5dqWqJTT-l3PWApmU6CQjr1J7HFkr8kJ6h3PtPo0eiF5KfbU1gp41jVy1KHsZ8_gKgGT2TQ1lmVWh-D00jqb2pEVsdwLQzh0qmWJPQzZ3iWq80HVDPSs182Cy04-vWVEwBizvQ96SjhOrehG_98MC>____
__ __
__ __
On Wed, Aug 26, 2015 at 3:16 PM, Matt Elliott via Outages <outages@outages.org<mailto:outages@outages.org> <mailto:outages@outages.org<mailto:outages@outages.org>>> wrote:____
Anyone else having issues on Level3? We’re seeing packet loss with anything that traverses Chicago.____
__ __
Thanks,____
Matt____
This message and attachment(s), if any, is intended for the sole use of the individual and/or entity for which it is addressed, and may contain information that is privileged, confidential and prohibited from disclosure under applicable law. If you are not the addressee, or authorized to receive this on behalf of the addressee, you are hereby notified that you may not use, copy, disclose or distribute to anyone this message or any part thereof. If you have received this in error, please immediately advise the sender by e-mail and delete this information and all attachments from your computer and network. Thank you. ____
_______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> <mailto:Outages@outages.org<mailto:Outages@outages.org>> https://puck.nether.net/mailman/listinfo/outages <http://cp.mcafee.com/d/avndy0QrhooKUqenbFI6XCQrFELCzBNd-VJ6WqbVEVsppudETjhvd7byrW9J6WrBTDNPX9IwjS5GSawFmGLMCSHfP_nVsSHlnUjrlDV_HYKrugsotgvvW_fccfL3CeLsKyCUqeuhs7c6zBCXDkhjmKCHtZ_BgY-F6lK1FJcSMrhWWl6O5bhAgYTsSjDdqymoIToHMd9_7wrwCHIcfBisEeROHsZ8_gTlKuAvEn8lrxrW0GHsZ8_jw09J5xcQsK6MnWhEwdbtmVWh-xFtd40lYPVXe0A1jh02vfZsQgiFhLW4zr9EVdBPdH>____
__ __
_______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> <mailto:Outages@outages.org<mailto:Outages@outages.org>> https://puck.nether.net/mailman/listinfo/outages <http://cp.mcafee.com/d/2DRPoQ938ArhooKUqenbFI6XCQrFELCzBNd-VJ6WqbVEVsppudETjhvd7byrW9J6WrBTDNPX9IwjS5GSawFmGLMCSHfP_nVsSHlnUjrlDV_HYKrugsotgvvW_fccfL3CeLsKyCUqeuhs7c6zBCXDkhjmKCHtZ_BgY-F6lK1FJASMrhWWl6O5bhAgYTsSjDdqymoIToHMd9_7wrwCHIcfBisEeROHsZ8_gTlKuAvEn8lrxrW0GHsZ8_jw09J5xcQsK6MnWhEwdbtmVWh-xFtd40lYPVXe0A1jh02vfZsQgiFhLW4zr9EVdWoUq>____
_______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> <mailto:Outages@outages.org<mailto:Outages@outages.org>> https://puck.nether.net/mailman/listinfo/outages <http://cp.mcafee.com/d/k-Kr41ESyMNtMQsKnjodTdETjhvd7byrZPqdQQnPhOUOOYrhKCy-qen4TQjqdQTbLfzDSjp0DIblIl1iJlvxdJmvD-LOVJmGLMCSHfP_nVsSYwUMWw-_R-uoovu7ctuVt5dMQsYyUeod7bdTeEyCJtdmXX_axVZicHs3jr1JwSzRQGdAamz8xVKVIDeqR4INpKNnwqj-f0T1dnoovaAVgtHBmVWh-xKHsZ8_gKgGT2TQ1lmVWh-D00jqb2pEVsdwLQzh0qmWJPQzZ3iWq80HVDPSs182Cy04-vWVEwBizvQ96SjhOrVbh4>
-- To him who is able to keep you from falling and to present you before his glorious presence without fault and with great joy
_______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> http://cp.mcafee.com/d/avndzhJ5xyXxEVsKC-rKrhKCy-qen4TXCQrFELCzBNBBUSztd5YQs...
The information contained in this E-mail message is privileged, confidential, and may be protected from disclosure; please be aware that any unauthorized use, printing, copying, disclosure or dissemination of this communication may be subject to legal restriction or sanction. If you think that you have received this E-mail message in error, please reply to the sender. This E-mail message and any attachments have been scanned for viruses and are believed to be free of any virus or other defect that might affect any computer system into which it is received and opened. It is the responsibility, however, of the recipient to ensure that it is virus free. _______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages
24 hours of on and off packet loss, more than just Chicago. More communication would be helpful. NOC can't seem to decide if it's transport or routing. Any insights?
On Aug 27, 2015, at 1:08 PM, Bobin Joseph via Outages <outages@outages.org> wrote:
We just got another update saying they still have issues.
IP services are affected across multiple markets. The IP NOC and IP and Transport Tier III are actively troubleshooting the issue. Operations Engineering is engaged and Field Services is on site in Chicago, IL investigating the issue. tw telecom has identified a network failure affecting your services. tw telecom tier-2 group is investigating the issue at this time.
Thanks, Bobin
From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Aaron Gross via Outages Sent: Thursday, August 27, 2015 10:55 AM To: Bob Hancock Cc: outages@outages.org Subject: Re: [outages] Level3/TW Telecom issues?
We are seeing the same things that happened yesterday on our circuits in Milwaukee and Chicago areas. Lots of packet loss and large latency through Chicago exchanges.
On Thu, Aug 27, 2015 at 12:33 PM, Bob Hancock via Outages <outages@outages.org <mailto:outages@outages.org>> wrote: We were seeing substantial loss through Chicago this morning but it seems to have subsided at this time.
Bob Hancock
On 08/27/2015 12:19 PM, Steven Fischer via Outages wrote:
we are still seeing significant packet loss to 3 of our sites in the Midwest.
On Thu, Aug 27, 2015 at 12:47 PM, Bobin Joseph via Outages <outages@outages.org <mailto:outages@outages.org> <mailto:outages@outages.org <mailto:outages@outages.org>>> wrote:
We got this from Level3____
__ __
Hello,
The network outage has been resolved.
*Event Conclusion Summary*
Start: August 26, 2015 19:15 GMT Stop: August 27, 2015 00:14 GMT
Root Cause: An unknown issue in Chicago, IL was impacting IP services. Fix Action: Adjustments were made to resolve the issue and restore services.
Summary: The IP NOC responded to an unknown issue in Chicago, IL that was impacting IP services. Tier III Technical Support were engaged to assist with investigations, and adjustments were made to resolve the issue and stabilize traffic. Monitoring continues to ensure stability. Should further issues be experienced, please contact the Level 3 Technical Service Center.
____
__ __
Thanks,____
Bobin____
__ __
*From:*Outages [mailto:outages-bounces@outages.org <mailto:outages-bounces@outages.org> <mailto:outages-bounces@outages.org <mailto:outages-bounces@outages.org>>] *On Behalf Of *Ryan Gelobter via Outages *Sent:* Thursday, August 27, 2015 9:44 AM *To:* Rafael Possamai *Cc:* outages@outages.org <mailto:outages@outages.org> <mailto:outages@outages.org <mailto:outages@outages.org>> *Subject:* Re: [outages] Level3/TW Telecom issues?____
__ __
Was anyone able to get some useful information out of Level3 regarding this yesterday? Also is anyone else seeing similar issues?____
On Aug 26, 2015 2:40 PM, "Rafael Possamai via Outages" <outages@outages.org <mailto:outages@outages.org> <mailto:outages@outages.org <mailto:outages@outages.org>>> wrote:____
Here's a graph showing some packet loss: http://imgur.com/KVAzetA <http://imgur.com/KVAzetA> <http://cp.mcafee.com/d/FZsSd38w83hJ5xyXxEVsKCMrKrhKCy-qen4TXCQrFELCzBNBBUSzt... <http://cp.mcafee.com/d/FZsSd38w83hJ5xyXxEVsKCMrKrhKCy-qen4TXCQrFELCzBNBBUSztd5YQsK9LECQrFKnuv7fICO1fomHoG2BqG_2rqI_fZvBPqJlvxdJmvD-LOVJV1NxR1Z_HYYMM-YeoWZOWarxEVV5MsMqemrKth5dqWqJTT-l3PWApmU6CQjr1J7HFkr8kJ6h3PtPo0eiF5KfbU1gp41jVy1KHsZ8_gKgGT2TQ1lmVWh-D00jqb2pEVsdwLQzh0qmWJPQzZ3iWq80HVDPSs182Cy04-vWVEwBizvQ96SjhOrehG_98MC>>____
__ __
__ __
On Wed, Aug 26, 2015 at 3:16 PM, Matt Elliott via Outages <outages@outages.org <mailto:outages@outages.org> <mailto:outages@outages.org <mailto:outages@outages.org>>> wrote:____
Anyone else having issues on Level3? We’re seeing packet loss with anything that traverses Chicago.____
__ __
Thanks,____
Matt____
This message and attachment(s), if any, is intended for the sole use of the individual and/or entity for which it is addressed, and may contain information that is privileged, confidential and prohibited from disclosure under applicable law. If you are not the addressee, or authorized to receive this on behalf of the addressee, you are hereby notified that you may not use, copy, disclose or distribute to anyone this message or any part thereof. If you have received this in error, please immediately advise the sender by e-mail and delete this information and all attachments from your computer and network. Thank you. ____
_______________________________________________ Outages mailing list Outages@outages.org <mailto:Outages@outages.org> <mailto:Outages@outages.org <mailto:Outages@outages.org>> https://puck.nether.net/mailman/listinfo/outages <https://puck.nether.net/mailman/listinfo/outages> <http://cp.mcafee.com/d/avndy0QrhooKUqenbFI6XCQrFELCzBNd-VJ6WqbVEVsppudETjhvd... <http://cp.mcafee.com/d/avndy0QrhooKUqenbFI6XCQrFELCzBNd-VJ6WqbVEVsppudETjhvd7byrW9J6WrBTDNPX9IwjS5GSawFmGLMCSHfP_nVsSHlnUjrlDV_HYKrugsotgvvW_fccfL3CeLsKyCUqeuhs7c6zBCXDkhjmKCHtZ_BgY-F6lK1FJcSMrhWWl6O5bhAgYTsSjDdqymoIToHMd9_7wrwCHIcfBisEeROHsZ8_gTlKuAvEn8lrxrW0GHsZ8_jw09J5xcQsK6MnWhEwdbtmVWh-xFtd40lYPVXe0A1jh02vfZsQgiFhLW4zr9EVdBPdH>>____
__ __
_______________________________________________ Outages mailing list Outages@outages.org <mailto:Outages@outages.org> <mailto:Outages@outages.org <mailto:Outages@outages.org>> https://puck.nether.net/mailman/listinfo/outages <https://puck.nether.net/mailman/listinfo/outages> <http://cp.mcafee.com/d/2DRPoQ938ArhooKUqenbFI6XCQrFELCzBNd-VJ6WqbVEVsppudETj... <http://cp.mcafee.com/d/2DRPoQ938ArhooKUqenbFI6XCQrFELCzBNd-VJ6WqbVEVsppudETjhvd7byrW9J6WrBTDNPX9IwjS5GSawFmGLMCSHfP_nVsSHlnUjrlDV_HYKrugsotgvvW_fccfL3CeLsKyCUqeuhs7c6zBCXDkhjmKCHtZ_BgY-F6lK1FJASMrhWWl6O5bhAgYTsSjDdqymoIToHMd9_7wrwCHIcfBisEeROHsZ8_gTlKuAvEn8lrxrW0GHsZ8_jw09J5xcQsK6MnWhEwdbtmVWh-xFtd40lYPVXe0A1jh02vfZsQgiFhLW4zr9EVdWoUq>>____
_______________________________________________ Outages mailing list Outages@outages.org <mailto:Outages@outages.org> <mailto:Outages@outages.org <mailto:Outages@outages.org>> https://puck.nether.net/mailman/listinfo/outages <https://puck.nether.net/mailman/listinfo/outages> <http://cp.mcafee.com/d/k-Kr41ESyMNtMQsKnjodTdETjhvd7byrZPqdQQnPhOUOOYrhKCy-q... <http://cp.mcafee.com/d/k-Kr41ESyMNtMQsKnjodTdETjhvd7byrZPqdQQnPhOUOOYrhKCy-qen4TQjqdQTbLfzDSjp0DIblIl1iJlvxdJmvD-LOVJmGLMCSHfP_nVsSYwUMWw-_R-uoovu7ctuVt5dMQsYyUeod7bdTeEyCJtdmXX_axVZicHs3jr1JwSzRQGdAamz8xVKVIDeqR4INpKNnwqj-f0T1dnoovaAVgtHBmVWh-xKHsZ8_gKgGT2TQ1lmVWh-D00jqb2pEVsdwLQzh0qmWJPQzZ3iWq80HVDPSs182Cy04-vWVEwBizvQ96SjhOrVbh4>>
-- To him who is able to keep you from falling and to present you before his glorious presence without fault and with great joy
_______________________________________________ Outages mailing list Outages@outages.org <mailto:Outages@outages.org> http://cp.mcafee.com/d/avndzhJ5xyXxEVsKC-rKrhKCy-qen4TXCQrFELCzBNBBUSztd5YQs... <http://cp.mcafee.com/d/avndzhJ5xyXxEVsKC-rKrhKCy-qen4TXCQrFELCzBNBBUSztd5YQsK9LECQrFKnuv7fICO1fomHoG2BqG_2rqI_fZvBPqJlvxdJmvD-LOVJV1NxR1Z_HYYMM-YeoWZOWarxEVV5MsMqemrKth5dqWqJTT-l3PWApmU6CQjq9J7HFkr8kJ6h3PtPpesRG9pyPtyL0QDYu1K2qKMM-l9OwXnaJPQzZ3tmVWh-xsxlK5LE2GJPQzZe00CQm4PhOUr1vF6y0QJRrDF7W6BQQg1nPfDIU2g5d409Y_RPh1aB6_EidICzASH16Ohmp>
The information contained in this E-mail message is privileged, confidential, and may be protected from disclosure; please be aware that any unauthorized use, printing, copying, disclosure or dissemination of this communication may be subject to legal restriction or sanction. If you think that you have received this E-mail message in error, please reply to the sender. This E-mail message and any attachments have been scanned for viruses and are believed to be free of any virus or other defect that might affect any computer system into which it is received and opened. It is the responsibility, however, of the recipient to ensure that it is virus free. _______________________________________________ Outages mailing list Outages@outages.org <mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages <https://puck.nether.net/mailman/listinfo/outages>
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Retracting my statement from yesterday this is still a solid issue here. Yes absolutely still continuing. Root Cause would be a great explanation at this point.
On Aug 27, 2015, at 13:10, Ryan Brooks via Outages <outages@outages.org> wrote:
24 hours of on and off packet loss, more than just Chicago. More communication would be helpful. NOC can't seem to decide if it's transport or routing. Any insights?
On Aug 27, 2015, at 1:08 PM, Bobin Joseph via Outages <outages@outages.org> wrote:
We just got another update saying they still have issues.
IP services are affected across multiple markets. The IP NOC and IP and Transport Tier III are actively troubleshooting the issue. Operations Engineering is engaged and Field Services is on site in Chicago, IL investigating the issue. tw telecom has identified a network failure affecting your services. tw telecom tier-2 group is investigating the issue at this time.
Thanks, Bobin
From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Aaron Gross via Outages Sent: Thursday, August 27, 2015 10:55 AM To: Bob Hancock Cc: outages@outages.org Subject: Re: [outages] Level3/TW Telecom issues?
We are seeing the same things that happened yesterday on our circuits in Milwaukee and Chicago areas. Lots of packet loss and large latency through Chicago exchanges.
On Thu, Aug 27, 2015 at 12:33 PM, Bob Hancock via Outages <outages@outages.org> wrote: We were seeing substantial loss through Chicago this morning but it seems to have subsided at this time.
Bob Hancock
On 08/27/2015 12:19 PM, Steven Fischer via Outages wrote:
we are still seeing significant packet loss to 3 of our sites in the Midwest.
On Thu, Aug 27, 2015 at 12:47 PM, Bobin Joseph via Outages <outages@outages.org <mailto:outages@outages.org>> wrote:
We got this from Level3____
__ __
Hello,
The network outage has been resolved.
*Event Conclusion Summary*
Start: August 26, 2015 19:15 GMT Stop: August 27, 2015 00:14 GMT
Root Cause: An unknown issue in Chicago, IL was impacting IP services. Fix Action: Adjustments were made to resolve the issue and restore services.
Summary: The IP NOC responded to an unknown issue in Chicago, IL that was impacting IP services. Tier III Technical Support were engaged to assist with investigations, and adjustments were made to resolve the issue and stabilize traffic. Monitoring continues to ensure stability. Should further issues be experienced, please contact the Level 3 Technical Service Center.
____
__ __
Thanks,____
Bobin____
__ __
*From:*Outages [mailto:outages-bounces@outages.org <mailto:outages-bounces@outages.org>] *On Behalf Of *Ryan Gelobter via Outages *Sent:* Thursday, August 27, 2015 9:44 AM *To:* Rafael Possamai *Cc:* outages@outages.org <mailto:outages@outages.org> *Subject:* Re: [outages] Level3/TW Telecom issues?____
__ __
Was anyone able to get some useful information out of Level3 regarding this yesterday? Also is anyone else seeing similar issues?____
On Aug 26, 2015 2:40 PM, "Rafael Possamai via Outages" <outages@outages.org <mailto:outages@outages.org>> wrote:____
Here's a graph showing some packet loss: http://imgur.com/KVAzetA <http://cp.mcafee.com/d/FZsSd38w83hJ5xyXxEVsKCMrKrhKCy-qen4TXCQrFELCzBNBBUSztd5YQsK9LECQrFKnuv7fICO1fomHoG2BqG_2rqI_fZvBPqJlvxdJmvD-LOVJV1NxR1Z_HYYMM-YeoWZOWarxEVV5MsMqemrKth5dqWqJTT-l3PWApmU6CQjr1J7HFkr8kJ6h3PtPo0eiF5KfbU1gp41jVy1KHsZ8_gKgGT2TQ1lmVWh-D00jqb2pEVsdwLQzh0qmWJPQzZ3iWq80HVDPSs182Cy04-vWVEwBizvQ96SjhOrehG_98MC>____
__ __
__ __
On Wed, Aug 26, 2015 at 3:16 PM, Matt Elliott via Outages <outages@outages.org <mailto:outages@outages.org>> wrote:____
Anyone else having issues on Level3? We’re seeing packet loss with anything that traverses Chicago.____
__ __
Thanks,____
Matt____
This message and attachment(s), if any, is intended for the sole use of the individual and/or entity for which it is addressed, and may contain information that is privileged, confidential and prohibited from disclosure under applicable law. If you are not the addressee, or authorized to receive this on behalf of the addressee, you are hereby notified that you may not use, copy, disclose or distribute to anyone this message or any part thereof. If you have received this in error, please immediately advise the sender by e-mail and delete this information and all attachments from your computer and network. Thank you. ____
_______________________________________________ Outages mailing list Outages@outages.org <mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages <http://cp.mcafee.com/d/avndy0QrhooKUqenbFI6XCQrFELCzBNd-VJ6WqbVEVsppudETjhvd7byrW9J6WrBTDNPX9IwjS5GSawFmGLMCSHfP_nVsSHlnUjrlDV_HYKrugsotgvvW_fccfL3CeLsKyCUqeuhs7c6zBCXDkhjmKCHtZ_BgY-F6lK1FJcSMrhWWl6O5bhAgYTsSjDdqymoIToHMd9_7wrwCHIcfBisEeROHsZ8_gTlKuAvEn8lrxrW0GHsZ8_jw09J5xcQsK6MnWhEwdbtmVWh-xFtd40lYPVXe0A1jh02vfZsQgiFhLW4zr9EVdBPdH>____
__ __
_______________________________________________ Outages mailing list Outages@outages.org <mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages <http://cp.mcafee.com/d/2DRPoQ938ArhooKUqenbFI6XCQrFELCzBNd-VJ6WqbVEVsppudETjhvd7byrW9J6WrBTDNPX9IwjS5GSawFmGLMCSHfP_nVsSHlnUjrlDV_HYKrugsotgvvW_fccfL3CeLsKyCUqeuhs7c6zBCXDkhjmKCHtZ_BgY-F6lK1FJASMrhWWl6O5bhAgYTsSjDdqymoIToHMd9_7wrwCHIcfBisEeROHsZ8_gTlKuAvEn8lrxrW0GHsZ8_jw09J5xcQsK6MnWhEwdbtmVWh-xFtd40lYPVXe0A1jh02vfZsQgiFhLW4zr9EVdWoUq>____
_______________________________________________ Outages mailing list Outages@outages.org <mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages <http://cp.mcafee.com/d/k-Kr41ESyMNtMQsKnjodTdETjhvd7byrZPqdQQnPhOUOOYrhKCy-qen4TQjqdQTbLfzDSjp0DIblIl1iJlvxdJmvD-LOVJmGLMCSHfP_nVsSYwUMWw-_R-uoovu7ctuVt5dMQsYyUeod7bdTeEyCJtdmXX_axVZicHs3jr1JwSzRQGdAamz8xVKVIDeqR4INpKNnwqj-f0T1dnoovaAVgtHBmVWh-xKHsZ8_gKgGT2TQ1lmVWh-D00jqb2pEVsdwLQzh0qmWJPQzZ3iWq80HVDPSs182Cy04-vWVEwBizvQ96SjhOrVbh4>
-- To him who is able to keep you from falling and to present you before his glorious presence without fault and with great joy
_______________________________________________ Outages mailing list Outages@outages.org http://cp.mcafee.com/d/avndzhJ5xyXxEVsKC-rKrhKCy-qen4TXCQrFELCzBNBBUSztd5YQs...
The information contained in this E-mail message is privileged, confidential, and may be protected from disclosure; please be aware that any unauthorized use, printing, copying, disclosure or dissemination of this communication may be subject to legal restriction or sanction. If you think that you have received this E-mail message in error, please reply to the sender. This E-mail message and any attachments have been scanned for viruses and are believed to be free of any virus or other defect that might affect any computer system into which it is received and opened. It is the responsibility, however, of the recipient to ensure that it is virus free. _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
-- Jason Hellenthal JJH48-ARIN
Seeing on and off in upstate NY Matthew Topper -----Original Message----- From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Jason Hellenthal via Outages Sent: Thursday, August 27, 2015 2:23 PM To: outages@outages.org Subject: Re: [outages] Level3/TW Telecom issues? Retracting my statement from yesterday this is still a solid issue here. Yes absolutely still continuing. Root Cause would be a great explanation at this point.
On Aug 27, 2015, at 13:10, Ryan Brooks via Outages <outages@outages.org> wrote:
24 hours of on and off packet loss, more than just Chicago. More communication would be helpful. NOC can't seem to decide if it's transport or routing. Any insights?
On Aug 27, 2015, at 1:08 PM, Bobin Joseph via Outages <outages@outages.org> wrote:
We just got another update saying they still have issues.
IP services are affected across multiple markets. The IP NOC and IP and Transport Tier III are actively troubleshooting the issue. Operations Engineering is engaged and Field Services is on site in Chicago, IL investigating the issue. tw telecom has identified a network failure affecting your services. tw telecom tier-2 group is investigating the issue at this time.
Thanks, Bobin
From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Aaron Gross via Outages Sent: Thursday, August 27, 2015 10:55 AM To: Bob Hancock Cc: outages@outages.org Subject: Re: [outages] Level3/TW Telecom issues?
We are seeing the same things that happened yesterday on our circuits in Milwaukee and Chicago areas. Lots of packet loss and large latency through Chicago exchanges.
On Thu, Aug 27, 2015 at 12:33 PM, Bob Hancock via Outages <outages@outages.org> wrote: We were seeing substantial loss through Chicago this morning but it seems to have subsided at this time.
Bob Hancock
On 08/27/2015 12:19 PM, Steven Fischer via Outages wrote:
we are still seeing significant packet loss to 3 of our sites in the Midwest.
On Thu, Aug 27, 2015 at 12:47 PM, Bobin Joseph via Outages <outages@outages.org <mailto:outages@outages.org>> wrote:
We got this from Level3____
__ __
Hello,
The network outage has been resolved.
*Event Conclusion Summary*
Start: August 26, 2015 19:15 GMT Stop: August 27, 2015 00:14 GMT
Root Cause: An unknown issue in Chicago, IL was impacting IP services. Fix Action: Adjustments were made to resolve the issue and restore services.
Summary: The IP NOC responded to an unknown issue in Chicago, IL that was impacting IP services. Tier III Technical Support were engaged to assist with investigations, and adjustments were made to resolve the issue and stabilize traffic. Monitoring continues to ensure stability. Should further issues be experienced, please contact the Level 3 Technical Service Center.
____
__ __
Thanks,____
Bobin____
__ __
*From:*Outages [mailto:outages-bounces@outages.org <mailto:outages-bounces@outages.org>] *On Behalf Of *Ryan Gelobter via Outages *Sent:* Thursday, August 27, 2015 9:44 AM *To:* Rafael Possamai *Cc:* outages@outages.org <mailto:outages@outages.org> *Subject:* Re: [outages] Level3/TW Telecom issues?____
__ __
Was anyone able to get some useful information out of Level3 regarding this yesterday? Also is anyone else seeing similar issues?____
On Aug 26, 2015 2:40 PM, "Rafael Possamai via Outages" <outages@outages.org <mailto:outages@outages.org>> wrote:____
Here's a graph showing some packet loss: http://imgur.com/KVAzetA
<http://cp.mcafee.com/d/FZsSd38w83hJ5xyXxEVsKCMrKrhKCy-qen4TXCQrFEL CzBNBBUSztd5YQsK9LECQrFKnuv7fICO1fomHoG2BqG_2rqI_fZvBPqJlvxdJmvD-LO VJV1NxR1Z_HYYMM-YeoWZOWarxEVV5MsMqemrKth5dqWqJTT-l3PWApmU6CQjr1J7HF kr8kJ6h3PtPo0eiF5KfbU1gp41jVy1KHsZ8_gKgGT2TQ1lmVWh-D00jqb2pEVsdwLQz h0qmWJPQzZ3iWq80HVDPSs182Cy04-vWVEwBizvQ96SjhOrehG_98MC>____
__ __
__ __
On Wed, Aug 26, 2015 at 3:16 PM, Matt Elliott via Outages <outages@outages.org <mailto:outages@outages.org>> wrote:____
Anyone else having issues on Level3? We’re seeing packet loss with anything that traverses Chicago.____
__ __
Thanks,____
Matt____
This message and attachment(s), if any, is intended for the sole use of the individual and/or entity for which it is addressed, and may contain information that is privileged, confidential and prohibited from disclosure under applicable law. If you are not the addressee, or authorized to receive this on behalf of the addressee, you are hereby notified that you may not use, copy, disclose or distribute to anyone this message or any part thereof. If you have received this in error, please immediately advise the sender by e-mail and delete this information and all attachments from your computer and network. Thank you. ____
_______________________________________________ Outages mailing list Outages@outages.org <mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages
<http://cp.mcafee.com/d/avndy0QrhooKUqenbFI6XCQrFELCzBNd-VJ6WqbVEVs ppudETjhvd7byrW9J6WrBTDNPX9IwjS5GSawFmGLMCSHfP_nVsSHlnUjrlDV_HYKrug sotgvvW_fccfL3CeLsKyCUqeuhs7c6zBCXDkhjmKCHtZ_BgY-F6lK1FJcSMrhWWl6O5 bhAgYTsSjDdqymoIToHMd9_7wrwCHIcfBisEeROHsZ8_gTlKuAvEn8lrxrW0GHsZ8_j w09J5xcQsK6MnWhEwdbtmVWh-xFtd40lYPVXe0A1jh02vfZsQgiFhLW4zr9EVdBPdH> ____
__ __
_______________________________________________ Outages mailing list Outages@outages.org <mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages
<http://cp.mcafee.com/d/2DRPoQ938ArhooKUqenbFI6XCQrFELCzBNd-VJ6WqbV EVsppudETjhvd7byrW9J6WrBTDNPX9IwjS5GSawFmGLMCSHfP_nVsSHlnUjrlDV_HYK rugsotgvvW_fccfL3CeLsKyCUqeuhs7c6zBCXDkhjmKCHtZ_BgY-F6lK1FJASMrhWWl 6O5bhAgYTsSjDdqymoIToHMd9_7wrwCHIcfBisEeROHsZ8_gTlKuAvEn8lrxrW0GHsZ 8_jw09J5xcQsK6MnWhEwdbtmVWh-xFtd40lYPVXe0A1jh02vfZsQgiFhLW4zr9EVdWo Uq>____
_______________________________________________ Outages mailing list Outages@outages.org <mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages
<http://cp.mcafee.com/d/k-Kr41ESyMNtMQsKnjodTdETjhvd7byrZPqdQQnPhOU OOYrhKCy-qen4TQjqdQTbLfzDSjp0DIblIl1iJlvxdJmvD-LOVJmGLMCSHfP_nVsSYw UMWw-_R-uoovu7ctuVt5dMQsYyUeod7bdTeEyCJtdmXX_axVZicHs3jr1JwSzRQGdAa mz8xVKVIDeqR4INpKNnwqj-f0T1dnoovaAVgtHBmVWh-xKHsZ8_gKgGT2TQ1lmVWh-D 00jqb2pEVsdwLQzh0qmWJPQzZ3iWq80HVDPSs182Cy04-vWVEwBizvQ96SjhOrVbh4>
-- To him who is able to keep you from falling and to present you before his glorious presence without fault and with great joy
_______________________________________________ Outages mailing list Outages@outages.org http://cp.mcafee.com/d/avndzhJ5xyXxEVsKC-rKrhKCy-qen4TXCQrFELCzBNBB USztd5YQsK9LECQrFKnuv7fICO1fomHoG2BqG_2rqI_fZvBPqJlvxdJmvD-LOVJV1Nx R1Z_HYYMM-YeoWZOWarxEVV5MsMqemrKth5dqWqJTT-l3PWApmU6CQjq9J7HFkr8kJ6 h3PtPpesRG9pyPtyL0QDYu1K2qKMM-l9OwXnaJPQzZ3tmVWh-xsxlK5LE2GJPQzZe00 CQm4PhOUr1vF6y0QJRrDF7W6BQQg1nPfDIU2g5d409Y_RPh1aB6_EidICzASH16Ohmp
The information contained in this E-mail message is privileged, confidential, and may be protected from disclosure; please be aware that any unauthorized use, printing, copying, disclosure or dissemination of this communication may be subject to legal restriction or sanction. If you think that you have received this E-mail message in error, please reply to the sender. This E-mail message and any attachments have been scanned for viruses and are believed to be free of any virus or other defect that might affect any computer system into which it is received and opened. It is the responsibility, however, of the recipient to ensure that it is virus free. _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
-- Jason Hellenthal JJH48-ARIN
On Thu, 27 Aug 2015, Matthew Topper via Outages wrote:
Seeing on and off in upstate NY
Matthew Topper
Confirmed upstate NY (Albany) as well. We've rerouted in the meanwhile; were seeing lost routes and split routing back to us. -- Jameel Akari
-----Original Message----- From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Jason Hellenthal via Outages Sent: Thursday, August 27, 2015 2:23 PM To: outages@outages.org Subject: Re: [outages] Level3/TW Telecom issues?
Retracting my statement from yesterday this is still a solid issue here. Yes absolutely still continuing. Root Cause would be a great explanation at this point.
On Aug 27, 2015, at 13:10, Ryan Brooks via Outages <outages@outages.org> wrote:
24 hours of on and off packet loss, more than just Chicago. More communication would be helpful. NOC can't seem to decide if it's transport or routing. Any insights?
On Aug 27, 2015, at 1:08 PM, Bobin Joseph via Outages <outages@outages.org> wrote:
We just got another update saying they still have issues.
IP services are affected across multiple markets. The IP NOC and IP and Transport Tier III are actively troubleshooting the issue. Operations Engineering is engaged and Field Services is on site in Chicago, IL investigating the issue. tw telecom has identified a network failure affecting your services. tw telecom tier-2 group is investigating the issue at this time.
Thanks, Bobin
From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Aaron Gross via Outages Sent: Thursday, August 27, 2015 10:55 AM To: Bob Hancock Cc: outages@outages.org Subject: Re: [outages] Level3/TW Telecom issues?
We are seeing the same things that happened yesterday on our circuits in Milwaukee and Chicago areas. Lots of packet loss and large latency through Chicago exchanges.
On Thu, Aug 27, 2015 at 12:33 PM, Bob Hancock via Outages <outages@outages.org> wrote: We were seeing substantial loss through Chicago this morning but it seems to have subsided at this time.
Bob Hancock
On 08/27/2015 12:19 PM, Steven Fischer via Outages wrote:
we are still seeing significant packet loss to 3 of our sites in the Midwest.
On Thu, Aug 27, 2015 at 12:47 PM, Bobin Joseph via Outages <outages@outages.org <mailto:outages@outages.org>> wrote:
We got this from Level3____
__ __
Hello,
The network outage has been resolved.
*Event Conclusion Summary*
Start: August 26, 2015 19:15 GMT Stop: August 27, 2015 00:14 GMT
Root Cause: An unknown issue in Chicago, IL was impacting IP services. Fix Action: Adjustments were made to resolve the issue and restore services.
Summary: The IP NOC responded to an unknown issue in Chicago, IL that was impacting IP services. Tier III Technical Support were engaged to assist with investigations, and adjustments were made to resolve the issue and stabilize traffic. Monitoring continues to ensure stability. Should further issues be experienced, please contact the Level 3 Technical Service Center.
____
__ __
Thanks,____
Bobin____
__ __
*From:*Outages [mailto:outages-bounces@outages.org <mailto:outages-bounces@outages.org>] *On Behalf Of *Ryan Gelobter via Outages *Sent:* Thursday, August 27, 2015 9:44 AM *To:* Rafael Possamai *Cc:* outages@outages.org <mailto:outages@outages.org> *Subject:* Re: [outages] Level3/TW Telecom issues?____
__ __
Was anyone able to get some useful information out of Level3 regarding this yesterday? Also is anyone else seeing similar issues?____
On Aug 26, 2015 2:40 PM, "Rafael Possamai via Outages" <outages@outages.org <mailto:outages@outages.org>> wrote:____
Here's a graph showing some packet loss: http://imgur.com/KVAzetA
<http://cp.mcafee.com/d/FZsSd38w83hJ5xyXxEVsKCMrKrhKCy-qen4TXCQrFEL CzBNBBUSztd5YQsK9LECQrFKnuv7fICO1fomHoG2BqG_2rqI_fZvBPqJlvxdJmvD-LO VJV1NxR1Z_HYYMM-YeoWZOWarxEVV5MsMqemrKth5dqWqJTT-l3PWApmU6CQjr1J7HF kr8kJ6h3PtPo0eiF5KfbU1gp41jVy1KHsZ8_gKgGT2TQ1lmVWh-D00jqb2pEVsdwLQz h0qmWJPQzZ3iWq80HVDPSs182Cy04-vWVEwBizvQ96SjhOrehG_98MC>____
__ __
__ __
On Wed, Aug 26, 2015 at 3:16 PM, Matt Elliott via Outages <outages@outages.org <mailto:outages@outages.org>> wrote:____
Anyone else having issues on Level3? We’re seeing packet loss with anything that traverses Chicago.____
__ __
Thanks,____
Matt____
This message and attachment(s), if any, is intended for the sole use of the individual and/or entity for which it is addressed, and may contain information that is privileged, confidential and prohibited from disclosure under applicable law. If you are not the addressee, or authorized to receive this on behalf of the addressee, you are hereby notified that you may not use, copy, disclose or distribute to anyone this message or any part thereof. If you have received this in error, please immediately advise the sender by e-mail and delete this information and all attachments from your computer and network. Thank you. ____
_______________________________________________ Outages mailing list Outages@outages.org <mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages
<http://cp.mcafee.com/d/avndy0QrhooKUqenbFI6XCQrFELCzBNd-VJ6WqbVEVs ppudETjhvd7byrW9J6WrBTDNPX9IwjS5GSawFmGLMCSHfP_nVsSHlnUjrlDV_HYKrug sotgvvW_fccfL3CeLsKyCUqeuhs7c6zBCXDkhjmKCHtZ_BgY-F6lK1FJcSMrhWWl6O5 bhAgYTsSjDdqymoIToHMd9_7wrwCHIcfBisEeROHsZ8_gTlKuAvEn8lrxrW0GHsZ8_j w09J5xcQsK6MnWhEwdbtmVWh-xFtd40lYPVXe0A1jh02vfZsQgiFhLW4zr9EVdBPdH> ____
__ __
_______________________________________________ Outages mailing list Outages@outages.org <mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages
<http://cp.mcafee.com/d/2DRPoQ938ArhooKUqenbFI6XCQrFELCzBNd-VJ6WqbV EVsppudETjhvd7byrW9J6WrBTDNPX9IwjS5GSawFmGLMCSHfP_nVsSHlnUjrlDV_HYK rugsotgvvW_fccfL3CeLsKyCUqeuhs7c6zBCXDkhjmKCHtZ_BgY-F6lK1FJASMrhWWl 6O5bhAgYTsSjDdqymoIToHMd9_7wrwCHIcfBisEeROHsZ8_gTlKuAvEn8lrxrW0GHsZ 8_jw09J5xcQsK6MnWhEwdbtmVWh-xFtd40lYPVXe0A1jh02vfZsQgiFhLW4zr9EVdWo Uq>____
_______________________________________________ Outages mailing list Outages@outages.org <mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages
<http://cp.mcafee.com/d/k-Kr41ESyMNtMQsKnjodTdETjhvd7byrZPqdQQnPhOU OOYrhKCy-qen4TQjqdQTbLfzDSjp0DIblIl1iJlvxdJmvD-LOVJmGLMCSHfP_nVsSYw UMWw-_R-uoovu7ctuVt5dMQsYyUeod7bdTeEyCJtdmXX_axVZicHs3jr1JwSzRQGdAa mz8xVKVIDeqR4INpKNnwqj-f0T1dnoovaAVgtHBmVWh-xKHsZ8_gKgGT2TQ1lmVWh-D 00jqb2pEVsdwLQzh0qmWJPQzZ3iWq80HVDPSs182Cy04-vWVEwBizvQ96SjhOrVbh4>
-- To him who is able to keep you from falling and to present you before his glorious presence without fault and with great joy
_______________________________________________ Outages mailing list Outages@outages.org http://cp.mcafee.com/d/avndzhJ5xyXxEVsKC-rKrhKCy-qen4TXCQrFELCzBNBB USztd5YQsK9LECQrFKnuv7fICO1fomHoG2BqG_2rqI_fZvBPqJlvxdJmvD-LOVJV1Nx R1Z_HYYMM-YeoWZOWarxEVV5MsMqemrKth5dqWqJTT-l3PWApmU6CQjq9J7HFkr8kJ6 h3PtPpesRG9pyPtyL0QDYu1K2qKMM-l9OwXnaJPQzZ3tmVWh-xsxlK5LE2GJPQzZe00 CQm4PhOUr1vF6y0QJRrDF7W6BQQg1nPfDIU2g5d409Y_RPh1aB6_EidICzASH16Ohmp
The information contained in this E-mail message is privileged, confidential, and may be protected from disclosure; please be aware that any unauthorized use, printing, copying, disclosure or dissemination of this communication may be subject to legal restriction or sanction. If you think that you have received this E-mail message in error, please reply to the sender. This E-mail message and any attachments have been scanned for viruses and are believed to be free of any virus or other defect that might affect any computer system into which it is received and opened. It is the responsibility, however, of the recipient to ensure that it is virus free. _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
-- Jason Hellenthal JJH48-ARIN
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
-- Jameel Akari
just got off the phone with our Level3 Service Manager - he indicated all should be well. Watching and Waiting... On Thu, Aug 27, 2015 at 2:33 PM, Jameel Akari via Outages < outages@outages.org> wrote:
On Thu, 27 Aug 2015, Matthew Topper via Outages wrote:
Seeing on and off in upstate NY
Matthew Topper
Confirmed upstate NY (Albany) as well. We've rerouted in the meanwhile; were seeing lost routes and split routing back to us.
-- Jameel Akari
-----Original Message----- From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Jason Hellenthal via Outages Sent: Thursday, August 27, 2015 2:23 PM To: outages@outages.org Subject: Re: [outages] Level3/TW Telecom issues?
Retracting my statement from yesterday this is still a solid issue here. Yes absolutely still continuing. Root Cause would be a great explanation at this point.
On Aug 27, 2015, at 13:10, Ryan Brooks via Outages <outages@outages.org>
wrote:
24 hours of on and off packet loss, more than just Chicago. More communication would be helpful. NOC can't seem to decide if it's transport or routing. Any insights?
On Aug 27, 2015, at 1:08 PM, Bobin Joseph via Outages <
outages@outages.org> wrote:
We just got another update saying they still have issues.
IP services are affected across multiple markets. The IP NOC and IP and Transport Tier III are actively troubleshooting the issue. Operations Engineering is engaged and Field Services is on site in Chicago, IL investigating the issue. tw telecom has identified a network failure affecting your services. tw telecom tier-2 group is investigating the issue at this time.
Thanks, Bobin
From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Aaron Gross via Outages Sent: Thursday, August 27, 2015 10:55 AM To: Bob Hancock Cc: outages@outages.org Subject: Re: [outages] Level3/TW Telecom issues?
We are seeing the same things that happened yesterday on our circuits in Milwaukee and Chicago areas. Lots of packet loss and large latency through Chicago exchanges.
On Thu, Aug 27, 2015 at 12:33 PM, Bob Hancock via Outages < outages@outages.org> wrote: We were seeing substantial loss through Chicago this morning but it seems to have subsided at this time.
Bob Hancock
On 08/27/2015 12:19 PM, Steven Fischer via Outages wrote:
we are still seeing significant packet loss to 3 of our sites in the Midwest.
On Thu, Aug 27, 2015 at 12:47 PM, Bobin Joseph via Outages <outages@outages.org <mailto:outages@outages.org>> wrote:
We got this from Level3____
__ __
Hello,
The network outage has been resolved.
*Event Conclusion Summary*
Start: August 26, 2015 19:15 GMT Stop: August 27, 2015 00:14 GMT
Root Cause: An unknown issue in Chicago, IL was impacting IP services. Fix Action: Adjustments were made to resolve the issue and restore services.
Summary: The IP NOC responded to an unknown issue in Chicago, IL that was impacting IP services. Tier III Technical Support were engaged to assist with investigations, and adjustments were made to resolve the issue and stabilize traffic. Monitoring continues to ensure stability. Should further issues be experienced, please contact the Level 3 Technical Service Center.
____
__ __
Thanks,____
Bobin____
__ __
*From:*Outages [mailto:outages-bounces@outages.org <mailto:outages-bounces@outages.org>] *On Behalf Of *Ryan Gelobter via Outages *Sent:* Thursday, August 27, 2015 9:44 AM *To:* Rafael Possamai *Cc:* outages@outages.org <mailto:outages@outages.org> *Subject:* Re: [outages] Level3/TW Telecom issues?____
__ __
Was anyone able to get some useful information out of Level3 regarding this yesterday? Also is anyone else seeing similar issues?____
On Aug 26, 2015 2:40 PM, "Rafael Possamai via Outages" <outages@outages.org <mailto:outages@outages.org>> wrote:____
Here's a graph showing some packet loss: http://imgur.com/KVAzetA
<http://cp.mcafee.com/d/FZsSd38w83hJ5xyXxEVsKCMrKrhKCy-qen4TXCQrFEL CzBNBBUSztd5YQsK9LECQrFKnuv7fICO1fomHoG2BqG_2rqI_fZvBPqJlvxdJmvD-LO VJV1NxR1Z_HYYMM-YeoWZOWarxEVV5MsMqemrKth5dqWqJTT-l3PWApmU6CQjr1J7HF kr8kJ6h3PtPo0eiF5KfbU1gp41jVy1KHsZ8_gKgGT2TQ1lmVWh-D00jqb2pEVsdwLQz h0qmWJPQzZ3iWq80HVDPSs182Cy04-vWVEwBizvQ96SjhOrehG_98MC>____
__ __
__ __
On Wed, Aug 26, 2015 at 3:16 PM, Matt Elliott via Outages <outages@outages.org <mailto:outages@outages.org>> wrote:____
Anyone else having issues on Level3? We’re seeing packet loss with anything that traverses Chicago.____
__ __
Thanks,____
Matt____
This message and attachment(s), if any, is intended for the sole use of the individual and/or entity for which it is addressed, and may contain information that is privileged, confidential and prohibited from disclosure under applicable law. If you are not the addressee, or authorized to receive this on behalf of the addressee, you are hereby notified that you may not use, copy, disclose or distribute to anyone this message or any part thereof. If you have received this in error, please immediately advise the sender by e-mail and delete this information and all attachments from your computer and network. Thank you. ____
_______________________________________________ Outages mailing list Outages@outages.org <mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages
<http://cp.mcafee.com/d/avndy0QrhooKUqenbFI6XCQrFELCzBNd-VJ6WqbVEVs ppudETjhvd7byrW9J6WrBTDNPX9IwjS5GSawFmGLMCSHfP_nVsSHlnUjrlDV_HYKrug sotgvvW_fccfL3CeLsKyCUqeuhs7c6zBCXDkhjmKCHtZ_BgY-F6lK1FJcSMrhWWl6O5 bhAgYTsSjDdqymoIToHMd9_7wrwCHIcfBisEeROHsZ8_gTlKuAvEn8lrxrW0GHsZ8_j w09J5xcQsK6MnWhEwdbtmVWh-xFtd40lYPVXe0A1jh02vfZsQgiFhLW4zr9EVdBPdH> ____
__ __
_______________________________________________ Outages mailing list Outages@outages.org <mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages
<http://cp.mcafee.com/d/2DRPoQ938ArhooKUqenbFI6XCQrFELCzBNd-VJ6WqbV EVsppudETjhvd7byrW9J6WrBTDNPX9IwjS5GSawFmGLMCSHfP_nVsSHlnUjrlDV_HYK rugsotgvvW_fccfL3CeLsKyCUqeuhs7c6zBCXDkhjmKCHtZ_BgY-F6lK1FJASMrhWWl 6O5bhAgYTsSjDdqymoIToHMd9_7wrwCHIcfBisEeROHsZ8_gTlKuAvEn8lrxrW0GHsZ 8_jw09J5xcQsK6MnWhEwdbtmVWh-xFtd40lYPVXe0A1jh02vfZsQgiFhLW4zr9EVdWo Uq>____
_______________________________________________ Outages mailing list Outages@outages.org <mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages
<http://cp.mcafee.com/d/k-Kr41ESyMNtMQsKnjodTdETjhvd7byrZPqdQQnPhOU OOYrhKCy-qen4TQjqdQTbLfzDSjp0DIblIl1iJlvxdJmvD-LOVJmGLMCSHfP_nVsSYw UMWw-_R-uoovu7ctuVt5dMQsYyUeod7bdTeEyCJtdmXX_axVZicHs3jr1JwSzRQGdAa mz8xVKVIDeqR4INpKNnwqj-f0T1dnoovaAVgtHBmVWh-xKHsZ8_gKgGT2TQ1lmVWh-D 00jqb2pEVsdwLQzh0qmWJPQzZ3iWq80HVDPSs182Cy04-vWVEwBizvQ96SjhOrVbh4>
-- To him who is able to keep you from falling and to present you before his glorious presence without fault and with great joy
_______________________________________________ Outages mailing list Outages@outages.org http://cp.mcafee.com/d/avndzhJ5xyXxEVsKC-rKrhKCy-qen4TXCQrFELCzBNBB USztd5YQsK9LECQrFKnuv7fICO1fomHoG2BqG_2rqI_fZvBPqJlvxdJmvD-LOVJV1Nx R1Z_HYYMM-YeoWZOWarxEVV5MsMqemrKth5dqWqJTT-l3PWApmU6CQjq9J7HFkr8kJ6 h3PtPpesRG9pyPtyL0QDYu1K2qKMM-l9OwXnaJPQzZ3tmVWh-xsxlK5LE2GJPQzZe00 CQm4PhOUr1vF6y0QJRrDF7W6BQQg1nPfDIU2g5d409Y_RPh1aB6_EidICzASH16Ohmp
The information contained in this E-mail message is privileged, confidential, and may be protected from disclosure; please be aware that any unauthorized use, printing, copying, disclosure or dissemination of this communication may be subject to legal restriction or sanction. If you think that you have received this E-mail message in error, please reply to the sender. This E-mail message and any attachments have been scanned for viruses and are believed to be free of any virus or other defect that might affect any computer system into which it is received and opened. It is the responsibility, however, of the recipient to ensure that it is virus free.
Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
-- Jason Hellenthal JJH48-ARIN
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
-- Jameel Akari _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
-- To him who is able to keep you from falling and to present you before his glorious presence without fault and with great joy
I'm seeing traffic outbound from Level 3 Tampa going to Frankfurt Germany currently, so they're definitely doing something wrong. 3 xe-10-3-1.bar1.Tampa1.Level3.net (4.53.172.29) 13.642 ms 13.650 ms 13.645 ms 4 ae-1-60.edge3.Frankfurt1.Level3.net (4.69.154.7) 117.129 ms 117.100 ms ae-2-70.edge3.Frankfurt1.Level3.net (4.69.154.71) 116.928 ms 5 ae-3-80.edge3.Frankfurt1.Level3.net (4.69.154.135) 116.724 ms ae-1-60.edge3.Frankfurt1.Level3.net (4.69.154.7) 117.079 ms ae-3-80.edge3.Frankfurt1.Level3.net (4.69.154.135) 116.756 ms
-----Original Message----- From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Matthew Topper via Outages Sent: Thursday, August 27, 2015 2:29 PM To: outages@outages.org Subject: Re: [outages] Level3/TW Telecom issues?
Seeing on and off in upstate NY
Matthew Topper
-----Original Message----- From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Jason Hellenthal via Outages Sent: Thursday, August 27, 2015 2:23 PM To: outages@outages.org Subject: Re: [outages] Level3/TW Telecom issues?
Retracting my statement from yesterday this is still a solid issue here. Yes absolutely still continuing. Root Cause would be a great explanation at this point.
On Aug 27, 2015, at 13:10, Ryan Brooks via Outages <outages@outages.org> wrote:
24 hours of on and off packet loss, more than just Chicago. More communication would be helpful. NOC can't seem to decide if it's transport or routing. Any insights?
On Aug 27, 2015, at 1:08 PM, Bobin Joseph via Outages <outages@outages.org> wrote:
We just got another update saying they still have issues.
IP services are affected across multiple markets. The IP NOC and IP and Transport Tier III are actively troubleshooting the issue. Operations Engineering is engaged and Field Services is on site in Chicago, IL investigating the issue. tw telecom has identified a network failure affecting your services. tw telecom tier-2 group is investigating the issue at this time.
Thanks, Bobin
From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Aaron Gross via Outages Sent: Thursday, August 27, 2015 10:55 AM To: Bob Hancock Cc: outages@outages.org Subject: Re: [outages] Level3/TW Telecom issues?
We are seeing the same things that happened yesterday on our circuits in Milwaukee and Chicago areas. Lots of packet loss and large latency through Chicago exchanges.
On Thu, Aug 27, 2015 at 12:33 PM, Bob Hancock via Outages <outages@outages.org> wrote: We were seeing substantial loss through Chicago this morning but it seems to have subsided at this time.
Bob Hancock
On 08/27/2015 12:19 PM, Steven Fischer via Outages wrote:
we are still seeing significant packet loss to 3 of our sites in the Midwest.
On Thu, Aug 27, 2015 at 12:47 PM, Bobin Joseph via Outages <outages@outages.org <mailto:outages@outages.org>> wrote:
We got this from Level3____
__ __
Hello,
The network outage has been resolved.
*Event Conclusion Summary*
Start: August 26, 2015 19:15 GMT Stop: August 27, 2015 00:14 GMT
Root Cause: An unknown issue in Chicago, IL was impacting IP services. Fix Action: Adjustments were made to resolve the issue and restore services.
Summary: The IP NOC responded to an unknown issue in Chicago, IL that was impacting IP services. Tier III Technical Support were engaged to assist with investigations, and adjustments were made to resolve the issue and stabilize traffic. Monitoring continues to ensure stability. Should further issues be experienced, please contact the Level 3 Technical Service Center.
____
__ __
Thanks,____
Bobin____
__ __
*From:*Outages [mailto:outages-bounces@outages.org <mailto:outages-bounces@outages.org>] *On Behalf Of *Ryan Gelobter via Outages *Sent:* Thursday, August 27, 2015 9:44 AM *To:* Rafael Possamai *Cc:* outages@outages.org <mailto:outages@outages.org> *Subject:* Re: [outages] Level3/TW Telecom issues?____
__ __
Was anyone able to get some useful information out of Level3 regarding this yesterday? Also is anyone else seeing similar issues?____
On Aug 26, 2015 2:40 PM, "Rafael Possamai via Outages" <outages@outages.org <mailto:outages@outages.org>> wrote:____
Here's a graph showing some packet loss: http://imgur.com/KVAzetA
<http://cp.mcafee.com/d/FZsSd38w83hJ5xyXxEVsKCMrKrhKCy-qen4TXCQrFEL
CzBNBBUSztd5YQsK9LECQrFKnuv7fICO1fomHoG2BqG_2rqI_fZvBPqJlvxdJmvD-LO
VJV1NxR1Z_HYYMM-YeoWZOWarxEVV5MsMqemrKth5dqWqJTT-l3PWApmU6CQjr1J7HF
kr8kJ6h3PtPo0eiF5KfbU1gp41jVy1KHsZ8_gKgGT2TQ1lmVWh-D00jqb2pEVsdwLQz
h0qmWJPQzZ3iWq80HVDPSs182Cy04-vWVEwBizvQ96SjhOrehG_98MC>____
__ __
__ __
On Wed, Aug 26, 2015 at 3:16 PM, Matt Elliott via Outages <outages@outages.org <mailto:outages@outages.org>> wrote:____
Anyone else having issues on Level3? We're seeing packet loss with anything that traverses Chicago.____
__ __
Thanks,____
Matt____
This message and attachment(s), if any, is intended for the sole use of the individual and/or entity for which it is addressed, and may contain information that is privileged, confidential and prohibited from disclosure under applicable law. If you are not the addressee, or authorized to receive this on behalf of the addressee, you are hereby notified that you may not use, copy, disclose or distribute to anyone this message or any part thereof. If you have received this in error, please immediately advise the sender by e-mail and delete this information and all attachments from your computer and network. Thank you. ____
_______________________________________________ Outages mailing list Outages@outages.org <mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages
<http://cp.mcafee.com/d/avndy0QrhooKUqenbFI6XCQrFELCzBNd-VJ6WqbVEVs
ppudETjhvd7byrW9J6WrBTDNPX9IwjS5GSawFmGLMCSHfP_nVsSHlnUjrlDV_HYKrug
sotgvvW_fccfL3CeLsKyCUqeuhs7c6zBCXDkhjmKCHtZ_BgY-F6lK1FJcSMrhWWl6O5
bhAgYTsSjDdqymoIToHMd9_7wrwCHIcfBisEeROHsZ8_gTlKuAvEn8lrxrW0GHsZ8_j
w09J5xcQsK6MnWhEwdbtmVWh-xFtd40lYPVXe0A1jh02vfZsQgiFhLW4zr9EVdBPdH>
____
__ __
_______________________________________________ Outages mailing list Outages@outages.org <mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages
<http://cp.mcafee.com/d/2DRPoQ938ArhooKUqenbFI6XCQrFELCzBNd-VJ6WqbV
EVsppudETjhvd7byrW9J6WrBTDNPX9IwjS5GSawFmGLMCSHfP_nVsSHlnUjrlDV_HYK
rugsotgvvW_fccfL3CeLsKyCUqeuhs7c6zBCXDkhjmKCHtZ_BgY-F6lK1FJASMrhWWl
6O5bhAgYTsSjDdqymoIToHMd9_7wrwCHIcfBisEeROHsZ8_gTlKuAvEn8lrxrW0GHsZ
8_jw09J5xcQsK6MnWhEwdbtmVWh-xFtd40lYPVXe0A1jh02vfZsQgiFhLW4zr9EVdWo
Uq>____
_______________________________________________ Outages mailing list Outages@outages.org <mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages
<http://cp.mcafee.com/d/k-Kr41ESyMNtMQsKnjodTdETjhvd7byrZPqdQQnPhOU
OOYrhKCy-qen4TQjqdQTbLfzDSjp0DIblIl1iJlvxdJmvD-LOVJmGLMCSHfP_nVsSYw
UMWw-_R-uoovu7ctuVt5dMQsYyUeod7bdTeEyCJtdmXX_axVZicHs3jr1JwSzRQGdAa
mz8xVKVIDeqR4INpKNnwqj-f0T1dnoovaAVgtHBmVWh-xKHsZ8_gKgGT2TQ1lmVWh-D
00jqb2pEVsdwLQzh0qmWJPQzZ3iWq80HVDPSs182Cy04-vWVEwBizvQ96SjhOrVbh4>
-- To him who is able to keep you from falling and to present you before his glorious presence without fault and with great joy
_______________________________________________ Outages mailing list Outages@outages.org
http://cp.mcafee.com/d/avndzhJ5xyXxEVsKC-rKrhKCy-qen4TXCQrFELCzBNBB
USztd5YQsK9LECQrFKnuv7fICO1fomHoG2BqG_2rqI_fZvBPqJlvxdJmvD-LOVJV1Nx
R1Z_HYYMM-YeoWZOWarxEVV5MsMqemrKth5dqWqJTT-l3PWApmU6CQjq9J7HFkr8kJ6
h3PtPpesRG9pyPtyL0QDYu1K2qKMM-l9OwXnaJPQzZ3tmVWh-xsxlK5LE2GJPQzZe00
CQm4PhOUr1vF6y0QJRrDF7W6BQQg1nPfDIU2g5d409Y_RPh1aB6_EidICzASH16Ohmp
The information contained in this E-mail message is privileged, confidential, and may be protected from disclosure; please be aware that any unauthorized use, printing, copying, disclosure or dissemination of this communication may be subject to legal restriction or sanction. If you think that you have received this E-mail message in error, please reply to the sender. This E-mail message and any attachments have been scanned for viruses and are believed to be free of any virus or other defect that might affect any computer system into which it is received and opened. It is the responsibility, however, of the recipient to ensure that it is virus free. _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
-- Jason Hellenthal JJH48-ARIN
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
On Aug 27, 2015, at 1:37 PM, David Hubbard via Outages <outages@outages.org> wrote:
I'm seeing traffic outbound from Level 3 Tampa going to Frankfurt Germany currently, so they're definitely doing something wrong.
3 xe-10-3-1.bar1.Tampa1.Level3.net (4.53.172.29) 13.642 ms 13.650 ms 13.645 ms 4 ae-1-60.edge3.Frankfurt1.Level3.net (4.69.154.7) 117.129 ms 117.100 ms ae-2-70.edge3.Frankfurt1.Level3.net (4.69.154.71) 116.928 ms 5 ae-3-80.edge3.Frankfurt1.Level3.net (4.69.154.135) 116.724 ms ae-1-60.edge3.Frankfurt1.Level3.net (4.69.154.7) 117.079 ms ae-3-80.edge3.Frankfurt1.Level3.net (4.69.154.135) 116.756 ms
And their noc says: Level3 has identified a fiber optic line failure affecting your services. Fiber crew is working to restore service, slack is being pulled now and splicing will begin soon. ETTR 2hr. - Some folks I've spoken with there say DWDM problems, other say peering. I'm sure someone knows, it's just not the NOC.
Anybody else having complete outage on Level3? We lost complete internet in DFW and SAN Thanks, Bobin -----Original Message----- From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Ryan Brooks via Outages Sent: Thursday, August 27, 2015 12:03 PM To: outages@outages.org Subject: Re: [outages] Level3/TW Telecom issues?
On Aug 27, 2015, at 1:37 PM, David Hubbard via Outages <outages@outages.org> wrote:
I'm seeing traffic outbound from Level 3 Tampa going to Frankfurt Germany currently, so they're definitely doing something wrong.
3 xe-10-3-1.bar1.Tampa1.Level3.net (4.53.172.29) 13.642 ms 13.650 ms 13.645 ms 4 ae-1-60.edge3.Frankfurt1.Level3.net (4.69.154.7) 117.129 ms 117.100 ms ae-2-70.edge3.Frankfurt1.Level3.net (4.69.154.71) 116.928 ms 5 ae-3-80.edge3.Frankfurt1.Level3.net (4.69.154.135) 116.724 ms ae-1-60.edge3.Frankfurt1.Level3.net (4.69.154.7) 117.079 ms ae-3-80.edge3.Frankfurt1.Level3.net (4.69.154.135) 116.756 ms
And their noc says: Level3 has identified a fiber optic line failure affecting your services. Fiber crew is working to restore service, slack is being pulled now and splicing will begin soon. ETTR 2hr. - Some folks I've spoken with there say DWDM problems, other say peering. I'm sure someone knows, it's just not the NOC. _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Was down in Boise, ID for about 15 minutes. Looks to have cleared up now though. [image: Interstate_TP_Logo]*Lucius Kalbfleisch* *IT -* Systems and Network Administrator (208) 442-7683 Direct (208) 284-7194 Mobile kalbfleischl@interstatecargo.com www.TrailersPlus.com <http://www.trailersplus.com/> [image: Facebook] Facebook.com/Trailersplus <http://www.facebook.com/Trailersplus> *TrailersPlus / Interstate Group* 3800 Airport Rd Nampa, ID 83687 On Thu, Aug 27, 2015 at 1:49 PM Bobin Joseph via Outages < outages@outages.org> wrote:
Anybody else having complete outage on Level3? We lost complete internet in DFW and SAN
Thanks, Bobin
-----Original Message----- From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Ryan Brooks via Outages Sent: Thursday, August 27, 2015 12:03 PM To: outages@outages.org Subject: Re: [outages] Level3/TW Telecom issues?
On Aug 27, 2015, at 1:37 PM, David Hubbard via Outages < outages@outages.org> wrote:
I'm seeing traffic outbound from Level 3 Tampa going to Frankfurt Germany currently, so they're definitely doing something wrong.
3 xe-10-3-1.bar1.Tampa1.Level3.net (4.53.172.29) 13.642 ms 13.650 ms 13.645 ms 4 ae-1-60.edge3.Frankfurt1.Level3.net (4.69.154.7) 117.129 ms 117.100 ms ae-2-70.edge3.Frankfurt1.Level3.net (4.69.154.71) 116.928 ms 5 ae-3-80.edge3.Frankfurt1.Level3.net (4.69.154.135) 116.724 ms ae-1-60.edge3.Frankfurt1.Level3.net (4.69.154.7) 117.079 ms ae-3-80.edge3.Frankfurt1.Level3.net (4.69.154.135) 116.756 ms
And their noc says:
Level3 has identified a fiber optic line failure affecting your services. Fiber crew is working to restore service, slack is being pulled now and splicing will begin soon. ETTR 2hr.
-
Some folks I've spoken with there say DWDM problems, other say peering. I'm sure someone knows, it's just not the NOC. _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Yep, complete outage at our DFW location aswell On Thu, Aug 27, 2015 at 2:48 PM, Bobin Joseph via Outages <outages@outages.org> wrote:
Anybody else having complete outage on Level3? We lost complete internet in DFW and SAN
Thanks, Bobin
-----Original Message----- From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Ryan Brooks via Outages Sent: Thursday, August 27, 2015 12:03 PM To: outages@outages.org Subject: Re: [outages] Level3/TW Telecom issues?
On Aug 27, 2015, at 1:37 PM, David Hubbard via Outages <outages@outages.org> wrote:
I'm seeing traffic outbound from Level 3 Tampa going to Frankfurt Germany currently, so they're definitely doing something wrong.
3 xe-10-3-1.bar1.Tampa1.Level3.net (4.53.172.29) 13.642 ms 13.650 ms 13.645 ms 4 ae-1-60.edge3.Frankfurt1.Level3.net (4.69.154.7) 117.129 ms 117.100 ms ae-2-70.edge3.Frankfurt1.Level3.net (4.69.154.71) 116.928 ms 5 ae-3-80.edge3.Frankfurt1.Level3.net (4.69.154.135) 116.724 ms ae-1-60.edge3.Frankfurt1.Level3.net (4.69.154.7) 117.079 ms ae-3-80.edge3.Frankfurt1.Level3.net (4.69.154.135) 116.756 ms
And their noc says:
Level3 has identified a fiber optic line failure affecting your services. Fiber crew is working to restore service, slack is being pulled now and splicing will begin soon. ETTR 2hr.
-
Some folks I've spoken with there say DWDM problems, other say peering. I'm sure someone knows, it's just not the NOC. _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
On 27 August 2015 at 12:48, Bobin Joseph via Outages <outages@outages.org> wrote:
Anybody else having complete outage on Level3? We lost complete internet in DFW and SAN
We were not down but significant routing issues in Las Vegas.
Thanks, Bobin
We had to shut down our Level3 circuit in the DFW area, had issues from 1406CST until we shut down our circuit. -----Original Message----- From: Outages [mailto:outages-bounces@outages.org] On Behalf Of jungle Boogie via Outages Sent: Thursday, August 27, 2015 2:56 PM To: Bobin Joseph Cc: outages@outages.org Subject: Re: [outages] Level3/TW Telecom issues? On 27 August 2015 at 12:48, Bobin Joseph via Outages <outages@outages.org> wrote:
Anybody else having complete outage on Level3? We lost complete internet in DFW and SAN
We were not down but significant routing issues in Las Vegas.
Thanks, Bobin
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Same goes here. Had to shut interfaces on our Level3 routers. Thanks, Bobin -----Original Message----- From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Eric Henson via Outages Sent: Thursday, August 27, 2015 12:59 PM To: outages@outages.org Subject: Re: [outages] Level3/TW Telecom issues? We had to shut down our Level3 circuit in the DFW area, had issues from 1406CST until we shut down our circuit. -----Original Message----- From: Outages [mailto:outages-bounces@outages.org] On Behalf Of jungle Boogie via Outages Sent: Thursday, August 27, 2015 2:56 PM To: Bobin Joseph Cc: outages@outages.org Subject: Re: [outages] Level3/TW Telecom issues? On 27 August 2015 at 12:48, Bobin Joseph via Outages <outages@outages.org> wrote:
Anybody else having complete outage on Level3? We lost complete internet in DFW and SAN
We were not down but significant routing issues in Las Vegas.
Thanks, Bobin
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Also had to kill our session with TW 4323 in Cocoa FL. Lots of intermittent issues. Nick Olsen Network Operations (855) FLSPEED x106 ---------------------------------------- From: "Eric Henson via Outages" <outages@outages.org> Sent: Thursday, August 27, 2015 4:00 PM To: "outages@outages.org" <outages@outages.org> Subject: Re: [outages] Level3/TW Telecom issues? We had to shut down our Level3 circuit in the DFW area, had issues from 1406CST until we shut down our circuit. -----Original Message----- From: Outages [mailto:outages-bounces@outages.org] On Behalf Of jungle Boogie via Outages Sent: Thursday, August 27, 2015 2:56 PM To: Bobin Joseph Cc: outages@outages.org Subject: Re: [outages] Level3/TW Telecom issues? On 27 August 2015 at 12:48, Bobin Joseph via Outages <outages@outages.org> wrote:
Anybody else having complete outage on Level3? We lost complete internet in DFW and SAN
We were not down but significant routing issues in Las Vegas.
Thanks, Bobin
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
If you have access to the Level3 portal you should see ticket #9639047 under Network Events now. *Event SummaryIP Network Event ~ Multiple Markets08/27/2015 6:36 PM GMT* IP and Transport Tier III, Operations Engineering and Field Services continue collaboratively working the issue. *08/27/2015 4:59 PM GMT* Operations Engineering is engaged and Field Services is on site in Chicago, IL investigating the issue. *08/27/2015 4:38 PM GMT* The engineers are currently migrating traffic in efforts of restoring services while troubleshooting continues. Field Services is being dispatched to a Chicago, IL site to assist. *08/27/2015 4:21 PM GMT* IP services are affected across multiple markets and the root cause is currently under investigation. The IP NOC and IP and Transport Tier III are actively troubleshooting and working to isolate the cause. The engineers have detected peering issues which are resulting in packet loss for customers. Please be advised that updates will be provided at minimum of hourly unless otherwise noted. On Thu, Aug 27, 2015 at 2:59 PM, Eric Henson via Outages < outages@outages.org> wrote:
We had to shut down our Level3 circuit in the DFW area, had issues from 1406CST until we shut down our circuit.
-----Original Message----- From: Outages [mailto:outages-bounces@outages.org] On Behalf Of jungle Boogie via Outages Sent: Thursday, August 27, 2015 2:56 PM To: Bobin Joseph Cc: outages@outages.org Subject: Re: [outages] Level3/TW Telecom issues?
On 27 August 2015 at 12:48, Bobin Joseph via Outages <outages@outages.org> wrote:
Anybody else having complete outage on Level3? We lost complete internet in DFW and SAN
We were not down but significant routing issues in Las Vegas.
Thanks, Bobin
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
*08/27/2015 8:05 PM GMT*Level 3 Tier III and Operations Engineering teams have identified Internet Protocols dropping, affecting customer services. Restoration efforts are in progress, but an estimated time of restoral is not available at this time. On Thu, Aug 27, 2015 at 3:06 PM, Ryan Gelobter <ryan.g@atwgpc.net> wrote:
If you have access to the Level3 portal you should see ticket #9639047 under Network Events now.
*Event SummaryIP Network Event ~ Multiple Markets08/27/2015 6:36 PM GMT*
IP and Transport Tier III, Operations Engineering and Field Services continue collaboratively working the issue.
*08/27/2015 4:59 PM GMT*
Operations Engineering is engaged and Field Services is on site in Chicago, IL investigating the issue.
*08/27/2015 4:38 PM GMT*
The engineers are currently migrating traffic in efforts of restoring services while troubleshooting continues. Field Services is being dispatched to a Chicago, IL site to assist.
*08/27/2015 4:21 PM GMT*
IP services are affected across multiple markets and the root cause is currently under investigation. The IP NOC and IP and Transport Tier III are actively troubleshooting and working to isolate the cause. The engineers have detected peering issues which are resulting in packet loss for customers. Please be advised that updates will be provided at minimum of hourly unless otherwise noted.
On Thu, Aug 27, 2015 at 2:59 PM, Eric Henson via Outages < outages@outages.org> wrote:
We had to shut down our Level3 circuit in the DFW area, had issues from 1406CST until we shut down our circuit.
-----Original Message----- From: Outages [mailto:outages-bounces@outages.org] On Behalf Of jungle Boogie via Outages Sent: Thursday, August 27, 2015 2:56 PM To: Bobin Joseph Cc: outages@outages.org Subject: Re: [outages] Level3/TW Telecom issues?
On 27 August 2015 at 12:48, Bobin Joseph via Outages <outages@outages.org> wrote:
Anybody else having complete outage on Level3? We lost complete internet in DFW and SAN
We were not down but significant routing issues in Las Vegas.
Thanks, Bobin
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
ICMP blazes through at times, but TCP completely falls apart.
On Aug 27, 2015, at 3:13 PM, Ryan Gelobter via Outages <outages@outages.org> wrote:
08/27/2015 8:05 PM GMT Level 3 Tier III and Operations Engineering teams have identified Internet Protocols dropping, affecting customer services. Restoration efforts are in progress, but an estimated time of restoral is not available at this time.
On Thu, Aug 27, 2015 at 3:06 PM, Ryan Gelobter <ryan.g@atwgpc.net <mailto:ryan.g@atwgpc.net>> wrote: If you have access to the Level3 portal you should see ticket #9639047 under Network Events now.
Event Summary IP Network Event ~ Multiple Markets
08/27/2015 6:36 PM GMT IP and Transport Tier III, Operations Engineering and Field Services continue collaboratively working the issue.
08/27/2015 4:59 PM GMT Operations Engineering is engaged and Field Services is on site in Chicago, IL investigating the issue.
08/27/2015 4:38 PM GMT The engineers are currently migrating traffic in efforts of restoring services while troubleshooting continues. Field Services is being dispatched to a Chicago, IL site to assist.
08/27/2015 4:21 PM GMT IP services are affected across multiple markets and the root cause is currently under investigation. The IP NOC and IP and Transport Tier III are actively troubleshooting and working to isolate the cause. The engineers have detected peering issues which are resulting in packet loss for customers. Please be advised that updates will be provided at minimum of hourly unless otherwise noted.
On Thu, Aug 27, 2015 at 2:59 PM, Eric Henson via Outages <outages@outages.org <mailto:outages@outages.org>> wrote: We had to shut down our Level3 circuit in the DFW area, had issues from 1406CST until we shut down our circuit.
-----Original Message----- From: Outages [mailto:outages-bounces@outages.org <mailto:outages-bounces@outages.org>] On Behalf Of jungle Boogie via Outages Sent: Thursday, August 27, 2015 2:56 PM To: Bobin Joseph Cc: outages@outages.org <mailto:outages@outages.org> Subject: Re: [outages] Level3/TW Telecom issues?
On 27 August 2015 at 12:48, Bobin Joseph via Outages <outages@outages.org <mailto:outages@outages.org>> wrote:
Anybody else having complete outage on Level3? We lost complete internet in DFW and SAN
We were not down but significant routing issues in Las Vegas.
Thanks, Bobin
_______________________________________________ Outages mailing list Outages@outages.org <mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages <https://puck.nether.net/mailman/listinfo/outages> _______________________________________________ Outages mailing list Outages@outages.org <mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages <https://puck.nether.net/mailman/listinfo/outages>
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Latest update..... *08/27/2015 9:49 PM GMT* Tier III Technical Support and Operations Engineering teams continue investigating the issue identified with the Internet Protocols. Restoration efforts are ongoing; however, there is currently no estimated restoral time. On Thu, Aug 27, 2015 at 3:14 PM, Ryan Brooks <rbrooks@stack41.com> wrote:
ICMP blazes through at times, but TCP completely falls apart.
On Aug 27, 2015, at 3:13 PM, Ryan Gelobter via Outages < outages@outages.org> wrote:
*08/27/2015 8:05 PM GMT*Level 3 Tier III and Operations Engineering teams have identified Internet Protocols dropping, affecting customer services. Restoration efforts are in progress, but an estimated time of restoral is not available at this time.
On Thu, Aug 27, 2015 at 3:06 PM, Ryan Gelobter <ryan.g@atwgpc.net> wrote:
If you have access to the Level3 portal you should see ticket #9639047 under Network Events now.
*Event SummaryIP Network Event ~ Multiple Markets08/27/2015 6:36 PM GMT*
IP and Transport Tier III, Operations Engineering and Field Services continue collaboratively working the issue.
*08/27/2015 4:59 PM GMT*
Operations Engineering is engaged and Field Services is on site in Chicago, IL investigating the issue.
*08/27/2015 4:38 PM GMT*
The engineers are currently migrating traffic in efforts of restoring services while troubleshooting continues. Field Services is being dispatched to a Chicago, IL site to assist.
*08/27/2015 4:21 PM GMT*
IP services are affected across multiple markets and the root cause is currently under investigation. The IP NOC and IP and Transport Tier III are actively troubleshooting and working to isolate the cause. The engineers have detected peering issues which are resulting in packet loss for customers. Please be advised that updates will be provided at minimum of hourly unless otherwise noted.
On Thu, Aug 27, 2015 at 2:59 PM, Eric Henson via Outages < outages@outages.org> wrote:
We had to shut down our Level3 circuit in the DFW area, had issues from 1406CST until we shut down our circuit.
-----Original Message----- From: Outages [mailto:outages-bounces@outages.org] On Behalf Of jungle Boogie via Outages Sent: Thursday, August 27, 2015 2:56 PM To: Bobin Joseph Cc: outages@outages.org Subject: Re: [outages] Level3/TW Telecom issues?
On 27 August 2015 at 12:48, Bobin Joseph via Outages < outages@outages.org> wrote:
Anybody else having complete outage on Level3? We lost complete internet in DFW and SAN
We were not down but significant routing issues in Las Vegas.
Thanks, Bobin
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
*08/27/2015 11:16 PM GMT* Multiple cards have been ordered, and will be installed on equipment in Chicago, IL in an effort to balance the traffic load through the site and alleviate the issue. Field Services are currently en route to the site with the cards, and are expected to arrive shortly. On Thu, Aug 27, 2015 at 5:02 PM, Ryan Gelobter <ryan.g@atwgpc.net> wrote:
Latest update..... *08/27/2015 9:49 PM GMT*
Tier III Technical Support and Operations Engineering teams continue investigating the issue identified with the Internet Protocols. Restoration efforts are ongoing; however, there is currently no estimated restoral time.
On Thu, Aug 27, 2015 at 3:14 PM, Ryan Brooks <rbrooks@stack41.com> wrote:
ICMP blazes through at times, but TCP completely falls apart.
On Aug 27, 2015, at 3:13 PM, Ryan Gelobter via Outages < outages@outages.org> wrote:
*08/27/2015 8:05 PM GMT*Level 3 Tier III and Operations Engineering teams have identified Internet Protocols dropping, affecting customer services. Restoration efforts are in progress, but an estimated time of restoral is not available at this time.
On Thu, Aug 27, 2015 at 3:06 PM, Ryan Gelobter <ryan.g@atwgpc.net> wrote:
If you have access to the Level3 portal you should see ticket #9639047 under Network Events now.
*Event SummaryIP Network Event ~ Multiple Markets08/27/2015 6:36 PM GMT*
IP and Transport Tier III, Operations Engineering and Field Services continue collaboratively working the issue.
*08/27/2015 4:59 PM GMT*
Operations Engineering is engaged and Field Services is on site in Chicago, IL investigating the issue.
*08/27/2015 4:38 PM GMT*
The engineers are currently migrating traffic in efforts of restoring services while troubleshooting continues. Field Services is being dispatched to a Chicago, IL site to assist.
*08/27/2015 4:21 PM GMT*
IP services are affected across multiple markets and the root cause is currently under investigation. The IP NOC and IP and Transport Tier III are actively troubleshooting and working to isolate the cause. The engineers have detected peering issues which are resulting in packet loss for customers. Please be advised that updates will be provided at minimum of hourly unless otherwise noted.
On Thu, Aug 27, 2015 at 2:59 PM, Eric Henson via Outages < outages@outages.org> wrote:
We had to shut down our Level3 circuit in the DFW area, had issues from 1406CST until we shut down our circuit.
-----Original Message----- From: Outages [mailto:outages-bounces@outages.org] On Behalf Of jungle Boogie via Outages Sent: Thursday, August 27, 2015 2:56 PM To: Bobin Joseph Cc: outages@outages.org Subject: Re: [outages] Level3/TW Telecom issues?
On 27 August 2015 at 12:48, Bobin Joseph via Outages < outages@outages.org> wrote:
Anybody else having complete outage on Level3? We lost complete internet in DFW and SAN
We were not down but significant routing issues in Las Vegas.
Thanks, Bobin
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Yesterday, fiber cut. Today, "Internet Protocol" problems. Tonight, capacity management?
On Aug 27, 2015, at 8:01 PM, Ryan Gelobter via Outages <outages@outages.org> wrote:
08/27/2015 11:16 PM GMT Multiple cards have been ordered, and will be installed on equipment in Chicago, IL in an effort to balance the traffic load through the site and alleviate the issue. Field Services are currently en route to the site with the cards, and are expected to arrive shortly.
On Thu, Aug 27, 2015 at 5:02 PM, Ryan Gelobter <ryan.g@atwgpc.net <mailto:ryan.g@atwgpc.net>> wrote: Latest update..... 08/27/2015 9:49 PM GMT Tier III Technical Support and Operations Engineering teams continue investigating the issue identified with the Internet Protocols. Restoration efforts are ongoing; however, there is currently no estimated restoral time.
On Thu, Aug 27, 2015 at 3:14 PM, Ryan Brooks <rbrooks@stack41.com <mailto:rbrooks@stack41.com>> wrote: ICMP blazes through at times, but TCP completely falls apart.
On Aug 27, 2015, at 3:13 PM, Ryan Gelobter via Outages <outages@outages.org <mailto:outages@outages.org>> wrote:
08/27/2015 8:05 PM GMT Level 3 Tier III and Operations Engineering teams have identified Internet Protocols dropping, affecting customer services. Restoration efforts are in progress, but an estimated time of restoral is not available at this time.
On Thu, Aug 27, 2015 at 3:06 PM, Ryan Gelobter <ryan.g@atwgpc.net <mailto:ryan.g@atwgpc.net>> wrote: If you have access to the Level3 portal you should see ticket #9639047 under Network Events now.
Event Summary IP Network Event ~ Multiple Markets
08/27/2015 6:36 PM GMT IP and Transport Tier III, Operations Engineering and Field Services continue collaboratively working the issue.
08/27/2015 4:59 PM GMT Operations Engineering is engaged and Field Services is on site in Chicago, IL investigating the issue.
08/27/2015 4:38 PM GMT The engineers are currently migrating traffic in efforts of restoring services while troubleshooting continues. Field Services is being dispatched to a Chicago, IL site to assist.
08/27/2015 4:21 PM GMT IP services are affected across multiple markets and the root cause is currently under investigation. The IP NOC and IP and Transport Tier III are actively troubleshooting and working to isolate the cause. The engineers have detected peering issues which are resulting in packet loss for customers. Please be advised that updates will be provided at minimum of hourly unless otherwise noted.
On Thu, Aug 27, 2015 at 2:59 PM, Eric Henson via Outages <outages@outages.org <mailto:outages@outages.org>> wrote: We had to shut down our Level3 circuit in the DFW area, had issues from 1406CST until we shut down our circuit.
-----Original Message----- From: Outages [mailto:outages-bounces@outages.org <mailto:outages-bounces@outages.org>] On Behalf Of jungle Boogie via Outages Sent: Thursday, August 27, 2015 2:56 PM To: Bobin Joseph Cc: outages@outages.org <mailto:outages@outages.org> Subject: Re: [outages] Level3/TW Telecom issues?
On 27 August 2015 at 12:48, Bobin Joseph via Outages <outages@outages.org <mailto:outages@outages.org>> wrote:
Anybody else having complete outage on Level3? We lost complete internet in DFW and SAN
We were not down but significant routing issues in Las Vegas.
Thanks, Bobin
_______________________________________________ Outages mailing list Outages@outages.org <mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages <https://puck.nether.net/mailman/listinfo/outages> _______________________________________________ Outages mailing list Outages@outages.org <mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages <https://puck.nether.net/mailman/listinfo/outages>
_______________________________________________ Outages mailing list Outages@outages.org <mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages <https://puck.nether.net/mailman/listinfo/outages>
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Rofl -- Jason Hellenthal JJH48-ARIN On Aug 27, 2015, at 20:08, Ryan Brooks via Outages <outages@outages.org> wrote: Yesterday, fiber cut. Today, "Internet Protocol" problems. Tonight, capacity management?
On Aug 27, 2015, at 8:01 PM, Ryan Gelobter via Outages <outages@outages.org> wrote:
08/27/2015 11:16 PM GMT Multiple cards have been ordered, and will be installed on equipment in Chicago, IL in an effort to balance the traffic load through the site and alleviate the issue. Field Services are currently en route to the site with the cards, and are expected to arrive shortly.
On Thu, Aug 27, 2015 at 5:02 PM, Ryan Gelobter <ryan.g@atwgpc.net> wrote:
Latest update..... 08/27/2015 9:49 PM GMT Tier III Technical Support and Operations Engineering teams continue investigating the issue identified with the Internet Protocols. Restoration efforts are ongoing; however, there is currently no estimated restoral time.
On Thu, Aug 27, 2015 at 3:14 PM, Ryan Brooks <rbrooks@stack41.com> wrote: ICMP blazes through at times, but TCP completely falls apart.
On Aug 27, 2015, at 3:13 PM, Ryan Gelobter via Outages <outages@outages.org> wrote:
08/27/2015 8:05 PM GMT Level 3 Tier III and Operations Engineering teams have identified Internet Protocols dropping, affecting customer services. Restoration efforts are in progress, but an estimated time of restoral is not available at this time.
On Thu, Aug 27, 2015 at 3:06 PM, Ryan Gelobter <ryan.g@atwgpc.net> wrote: If you have access to the Level3 portal you should see ticket #9639047 under Network Events now.
Event Summary IP Network Event ~ Multiple Markets
08/27/2015 6:36 PM GMT IP and Transport Tier III, Operations Engineering and Field Services continue collaboratively working the issue.
08/27/2015 4:59 PM GMT Operations Engineering is engaged and Field Services is on site in Chicago, IL investigating the issue.
08/27/2015 4:38 PM GMT The engineers are currently migrating traffic in efforts of restoring services while troubleshooting continues. Field Services is being dispatched to a Chicago, IL site to assist.
08/27/2015 4:21 PM GMT IP services are affected across multiple markets and the root cause is currently under investigation. The IP NOC and IP and Transport Tier III are actively troubleshooting and working to isolate the cause. The engineers have detected peering issues which are resulting in packet loss for customers. Please be advised that updates will be provided at minimum of hourly unless otherwise noted.
On Thu, Aug 27, 2015 at 2:59 PM, Eric Henson via Outages <outages@outages.org> wrote: We had to shut down our Level3 circuit in the DFW area, had issues from 1406CST until we shut down our circuit.
-----Original Message----- From: Outages [mailto:outages-bounces@outages.org] On Behalf Of jungle Boogie via Outages Sent: Thursday, August 27, 2015 2:56 PM To: Bobin Joseph Cc: outages@outages.org Subject: Re: [outages] Level3/TW Telecom issues?
On 27 August 2015 at 12:48, Bobin Joseph via Outages <outages@outages.org> wrote: > Anybody else having complete outage on Level3? We lost complete > internet in DFW and SAN >
We were not down but significant routing issues in Las Vegas.
> Thanks, > Bobin > _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Tonight..we did nothing and waited for everyone to go to bed. 08/28/2015 3:08 AM GMT Event Conclusion Summary Start: August 27, 2015 13:20 GMT Stop: August 28, 2015 00:00 GMT Root Cause: A protocol issue impacted IP services in multiple markets. Fix Action: Adjustments were made to clear the errors. Summary: The IP NOC began investigating the root cause with Tier III Technical Support. It was reported that the issue was causing packet loss for customers. Operations Engineering teams were engaged, and Field Services were dispatched to a site in Chicago, IL to assist with investigations. Troubleshooting identified a protocol issue, and Operations Engineering worked with Tier III Technical Support to perform adjustments on the links. It was confirmed that the errors cleared. The traffic load was also lowered on cards in Chicago to alleviate any further issues. Should any additional impact be experienced, please contact the Level 3 Technical Service Center. On Aug 27, 2015 8:07 PM, "Ryan Brooks via Outages" <outages@outages.org> wrote:
Yesterday, fiber cut. Today, "Internet Protocol" problems. Tonight, capacity management?
On Aug 27, 2015, at 8:01 PM, Ryan Gelobter via Outages < outages@outages.org> wrote:
*08/27/2015 11:16 PM GMT*
Multiple cards have been ordered, and will be installed on equipment in Chicago, IL in an effort to balance the traffic load through the site and alleviate the issue. Field Services are currently en route to the site with the cards, and are expected to arrive shortly.
On Thu, Aug 27, 2015 at 5:02 PM, Ryan Gelobter <ryan.g@atwgpc.net> wrote:
Latest update..... *08/27/2015 9:49 PM GMT*
Tier III Technical Support and Operations Engineering teams continue investigating the issue identified with the Internet Protocols. Restoration efforts are ongoing; however, there is currently no estimated restoral time.
On Thu, Aug 27, 2015 at 3:14 PM, Ryan Brooks <rbrooks@stack41.com> wrote:
ICMP blazes through at times, but TCP completely falls apart.
On Aug 27, 2015, at 3:13 PM, Ryan Gelobter via Outages < outages@outages.org> wrote:
*08/27/2015 8:05 PM GMT*Level 3 Tier III and Operations Engineering teams have identified Internet Protocols dropping, affecting customer services. Restoration efforts are in progress, but an estimated time of restoral is not available at this time.
On Thu, Aug 27, 2015 at 3:06 PM, Ryan Gelobter <ryan.g@atwgpc.net> wrote:
If you have access to the Level3 portal you should see ticket #9639047 under Network Events now.
*Event SummaryIP Network Event ~ Multiple Markets08/27/2015 6:36 PM GMT*
IP and Transport Tier III, Operations Engineering and Field Services continue collaboratively working the issue.
*08/27/2015 4:59 PM GMT*
Operations Engineering is engaged and Field Services is on site in Chicago, IL investigating the issue.
*08/27/2015 4:38 PM GMT*
The engineers are currently migrating traffic in efforts of restoring services while troubleshooting continues. Field Services is being dispatched to a Chicago, IL site to assist.
*08/27/2015 4:21 PM GMT*
IP services are affected across multiple markets and the root cause is currently under investigation. The IP NOC and IP and Transport Tier III are actively troubleshooting and working to isolate the cause. The engineers have detected peering issues which are resulting in packet loss for customers. Please be advised that updates will be provided at minimum of hourly unless otherwise noted.
On Thu, Aug 27, 2015 at 2:59 PM, Eric Henson via Outages < outages@outages.org> wrote:
We had to shut down our Level3 circuit in the DFW area, had issues from 1406CST until we shut down our circuit.
-----Original Message----- From: Outages [mailto:outages-bounces@outages.org] On Behalf Of jungle Boogie via Outages Sent: Thursday, August 27, 2015 2:56 PM To: Bobin Joseph Cc: outages@outages.org Subject: Re: [outages] Level3/TW Telecom issues?
On 27 August 2015 at 12:48, Bobin Joseph via Outages < outages@outages.org> wrote:
Anybody else having complete outage on Level3? We lost complete internet in DFW and SAN
We were not down but significant routing issues in Las Vegas.
Thanks, Bobin
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
All my tickets got closed w/o confirmation. Good thing they turned down the tcp-disruption settings / "perform adjustments on the links". It is also very disconcerting that ICMP and IPSec seemed to do better during this capacity problem; perception is important w/ traffic engineering, but not so much with NOC mad-libs. Simultaneous to the below update, another notice indicated that OSP completed splice work which fixed the problem. So we're double-fixed now.
On Aug 28, 2015, at 1:13 AM, Ryan Gelobter <ryan.g@atwgpc.net> wrote:
Tonight..we did nothing and waited for everyone to go to bed.
08/28/2015 3:08 AM GMT Event Conclusion Summary
Start: August 27, 2015 13:20 GMT Stop: August 28, 2015 00:00 GMT
Root Cause: A protocol issue impacted IP services in multiple markets. Fix Action: Adjustments were made to clear the errors.
Summary: The IP NOC began investigating the root cause with Tier III Technical Support. It was reported that the issue was causing packet loss for customers. Operations Engineering teams were engaged, and Field Services were dispatched to a site in Chicago, IL to assist with investigations. Troubleshooting identified a protocol issue, and Operations Engineering worked with Tier III Technical Support to perform adjustments on the links. It was confirmed that the errors cleared. The traffic load was also lowered on cards in Chicago to alleviate any further issues. Should any additional impact be experienced, please contact the Level 3 Technical Service Center. On Aug 27, 2015 8:07 PM, "Ryan Brooks via Outages" <outages@outages.org <mailto:outages@outages.org>> wrote: Yesterday, fiber cut. Today, "Internet Protocol" problems. Tonight, capacity management?
On Aug 27, 2015, at 8:01 PM, Ryan Gelobter via Outages <outages@outages.org <mailto:outages@outages.org>> wrote:
08/27/2015 11:16 PM GMT Multiple cards have been ordered, and will be installed on equipment in Chicago, IL in an effort to balance the traffic load through the site and alleviate the issue. Field Services are currently en route to the site with the cards, and are expected to arrive shortly.
On Thu, Aug 27, 2015 at 5:02 PM, Ryan Gelobter <ryan.g@atwgpc.net <mailto:ryan.g@atwgpc.net>> wrote: Latest update..... 08/27/2015 9:49 PM GMT Tier III Technical Support and Operations Engineering teams continue investigating the issue identified with the Internet Protocols. Restoration efforts are ongoing; however, there is currently no estimated restoral time.
On Thu, Aug 27, 2015 at 3:14 PM, Ryan Brooks <rbrooks@stack41.com <mailto:rbrooks@stack41.com>> wrote: ICMP blazes through at times, but TCP completely falls apart.
On Aug 27, 2015, at 3:13 PM, Ryan Gelobter via Outages <outages@outages.org <mailto:outages@outages.org>> wrote:
08/27/2015 8:05 PM GMT Level 3 Tier III and Operations Engineering teams have identified Internet Protocols dropping, affecting customer services. Restoration efforts are in progress, but an estimated time of restoral is not available at this time.
On Thu, Aug 27, 2015 at 3:06 PM, Ryan Gelobter <ryan.g@atwgpc.net <mailto:ryan.g@atwgpc.net>> wrote: If you have access to the Level3 portal you should see ticket #9639047 under Network Events now.
Event Summary IP Network Event ~ Multiple Markets
08/27/2015 6:36 PM GMT IP and Transport Tier III, Operations Engineering and Field Services continue collaboratively working the issue.
08/27/2015 4:59 PM GMT Operations Engineering is engaged and Field Services is on site in Chicago, IL investigating the issue.
08/27/2015 4:38 PM GMT The engineers are currently migrating traffic in efforts of restoring services while troubleshooting continues. Field Services is being dispatched to a Chicago, IL site to assist.
08/27/2015 4:21 PM GMT IP services are affected across multiple markets and the root cause is currently under investigation. The IP NOC and IP and Transport Tier III are actively troubleshooting and working to isolate the cause. The engineers have detected peering issues which are resulting in packet loss for customers. Please be advised that updates will be provided at minimum of hourly unless otherwise noted.
On Thu, Aug 27, 2015 at 2:59 PM, Eric Henson via Outages <outages@outages.org <mailto:outages@outages.org>> wrote: We had to shut down our Level3 circuit in the DFW area, had issues from 1406CST until we shut down our circuit.
-----Original Message----- From: Outages [mailto:outages-bounces@outages.org <mailto:outages-bounces@outages.org>] On Behalf Of jungle Boogie via Outages Sent: Thursday, August 27, 2015 2:56 PM To: Bobin Joseph Cc: outages@outages.org <mailto:outages@outages.org> Subject: Re: [outages] Level3/TW Telecom issues?
On 27 August 2015 at 12:48, Bobin Joseph via Outages <outages@outages.org <mailto:outages@outages.org>> wrote:
Anybody else having complete outage on Level3? We lost complete internet in DFW and SAN
We were not down but significant routing issues in Las Vegas.
Thanks, Bobin
_______________________________________________ Outages mailing list Outages@outages.org <mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages <https://puck.nether.net/mailman/listinfo/outages> _______________________________________________ Outages mailing list Outages@outages.org <mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages <https://puck.nether.net/mailman/listinfo/outages>
_______________________________________________ Outages mailing list Outages@outages.org <mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages <https://puck.nether.net/mailman/listinfo/outages>
_______________________________________________ Outages mailing list Outages@outages.org <mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages <https://puck.nether.net/mailman/listinfo/outages>
_______________________________________________ Outages mailing list Outages@outages.org <mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages <https://puck.nether.net/mailman/listinfo/outages>
August 27th, 2015: the day everyone routed around L3 On Fri, Aug 28, 2015 at 1:27 AM, Ryan Brooks via Outages < outages@outages.org> wrote:
All my tickets got closed w/o confirmation. Good thing they turned down the tcp-disruption settings / "perform adjustments on the links". It is also very disconcerting that ICMP and IPSec seemed to do better during this capacity problem; perception is important w/ traffic engineering, but not so much with NOC mad-libs.
Simultaneous to the below update, another notice indicated that OSP completed splice work which fixed the problem. So we're double-fixed now.
On Aug 28, 2015, at 1:13 AM, Ryan Gelobter <ryan.g@atwgpc.net> wrote:
Tonight..we did nothing and waited for everyone to go to bed.
08/28/2015 3:08 AM GMT Event Conclusion Summary
Start: August 27, 2015 13:20 GMT Stop: August 28, 2015 00:00 GMT
Root Cause: A protocol issue impacted IP services in multiple markets. Fix Action: Adjustments were made to clear the errors.
Summary: The IP NOC began investigating the root cause with Tier III Technical Support. It was reported that the issue was causing packet loss for customers. Operations Engineering teams were engaged, and Field Services were dispatched to a site in Chicago, IL to assist with investigations. Troubleshooting identified a protocol issue, and Operations Engineering worked with Tier III Technical Support to perform adjustments on the links. It was confirmed that the errors cleared. The traffic load was also lowered on cards in Chicago to alleviate any further issues. Should any additional impact be experienced, please contact the Level 3 Technical Service Center. On Aug 27, 2015 8:07 PM, "Ryan Brooks via Outages" <outages@outages.org> wrote:
Yesterday, fiber cut. Today, "Internet Protocol" problems. Tonight, capacity management?
On Aug 27, 2015, at 8:01 PM, Ryan Gelobter via Outages < outages@outages.org> wrote:
*08/27/2015 11:16 PM GMT*
Multiple cards have been ordered, and will be installed on equipment in Chicago, IL in an effort to balance the traffic load through the site and alleviate the issue. Field Services are currently en route to the site with the cards, and are expected to arrive shortly.
On Thu, Aug 27, 2015 at 5:02 PM, Ryan Gelobter <ryan.g@atwgpc.net> wrote:
Latest update..... *08/27/2015 9:49 PM GMT*
Tier III Technical Support and Operations Engineering teams continue investigating the issue identified with the Internet Protocols. Restoration efforts are ongoing; however, there is currently no estimated restoral time.
On Thu, Aug 27, 2015 at 3:14 PM, Ryan Brooks <rbrooks@stack41.com> wrote:
ICMP blazes through at times, but TCP completely falls apart.
On Aug 27, 2015, at 3:13 PM, Ryan Gelobter via Outages < outages@outages.org> wrote:
*08/27/2015 8:05 PM GMT*Level 3 Tier III and Operations Engineering teams have identified Internet Protocols dropping, affecting customer services. Restoration efforts are in progress, but an estimated time of restoral is not available at this time.
On Thu, Aug 27, 2015 at 3:06 PM, Ryan Gelobter <ryan.g@atwgpc.net> wrote:
If you have access to the Level3 portal you should see ticket #9639047 under Network Events now.
*Event SummaryIP Network Event ~ Multiple Markets08/27/2015 6:36 PM GMT*
IP and Transport Tier III, Operations Engineering and Field Services continue collaboratively working the issue.
*08/27/2015 4:59 PM GMT*
Operations Engineering is engaged and Field Services is on site in Chicago, IL investigating the issue.
*08/27/2015 4:38 PM GMT*
The engineers are currently migrating traffic in efforts of restoring services while troubleshooting continues. Field Services is being dispatched to a Chicago, IL site to assist.
*08/27/2015 4:21 PM GMT*
IP services are affected across multiple markets and the root cause is currently under investigation. The IP NOC and IP and Transport Tier III are actively troubleshooting and working to isolate the cause. The engineers have detected peering issues which are resulting in packet loss for customers. Please be advised that updates will be provided at minimum of hourly unless otherwise noted.
On Thu, Aug 27, 2015 at 2:59 PM, Eric Henson via Outages < outages@outages.org> wrote:
We had to shut down our Level3 circuit in the DFW area, had issues from 1406CST until we shut down our circuit.
-----Original Message----- From: Outages [mailto:outages-bounces@outages.org] On Behalf Of jungle Boogie via Outages Sent: Thursday, August 27, 2015 2:56 PM To: Bobin Joseph Cc: outages@outages.org Subject: Re: [outages] Level3/TW Telecom issues?
On 27 August 2015 at 12:48, Bobin Joseph via Outages < outages@outages.org> wrote: > Anybody else having complete outage on Level3? We lost complete > internet in DFW and SAN >
We were not down but significant routing issues in Las Vegas.
> Thanks, > Bobin > _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
So that is how they lowered the traffic on the cards in Chicago. Thanks, Bobin From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Rafael Possamai via Outages Sent: Friday, August 28, 2015 6:26 AM To: Ryan Brooks Cc: outages@outages.org Subject: Re: [outages] Level3/TW Telecom issues? August 27th, 2015: the day everyone routed around L3 On Fri, Aug 28, 2015 at 1:27 AM, Ryan Brooks via Outages <outages@outages.org<mailto:outages@outages.org>> wrote: All my tickets got closed w/o confirmation. Good thing they turned down the tcp-disruption settings / "perform adjustments on the links". It is also very disconcerting that ICMP and IPSec seemed to do better during this capacity problem; perception is important w/ traffic engineering, but not so much with NOC mad-libs. Simultaneous to the below update, another notice indicated that OSP completed splice work which fixed the problem. So we're double-fixed now. On Aug 28, 2015, at 1:13 AM, Ryan Gelobter <ryan.g@atwgpc.net<mailto:ryan.g@atwgpc.net>> wrote: Tonight..we did nothing and waited for everyone to go to bed. 08/28/2015 3:08 AM GMT Event Conclusion Summary Start: August 27, 2015 13:20 GMT Stop: August 28, 2015 00:00 GMT Root Cause: A protocol issue impacted IP services in multiple markets. Fix Action: Adjustments were made to clear the errors. Summary: The IP NOC began investigating the root cause with Tier III Technical Support. It was reported that the issue was causing packet loss for customers. Operations Engineering teams were engaged, and Field Services were dispatched to a site in Chicago, IL to assist with investigations. Troubleshooting identified a protocol issue, and Operations Engineering worked with Tier III Technical Support to perform adjustments on the links. It was confirmed that the errors cleared. The traffic load was also lowered on cards in Chicago to alleviate any further issues. Should any additional impact be experienced, please contact the Level 3 Technical Service Center. On Aug 27, 2015 8:07 PM, "Ryan Brooks via Outages" <outages@outages.org<mailto:outages@outages.org>> wrote: Yesterday, fiber cut. Today, "Internet Protocol" problems. Tonight, capacity management? On Aug 27, 2015, at 8:01 PM, Ryan Gelobter via Outages <outages@outages.org<mailto:outages@outages.org>> wrote: 08/27/2015 11:16 PM GMT Multiple cards have been ordered, and will be installed on equipment in Chicago, IL in an effort to balance the traffic load through the site and alleviate the issue. Field Services are currently en route to the site with the cards, and are expected to arrive shortly. On Thu, Aug 27, 2015 at 5:02 PM, Ryan Gelobter <ryan.g@atwgpc.net<mailto:ryan.g@atwgpc.net>> wrote: Latest update..... 08/27/2015 9:49 PM GMT Tier III Technical Support and Operations Engineering teams continue investigating the issue identified with the Internet Protocols. Restoration efforts are ongoing; however, there is currently no estimated restoral time. On Thu, Aug 27, 2015 at 3:14 PM, Ryan Brooks <rbrooks@stack41.com<mailto:rbrooks@stack41.com>> wrote: ICMP blazes through at times, but TCP completely falls apart. On Aug 27, 2015, at 3:13 PM, Ryan Gelobter via Outages <outages@outages.org<mailto:outages@outages.org>> wrote: 08/27/2015 8:05 PM GMT Level 3 Tier III and Operations Engineering teams have identified Internet Protocols dropping, affecting customer services. Restoration efforts are in progress, but an estimated time of restoral is not available at this time. On Thu, Aug 27, 2015 at 3:06 PM, Ryan Gelobter <ryan.g@atwgpc.net<mailto:ryan.g@atwgpc.net>> wrote: If you have access to the Level3 portal you should see ticket #9639047 under Network Events now. Event Summary IP Network Event ~ Multiple Markets 08/27/2015 6:36 PM GMT IP and Transport Tier III, Operations Engineering and Field Services continue collaboratively working the issue. 08/27/2015 4:59 PM GMT Operations Engineering is engaged and Field Services is on site in Chicago, IL investigating the issue. 08/27/2015 4:38 PM GMT The engineers are currently migrating traffic in efforts of restoring services while troubleshooting continues. Field Services is being dispatched to a Chicago, IL site to assist. 08/27/2015 4:21 PM GMT IP services are affected across multiple markets and the root cause is currently under investigation. The IP NOC and IP and Transport Tier III are actively troubleshooting and working to isolate the cause. The engineers have detected peering issues which are resulting in packet loss for customers. Please be advised that updates will be provided at minimum of hourly unless otherwise noted. On Thu, Aug 27, 2015 at 2:59 PM, Eric Henson via Outages <outages@outages.org<mailto:outages@outages.org>> wrote: We had to shut down our Level3 circuit in the DFW area, had issues from 1406CST until we shut down our circuit. -----Original Message----- From: Outages [mailto:outages-bounces@outages.org<mailto:outages-bounces@outages.org>] On Behalf Of jungle Boogie via Outages Sent: Thursday, August 27, 2015 2:56 PM To: Bobin Joseph Cc: outages@outages.org<mailto:outages@outages.org> Subject: Re: [outages] Level3/TW Telecom issues? On 27 August 2015 at 12:48, Bobin Joseph via Outages <outages@outages.org<mailto:outages@outages.org>> wrote:
Anybody else having complete outage on Level3? We lost complete internet in DFW and SAN
We were not down but significant routing issues in Las Vegas.
Thanks, Bobin
_______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages _______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages _______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages _______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages _______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages _______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages
Not trying to beat a dead horse, but any ongoing issues with Level 3 ? Screenshot http://i.imgur.com/VuO9nrh.png http://www1.internetpulse.net/ shows packet loss Thanks, Bobin From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Bobin Joseph via Outages Sent: Friday, August 28, 2015 6:57 AM To: outages@outages.org Subject: Re: [outages] Level3/TW Telecom issues? So that is how they lowered the traffic on the cards in Chicago. Thanks, Bobin From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Rafael Possamai via Outages Sent: Friday, August 28, 2015 6:26 AM To: Ryan Brooks Cc: outages@outages.org<mailto:outages@outages.org> Subject: Re: [outages] Level3/TW Telecom issues? August 27th, 2015: the day everyone routed around L3 On Fri, Aug 28, 2015 at 1:27 AM, Ryan Brooks via Outages <outages@outages.org<mailto:outages@outages.org>> wrote: All my tickets got closed w/o confirmation. Good thing they turned down the tcp-disruption settings / "perform adjustments on the links". It is also very disconcerting that ICMP and IPSec seemed to do better during this capacity problem; perception is important w/ traffic engineering, but not so much with NOC mad-libs. Simultaneous to the below update, another notice indicated that OSP completed splice work which fixed the problem. So we're double-fixed now. On Aug 28, 2015, at 1:13 AM, Ryan Gelobter <ryan.g@atwgpc.net<mailto:ryan.g@atwgpc.net>> wrote: Tonight..we did nothing and waited for everyone to go to bed. 08/28/2015 3:08 AM GMT Event Conclusion Summary Start: August 27, 2015 13:20 GMT Stop: August 28, 2015 00:00 GMT Root Cause: A protocol issue impacted IP services in multiple markets. Fix Action: Adjustments were made to clear the errors. Summary: The IP NOC began investigating the root cause with Tier III Technical Support. It was reported that the issue was causing packet loss for customers. Operations Engineering teams were engaged, and Field Services were dispatched to a site in Chicago, IL to assist with investigations. Troubleshooting identified a protocol issue, and Operations Engineering worked with Tier III Technical Support to perform adjustments on the links. It was confirmed that the errors cleared. The traffic load was also lowered on cards in Chicago to alleviate any further issues. Should any additional impact be experienced, please contact the Level 3 Technical Service Center. On Aug 27, 2015 8:07 PM, "Ryan Brooks via Outages" <outages@outages.org<mailto:outages@outages.org>> wrote: Yesterday, fiber cut. Today, "Internet Protocol" problems. Tonight, capacity management? On Aug 27, 2015, at 8:01 PM, Ryan Gelobter via Outages <outages@outages.org<mailto:outages@outages.org>> wrote: 08/27/2015 11:16 PM GMT Multiple cards have been ordered, and will be installed on equipment in Chicago, IL in an effort to balance the traffic load through the site and alleviate the issue. Field Services are currently en route to the site with the cards, and are expected to arrive shortly. On Thu, Aug 27, 2015 at 5:02 PM, Ryan Gelobter <ryan.g@atwgpc.net<mailto:ryan.g@atwgpc.net>> wrote: Latest update..... 08/27/2015 9:49 PM GMT Tier III Technical Support and Operations Engineering teams continue investigating the issue identified with the Internet Protocols. Restoration efforts are ongoing; however, there is currently no estimated restoral time. On Thu, Aug 27, 2015 at 3:14 PM, Ryan Brooks <rbrooks@stack41.com<mailto:rbrooks@stack41.com>> wrote: ICMP blazes through at times, but TCP completely falls apart. On Aug 27, 2015, at 3:13 PM, Ryan Gelobter via Outages <outages@outages.org<mailto:outages@outages.org>> wrote: 08/27/2015 8:05 PM GMT Level 3 Tier III and Operations Engineering teams have identified Internet Protocols dropping, affecting customer services. Restoration efforts are in progress, but an estimated time of restoral is not available at this time. On Thu, Aug 27, 2015 at 3:06 PM, Ryan Gelobter <ryan.g@atwgpc.net<mailto:ryan.g@atwgpc.net>> wrote: If you have access to the Level3 portal you should see ticket #9639047 under Network Events now. Event Summary IP Network Event ~ Multiple Markets 08/27/2015 6:36 PM GMT IP and Transport Tier III, Operations Engineering and Field Services continue collaboratively working the issue. 08/27/2015 4:59 PM GMT Operations Engineering is engaged and Field Services is on site in Chicago, IL investigating the issue. 08/27/2015 4:38 PM GMT The engineers are currently migrating traffic in efforts of restoring services while troubleshooting continues. Field Services is being dispatched to a Chicago, IL site to assist. 08/27/2015 4:21 PM GMT IP services are affected across multiple markets and the root cause is currently under investigation. The IP NOC and IP and Transport Tier III are actively troubleshooting and working to isolate the cause. The engineers have detected peering issues which are resulting in packet loss for customers. Please be advised that updates will be provided at minimum of hourly unless otherwise noted. On Thu, Aug 27, 2015 at 2:59 PM, Eric Henson via Outages <outages@outages.org<mailto:outages@outages.org>> wrote: We had to shut down our Level3 circuit in the DFW area, had issues from 1406CST until we shut down our circuit. -----Original Message----- From: Outages [mailto:outages-bounces@outages.org<mailto:outages-bounces@outages.org>] On Behalf Of jungle Boogie via Outages Sent: Thursday, August 27, 2015 2:56 PM To: Bobin Joseph Cc: outages@outages.org<mailto:outages@outages.org> Subject: Re: [outages] Level3/TW Telecom issues? On 27 August 2015 at 12:48, Bobin Joseph via Outages <outages@outages.org<mailto:outages@outages.org>> wrote:
Anybody else having complete outage on Level3? We lost complete internet in DFW and SAN
We were not down but significant routing issues in Las Vegas.
Thanks, Bobin
_______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages _______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages _______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages _______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages _______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages _______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages
We switched back to Level 3 about 2 hours ago and haven’t had any issues. They closed our ticket saying the issue was resolved. From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Bobin Joseph via Outages Sent: Friday, August 28, 2015 11:16 AM To: 'outages@outages.org' Subject: Re: [outages] Level3/TW Telecom issues? Not trying to beat a dead horse, but any ongoing issues with Level 3 ? Screenshot http://i.imgur.com/VuO9nrh.png http://www1.internetpulse.net/ shows packet loss Thanks, Bobin From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Bobin Joseph via Outages Sent: Friday, August 28, 2015 6:57 AM To: outages@outages.org<mailto:outages@outages.org> Subject: Re: [outages] Level3/TW Telecom issues? So that is how they lowered the traffic on the cards in Chicago. Thanks, Bobin From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Rafael Possamai via Outages Sent: Friday, August 28, 2015 6:26 AM To: Ryan Brooks Cc: outages@outages.org<mailto:outages@outages.org> Subject: Re: [outages] Level3/TW Telecom issues? August 27th, 2015: the day everyone routed around L3 On Fri, Aug 28, 2015 at 1:27 AM, Ryan Brooks via Outages <outages@outages.org<mailto:outages@outages.org>> wrote: All my tickets got closed w/o confirmation. Good thing they turned down the tcp-disruption settings / "perform adjustments on the links". It is also very disconcerting that ICMP and IPSec seemed to do better during this capacity problem; perception is important w/ traffic engineering, but not so much with NOC mad-libs. Simultaneous to the below update, another notice indicated that OSP completed splice work which fixed the problem. So we're double-fixed now. On Aug 28, 2015, at 1:13 AM, Ryan Gelobter <ryan.g@atwgpc.net<mailto:ryan.g@atwgpc.net>> wrote: Tonight..we did nothing and waited for everyone to go to bed. 08/28/2015 3:08 AM GMT Event Conclusion Summary Start: August 27, 2015 13:20 GMT Stop: August 28, 2015 00:00 GMT Root Cause: A protocol issue impacted IP services in multiple markets. Fix Action: Adjustments were made to clear the errors. Summary: The IP NOC began investigating the root cause with Tier III Technical Support. It was reported that the issue was causing packet loss for customers. Operations Engineering teams were engaged, and Field Services were dispatched to a site in Chicago, IL to assist with investigations. Troubleshooting identified a protocol issue, and Operations Engineering worked with Tier III Technical Support to perform adjustments on the links. It was confirmed that the errors cleared. The traffic load was also lowered on cards in Chicago to alleviate any further issues. Should any additional impact be experienced, please contact the Level 3 Technical Service Center. On Aug 27, 2015 8:07 PM, "Ryan Brooks via Outages" <outages@outages.org<mailto:outages@outages.org>> wrote: Yesterday, fiber cut. Today, "Internet Protocol" problems. Tonight, capacity management? On Aug 27, 2015, at 8:01 PM, Ryan Gelobter via Outages <outages@outages.org<mailto:outages@outages.org>> wrote: 08/27/2015 11:16 PM GMT Multiple cards have been ordered, and will be installed on equipment in Chicago, IL in an effort to balance the traffic load through the site and alleviate the issue. Field Services are currently en route to the site with the cards, and are expected to arrive shortly. On Thu, Aug 27, 2015 at 5:02 PM, Ryan Gelobter <ryan.g@atwgpc.net<mailto:ryan.g@atwgpc.net>> wrote: Latest update..... 08/27/2015 9:49 PM GMT Tier III Technical Support and Operations Engineering teams continue investigating the issue identified with the Internet Protocols. Restoration efforts are ongoing; however, there is currently no estimated restoral time. On Thu, Aug 27, 2015 at 3:14 PM, Ryan Brooks <rbrooks@stack41.com<mailto:rbrooks@stack41.com>> wrote: ICMP blazes through at times, but TCP completely falls apart. On Aug 27, 2015, at 3:13 PM, Ryan Gelobter via Outages <outages@outages.org<mailto:outages@outages.org>> wrote: 08/27/2015 8:05 PM GMT Level 3 Tier III and Operations Engineering teams have identified Internet Protocols dropping, affecting customer services. Restoration efforts are in progress, but an estimated time of restoral is not available at this time. On Thu, Aug 27, 2015 at 3:06 PM, Ryan Gelobter <ryan.g@atwgpc.net<mailto:ryan.g@atwgpc.net>> wrote: If you have access to the Level3 portal you should see ticket #9639047 under Network Events now. Event Summary IP Network Event ~ Multiple Markets 08/27/2015 6:36 PM GMT IP and Transport Tier III, Operations Engineering and Field Services continue collaboratively working the issue. 08/27/2015 4:59 PM GMT Operations Engineering is engaged and Field Services is on site in Chicago, IL investigating the issue. 08/27/2015 4:38 PM GMT The engineers are currently migrating traffic in efforts of restoring services while troubleshooting continues. Field Services is being dispatched to a Chicago, IL site to assist. 08/27/2015 4:21 PM GMT IP services are affected across multiple markets and the root cause is currently under investigation. The IP NOC and IP and Transport Tier III are actively troubleshooting and working to isolate the cause. The engineers have detected peering issues which are resulting in packet loss for customers. Please be advised that updates will be provided at minimum of hourly unless otherwise noted. On Thu, Aug 27, 2015 at 2:59 PM, Eric Henson via Outages <outages@outages.org<mailto:outages@outages.org>> wrote: We had to shut down our Level3 circuit in the DFW area, had issues from 1406CST until we shut down our circuit. -----Original Message----- From: Outages [mailto:outages-bounces@outages.org<mailto:outages-bounces@outages.org>] On Behalf Of jungle Boogie via Outages Sent: Thursday, August 27, 2015 2:56 PM To: Bobin Joseph Cc: outages@outages.org<mailto:outages@outages.org> Subject: Re: [outages] Level3/TW Telecom issues? On 27 August 2015 at 12:48, Bobin Joseph via Outages <outages@outages.org<mailto:outages@outages.org>> wrote:
Anybody else having complete outage on Level3? We lost complete internet in DFW and SAN
We were not down but significant routing issues in Las Vegas.
Thanks, Bobin
_______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages _______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages _______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages _______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages _______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages _______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages
Oddly enough, we were seeing IPsec unaffected too.
-----Original Message----- From: Outages [mailto:outages-bounces@outages.org] On Behalf Of Ryan Brooks via Outages Sent: Thursday, August 27, 2015 4:15 PM To: Ryan Gelobter Cc: outages@outages.org Subject: Re: [outages] Level3/TW Telecom issues?
ICMP blazes through at times, but TCP completely falls apart.
On Aug 27, 2015, at 3:13 PM, Ryan Gelobter via Outages <outages@outages.org> wrote:
08/27/2015 8:05 PM GMT Level 3 Tier III and Operations Engineering teams have identified Internet Protocols dropping, affecting customer services. Restoration efforts are in progress, but an estimated time of restoral is not available at this time.
On Thu, Aug 27, 2015 at 3:06 PM, Ryan Gelobter <ryan.g@atwgpc.net> wrote:
If you have access to the Level3 portal you should see ticket #9639047 under Network Events now.
Event Summary IP Network Event ~ Multiple Markets
08/27/2015 6:36 PM GMT IP and Transport Tier III, Operations Engineering and Field Services continue collaboratively working the issue.
08/27/2015 4:59 PM GMT Operations Engineering is engaged and Field Services is on site in Chicago, IL investigating the issue.
08/27/2015 4:38 PM GMT The engineers are currently migrating traffic in efforts of restoring services while troubleshooting continues. Field Services is being dispatched to a Chicago, IL site to assist.
08/27/2015 4:21 PM GMT IP services are affected across multiple markets and the root cause is currently under investigation. The IP NOC and IP and Transport Tier III are actively troubleshooting and working to isolate the cause. The engineers have detected peering issues which are resulting in packet loss for customers. Please be advised that updates will be provided at minimum of hourly unless otherwise noted.
On Thu, Aug 27, 2015 at 2:59 PM, Eric Henson via Outages <outages@outages.org> wrote:
We had to shut down our Level3 circuit in the DFW area, had issues from 1406CST until we shut down our circuit.
-----Original Message----- From: Outages [mailto:outages-bounces@outages.org] On Behalf Of jungle Boogie via Outages Sent: Thursday, August 27, 2015 2:56 PM To: Bobin Joseph Cc: outages@outages.org Subject: Re: [outages] Level3/TW Telecom issues?
On 27 August 2015 at 12:48, Bobin Joseph via Outages <outages@outages.org> wrote: > Anybody else having complete outage on Level3? We lost complete > internet in DFW and SAN >
We were not down but significant routing issues in Las Vegas.
> Thanks, > Bobin > _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
one of my clients with TW Telecom has been experiencing packet loss since 10Am ish located in NYC On Thu, Aug 27, 2015 at 3:55 PM, jungle Boogie via Outages < outages@outages.org> wrote:
On 27 August 2015 at 12:48, Bobin Joseph via Outages <outages@outages.org> wrote:
Anybody else having complete outage on Level3? We lost complete internet in DFW and SAN
We were not down but significant routing issues in Las Vegas.
Thanks, Bobin
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
-- Izzy Goldstein Main: (212) 477.1000 Fax: (212) 477.8900 igoldstein@telego.com Website <http://www.telego.com/> | LinkedIn <https://www.linkedin.com/company/telego> | Blog <http://www.telego.com/blog> | Confidentially Notice: This e-mail may contain confidential and/or privileged information. If you are not the intended recipient or have received this e-mail in error please notify us immediately by email reply and destroy this e-mail. Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden. Any views or opinions presented in this email are solely those of the author and do not necessarily represent those of TeleGo (T). Employees of TeleGo are expressly required not to make defamatory statements and not to infringe or authorize any infringement of copyright or any other legal right by email communications. Any such communication is contrary to TeleGo policy and outside the scope of the employment of the individual concerned. TeleGo will not accept any liability in respect of such communication, and the employee responsible will be personally liable for any damages or other liability arising. Think before you print.
I really don't understand the logic of whoever wrote that status update. How does that update has any value at all its so vague and lacks any information "uh yeah we can confirm many hours later that there was a issue in an area that is unknown or too important to tell you anything specific, we applied a change that fixed said unknown problem and we will continue to monitor it even though it took many hours for us to react the first time around" i know you cant always be too specific but its really disappointing to see this kind of communication become somewhat the norm to customers who are paying that extra premium for an SLA. Problems happen but customers are a lot more understanding when theres good communication. This doesn't tell the customer anything chris On Thu, Aug 27, 2015 at 12:47 PM, Bobin Joseph via Outages < outages@outages.org> wrote:
We got this from Level3
Hello,
The network outage has been resolved.
*Event Conclusion Summary*
Start: August 26, 2015 19:15 GMT Stop: August 27, 2015 00:14 GMT
Root Cause: An unknown issue in Chicago, IL was impacting IP services. Fix Action: Adjustments were made to resolve the issue and restore services.
Summary: The IP NOC responded to an unknown issue in Chicago, IL that was impacting IP services. Tier III Technical Support were engaged to assist with investigations, and adjustments were made to resolve the issue and stabilize traffic. Monitoring continues to ensure stability. Should further issues be experienced, please contact the Level 3 Technical Service Center.
Thanks,
Bobin
*From:* Outages [mailto:outages-bounces@outages.org] *On Behalf Of *Ryan Gelobter via Outages *Sent:* Thursday, August 27, 2015 9:44 AM *To:* Rafael Possamai *Cc:* outages@outages.org *Subject:* Re: [outages] Level3/TW Telecom issues?
Was anyone able to get some useful information out of Level3 regarding this yesterday? Also is anyone else seeing similar issues?
On Aug 26, 2015 2:40 PM, "Rafael Possamai via Outages" < outages@outages.org> wrote:
Here's a graph showing some packet loss: http://imgur.com/KVAzetA
On Wed, Aug 26, 2015 at 3:16 PM, Matt Elliott via Outages < outages@outages.org> wrote:
Anyone else having issues on Level3? We’re seeing packet loss with anything that traverses Chicago.
Thanks,
Matt
This message and attachment(s), if any, is intended for the sole use of the individual and/or entity for which it is addressed, and may contain information that is privileged, confidential and prohibited from disclosure under applicable law. If you are not the addressee, or authorized to receive this on behalf of the addressee, you are hereby notified that you may not use, copy, disclose or distribute to anyone this message or any part thereof. If you have received this in error, please immediately advise the sender by e-mail and delete this information and all attachments from your computer and network. Thank you.
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
For me, issues seemed to clear up around 1700 CDT yesterday, but once again I am getting packet loss to the level3 network.. Signature To ensure prompt service, please use "Reply All" when responding Robert Colon | TCSA Tier 3 | EthoStream LLC | P: (877) 282-2519 x2114 | F: (414) 258-8307 | 20800 Swenson Drive, Suite 175 Waukesha, WI 53186 | www.telkonet.com <http://www.telkonet.com/> | www.ethostream.com <http://www.ethostream.com/> | @Telkonet <https://twitter.com/telkonet> On 8/27/2015 11:44 AM, Ryan Gelobter via Outages wrote:
Was anyone able to get some useful information out of Level3 regarding this yesterday? Also is anyone else seeing similar issues?
On Aug 26, 2015 2:40 PM, "Rafael Possamai via Outages" <outages@outages.org <mailto:outages@outages.org>> wrote:
Here's a graph showing some packet loss: http://imgur.com/KVAzetA
On Wed, Aug 26, 2015 at 3:16 PM, Matt Elliott via Outages <outages@outages.org <mailto:outages@outages.org>> wrote:
Anyone else having issues on Level3? We’re seeing packet loss with anything that traverses Chicago.
Thanks, Matt This message and attachment(s), if any, is intended for the sole use of the individual and/or entity for which it is addressed, and may contain information that is privileged, confidential and prohibited from disclosure under applicable law. If you are not the addressee, or authorized to receive this on behalf of the addressee, you are hereby notified that you may not use, copy, disclose or distribute to anyone this message or any part thereof. If you have received this in error, please immediately advise the sender by e-mail and delete this information and all attachments from your computer and network. Thank you.
_______________________________________________ Outages mailing list Outages@outages.org <mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org <mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
-- ------------------------------ <http://www.telkonet.com> <http://www.ethostream.com> This e-mail and any files transmitted with it are private and confidential and are solely for the use of the addressee. It may contain material which is legally privileged. If you are not the addressee or the person responsible for delivering to the addressee, be advised that you have received this e-mail in error and that any use of it is strictly prohibited. If you have received this e-mail in error please contact us at (414) 223-0473.
participants (22)
-
Aaron Gross -
Bob Colon -
Bob Hancock -
Bobin Joseph -
chris -
Craig Gill -
David Hubbard -
Eric Henson -
Izzy Goldstein - TeleGo -
Jameel Akari -
Jason Hellenthal -
jungle Boogie -
Lucius Kalbfleisch -
Matt Elliott -
Matthew Topper -
Nick Olsen -
Rafael Possamai -
Ryan Brooks -
Ryan Gelobter -
Steven Fischer -
Tim Glen -
Tyler Applebaum