Starting at 1:36 PM UTC TATA Communications is having major issues in Los Angeles. We lost light on our ports and this appears to be affecting their trans-pacific traffic significantly as well. No update from them yet. Sincerely, Charles Henry Network Operations Team Total Uptime Technologies, LLC Support Portal: https://support.totaluptime.com +1 800-584-1514 - TEL (US & Canada) +44 330 808 0228 - TEL (United Kingdom) +1 888-584-1194 - FAX Privileged/Confidential Information may be contained in this message. If you are not the addressee indicated in this message (or responsible for delivery of the message to such person), you may not copy or deliver this message to anyone. In such case, you should destroy this message and kindly notify the sender by reply email. Please advise immediately if you or your employer does not consent to email or messages of this kind. Opinions, conclusions and other information in this message that do not relate to the official business of the firm or its subsidiaries or business units shall be understood as neither given nor endorsed by it.
I'm seeing traceroutes from Ashburn to Toronto, Montreal, and LAX taking a detour through Sydney Australia right now. On Sat, Oct 3, 2020 at 7:07 AM Total Uptime Netops via Outages < outages@outages.org> wrote:
Starting at 1:36 PM UTC TATA Communications is having major issues in Los Angeles. We lost light on our ports and this appears to be affecting their trans-pacific traffic significantly as well. No update from them yet.
Sincerely,
Charles Henry Network Operations Team Total Uptime Technologies, LLC Support Portal: https://support.totaluptime.com
+1 800-584-1514 - TEL (US & Canada) +44 330 808 0228 - TEL (United Kingdom) +1 888-584-1194 - FAX
Privileged/Confidential Information may be contained in this message. If you are not the addressee indicated in this message (or responsible for delivery of the message to such person), you may not copy or deliver this message to anyone. In such case, you should destroy this message and kindly notify the sender by reply email. Please advise immediately if you or your employer does not consent to email or messages of this kind. Opinions, conclusions and other information in this message that do not relate to the official business of the firm or its subsidiaries or business units shall be understood as neither given nor endorsed by it. _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Here is are some traceroutes. Looks like some packet loss is starting to happen. Ashburn > LAX [root@core0 ~]# mtr 209.58.46.11 --report HOST: core0.vocaltransit.net Loss% Snt Last Avg Best Wrst StDev 1. rtr.rtc.vocaltransit.net 0.0% 10 0.1 0.2 0.1 0.4 0.1 2. ge-102-0-0-0.r05.asbnva02.us 0.0% 10 1.2 1.3 1.2 1.6 0.1 3. ae-7.r25.asbnva02.us.bb.gin. 90.0% 10 2.9 2.9 2.9 2.9 0.0 4. ae-2.r25.lsanca07.us.bb.gin. 20.0% 10 66.5 66.1 64.9 67.7 0.9 5. ae-7.r22.lsanca07.us.bb.gin. 0.0% 10 65.1 65.1 65.0 65.3 0.1 6. ae-5.r20.sydnau02.au.bb.gin. 0.0% 10 235.3 230.4 226.4 250.6 7.6 7. ix-ae-12-0.thar1.1mh-sydney. 0.0% 10 217.5 218.0 217.4 220.2 1.1 8. if-ae-2-2.tcore1.0pp-sydney. 0.0% 10 251.9 252.0 251.7 252.4 0.2 9. if-ae-5-2.thar1.pv4-piti.as6 0.0% 10 245.5 247.9 245.5 255.6 3.9 10. if-ae-21-2.tcore2.tv2-tokyo. 0.0% 10 250.1 247.5 245.6 250.3 2.0 11. if-et-24-2.hcore2.kv8-chiba. 20.0% 10 305.5 305.7 305.3 306.5 0.5 12. if-ae-20-2.tcore2.lvw-losang 80.0% 10 245.5 246.1 245.5 246.6 0.7 13. if-ae-2-2.tcore1.lvw-losange 10.0% 10 245.8 249.2 245.4 257.2 4.5 14. if-ae-14-2.vpe2.laa-losangel 10.0% 10 250.4 250.4 245.2 263.4 5.9 15. 209.58.46.11 10.0% 10 245.3 247.3 245.3 258.5 4.3 Ashburn > Toronto [root@core0 ~]# mtr 209.58.46.109 --report HOST: core0.vocaltransit.net Loss% Snt Last Avg Best Wrst StDev 1. rtr.rtc.vocaltransit.net 0.0% 10 0.2 0.2 0.1 0.3 0.1 2. ge-102-0-0-0.r05.asbnva02.us 0.0% 10 1.3 1.3 1.1 1.7 0.2 3. ae-7.r25.asbnva02.us.bb.gin. 0.0% 10 1.9 1.7 1.2 2.9 0.5 4. ae-2.r25.lsanca07.us.bb.gin. 0.0% 10 65.1 66.1 65.0 69.0 1.5 5. ae-7.r22.lsanca07.us.bb.gin. 0.0% 10 65.1 66.0 64.9 71.3 1.9 6. ae-5.r20.sydnau02.au.bb.gin. 0.0% 10 240.4 227.9 226.3 240.4 4.4 7. ix-ae-12-0.thar1.1mh-sydney. 0.0% 10 217.6 217.8 217.4 220.7 1.0 8. if-ae-2-2.tcore1.0pp-sydney. 0.0% 10 249.9 251.0 249.8 255.9 2.2 9. if-ae-5-2.thar1.pv4-piti.as6 0.0% 10 248.4 249.1 248.3 251.2 1.1 10. if-ae-3-2.tcore1.tv2-tokyo.a 0.0% 10 248.4 248.6 248.3 249.2 0.4 11. if-et-21-2.hcore1.kv8-chiba. 10.0% 10 307.1 312.3 307.0 342.1 11.5 12. if-ae-5-2.tcore2.sv1-santacl 80.0% 10 250.8 250.7 250.5 250.8 0.2 13. if-ae-18-4.tcore1.sqn-sanjos 0.0% 10 250.5 251.0 250.5 252.7 0.6 14. if-ae-1-2.tcore2.sqn-sanjose 0.0% 10 250.0 250.1 250.0 250.3 0.1 15. if-ae-51-2.tcore2.ct8-chicag 0.0% 10 252.6 251.2 250.6 252.6 0.8 16. if-ae-22-2.tcore1.ct8-chicag 0.0% 10 250.2 252.3 250.0 265.0 4.6 17. if-ae-8-2.tcore2.tnk-toronto 10.0% 10 249.9 250.0 249.8 251.4 0.5 18. if-ae-2-2.tcore1.tnk-toronto 10.0% 10 250.7 250.8 250.6 251.0 0.1 19. if-ae-25-2.tcore1.ttt-toront 10.0% 10 250.7 250.6 250.2 251.4 0.4 20. if-ae-1-2.vpe1.ttt-toronto.a 10.0% 10 249.7 249.6 249.5 249.7 0.1 21. 209.58.46.109 10.0% 10 249.7 249.7 249.6 249.8 0.1 Ashburn > Montreal [root@core0 ~]# mtr 209.58.46.142 --report HOST: core0.vocaltransit.net Loss% Snt Last Avg Best Wrst StDev 1. rtr.rtc.vocaltransit.net 0.0% 10 0.3 0.3 0.1 0.5 0.1 2. ge-102-0-0-0.r05.asbnva02.us 0.0% 10 1.2 1.2 1.1 1.3 0.1 3. ae-7.r25.asbnva02.us.bb.gin. 0.0% 10 3.8 2.3 1.1 7.6 2.1 4. ae-2.r25.lsanca07.us.bb.gin. 70.0% 10 61.1 61.3 60.9 61.9 0.5 5. ae-7.r22.lsanca07.us.bb.gin. 0.0% 10 61.0 62.5 60.8 75.5 4.5 6. ae-5.r20.sydnau02.au.bb.gin. 0.0% 10 222.6 222.6 222.3 224.0 0.5 7. ix-ae-12-0.thar1.1mh-sydney. 0.0% 10 213.6 213.4 213.3 213.6 0.1 8. if-ae-2-2.tcore1.0pp-sydney. 0.0% 10 251.7 249.9 249.0 251.7 1.1 9. if-ae-5-2.thar1.pv4-piti.as6 0.0% 10 248.8 249.0 248.5 251.5 0.9 10. if-ae-3-2.tcore1.tv2-tokyo.a 0.0% 10 249.1 248.6 248.2 249.1 0.3 11. if-et-21-2.hcore1.kv8-chiba. 10.0% 10 303.1 303.7 302.9 306.8 1.2 12. if-ae-5-2.tcore2.sv1-santacl 60.0% 10 249.6 249.7 249.5 250.1 0.3 13. if-ae-18-2.tcore1.sqn-sanjos 0.0% 10 249.4 249.3 249.2 249.4 0.1 14. if-ae-1-2.tcore2.sqn-sanjose 0.0% 10 249.2 249.2 249.0 249.5 0.2 15. if-ae-51-2.tcore2.ct8-chicag 0.0% 10 249.5 251.4 249.1 267.4 5.6 16. if-ae-30-2.tcore1.w6c-montre 10.0% 10 249.6 249.3 249.0 249.9 0.3 17. if-ae-12-2.tcore1.mtt-montre 10.0% 10 249.0 249.4 248.7 250.9 0.8 18. if-ae-1-2.vpe1.mtt-montreal. 10.0% 10 248.9 248.9 248.7 249.3 0.2 19. 209.58.46.142 10.0% 10 248.9 248.9 248.7 249.1 0.1 On Sat, Oct 3, 2020 at 9:52 AM Jared Geiger <jared@compuwizz.net> wrote:
I'm seeing traceroutes from Ashburn to Toronto, Montreal, and LAX taking a detour through Sydney Australia right now.
On Sat, Oct 3, 2020 at 7:07 AM Total Uptime Netops via Outages < outages@outages.org> wrote:
Starting at 1:36 PM UTC TATA Communications is having major issues in Los Angeles. We lost light on our ports and this appears to be affecting their trans-pacific traffic significantly as well. No update from them yet.
Sincerely,
Charles Henry Network Operations Team Total Uptime Technologies, LLC Support Portal: https://support.totaluptime.com
+1 800-584-1514 - TEL (US & Canada) +44 330 808 0228 - TEL (United Kingdom) +1 888-584-1194 - FAX
Privileged/Confidential Information may be contained in this message. If you are not the addressee indicated in this message (or responsible for delivery of the message to such person), you may not copy or deliver this message to anyone. In such case, you should destroy this message and kindly notify the sender by reply email. Please advise immediately if you or your employer does not consent to email or messages of this kind. Opinions, conclusions and other information in this message that do not relate to the official business of the firm or its subsidiaries or business units shall be understood as neither given nor endorsed by it. _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
At least your packets are transiting down to Melbourne where they’d catch COVID-19 on ingress. 😂
On 4 Oct 2020, at 2:58 am, Jared Geiger via Outages <outages@outages.org> wrote:
I'm seeing traceroutes from Ashburn to Toronto, Montreal, and LAX taking a detour through Sydney Australia right now.
On Sat, Oct 3, 2020 at 7:07 AM Total Uptime Netops via Outages <outages@outages.org> wrote: Starting at 1:36 PM UTC TATA Communications is having major issues in Los Angeles. We lost light on our ports and this appears to be affecting their trans-pacific traffic significantly as well. No update from them yet.
Sincerely,
Charles Henry Network Operations Team Total Uptime Technologies, LLC Support Portal: https://support.totaluptime.com
+1 800-584-1514 - TEL (US & Canada) +44 330 808 0228 - TEL (United Kingdom) +1 888-584-1194 - FAX
Privileged/Confidential Information may be contained in this message. If you are not the addressee indicated in this message (or responsible for delivery of the message to such person), you may not copy or deliver this message to anyone. In such case, you should destroy this message and kindly notify the sender by reply email. Please advise immediately if you or your employer does not consent to email or messages of this kind. Opinions, conclusions and other information in this message that do not relate to the official business of the firm or its subsidiaries or business units shall be understood as neither given nor endorsed by it. _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
If anyone at TATA 6453 is watching, you're announcing 209.58.46.0/24 to NTT 2914 in Sydney causing traffic to your North American VOIP network to go through Australia. The rest of your announcements to your peers are for 209.58.0.0/17 which correctly keeps the traffic in North America. Filtering the /24 out of NTT's BGP feed fixed my latency and packet loss problem. NTT contacted the TATA NOC on my behalf but no one has fixed it yet. On Sat, Oct 3, 2020 at 3:05 PM Dave Browning via Outages < outages@outages.org> wrote:
At least your packets are transiting down to Melbourne where they’d catch COVID-19 on ingress. 😂
On 4 Oct 2020, at 2:58 am, Jared Geiger via Outages <outages@outages.org> wrote:
I'm seeing traceroutes from Ashburn to Toronto, Montreal, and LAX taking a detour through Sydney Australia right now.
On Sat, Oct 3, 2020 at 7:07 AM Total Uptime Netops via Outages < outages@outages.org> wrote:
Starting at 1:36 PM UTC TATA Communications is having major issues in Los Angeles. We lost light on our ports and this appears to be affecting their trans-pacific traffic significantly as well. No update from them yet.
Sincerely,
Charles Henry Network Operations Team Total Uptime Technologies, LLC Support Portal: https://support.totaluptime.com
+1 800-584-1514 - TEL (US & Canada) +44 330 808 0228 - TEL (United Kingdom) +1 888-584-1194 - FAX
Privileged/Confidential Information may be contained in this message. If you are not the addressee indicated in this message (or responsible for delivery of the message to such person), you may not copy or deliver this message to anyone. In such case, you should destroy this message and kindly notify the sender by reply email. Please advise immediately if you or your employer does not consent to email or messages of this kind. Opinions, conclusions and other information in this message that do not relate to the official business of the firm or its subsidiaries or business units shall be understood as neither given nor endorsed by it. _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
participants (3)
-
Dave Browning -
Jared Geiger -
Total Uptime Netops