Twelve99 - Major packet loss
Hi folks, Seeing major packet loss on 1299. Somewhere between Seattle and Chicago. mtr below between AWS usw2 and Hetzner Germany. My traceroute [v0.95] ** (**) -> hc-ping.com (176.9.71.146) 2026-07-02T09:06:23+0000 Keys: Help Display mode Restart statistics Order of fields quit Packets Pings Host Loss% Snt Last Avg Best Wrst StDev 1. ip-**.us-west-2.compute.internal 0.0% 81 0.2 0.2 0.2 0.3 0.0 2. 242.11.42.7 0.0% 81 8.2 9.4 8.0 20.0 2.6 242.11.42.5 242.11.43.5 242.11.43.135 242.11.42.129 242.11.43.3 242.11.42.131 242.11.42.3 3. amazon-ic-341989.ip.twelve99-cust.net 0.0% 81 7.6 7.6 7.5 7.9 0.1 4. sea-b1-link.ip.twelve99.net 0.0% 80 9.4 9.1 8.3 14.5 1.1 5. chi-bb2-link.ip.twelve99.net 67.1% 80 175.5 174.2 171.2 176.8 1.8 6. ewr-bb2-link.ip.twelve99.net 56.2% 80 112.3 112.3 110.2 113.4 0.7 7. ldn-bb2-link.ip.twelve99.net 54.4% 80 205.7 205.6 204.7 207.3 0.6 8. hbg-bb2-link.ip.twelve99.net 60.8% 80 204.2 204.0 202.6 205.5 0.7 9. hbg-b2-link.ip.twelve99.net 62.0% 80 191.9 192.6 190.4 203.8 2.7 10. hetzner-ic-383976.ip.twelve99-cust.net 55.7% 80 201.4 201.3 198.8 203.0 0.8 11. core21.fsn1.hetzner.com 38.8% 80 201.1 200.8 199.6 202.0 0.6 12. ex9k2.dc6.fsn1.hetzner.com 55.7% 80 201.4 201.4 199.7 206.8 1.1 13. static.146.71.9.176.clients.your-server.de 41.2% 80 201.7 201.3 199.2 202.6 0.7
a few others + us (AS400940) have been seeing the same since 06:04 UTC for many arelion paths - their noc is not acknowledging the incident over email comms. very frustrating My traceroute [v0.95] production-hikari-dpjfk-1 (152.233.47.66) -> 162.220.232.255 (162.22026-07-02T06:13:31+0000 Keys: Help Display mode Restart statistics Order of fields quit Packets Pings Host Loss% Snt Last Avg Best Wrst StDev 1. unn-152-233-47-76.datapacket.com 38.9% 285 0.1 0.1 0.1 0.3 0.0 2. vl223.nyc-drt12-core-1.cdn77.com 0.0% 285 0.9 0.9 0.8 1.5 0.1 3. ewr-b6-link.ip.twelve99.net 0.0% 285 3.4 3.8 2.1 12.2 1.0 4. nyk-bb5-link.ip.twelve99.net 62.1% 285 1.5 1.6 1.5 4.1 0.3 5. rest-bb1-link.ip.twelve99.net 87.0% 285 8.2 8.3 8.1 11.1 0.5 6. atl-bb1-link.ip.twelve99.net 64.1% 284 23.8 23.9 23.8 25.5 0.2 7. atl-b24-link.ip.twelve99.net 96.5% 284 24.0 24.1 23.9 24.3 0.1 8. atl-bb2-link.ip.twelve99.net 57.0% 284 24.0 23.9 23.9 24.3 0.1 9. hou-bb4-link.ip.twelve99.net 84.5% 284 41.7 42.6 41.0 44.8 0.8 10. dls-b23-link.ip.twelve99.net 59.5% 284 52.2 52.2 52.0 59.0 0.6 11. lax-bb2-link.ip.twelve99.net 89.8% 284 99.8 99.2 98.1 99.9 0.4 12. palo-bb4-link.ip.twelve99.net 18.4% 284 116.5 120.3 111.8 135.3 3.8 13. sjo-b23-link.ip.twelve99.net 89.8% 284 125.5 122.6 116.7 127.3 3.7 14. railwaycorp-ic-390075.ip.twelve99-cust.net 9.2% 284 118.7 117.5 111.6 123.9 3.6 15. 162.220.232.255 5.7% 284 117.6 117.6 116.3 123.3 0.8 On Thu, Jul 2, 2026 at 6:43 PM Phil Lavin via Outages <outages@outages.org> wrote:
Hi folks,
Seeing major packet loss on 1299. Somewhere between Seattle and Chicago. mtr below between AWS usw2 and Hetzner Germany.
My traceroute [v0.95] ** (**) -> hc-ping.com (176.9.71.146) 2026-07-02T09:06:23+0000 Keys: Help Display mode Restart statistics Order of fields quit Packets Pings Host Loss% Snt Last Avg Best Wrst StDev 1. ip-**.us-west-2.compute.internal 0.0% 81 0.2 0.2 0.2 0.3 0.0 2. 242.11.42.7 0.0% 81 8.2 9.4 8.0 20.0 2.6 242.11.42.5 242.11.43.5 242.11.43.135 242.11.42.129 242.11.43.3 242.11.42.131 242.11.42.3 3. amazon-ic-341989.ip.twelve99-cust.net 0.0% 81 7.6 7.6 7.5 7.9 0.1 4. sea-b1-link.ip.twelve99.net 0.0% 80 9.4 9.1 8.3 14.5 1.1 5. chi-bb2-link.ip.twelve99.net 67.1% 80 175.5 174.2 171.2 176.8 1.8 6. ewr-bb2-link.ip.twelve99.net 56.2% 80 112.3 112.3 110.2 113.4 0.7 7. ldn-bb2-link.ip.twelve99.net 54.4% 80 205.7 205.6 204.7 207.3 0.6 8. hbg-bb2-link.ip.twelve99.net 60.8% 80 204.2 204.0 202.6 205.5 0.7 9. hbg-b2-link.ip.twelve99.net 62.0% 80 191.9 192.6 190.4 203.8 2.7 10. hetzner-ic-383976.ip.twelve99-cust.net 55.7% 80 201.4 201.3 198.8 203.0 0.8 11. core21.fsn1.hetzner.com 38.8% 80 201.1 200.8 199.6 202.0 0.6 12. ex9k2.dc6.fsn1.hetzner.com 55.7% 80 201.4 201.4 199.7 206.8 1.1 13. static.146.71.9.176.clients.your-server.de 41.2% 80 201.7 201.3 199.2 202.6 0.7
______________________________________________ Outages mailing list outages@outages.org Sign up for an account https://lists.outages.org/accounts/signup/ To subscribe send an email to outages-join@outages.org To unsubscribe send an email to outages-leave@outages.org To contact the list owners outages-owner@outages.org Archives https://lists.outages.org/archives/list/outages@outages.org/
Thank you for using Outages Lists! For post outage discussion, please redirect to outages-discussion@outages.org
I'm not seeing return-path mtrs/traceroutes provided. You guys, thusly, are only providing half the picture. Initial post shows possible loss between Chicago and Newark, follow-up shows possible loss between Newark and New York. But we need to see the return path. You may want to try https://lg.twelve99.net/ (Arelion looking glass) if you do not have access to the destination. Where I got that from: https://www.peeringdb.com/net/10 Reading material in the meantime: https://www.nanog.org/meetings/nanog47/presentations/Sunday/RAS_Traceroute_N... -- | Jeremy Chadwick jdc@koitsu.org | | UNIX Systems Administrator PGP 0x2A389531 | | Making life hard for others since 1977. | On Thu, Jul 02, 2026 at 06:46:20PM +0900, Phineas Walton via Outages wrote:
a few others + us (AS400940) have been seeing the same since 06:04 UTC for many arelion paths - their noc is not acknowledging the incident over email comms. very frustrating
My traceroute [v0.95] production-hikari-dpjfk-1 (152.233.47.66) -> 162.220.232.255 (162.22026-07-02T06:13:31+0000 Keys: Help Display mode Restart statistics Order of fields quit Packets Pings Host Loss% Snt Last Avg Best Wrst StDev 1. unn-152-233-47-76.datapacket.com 38.9% 285 0.1 0.1 0.1 0.3 0.0 2. vl223.nyc-drt12-core-1.cdn77.com 0.0% 285 0.9 0.9 0.8 1.5 0.1 3. ewr-b6-link.ip.twelve99.net 0.0% 285 3.4 3.8 2.1 12.2 1.0 4. nyk-bb5-link.ip.twelve99.net 62.1% 285 1.5 1.6 1.5 4.1 0.3 5. rest-bb1-link.ip.twelve99.net 87.0% 285 8.2 8.3 8.1 11.1 0.5 6. atl-bb1-link.ip.twelve99.net 64.1% 284 23.8 23.9 23.8 25.5 0.2 7. atl-b24-link.ip.twelve99.net 96.5% 284 24.0 24.1 23.9 24.3 0.1 8. atl-bb2-link.ip.twelve99.net 57.0% 284 24.0 23.9 23.9 24.3 0.1 9. hou-bb4-link.ip.twelve99.net 84.5% 284 41.7 42.6 41.0 44.8 0.8 10. dls-b23-link.ip.twelve99.net 59.5% 284 52.2 52.2 52.0 59.0 0.6 11. lax-bb2-link.ip.twelve99.net 89.8% 284 99.8 99.2 98.1 99.9 0.4 12. palo-bb4-link.ip.twelve99.net 18.4% 284 116.5 120.3 111.8 135.3 3.8 13. sjo-b23-link.ip.twelve99.net 89.8% 284 125.5 122.6 116.7 127.3 3.7 14. railwaycorp-ic-390075.ip.twelve99-cust.net 9.2% 284 118.7 117.5 111.6 123.9 3.6 15. 162.220.232.255 5.7% 284 117.6 117.6 116.3 123.3 0.8
On Thu, Jul 2, 2026 at 6:43 PM Phil Lavin via Outages <outages@outages.org> wrote:
Hi folks,
Seeing major packet loss on 1299. Somewhere between Seattle and Chicago. mtr below between AWS usw2 and Hetzner Germany.
My traceroute [v0.95] ** (**) -> hc-ping.com (176.9.71.146) 2026-07-02T09:06:23+0000 Keys: Help Display mode Restart statistics Order of fields quit Packets Pings Host Loss% Snt Last Avg Best Wrst StDev 1. ip-**.us-west-2.compute.internal 0.0% 81 0.2 0.2 0.2 0.3 0.0 2. 242.11.42.7 0.0% 81 8.2 9.4 8.0 20.0 2.6 242.11.42.5 242.11.43.5 242.11.43.135 242.11.42.129 242.11.43.3 242.11.42.131 242.11.42.3 3. amazon-ic-341989.ip.twelve99-cust.net 0.0% 81 7.6 7.6 7.5 7.9 0.1 4. sea-b1-link.ip.twelve99.net 0.0% 80 9.4 9.1 8.3 14.5 1.1 5. chi-bb2-link.ip.twelve99.net 67.1% 80 175.5 174.2 171.2 176.8 1.8 6. ewr-bb2-link.ip.twelve99.net 56.2% 80 112.3 112.3 110.2 113.4 0.7 7. ldn-bb2-link.ip.twelve99.net 54.4% 80 205.7 205.6 204.7 207.3 0.6 8. hbg-bb2-link.ip.twelve99.net 60.8% 80 204.2 204.0 202.6 205.5 0.7 9. hbg-b2-link.ip.twelve99.net 62.0% 80 191.9 192.6 190.4 203.8 2.7 10. hetzner-ic-383976.ip.twelve99-cust.net 55.7% 80 201.4 201.3 198.8 203.0 0.8 11. core21.fsn1.hetzner.com 38.8% 80 201.1 200.8 199.6 202.0 0.6 12. ex9k2.dc6.fsn1.hetzner.com 55.7% 80 201.4 201.4 199.7 206.8 1.1 13. static.146.71.9.176.clients.your-server.de 41.2% 80 201.7 201.3 199.2 202.6 0.7
______________________________________________ Outages mailing list outages@outages.org Sign up for an account https://lists.outages.org/accounts/signup/ To subscribe send an email to outages-join@outages.org To unsubscribe send an email to outages-leave@outages.org To contact the list owners outages-owner@outages.org Archives https://lists.outages.org/archives/list/outages@outages.org/
Thank you for using Outages Lists! For post outage discussion, please redirect to outages-discussion@outages.org
______________________________________________ Outages mailing list outages@outages.org Sign up for an account https://lists.outages.org/accounts/signup/ To subscribe send an email to outages-join@outages.org To unsubscribe send an email to outages-leave@outages.org To contact the list owners outages-owner@outages.org Archives https://lists.outages.org/archives/list/outages@outages.org/
Thank you for using Outages Lists! For post outage discussion, please redirect to outages-discussion@outages.org
We've been tracking some packet loss since last night. Explanation from 1299 was congestion due to a fibre cut. They've reported to us a short time ago that they've been able to clear the congestion using some traffic engineering. That said, it's relatively early in the day in most of the USA, so I wouldn't be surprised to see things decay again unless they've been able to restore capacity. We have trace routes showing massive packet loss between Dallas and Houston between 2100 and 0410 UTC with some light packet loss a bit later in various places. (probably below most people's detection threshold). Tyler Automattic Network Operations On Thu, Jul 2, 2026 at 11:39 AM Jeremy Chadwick via Outages < outages@outages.org> wrote:
I'm not seeing return-path mtrs/traceroutes provided. You guys, thusly, are only providing half the picture.
Initial post shows possible loss between Chicago and Newark, follow-up shows possible loss between Newark and New York. But we need to see the return path.
You may want to try https://lg.twelve99.net/ (Arelion looking glass) if you do not have access to the destination. Where I got that from: https://www.peeringdb.com/net/10
Reading material in the meantime:
https://www.nanog.org/meetings/nanog47/presentations/Sunday/RAS_Traceroute_N...
-- | Jeremy Chadwick jdc@koitsu.org | | UNIX Systems Administrator PGP 0x2A389531 | | Making life hard for others since 1977. |
On Thu, Jul 02, 2026 at 06:46:20PM +0900, Phineas Walton via Outages wrote:
a few others + us (AS400940) have been seeing the same since 06:04 UTC for many arelion paths - their noc is not acknowledging the incident over email comms. very frustrating
My traceroute [v0.95] production-hikari-dpjfk-1 (152.233.47.66) -> 162.220.232.255 (162.22026-07-02T06:13:31+0000 Keys: Help Display mode Restart statistics Order of fields quit Packets Pings Host Loss% Snt Last Avg Best Wrst StDev 1. unn-152-233-47-76.datapacket.com 38.9% 285 0.1 0.1 0.1 0.3 0.0 2. vl223.nyc-drt12-core-1.cdn77.com 0.0% 285 0.9 0.9 0.8 1.5 0.1 3. ewr-b6-link.ip.twelve99.net 0.0% 285 3.4 3.8 2.1 12.2 1.0 4. nyk-bb5-link.ip.twelve99.net 62.1% 285 1.5 1.6 1.5 4.1 0.3 5. rest-bb1-link.ip.twelve99.net 87.0% 285 8.2 8.3 8.1 11.1 0.5 6. atl-bb1-link.ip.twelve99.net 64.1% 284 23.8 23.9 23.8 25.5 0.2 7. atl-b24-link.ip.twelve99.net 96.5% 284 24.0 24.1 23.9 24.3 0.1 8. atl-bb2-link.ip.twelve99.net 57.0% 284 24.0 23.9 23.9 24.3 0.1 9. hou-bb4-link.ip.twelve99.net 84.5% 284 41.7 42.6 41.0 44.8 0.8 10. dls-b23-link.ip.twelve99.net 59.5% 284 52.2 52.2 52.0 59.0 0.6 11. lax-bb2-link.ip.twelve99.net 89.8% 284 99.8 99.2 98.1 99.9 0.4 12. palo-bb4-link.ip.twelve99.net 18.4% 284 116.5 120.3 111.8 135.3 3.8 13. sjo-b23-link.ip.twelve99.net 89.8% 284 125.5 122.6 116.7 127.3 3.7 14. railwaycorp-ic-390075.ip.twelve99-cust.net 9.2% 284 118.7 117.5 111.6 123.9 3.6 15. 162.220.232.255 5.7% 284 117.6 117.6 116.3 123.3 0.8
On Thu, Jul 2, 2026 at 6:43 PM Phil Lavin via Outages < outages@outages.org> wrote:
Hi folks,
Seeing major packet loss on 1299. Somewhere between Seattle and Chicago. mtr below between AWS usw2 and Hetzner Germany.
My traceroute [v0.95] ** (**) -> hc-ping.com (176.9.71.146) 2026-07-02T09:06:23+0000 Keys: Help Display mode Restart statistics Order of fields quit Packets Pings Host Loss% Snt Last Avg Best Wrst StDev 1. ip-**.us-west-2.compute.internal 0.0% 81 0.2 0.2 0.2 0.3 0.0 2. 242.11.42.7 0.0% 81 8.2 9.4 8.0 20.0 2.6 242.11.42.5 242.11.43.5 242.11.43.135 242.11.42.129 242.11.43.3 242.11.42.131 242.11.42.3 3. amazon-ic-341989.ip.twelve99-cust.net 0.0% 81 7.6 7.6 7.5 7.9 0.1 4. sea-b1-link.ip.twelve99.net 0.0% 80 9.4 9.1 8.3 14.5 1.1 5. chi-bb2-link.ip.twelve99.net 67.1% 80 175.5 174.2 171.2 176.8 1.8 6. ewr-bb2-link.ip.twelve99.net 56.2% 80 112.3 112.3 110.2 113.4 0.7 7. ldn-bb2-link.ip.twelve99.net 54.4% 80 205.7 205.6 204.7 207.3 0.6 8. hbg-bb2-link.ip.twelve99.net 60.8% 80 204.2 204.0 202.6 205.5 0.7 9. hbg-b2-link.ip.twelve99.net 62.0% 80 191.9 192.6 190.4 203.8 2.7 10. hetzner-ic-383976.ip.twelve99-cust.net 55.7% 80 201.4 201.3 198.8 203.0 0.8 11. core21.fsn1.hetzner.com 38.8% 80 201.1 200.8 199.6 202.0 0.6 12. ex9k2.dc6.fsn1.hetzner.com 55.7% 80 201.4 201.4 199.7 206.8 1.1 13. static.146.71.9.176.clients.your-server.de 41.2% 80 201.7 201.3 199.2 202.6 0.7
______________________________________________ Outages mailing list outages@outages.org Sign up for an account https://lists.outages.org/accounts/signup/ To subscribe send an email to outages-join@outages.org To unsubscribe send an email to outages-leave@outages.org To contact the list owners outages-owner@outages.org Archives https://lists.outages.org/archives/list/outages@outages.org/
Thank you for using Outages Lists! For post outage discussion, please redirect to outages-discussion@outages.org
______________________________________________ Outages mailing list outages@outages.org Sign up for an account https://lists.outages.org/accounts/signup/ To subscribe send an email to outages-join@outages.org To unsubscribe send an email to outages-leave@outages.org To contact the list owners outages-owner@outages.org Archives https://lists.outages.org/archives/list/outages@outages.org/
Thank you for using Outages Lists! For post outage discussion, please redirect to outages-discussion@outages.org
Outages mailing list outages@outages.org Sign up for an account https://lists.outages.org/accounts/signup/ To subscribe send an email to outages-join@outages.org To unsubscribe send an email to outages-leave@outages.org To contact the list owners outages-owner@outages.org Archives https://lists.outages.org/archives/list/outages@outages.org/
Thank you for using Outages Lists! For post outage discussion, please redirect to outages-discussion@outages.org
Adding to my previous note. We also tracked packet loss until ~ 1030 UTC in Los Angeles. Hops 4 and 5 are both in LA if DNS is to be believed. MTR snippit below: HOST: nxx.xxx.wordpress.com Loss% Drop Rcv Snt Last Best Avg Wrst StDev 4.|-- 62.115.61.177 0.0% 0 25 25 0.6 0.5 0.7 1.0 0.1 5.|-- 62.115.140.156 68.0% 17 8 25 0.9 0.5 0.7 1.0 0.2 6.|-- 62.115.140.229 88.0% 22 3 25 189.3 174.6 179.9 189.3 8.1 7.|-- 62.115.137.38 40.0% 10 15 25 88.4 75.6 210.5 305.3 105.2 Tyler Automattic Network Operations On Thu, Jul 2, 2026 at 12:06 PM Automattic NOC <noc@a8c.com> wrote:
We've been tracking some packet loss since last night. Explanation from 1299 was congestion due to a fibre cut. They've reported to us a short time ago that they've been able to clear the congestion using some traffic engineering. That said, it's relatively early in the day in most of the USA, so I wouldn't be surprised to see things decay again unless they've been able to restore capacity.
We have trace routes showing massive packet loss between Dallas and Houston between 2100 and 0410 UTC with some light packet loss a bit later in various places. (probably below most people's detection threshold).
Tyler Automattic Network Operations
On Thu, Jul 2, 2026 at 11:39 AM Jeremy Chadwick via Outages < outages@outages.org> wrote:
I'm not seeing return-path mtrs/traceroutes provided. You guys, thusly, are only providing half the picture.
Initial post shows possible loss between Chicago and Newark, follow-up shows possible loss between Newark and New York. But we need to see the return path.
You may want to try https://lg.twelve99.net/ (Arelion looking glass) if you do not have access to the destination. Where I got that from: https://www.peeringdb.com/net/10
Reading material in the meantime:
https://www.nanog.org/meetings/nanog47/presentations/Sunday/RAS_Traceroute_N...
-- | Jeremy Chadwick jdc@koitsu.org | | UNIX Systems Administrator PGP 0x2A389531 | | Making life hard for others since 1977. |
On Thu, Jul 02, 2026 at 06:46:20PM +0900, Phineas Walton via Outages wrote:
a few others + us (AS400940) have been seeing the same since 06:04 UTC for many arelion paths - their noc is not acknowledging the incident over email comms. very frustrating
My traceroute [v0.95] production-hikari-dpjfk-1 (152.233.47.66) -> 162.220.232.255 (162.22026-07-02T06:13:31+0000 Keys: Help Display mode Restart statistics Order of fields quit Packets Pings Host Loss% Snt Last Avg Best Wrst StDev 1. unn-152-233-47-76.datapacket.com 38.9% 285 0.1 0.1 0.1 0.3 0.0 2. vl223.nyc-drt12-core-1.cdn77.com 0.0% 285 0.9 0.9 0.8 1.5 0.1 3. ewr-b6-link.ip.twelve99.net 0.0% 285 3.4 3.8 2.1 12.2 1.0 4. nyk-bb5-link.ip.twelve99.net 62.1% 285 1.5 1.6 1.5 4.1 0.3 5. rest-bb1-link.ip.twelve99.net 87.0% 285 8.2 8.3 8.1 11.1 0.5 6. atl-bb1-link.ip.twelve99.net 64.1% 284 23.8 23.9 23.8 25.5 0.2 7. atl-b24-link.ip.twelve99.net 96.5% 284 24.0 24.1 23.9 24.3 0.1 8. atl-bb2-link.ip.twelve99.net 57.0% 284 24.0 23.9 23.9 24.3 0.1 9. hou-bb4-link.ip.twelve99.net 84.5% 284 41.7 42.6 41.0 44.8 0.8 10. dls-b23-link.ip.twelve99.net 59.5% 284 52.2 52.2 52.0 59.0 0.6 11. lax-bb2-link.ip.twelve99.net 89.8% 284 99.8 99.2 98.1 99.9 0.4 12. palo-bb4-link.ip.twelve99.net 18.4% 284 116.5 120.3 111.8 135.3 3.8 13. sjo-b23-link.ip.twelve99.net 89.8% 284 125.5 122.6 116.7 127.3 3.7 14. railwaycorp-ic-390075.ip.twelve99-cust.net 9.2% 284 118.7 117.5 111.6 123.9 3.6 15. 162.220.232.255 5.7% 284 117.6 117.6 116.3 123.3 0.8
On Thu, Jul 2, 2026 at 6:43 PM Phil Lavin via Outages < outages@outages.org> wrote:
Hi folks,
Seeing major packet loss on 1299. Somewhere between Seattle and Chicago. mtr below between AWS usw2 and Hetzner Germany.
My traceroute [v0.95] ** (**) -> hc-ping.com (176.9.71.146) 2026-07-02T09:06:23+0000 Keys: Help Display mode Restart statistics Order of fields quit Packets Pings Host Loss% Snt Last Avg Best Wrst StDev 1. ip-**.us-west-2.compute.internal 0.0% 81 0.2 0.2 0.2 0.3 0.0 2. 242.11.42.7 0.0% 81 8.2 9.4 8.0 20.0 2.6 242.11.42.5 242.11.43.5 242.11.43.135 242.11.42.129 242.11.43.3 242.11.42.131 242.11.42.3 3. amazon-ic-341989.ip.twelve99-cust.net 0.0% 81 7.6 7.6 7.5 7.9 0.1 4. sea-b1-link.ip.twelve99.net 0.0% 80 9.4 9.1 8.3 14.5 1.1 5. chi-bb2-link.ip.twelve99.net 67.1% 80 175.5 174.2 171.2 176.8 1.8 6. ewr-bb2-link.ip.twelve99.net 56.2% 80 112.3 112.3 110.2 113.4 0.7 7. ldn-bb2-link.ip.twelve99.net 54.4% 80 205.7 205.6 204.7 207.3 0.6 8. hbg-bb2-link.ip.twelve99.net 60.8% 80 204.2 204.0 202.6 205.5 0.7 9. hbg-b2-link.ip.twelve99.net 62.0% 80 191.9 192.6 190.4 203.8 2.7 10. hetzner-ic-383976.ip.twelve99-cust.net 55.7% 80 201.4 201.3 198.8 203.0 0.8 11. core21.fsn1.hetzner.com 38.8% 80 201.1 200.8 199.6 202.0 0.6 12. ex9k2.dc6.fsn1.hetzner.com 55.7% 80 201.4 201.4 199.7 206.8 1.1 13. static.146.71.9.176.clients.your-server.de 41.2% 80 201.7 201.3 199.2 202.6 0.7
______________________________________________ Outages mailing list outages@outages.org Sign up for an account https://lists.outages.org/accounts/signup/ To subscribe send an email to outages-join@outages.org To unsubscribe send an email to outages-leave@outages.org To contact the list owners outages-owner@outages.org Archives https://lists.outages.org/archives/list/outages@outages.org/
Thank you for using Outages Lists! For post outage discussion, please redirect to outages-discussion@outages.org
______________________________________________ Outages mailing list outages@outages.org Sign up for an account https://lists.outages.org/accounts/signup/ To subscribe send an email to outages-join@outages.org To unsubscribe send an email to outages-leave@outages.org To contact the list owners outages-owner@outages.org Archives https://lists.outages.org/archives/list/outages@outages.org/
Thank you for using Outages Lists! For post outage discussion, please redirect to outages-discussion@outages.org
Outages mailing list outages@outages.org Sign up for an account https://lists.outages.org/accounts/signup/ To subscribe send an email to outages-join@outages.org To unsubscribe send an email to outages-leave@outages.org To contact the list owners outages-owner@outages.org Archives https://lists.outages.org/archives/list/outages@outages.org/
Thank you for using Outages Lists! For post outage discussion, please redirect to outages-discussion@outages.org
I wonder if it's the return. I used 1299's lg to our IP's in the EU. All trachea show SEA -> CHI in the first part of the path with no PL. Tracing from their LG to 146.71.9.176 seems to show it dying in YUL. On Thu, Jul 2, 2026 at 5:17 AM Phil Lavin via Outages <outages@outages.org> wrote:
Hi folks,
Seeing major packet loss on 1299. Somewhere between Seattle and Chicago. mtr below between AWS usw2 and Hetzner Germany.
My traceroute [v0.95] ** (**) -> hc-ping.com (176.9.71.146) 2026-07-02T09:06:23+0000 Keys: Help Display mode Restart statistics Order of fields quit Packets Pings Host Loss% Snt Last Avg Best Wrst StDev 1. ip-**.us-west-2.compute.internal 0.0% 81 0.2 0.2 0.2 0.3 0.0 2. 242.11.42.7 0.0% 81 8.2 9.4 8.0 20.0 2.6 242.11.42.5 242.11.43.5 242.11.43.135 242.11.42.129 242.11.43.3 242.11.42.131 242.11.42.3 3. amazon-ic-341989.ip.twelve99-cust.net 0.0% 81 7.6 7.6 7.5 7.9 0.1 4. sea-b1-link.ip.twelve99.net 0.0% 80 9.4 9.1 8.3 14.5 1.1 5. chi-bb2-link.ip.twelve99.net 67.1% 80 175.5 174.2 171.2 176.8 1.8 6. ewr-bb2-link.ip.twelve99.net 56.2% 80 112.3 112.3 110.2 113.4 0.7 7. ldn-bb2-link.ip.twelve99.net 54.4% 80 205.7 205.6 204.7 207.3 0.6 8. hbg-bb2-link.ip.twelve99.net 60.8% 80 204.2 204.0 202.6 205.5 0.7 9. hbg-b2-link.ip.twelve99.net 62.0% 80 191.9 192.6 190.4 203.8 2.7 10. hetzner-ic-383976.ip.twelve99-cust.net 55.7% 80 201.4 201.3 198.8 203.0 0.8 11. core21.fsn1.hetzner.com 38.8% 80 201.1 200.8 199.6 202.0 0.6 12. ex9k2.dc6.fsn1.hetzner.com 55.7% 80 201.4 201.4 199.7 206.8 1.1 13. static.146.71.9.176.clients.your-server.de 41.2% 80 201.7 201.3 199.2 202.6 0.7
______________________________________________ Outages mailing list outages@outages.org Sign up for an account https://lists.outages.org/accounts/signup/ To subscribe send an email to outages-join@outages.org To unsubscribe send an email to outages-leave@outages.org To contact the list owners outages-owner@outages.org Archives https://lists.outages.org/archives/list/outages@outages.org/
Thank you for using Outages Lists! For post outage discussion, please redirect to outages-discussion@outages.org
participants (5)
-
Automattic NOC -
Dovid Bender -
Jeremy Chadwick -
Phil Lavin -
Phineas Walton