Re: [outages] Twelve99 / AWS usw2 significant loss
Thanks, ytti. I have raised a case with AWS but I expect it to be as unproductive as usual. In this type of issue, there is often a friendly Engineer keeping an eye on NANOG list or IRC channel who can mitigate it. Hoping that to be the case this time, also.
On 26 Jan 2024, at 08:40, Saku Ytti <saku@ytti.fi> wrote:
On Fri, 26 Jan 2024 at 10:23, Phil Lavin via NANOG <nanog@nanog.org> wrote:
88.99.88.67 to 216.147.3.209: Host Loss% Snt Last Avg Best Wrst StDev 1. 10.88.10.254 0.0% 176 0.2 0.1 0.1 0.3 0.1 7. nug-b1-link.ip.twelve99.net 0.0% 176 3.3 3.5 3.1 24.1 1.6 8. hbg-bb2-link.ip.twelve99.net 86.9% 175 18.9 18.9 18.7 19.2 0.1 9. ldn-bb2-link.ip.twelve99.net 92.0% 175 30.5 30.6 30.4 30.8 0.1 10. nyk-bb1-link.ip.twelve99.net 4.6% 175 99.5 99.5 99.3 100.1 0.2 11. sjo-b23-link.ip.twelve99.net 56.3% 175 296.8 306.0 289.7 315.0 5.5 12. amazon-ic-366608.ip.twelve99-cust.net 80.5% 175 510.0 513.5 500.7 539.7 8.4
This implies the problem is not on this path, because #10 is not experiencing it, possibly because it happens to return a packet via another option, but certainly shows the problem didn't happen in this direction yet at #10, but because #8 and #9 saw it, they already saw it on the other direction.
44.236.47.236 to 178.63.26.145: Host Loss% Snt Last Avg Best Wrst StDev 1. ip-10-96-50-153.us-west-2.compute.internal 0.0% 267 0.2 0.2 0.2 0.4 0.0 11. port-b3-link.ip.twelve99.net 0.0% 267 5.8 5.9 5.6 11.8 0.5 12. palo-b24-link.ip.twelve99.net 4.9% 267 21.1 21.5 21.0 58.4 3.1 13. sjo-b23-link.ip.twelve99.net 0.0% 266 21.4 22.7 21.3 86.2 6.5 14. nyk-bb1-link.ip.twelve99.net 58.1% 266 432.7 422.7 407.2 438.5 6.5 15. ldn-bb2-link.ip.twelve99.net 98.1% 266 485.6 485.4 481.6 491.1 3.9 16. hbg-bb2-link.ip.twelve99.net 92.5% 266 504.1 499.8 489.8 510.1 5.9 17. nug-b1-link.ip.twelve99.net 55.5% 266 523.5 519.6 504.4 561.7 7.6 18. hetzner-ic-340780.ip.twelve99-cust.net 53.6% 266 524.4 519.2 506.0 545.5 6.9 19. core22.fsn1.hetzner.com 70.2% 266 521.7 519.2 498.5 531.7 6.6 20. static.213-239-254-150.clients.your-server.de 33.2% 266 382.4 375.4 364.9 396.5 4.1 21. static.145.26.63.178.clients.your-server.de 62.0% 266 529.9 518.4 506.9 531.3 6.1
This suggests the congestion point is from sjo to nyk, in 1299, not AWS at all.
You could try to fix SPORT/DPORT, and do several SPORT options, to see if loss goes away with some, to determine if all LAG members are full or just one.
At any rate, this seems business as usual, sometimes internet is very lossy, you should contact your service provider, which I guess is AWS here, so they can contact their service provider or 1299.
-- ++ytti
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Seems like the destination is in Hetzner, they could also raise it with Twelve99 or prepend routes to use an alternate path. On Fri, Jan 26, 2024 at 7:46 PM Phil Lavin via Outages <outages@outages.org> wrote:
Thanks, ytti. I have raised a case with AWS but I expect it to be as unproductive as usual. In this type of issue, there is often a friendly Engineer keeping an eye on NANOG list or IRC channel who can mitigate it. Hoping that to be the case this time, also.
On 26 Jan 2024, at 08:40, Saku Ytti <saku@ytti.fi> wrote:
On Fri, 26 Jan 2024 at 10:23, Phil Lavin via NANOG <nanog@nanog.org> wrote:
88.99.88.67 to 216.147.3.209: Host Loss% Snt Last Avg Best Wrst StDev 1. 10.88.10.254 0.0% 176 0.2 0.1 0.1 0.3 0.1 7. nug-b1-link.ip.twelve99.net 0.0% 176 3.3 3.5 3.1 24.1 1.6 8. hbg-bb2-link.ip.twelve99.net 86.9% 175 18.9 18.9 18.7 19.2 0.1 9. ldn-bb2-link.ip.twelve99.net 92.0% 175 30.5 30.6 30.4 30.8 0.1 10. nyk-bb1-link.ip.twelve99.net 4.6% 175 99.5 99.5 99.3 100.1 0.2 11. sjo-b23-link.ip.twelve99.net 56.3% 175 296.8 306.0 289.7 315.0 5.5 12. amazon-ic-366608.ip.twelve99-cust.net 80.5% 175 510.0 513.5 500.7 539.7 8.4
This implies the problem is not on this path, because #10 is not experiencing it, possibly because it happens to return a packet via another option, but certainly shows the problem didn't happen in this direction yet at #10, but because #8 and #9 saw it, they already saw it on the other direction.
44.236.47.236 to 178.63.26.145: Host Loss% Snt Last Avg Best Wrst StDev 1. ip-10-96-50-153.us-west-2.compute.internal 0.0% 267 0.2 0.2 0.2 0.4 0.0 11. port-b3-link.ip.twelve99.net 0.0% 267 5.8 5.9 5.6 11.8 0.5 12. palo-b24-link.ip.twelve99.net 4.9% 267 21.1 21.5 21.0 58.4 3.1 13. sjo-b23-link.ip.twelve99.net 0.0% 266 21.4 22.7 21.3 86.2 6.5 14. nyk-bb1-link.ip.twelve99.net 58.1% 266 432.7 422.7 407.2 438.5 6.5 15. ldn-bb2-link.ip.twelve99.net 98.1% 266 485.6 485.4 481.6 491.1 3.9 16. hbg-bb2-link.ip.twelve99.net 92.5% 266 504.1 499.8 489.8 510.1 5.9 17. nug-b1-link.ip.twelve99.net 55.5% 266 523.5 519.6 504.4 561.7 7.6 18. hetzner-ic-340780.ip.twelve99-cust.net 53.6% 266 524.4 519.2 506.0 545.5 6.9 19. core22.fsn1.hetzner.com 70.2% 266 521.7 519.2 498.5 531.7 6.6 20. static.213-239-254-150.clients.your-server.de 33.2% 266 382.4 375.4 364.9 396.5 4.1 21. static.145.26.63.178.clients.your-server.de 62.0% 266 529.9 518.4 506.9 531.3 6.1
This suggests the congestion point is from sjo to nyk, in 1299, not AWS at all.
You could try to fix SPORT/DPORT, and do several SPORT options, to see if loss goes away with some, to determine if all LAG members are full or just one.
At any rate, this seems business as usual, sometimes internet is very lossy, you should contact your service provider, which I guess is AWS here, so they can contact their service provider or 1299.
-- ++ytti
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
participants (2)
-
Andras Toth via Outages -
Phil Lavin via Outages