Twelve99 packet loss around London

Seeing significant loss and latency to 8.8.8.8 over Twelve99. Looks to be around London/Dublin: My traceroute [v0.95] VONMVPCVXYVFD (172.20.0.76) -> 8.8.8.8 (8.8.8.8) 2024-08-12T14:32:13+0100 Keys: Help Display mode Restart statistics Order of fields quit Packets Pings Host Loss% Snt Last Avg Best Wrst StDev 1. 172.20.0.1 0.0% 79 1.8 1.2 0.6 3.7 0.5 2. lo0-0.bng8.thn-lon.zen.net.uk 0.0% 79 6.8 7.3 5.1 30.1 3.3 3. 51-148-244-20.dsl.zen.co.uk 0.0% 79 5.9 6.5 5.2 9.8 1.0 4. ldn-b3-link.ip.twelve99.net 1.3% 78 5.6 6.7 5.3 14.1 1.3 5. ldn-bb2-link.ip.twelve99.net 1.3% 78 7.1 7.0 5.4 17.3 1.4 6. dln-b3-link.ip.twelve99.net 42.3% 78 15.6 15.9 14.9 18.2 0.7 7. 72.14.243.178 55.8% 78 32.4 38.4 16.6 54.5 11.4 8. 108.170.227.21 17.9% 78 32.3 34.7 16.7 54.4 11.2 9. 209.85.244.231 32.5% 78 28.3 35.1 18.6 53.6 10.1 10. dns.google 25.6% 78 38.2 34.0 16.2 54.4 10.3 _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages

I am hearing that GlobalSwitch just had a building-scale outage as a result of some maintenance gone wrong. On Mon, 12 Aug 2024 at 14:38, Phil Lavin via Outages <outages@outages.org> wrote:
Seeing significant loss and latency to 8.8.8.8 over Twelve99. Looks to be around London/Dublin:
My traceroute [v0.95] VONMVPCVXYVFD (172.20.0.76) -> 8.8.8.8 (8.8.8.8) 2024-08-12T14:32:13+0100 Keys: Help Display mode Restart statistics Order of fields quit Packets Pings Host Loss% Snt Last Avg Best Wrst StDev 1. 172.20.0.1 0.0% 79 1.8 1.2 0.6 3.7 0.5 2. lo0-0.bng8.thn-lon.zen.net.uk 0.0% 79 6.8 7.3 5.1 30.1 3.3 3. 51-148-244-20.dsl.zen.co.uk 0.0% 79 5.9 6.5 5.2 9.8 1.0 4. ldn-b3-link.ip.twelve99.net 1.3% 78 5.6 6.7 5.3 14.1 1.3 5. ldn-bb2-link.ip.twelve99.net 1.3% 78 7.1 7.0 5.4 17.3 1.4 6. dln-b3-link.ip.twelve99.net 42.3% 78 15.6 15.9 14.9 18.2 0.7 7. 72.14.243.178 55.8% 78 32.4 38.4 16.6 54.5 11.4 8. 108.170.227.21 17.9% 78 32.3 34.7 16.7 54.4 11.2 9. 209.85.244.231 32.5% 78 28.3 35.1 18.6 53.6 10.1 10. dns.google 25.6% 78 38.2 34.0 16.2 54.4 10.3 _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages

I should clarify, that would be Global Switch North in London. On Mon, 12 Aug 2024 at 14:54, Tom Storey <tom@snnap.net> wrote:
I am hearing that GlobalSwitch just had a building-scale outage as a result of some maintenance gone wrong.
On Mon, 12 Aug 2024 at 14:38, Phil Lavin via Outages <outages@outages.org> wrote:
Seeing significant loss and latency to 8.8.8.8 over Twelve99. Looks to be around London/Dublin:
My traceroute [v0.95] VONMVPCVXYVFD (172.20.0.76) -> 8.8.8.8 (8.8.8.8) 2024-08-12T14:32:13+0100 Keys: Help Display mode Restart statistics Order of fields quit Packets Pings Host Loss% Snt Last Avg Best Wrst StDev 1. 172.20.0.1 0.0% 79 1.8 1.2 0.6 3.7 0.5 2. lo0-0.bng8.thn-lon.zen.net.uk 0.0% 79 6.8 7.3 5.1 30.1 3.3 3. 51-148-244-20.dsl.zen.co.uk 0.0% 79 5.9 6.5 5.2 9.8 1.0 4. ldn-b3-link.ip.twelve99.net 1.3% 78 5.6 6.7 5.3 14.1 1.3 5. ldn-bb2-link.ip.twelve99.net 1.3% 78 7.1 7.0 5.4 17.3 1.4 6. dln-b3-link.ip.twelve99.net 42.3% 78 15.6 15.9 14.9 18.2 0.7 7. 72.14.243.178 55.8% 78 32.4 38.4 16.6 54.5 11.4 8. 108.170.227.21 17.9% 78 32.3 34.7 16.7 54.4 11.2 9. 209.85.244.231 32.5% 78 28.3 35.1 18.6 53.6 10.1 10. dns.google 25.6% 78 38.2 34.0 16.2 54.4 10.3 _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages

It's a very interesting situation this. The information I have this morning suggests that the work was being carried out in GS East, and covering several floors there. But we are in GS North and were impacted as well. In any case, someone was having a very bad day yesterday. On Mon, 12 Aug 2024 at 14:57, Tom Storey <tom@snnap.net> wrote:
I should clarify, that would be Global Switch North in London.
On Mon, 12 Aug 2024 at 14:54, Tom Storey <tom@snnap.net> wrote:
I am hearing that GlobalSwitch just had a building-scale outage as a result of some maintenance gone wrong.
On Mon, 12 Aug 2024 at 14:38, Phil Lavin via Outages <outages@outages.org> wrote:
Seeing significant loss and latency to 8.8.8.8 over Twelve99. Looks to be around London/Dublin:
My traceroute [v0.95] VONMVPCVXYVFD (172.20.0.76) -> 8.8.8.8 (8.8.8.8) 2024-08-12T14:32:13+0100 Keys: Help Display mode Restart statistics Order of fields quit Packets Pings Host Loss% Snt Last Avg Best Wrst StDev 1. 172.20.0.1 0.0% 79 1.8 1.2 0.6 3.7 0.5 2. lo0-0.bng8.thn-lon.zen.net.uk 0.0% 79 6.8 7.3 5.1 30.1 3.3 3. 51-148-244-20.dsl.zen.co.uk 0.0% 79 5.9 6.5 5.2 9.8 1.0 4. ldn-b3-link.ip.twelve99.net 1.3% 78 5.6 6.7 5.3 14.1 1.3 5. ldn-bb2-link.ip.twelve99.net 1.3% 78 7.1 7.0 5.4 17.3 1.4 6. dln-b3-link.ip.twelve99.net 42.3% 78 15.6 15.9 14.9 18.2 0.7 7. 72.14.243.178 55.8% 78 32.4 38.4 16.6 54.5 11.4 8. 108.170.227.21 17.9% 78 32.3 34.7 16.7 54.4 11.2 9. 209.85.244.231 32.5% 78 28.3 35.1 18.6 53.6 10.1 10. dns.google 25.6% 78 38.2 34.0 16.2 54.4 10.3 _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages

Maybe related, Google reported a network connectivity issue on their UK GCP location (eu-west2) yesterday: https://status.cloud.google.com/incidents/ETJGhvY9Xaktw7tgi8dF On Tue, Aug 13, 2024, at 11:45, Tom Storey via Outages wrote:
It's a very interesting situation this.
The information I have this morning suggests that the work was being carried out in GS East, and covering several floors there. But we are in GS North and were impacted as well.
In any case, someone was having a very bad day yesterday.
On Mon, 12 Aug 2024 at 14:57, Tom Storey <tom@snnap.net> wrote:
I should clarify, that would be Global Switch North in London.
On Mon, 12 Aug 2024 at 14:54, Tom Storey <tom@snnap.net> wrote:
I am hearing that GlobalSwitch just had a building-scale outage as a result of some maintenance gone wrong.
On Mon, 12 Aug 2024 at 14:38, Phil Lavin via Outages <outages@outages.org> wrote:
Seeing significant loss and latency to 8.8.8.8 over Twelve99. Looks to be around London/Dublin:
My traceroute [v0.95] VONMVPCVXYVFD (172.20.0.76) -> 8.8.8.8 (8.8.8.8) 2024-08-12T14:32:13+0100 Keys: Help Display mode Restart statistics Order of fields quit Packets Pings Host Loss% Snt Last Avg Best Wrst StDev 1. 172.20.0.1 0.0% 79 1.8 1.2 0.6 3.7 0.5 2. lo0-0.bng8.thn-lon.zen.net.uk 0.0% 79 6.8 7.3 5.1 30.1 3.3 3. 51-148-244-20.dsl.zen.co.uk 0.0% 79 5.9 6.5 5.2 9.8 1.0 4. ldn-b3-link.ip.twelve99.net 1.3% 78 5.6 6.7 5.3 14.1 1.3 5. ldn-bb2-link.ip.twelve99.net 1.3% 78 7.1 7.0 5.4 17.3 1.4 6. dln-b3-link.ip.twelve99.net 42.3% 78 15.6 15.9 14.9 18.2 0.7 7. 72.14.243.178 55.8% 78 32.4 38.4 16.6 54.5 11.4 8. 108.170.227.21 17.9% 78 32.3 34.7 16.7 54.4 11.2 9. 209.85.244.231 32.5% 78 28.3 35.1 18.6 53.6 10.1 10. dns.google 25.6% 78 38.2 34.0 16.2 54.4 10.3 _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Kevin Decherf - @Kdecherf GPG 0x108ABD75A81E6E2F https://kdecherf.com _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
participants (3)
-
Kevin Decherf via Outages
-
Phil Lavin via Outages
-
Tom Storey via Outages