Had some alarms go off this morning regarding an apparent problem with Level3 transiting Denver. Anyone else see anything? Level 3 seems to be unaware, still, at least according to their support desk. tracing Illinois-California: [08:11] 8 ae-3-3.ebr2.Denver1.Level3.net (4.69.132.61) [AS 3356] [MPLS: Label 1350 Exp 0] 36 msec 32 msec 44 msec [08:11] 9 ae-1-100.ebr1.Denver1.Level3.net (4.69.151.181) [AS 3356] [MPLS: Label 1360 Exp 0] 32 msec 32 msec 40 msec [08:11] 10 * * * [08:11] 11 * * * And in reverse: [08:12] 4.69.152.209 [AS 3356] 4 msec [08:12] 4.69.152.81 [AS 3356] 0 msec [08:12] 3 * * * [08:12] 4 * * * 7:45-8:10 or so, US Central time. -Steve
Yes, having a lot of packet loss to a Salt Lake City HSIP circuit I have with them.. No problem with my Denver HSIP circuit though. I have a ticket open with them and the tech was not aware of an outage. On 7/19/12 6:23 AM, "Steve Dispensa" <dispensa@phonefactor.com> wrote:
Had some alarms go off this morning regarding an apparent problem with Level3 transiting Denver. Anyone else see anything? Level 3 seems to be unaware, still, at least according to their support desk.
tracing Illinois-California:
[08:11] 8 ae-3-3.ebr2.Denver1.Level3.net (4.69.132.61) [AS 3356] [MPLS: Label 1350 Exp 0] 36 msec 32 msec 44 msec [08:11] 9 ae-1-100.ebr1.Denver1.Level3.net (4.69.151.181) [AS 3356] [MPLS: Label 1360 Exp 0] 32 msec 32 msec 40 msec [08:11] 10 * * * [08:11] 11 * * *
And in reverse:
[08:12] 4.69.152.209 [AS 3356] 4 msec [08:12] 4.69.152.81 [AS 3356] 0 msec [08:12] 3 * * * [08:12] 4 * * *
7:45-8:10 or so, US Central time.
-Steve
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Just happened again, and we've been seeing crazy packet loss through slc on a L3 route from Kansas City to San Francisco as well. Something has been broken in SLC since at least 8:50 AM Central. -sd
-----Original Message----- From: Micah Byers [mailto:mbyers@telesphere.com] Sent: Thursday, July 19, 2012 9:36 AM To: Steve Dispensa; outages@outages.org Subject: Re: [outages] L3 Denver issue
Yes, having a lot of packet loss to a Salt Lake City HSIP circuit I have with them.. No problem with my Denver HSIP circuit though. I have a ticket open with them and the tech was not aware of an outage.
On 7/19/12 6:23 AM, "Steve Dispensa" <dispensa@phonefactor.com> wrote:
Had some alarms go off this morning regarding an apparent problem with Level3 transiting Denver. Anyone else see anything? Level 3 seems to be unaware, still, at least according to their support desk.
tracing Illinois-California:
[08:11] 8 ae-3-3.ebr2.Denver1.Level3.net (4.69.132.61) [AS 3356] [MPLS: Label 1350 Exp 0] 36 msec 32 msec 44 msec [08:11] 9 ae-1-100.ebr1.Denver1.Level3.net (4.69.151.181) [AS 3356] [MPLS: Label 1360 Exp 0] 32 msec 32 msec 40 msec [08:11] 10 * * * [08:11] 11 * * *
And in reverse:
[08:12] 4.69.152.209 [AS 3356] 4 msec [08:12] 4.69.152.81 [AS 3356] 0 msec [08:12] 3 * * * [08:12] 4 * * *
7:45-8:10 or so, US Central time.
-Steve
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
I am not sure if it is related to your packet loss, but we received word that L3 has a fiber mess in Tucson, AZ where a splice case and fiber were stolen around 1:20 PST. No ETR on that yet. Possible congestion on remaining routes?
Update I got from Level3.. Summary A link is at 100% utilization between Salt Lake City, UT and Denver, CO, which is affecting IP traffic. Event Case ID: 5781942 Location: Between Salt Lake City, UT to Denver, CO Impacted For: 3 hours 48 minutes ETR: Unknown Bridge: IP *** CASCADED EXTERNAL NOTES 19-Jul-2012 18:46:28 GMT saldanam From CASE: 5781942 - Event The IP NOC continues to isolate and investigate this issue to determine the root cause with no new information available. On 7/19/12 9:54 AM, "Steve Dispensa" <dispensa@phonefactor.com> wrote:
Just happened again, and we've been seeing crazy packet loss through slc on a L3 route from Kansas City to San Francisco as well. Something has been broken in SLC since at least 8:50 AM Central.
-sd
-----Original Message----- From: Micah Byers [mailto:mbyers@telesphere.com] Sent: Thursday, July 19, 2012 9:36 AM To: Steve Dispensa; outages@outages.org Subject: Re: [outages] L3 Denver issue
Yes, having a lot of packet loss to a Salt Lake City HSIP circuit I have with them.. No problem with my Denver HSIP circuit though. I have a ticket open with them and the tech was not aware of an outage.
On 7/19/12 6:23 AM, "Steve Dispensa" <dispensa@phonefactor.com> wrote:
Had some alarms go off this morning regarding an apparent problem with Level3 transiting Denver. Anyone else see anything? Level 3 seems to be unaware, still, at least according to their support desk.
tracing Illinois-California:
[08:11] 8 ae-3-3.ebr2.Denver1.Level3.net (4.69.132.61) [AS 3356] [MPLS: Label 1350 Exp 0] 36 msec 32 msec 44 msec [08:11] 9 ae-1-100.ebr1.Denver1.Level3.net (4.69.151.181) [AS 3356] [MPLS: Label 1360 Exp 0] 32 msec 32 msec 40 msec [08:11] 10 * * * [08:11] 11 * * *
And in reverse:
[08:12] 4.69.152.209 [AS 3356] 4 msec [08:12] 4.69.152.81 [AS 3356] 0 msec [08:12] 3 * * * [08:12] 4 * * *
7:45-8:10 or so, US Central time.
-Steve
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Thanks; that's consistent with what I'm (still) seeing, and more than we've gotten out of them. -Steve
-----Original Message----- From: Micah Byers [mailto:mbyers@telesphere.com] Sent: Thursday, July 19, 2012 1:49 PM To: Steve Dispensa; outages@outages.org Subject: Re: [outages] L3 Denver issue
Update I got from Level3..
Summary A link is at 100% utilization between Salt Lake City, UT and Denver, CO, which is affecting IP traffic.
Event Case ID: 5781942 Location: Between Salt Lake City, UT to Denver, CO Impacted For: 3 hours 48 minutes ETR: Unknown Bridge: IP
*** CASCADED EXTERNAL NOTES 19-Jul-2012 18:46:28 GMT saldanam From CASE: 5781942 - Event The IP NOC continues to isolate and investigate this issue to determine the root cause with no new information available.
On 7/19/12 9:54 AM, "Steve Dispensa" <dispensa@phonefactor.com> wrote:
Just happened again, and we've been seeing crazy packet loss through slc on a L3 route from Kansas City to San Francisco as well. Something has been broken in SLC since at least 8:50 AM Central.
-sd
-----Original Message----- From: Micah Byers [mailto:mbyers@telesphere.com] Sent: Thursday, July 19, 2012 9:36 AM To: Steve Dispensa; outages@outages.org Subject: Re: [outages] L3 Denver issue
Yes, having a lot of packet loss to a Salt Lake City HSIP circuit I have with them.. No problem with my Denver HSIP circuit though. I have a ticket open with them and the tech was not aware of an outage.
On 7/19/12 6:23 AM, "Steve Dispensa" <dispensa@phonefactor.com> wrote:
Had some alarms go off this morning regarding an apparent problem with Level3 transiting Denver. Anyone else see anything? Level 3 seems to be unaware, still, at least according to their support desk.
tracing Illinois-California:
[08:11] 8 ae-3-3.ebr2.Denver1.Level3.net (4.69.132.61) [AS 3356] [MPLS: Label 1350 Exp 0] 36 msec 32 msec 44 msec [08:11] 9 ae-1-100.ebr1.Denver1.Level3.net (4.69.151.181) [AS 3356] [MPLS: Label 1360 Exp 0] 32 msec 32 msec 40 msec [08:11] 10 * * * [08:11] 11 * * *
And in reverse:
[08:12] 4.69.152.209 [AS 3356] 4 msec [08:12] 4.69.152.81 [AS 3356] 0 msec [08:12] 3 * * * [08:12] 4 * * *
7:45-8:10 or so, US Central time.
-Steve
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Steve, On 07/19/2012 07:23 AM, Steve Dispensa wrote:
Had some alarms go off this morning regarding an apparent problem with Level3 transiting Denver. Anyone else see anything? Level 3 seems to be unaware, still, at least according to their support desk.
We're in Denver and have a L3 pipe - not seeing any problems: [root@smtp-out ~]# mtr --report --report-cycles=10 ipv6.google.com smtp-out.axint.net Snt: 10 Loss% Last Avg Best Wrst StDev ip6-gw.axint.net 0.0% 0.7 0.6 0.3 0.7 0.1 ge-6-24-576.car1.Denver1.Level3.net 10.0% 1.8 1.3 1.2 1.8 0.2 vl-4040.car1.Dallas1.Level3.net 0.0% 15.5 30.4 15.4 161.3 46.0 vl-70.edge2.Dallas3.Level3.net 0.0% 15.9 15.9 15.6 16.5 0.2 2001:1900:4:3::256 0.0% 15.9 25.4 15.8 55.9 14.4 2001:4860::1:0:dd7 0.0% 15.9 31.1 15.8 80.0 25.8 2001:4860::8:0:2c9c 0.0% 16.2 16.1 15.9 16.2 0.1 2001:4860::8:0:29ef 0.0% 23.2 23.9 23.1 29.5 2.0 2001:4860::8:0:281d 0.0% 42.7 44.5 42.5 57.3 4.6 2001:4860::8:0:3426 0.0% 62.7 62.4 62.1 62.7 0.2 2001:4860::1:0:7a4 0.0% 62.3 62.3 62.0 62.5 0.1 2001:4860:0:1::593 0.0% 62.5 62.5 62.4 62.5 0.0 den03s06-in-x13.1e100.net 0.0% 62.3 62.2 62.1 62.3 0.1 [root@venus ~]# mtr --report-cycles=10 --report 208.69.73.1 HOST: venus.axint.net Loss% Snt Last Avg Best Wrst StDev 1.|-- parked.peakcolo.com 0.0% 10 1.0 1.1 0.9 1.2 0.1 2.|-- 173.247.71.62 0.0% 10 1.0 1.4 0.8 4.5 1.1 3.|-- ge-8-33.car1.Denver1.Leve 0.0% 10 1.1 1.1 1.0 1.2 0.1 4.|-- axis-core-vrrp-10.axint.n 0.0% 10 1.2 1.1 1.0 1.2 0.1 Chris
participants (4)
-
Andy Koch -
Chris Stone -
Micah Byers -
Steve Dispensa