Unfortunately one of our critical destinations is now advertising a route through Cogent, and they are having some problems it seems. Anyone else? --- 66.62.162.31 ping statistics --- 62 packets transmitted, 40 received, 35% packet loss, time 61016ms rtt min/avg/max/mdev = 53.803/57.115/62.127/1.730 ms This is at the same time that latency to the last router before cogent is perfect (1.7-1.9ms). Relevant route: 3 204.15.220.81 (204.15.220.81) 1.268 ms 1.303 ms 1.289 ms 4 gi3-12.na41.b022559-0.phx02.atlas.cogentco.com (38.122.88.13) 1.497 ms 1.881 ms 1.889 ms 5 vl3810.mag02.phx02.atlas.cogentco.com (66.28.64.209) 12.983 ms 12.979 ms 12.964 ms 6 * te7-4.ccr01.phx02.atlas.cogentco.com (154.54.85.85) 195.832 ms * 7 te0-1-0-7.ccr22.lax01.atlas.cogentco.com (154.54.80.205) 13.472 ms te0-3-0-0.mpd22.lax01.atlas.cogentco.com (154.54.29.25) 12.875 ms te0-1-0-7.ccr22.lax01.atlas.cogentco.com (154.54.80.205) 13.028 ms 8 te3-4.ccr02.lax05.atlas.cogentco.com (154.54.3.10) 151.705 ms te9-8.ccr02.lax05.atlas.cogentco.com (154.54.44.134) 151.715 ms te4-8.ccr02.lax05.atlas.cogentco.com (154.54.30.194) 151.691 ms 9 xe-1-2-1.mpr1.lax12.us.above.net (64.125.13.169) 23.115 ms 23.105 ms 23.090 ms 10 xe-3-0-0.cr1.lax112.us.above.net (64.125.30.17) 13.148 ms 13.090 ms 13.117 ms 11 xe-0-2-0.cr1.sjc2.us.above.net (64.125.26.26) 21.132 ms 21.585 ms 20.958 ms 12 xe-1-0-1.mpr1.den1.us.above.net (64.125.30.133) 50.248 ms 50.045 ms 50.001 ms 13 (208.185.20.58) 50.859 ms 55.178 ms 55.035 ms -- Carlos Alvarez TelEvolve 602-889-3003
I saw packet loss on Cogent today as well. We have a Boston and NYC site, both saw high latency and some packet loss to almost all destination (even to other Cogent on-net POPs) It seems to have settled down, but I have not gotten a response to support requests. On Wed, Oct 10, 2012 at 6:16 PM, Carlos Alvarez <carlos@televolve.com> wrote:
Unfortunately one of our critical destinations is now advertising a route through Cogent, and they are having some problems it seems. Anyone else?
--- 66.62.162.31 ping statistics --- 62 packets transmitted, 40 received, 35% packet loss, time 61016ms rtt min/avg/max/mdev = 53.803/57.115/62.127/1.730 ms
This is at the same time that latency to the last router before cogent is perfect (1.7-1.9ms).
Relevant route:
3 204.15.220.81 (204.15.220.81) 1.268 ms 1.303 ms 1.289 ms 4 gi3-12.na41.b022559-0.phx02.atlas.cogentco.com (38.122.88.13) 1.497 ms 1.881 ms 1.889 ms 5 vl3810.mag02.phx02.atlas.cogentco.com (66.28.64.209) 12.983 ms 12.979 ms 12.964 ms 6 * te7-4.ccr01.phx02.atlas.cogentco.com (154.54.85.85) 195.832 ms * 7 te0-1-0-7.ccr22.lax01.atlas.cogentco.com (154.54.80.205) 13.472 ms te0-3-0-0.mpd22.lax01.atlas.cogentco.com (154.54.29.25) 12.875 ms te0-1-0-7.ccr22.lax01.atlas.cogentco.com (154.54.80.205) 13.028 ms 8 te3-4.ccr02.lax05.atlas.cogentco.com (154.54.3.10) 151.705 ms te9-8.ccr02.lax05.atlas.cogentco.com (154.54.44.134) 151.715 ms te4-8.ccr02.lax05.atlas.cogentco.com (154.54.30.194) 151.691 ms 9 xe-1-2-1.mpr1.lax12.us.above.net (64.125.13.169) 23.115 ms 23.105 ms 23.090 ms 10 xe-3-0-0.cr1.lax112.us.above.net (64.125.30.17) 13.148 ms 13.090 ms 13.117 ms 11 xe-0-2-0.cr1.sjc2.us.above.net (64.125.26.26) 21.132 ms 21.585 ms 20.958 ms 12 xe-1-0-1.mpr1.den1.us.above.net (64.125.30.133) 50.248 ms 50.045 ms 50.001 ms 13 (208.185.20.58) 50.859 ms 55.178 ms 55.035 ms
-- Carlos Alvarez TelEvolve 602-889-3003
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
-- Brent Jones brent@brentrjones.com
On Wed, Oct 10, 2012 at 6:23 PM, Brent Jones <brent@brentrjones.com> wrote:
(even to other Cogent on-net POPs)
This right here is frustrating, since they are in the same city and same network: 5 vl3810.mag02.phx02.atlas.cogentco.com (66.28.64.209) 12.983 ms 12.979 ms 12.964 ms 6 * te7-4.ccr01.phx02.atlas.cogentco.com (154.54.85.85) 195.832 ms * As you might imagine, we have a lot of complaints. -- Carlos Alvarez TelEvolve 602-889-3003
Carlos, How about a return-path traceroute? The ingress interface may be (and probably is) different than the egress. It would also help if you could provide source *and* destination IPs of the traceroutes. Details: http://www.nanog.org/meetings/nanog47/presentations/Sunday/RAS_Traceroute_N4... -- | Jeremy Chadwick jdc@koitsu.org | | UNIX Systems Administrator http://jdc.koitsu.org/ | | Mountain View, CA, US | | Making life hard for others since 1977. PGP 4BD6C0CB | On Wed, Oct 10, 2012 at 06:30:04PM -0700, Carlos Alvarez wrote:
On Wed, Oct 10, 2012 at 6:23 PM, Brent Jones <brent@brentrjones.com> wrote:
(even to other Cogent on-net POPs)
This right here is frustrating, since they are in the same city and same network:
5 vl3810.mag02.phx02.atlas.cogentco.com (66.28.64.209) 12.983 ms 12.979 ms 12.964 ms 6 * te7-4.ccr01.phx02.atlas.cogentco.com (154.54.85.85) 195.832 ms *
As you might imagine, we have a lot of complaints.
-- Carlos Alvarez TelEvolve 602-889-3003
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
On Wed, Oct 10, 2012 at 6:37 PM, Jeremy Chadwick <jdc@koitsu.org> wrote:
Carlos,
How about a return-path traceroute? The ingress interface may be (and probably is) different than the egress.
I don't have access to that device. No response from them yet.
It would also help if you could provide source *and* destination IPs of the traceroutes.
208.86.121.129 to 66.62.162.31 -- Carlos Alvarez TelEvolve 602-889-3003
I take it back, still seeing high latency, and very poor transfer performance :( 38.122.252.185 to 154.54.0.89 (BOS to PDX, Cogent on-net POPs), spiking to 150ms latency. On Wed, Oct 10, 2012 at 6:30 PM, Carlos Alvarez <carlos@televolve.com> wrote:
On Wed, Oct 10, 2012 at 6:23 PM, Brent Jones <brent@brentrjones.com> wrote:
(even to other Cogent on-net POPs)
This right here is frustrating, since they are in the same city and same network:
5 vl3810.mag02.phx02.atlas.cogentco.com (66.28.64.209) 12.983 ms 12.979 ms 12.964 ms 6 * te7-4.ccr01.phx02.atlas.cogentco.com (154.54.85.85) 195.832 ms *
As you might imagine, we have a lot of complaints.
-- Carlos Alvarez TelEvolve 602-889-3003
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
-- Brent Jones brent@brentrjones.com
Internet health report report latency between cogent and Level 3 http://www.internetpulse.net/ Ujjval K On Oct 10, 2012, at 8:10 PM, Brent Jones <brent@brentrjones.com> wrote:
I take it back, still seeing high latency, and very poor transfer performance :(
38.122.252.185 to 154.54.0.89 (BOS to PDX, Cogent on-net POPs), spiking to 150ms latency.
On Wed, Oct 10, 2012 at 6:30 PM, Carlos Alvarez <carlos@televolve.com> wrote:
On Wed, Oct 10, 2012 at 6:23 PM, Brent Jones <brent@brentrjones.com> wrote:
(even to other Cogent on-net POPs)
This right here is frustrating, since they are in the same city and same network:
5 vl3810.mag02.phx02.atlas.cogentco.com (66.28.64.209) 12.983 ms 12.979 ms 12.964 ms 6 * te7-4.ccr01.phx02.atlas.cogentco.com (154.54.85.85) 195.832 ms *
As you might imagine, we have a lot of complaints.
-- Carlos Alvarez TelEvolve 602-889-3003
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
-- Brent Jones brent@brentrjones.com _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Just a thought have you both ruled out ICMP priortization? Cogent does this... Also, have you called or just submitted a ticket? Everytime, I've opened a ticket....I've gotten a response rather quickly and for sure when I call. -----Original Message----- From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On Behalf Of Brent Jones Sent: Wednesday, October 10, 2012 9:10 PM To: Carlos Alvarez Cc: outages@outages.org Subject: Re: [outages] Cogent packet loss and jitter I take it back, still seeing high latency, and very poor transfer performance :( 38.122.252.185 to 154.54.0.89 (BOS to PDX, Cogent on-net POPs), spiking to 150ms latency. On Wed, Oct 10, 2012 at 6:30 PM, Carlos Alvarez <carlos@televolve.com> wrote:
On Wed, Oct 10, 2012 at 6:23 PM, Brent Jones <brent@brentrjones.com> wrote:
(even to other Cogent on-net POPs)
This right here is frustrating, since they are in the same city and same network:
5 vl3810.mag02.phx02.atlas.cogentco.com (66.28.64.209) 12.983 ms 12.979 ms 12.964 ms 6 * te7-4.ccr01.phx02.atlas.cogentco.com (154.54.85.85) 195.832 ms *
As you might imagine, we have a lot of complaints.
-- Carlos Alvarez TelEvolve 602-889-3003
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
-- Brent Jones brent@brentrjones.com _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Sorry Otis for a double response, I hit reply to on accident (getting tired). I noticed a lot of latency and failure to establish connections about the time the OP posted this message. Most of the sites in question were routing through cogent as well. I'd bet something happened on their end, but I do not see this happening at the moment anymore. Ryan On Wed, Oct 10, 2012 at 10:43 PM, Otis L. Surratt, Jr. <otis@ocosa.com>wrote:
Just a thought have you both ruled out ICMP priortization? Cogent does this... Also, have you called or just submitted a ticket?
Everytime, I've opened a ticket....I've gotten a response rather quickly and for sure when I call.
-----Original Message----- From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On Behalf Of Brent Jones Sent: Wednesday, October 10, 2012 9:10 PM To: Carlos Alvarez Cc: outages@outages.org Subject: Re: [outages] Cogent packet loss and jitter
I take it back, still seeing high latency, and very poor transfer performance :(
38.122.252.185 to 154.54.0.89 (BOS to PDX, Cogent on-net POPs), spiking to 150ms latency.
On Wed, Oct 10, 2012 at 6:30 PM, Carlos Alvarez <carlos@televolve.com> wrote:
On Wed, Oct 10, 2012 at 6:23 PM, Brent Jones <brent@brentrjones.com> wrote:
(even to other Cogent on-net POPs)
This right here is frustrating, since they are in the same city and same network:
5 vl3810.mag02.phx02.atlas.cogentco.com (66.28.64.209) 12.983 ms 12.979 ms 12.964 ms 6 * te7-4.ccr01.phx02.atlas.cogentco.com (154.54.85.85) 195.832 ms *
As you might imagine, we have a lot of complaints.
-- Carlos Alvarez TelEvolve 602-889-3003
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
-- Brent Jones brent@brentrjones.com _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Otis et al, I believe Carlos ruled out ICMP prio as a result of doing a native server-to-server ping, from 208.86.121.129 to 66.62.162.31, as indicated here:
--- 66.62.162.31 ping statistics --- 62 packets transmitted, 40 received, 35% packet loss, time 61016ms rtt min/avg/max/mdev = 53.803/57.115/62.127/1.730 ms
ICMP prio doesn't affect src-to-dst RTT or packet loss; it affects src-to-administrative-router-IP situations (individual hop probes in traceroute, sans the destination, etc.). I think it's more likely that there is a saturated network link somewhere, given the behaviour shown. Seeing the return path (not for latency or packet loss reasons, but to see the peering points) would be useful though. -- | Jeremy Chadwick jdc@koitsu.org | | UNIX Systems Administrator http://jdc.koitsu.org/ | | Mountain View, CA, US | | Making life hard for others since 1977. PGP 4BD6C0CB | On Wed, Oct 10, 2012 at 09:43:15PM -0500, Otis L. Surratt, Jr. wrote:
Just a thought have you both ruled out ICMP priortization? Cogent does this... Also, have you called or just submitted a ticket?
Everytime, I've opened a ticket....I've gotten a response rather quickly and for sure when I call.
-----Original Message----- From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On Behalf Of Brent Jones Sent: Wednesday, October 10, 2012 9:10 PM To: Carlos Alvarez Cc: outages@outages.org Subject: Re: [outages] Cogent packet loss and jitter
I take it back, still seeing high latency, and very poor transfer performance :(
38.122.252.185 to 154.54.0.89 (BOS to PDX, Cogent on-net POPs), spiking to 150ms latency.
On Wed, Oct 10, 2012 at 6:30 PM, Carlos Alvarez <carlos@televolve.com> wrote:
On Wed, Oct 10, 2012 at 6:23 PM, Brent Jones <brent@brentrjones.com> wrote:
(even to other Cogent on-net POPs)
This right here is frustrating, since they are in the same city and same network:
5 vl3810.mag02.phx02.atlas.cogentco.com (66.28.64.209) 12.983 ms 12.979 ms 12.964 ms 6 * te7-4.ccr01.phx02.atlas.cogentco.com (154.54.85.85) 195.832 ms *
As you might imagine, we have a lot of complaints.
-- Carlos Alvarez TelEvolve 602-889-3003
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
-- Brent Jones brent@brentrjones.com _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
The latency at hop 6 isn't necessarily related to the end-to-end problem (as pointed out in the slides Jeremy sent)- it's not carrying through to any later hops, so it could just be a bad asymmetric reverse path from that particular router or that router being slow on traceroute responses. That hop is perfectly responsive when I ping it from a variety of locations, even if the responses are sometimes slow. The latency to hop 10 (after Cogent) is about the same as the latency to hop 5 (inside Cogent), and the latency doesn't really go up until the hop in Denver (when suddenly you have a much longer path). None of my comments say anything about the loss, however. On Wed, Oct 10, 2012 at 6:30 PM, Carlos Alvarez <carlos@televolve.com>wrote:
On Wed, Oct 10, 2012 at 6:23 PM, Brent Jones <brent@brentrjones.com>wrote:
(even to other Cogent on-net POPs)
This right here is frustrating, since they are in the same city and same network:
5 vl3810.mag02.phx02.atlas.cogentco.com (66.28.64.209) 12.983 ms 12.979 ms 12.964 ms 6 * te7-4.ccr01.phx02.atlas.cogentco.com (154.54.85.85) 195.832 ms *
As you might imagine, we have a lot of complaints.
-- Carlos Alvarez TelEvolve 602-889-3003
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
participants (7)
-
Brent Jones -
Carlos Alvarez -
Ethan Katz-Bassett -
Jeremy Chadwick -
Otis L. Surratt, Jr. -
Ryan McIntosh -
Ss Gmail