About 10 minutes ago we stopped being able to pass traffic through cogent. I de-peered us from Cogent, and everything appears better. When I call cogent, all I get is a busy signal (must be a major outage). Anyone else seeing anything? ---- Matthew Huff | 1 Manhattanville Rd Director of Operations | Purchase, NY 10577 OTA Management LLC | Phone: 914-460-4039 aim: matthewbhuff | Fax: 914-460-4139
On Dec 6, 2012, at 11:12 , Matthew Huff <mhuff@ox.com> wrote:
About 10 minutes ago we stopped being able to pass traffic through cogent. I de-peered us from Cogent, and everything appears better. When I call cogent, all I get is a busy signal (must be a major outage). Anyone else seeing anything?
Our Cogent peering is still up and passing normal levels of traffic in both directions. -- Julian Y. Koh Manager, Network Transport, Telecommunications and Network Services Northwestern University Information Technology (NUIT) 2001 Sheridan Road #G-166 Evanston, IL 60208 847-467-5780 NUIT Web Site: <http://www.it.northwestern.edu/> PGP Public Key:<http://bt.ittns.northwestern.edu/julian/pgppubkey.html>
On Dec 6, 2012, at 11:12 AM, Matthew Huff wrote:
About 10 minutes ago we stopped being able to pass traffic through cogent. I de-peered us from Cogent, and everything appears better. When I call cogent, all I get is a busy signal (must be a major outage). Anyone else seeing anything?
I saw packet loss to a destination in Troy, NY which has Cogent as one of its upstreams as well (and it was the Cogent path which I saw packet loss/disruption on). This was observed from a non-Cogent multihomed network in Reston, VA. Started at at 11:54 Eastern and ended about 20 minutes later.
On Dec 6, 2012, at 11:17 AM, Ryan Rawdon wrote:
On Dec 6, 2012, at 11:12 AM, Matthew Huff wrote:
About 10 minutes ago we stopped being able to pass traffic through cogent. I de-peered us from Cogent, and everything appears better. When I call cogent, all I get is a busy signal (must be a major outage). Anyone else seeing anything?
I saw packet loss to a destination in Troy, NY which has Cogent as one of its upstreams as well (and it was the Cogent path which I saw packet loss/disruption on). This was observed from a non-Cogent multihomed network in Reston, VA. Started at at 11:54 Eastern and ended about 20 minutes later.
Replying to myself, here is the path on which I saw disruption (no problems now/at the time of the mtr -r below): ryan@vicious:~$ mtr -r -w rpi.lily.org HOST: vicious Loss% Snt Last Avg Best Wrst StDev 1. 192.168.152.199 0.0% 10 0.3 0.2 0.2 0.3 0.0 2. 67.23.118.177 0.0% 10 0.8 0.8 0.6 1.0 0.1 3. te0-1-0-0.nvarsx-COR00.yellowfiber.net 0.0% 10 1.5 1.4 1.2 1.7 0.2 4. te0-0-0-1.nvarsx-cor01.yellowfiber.net 0.0% 10 1.3 1.4 1.2 1.8 0.2 5. te0-0-0-2.nvaiad-cor00.yellowfiber.net 0.0% 10 1.5 1.4 1.2 1.7 0.2 6. xe-9-0-6.ar1.iad1.us.nlayer.net 0.0% 10 4.7 3.3 1.9 5.2 1.2 7. ae0-30g.ar2.iad1.us.nlayer.net 0.0% 10 3.9 3.7 2.4 7.5 1.7 8. 144.228.205.97 0.0% 10 1.7 3.2 1.2 17.3 5.0 9. sl-st31-ash-0-8-0-2.sprintlink.net 0.0% 10 2.5 2.5 1.7 5.7 1.1 10. 144.232.8.14 0.0% 10 1.7 1.8 1.6 2.1 0.2 11. te0-2-0-5.mpd22.dca01.atlas.cogentco.com 0.0% 10 3.0 2.8 2.5 3.1 0.2 12. te0-3-0-4.mpd22.ord01.atlas.cogentco.com 0.0% 10 21.7 21.5 21.1 21.7 0.2 13. te0-6-0-17.mpd22.jfk02.atlas.cogentco.com 0.0% 10 47.4 48.2 47.1 49.6 0.9 14. te3-3.ccr01.jfk01.atlas.cogentco.com 0.0% 10 46.6 44.3 40.0 48.7 3.7 15. te7-1.mag02.jfk01.atlas.cogentco.com 0.0% 10 47.3 47.7 46.1 49.4 1.0 16. 38.122.9.54 0.0% 10 51.6 50.7 46.6 52.5 1.6 17. vccfr7-39-33.net.rpi.edu 10.0% 10 50.8 53.6 48.1 85.2 11.9 18. ??? 100.0 10 0.0 0.0 0.0 0.0 0.0 ryan@vicious:~$
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Where do you peer? Frank -----Original Message----- From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On Behalf Of Matthew Huff Sent: Thursday, December 06, 2012 11:12 AM To: 'Outages@outages.org' Subject: [outages] Cogent outage ??? About 10 minutes ago we stopped being able to pass traffic through cogent. I de-peered us from Cogent, and everything appears better. When I call cogent, all I get is a busy signal (must be a major outage). Anyone else seeing anything? ---- Matthew Huff | 1 Manhattanville Rd Director of Operations | Purchase, NY 10577 OTA Management LLC | Phone: 914-460-4039 aim: matthewbhuff | Fax: 914-460-4139
Looks like cogent is having peering issues, see the picture from the internet health report. http://www.internetpulse.net/ On Thu, Dec 6, 2012 at 12:17 PM, Frank Bulk <frnkblk@iname.com> wrote:
Where do you peer?
Frank
-----Original Message----- From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On Behalf Of Matthew Huff Sent: Thursday, December 06, 2012 11:12 AM To: 'Outages@outages.org' Subject: [outages] Cogent outage ???
About 10 minutes ago we stopped being able to pass traffic through cogent. I de-peered us from Cogent, and everything appears better. When I call cogent, all I get is a busy signal (must be a major outage). Anyone else seeing anything?
---- Matthew Huff | 1 Manhattanville Rd Director of Operations | Purchase, NY 10577 OTA Management LLC | Phone: 914-460-4039 aim: matthewbhuff | Fax: 914-460-4139
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Quote from CogentCo; haven't verified yet. You were impacted by a mass outage in the New York/New Jersey area. We are now seeing this issue restored if you continue to have any issues please contact our support department 877-726-4368 opt2. From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On Behalf Of Mitch Sent: Thursday, December 06, 2012 12:20 PM To: Frank Bulk Cc: Outages@outages.org Subject: Re: [outages] Cogent outage ??? Looks like cogent is having peering issues, see the picture from the internet health report. http://www.internetpulse.net/ On Thu, Dec 6, 2012 at 12:17 PM, Frank Bulk <frnkblk@iname.com<mailto:frnkblk@iname.com>> wrote: Where do you peer? Frank -----Original Message----- From: outages-bounces@outages.org<mailto:outages-bounces@outages.org> [mailto:outages-bounces@outages.org<mailto:outages-bounces@outages.org>] On Behalf Of Matthew Huff Sent: Thursday, December 06, 2012 11:12 AM To: 'Outages@outages.org<mailto:Outages@outages.org>' Subject: [outages] Cogent outage ??? About 10 minutes ago we stopped being able to pass traffic through cogent. I de-peered us from Cogent, and everything appears better. When I call cogent, all I get is a busy signal (must be a major outage). Anyone else seeing anything? ---- Matthew Huff | 1 Manhattanville Rd Director of Operations | Purchase, NY 10577 OTA Management LLC | Phone: 914-460-4039<tel:914-460-4039> aim: matthewbhuff | Fax: 914-460-4139<tel:914-460-4139> _______________________________________________ Outages mailing list Outages@outages.org<mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages ---------------------------------------------------------------------- CONFIDENTIALITY NOTICE: This e-mail transmission may contain confidential information. This information is solely for the use of the individual(s) or entity to whom or which it was intended. If not an intended recipient, any review, copying, printing, disclosure, distribution or any other use is strictly prohibited. If you have received this email in error, please immediately notify the sender by reply e-mail. Please delete this e-mail from your files if you are not the intended recipient. Thank you.
I am seeing issues between JFK and Dallas Had to reroute traffic over L3 SW1.AMS1#traceroute 199.30.108.0 Type escape sequence to abort. Tracing the route to bb1.phx1.umedia.com (199.30.108.0) 1 37.140.232.17 0 msec 8 msec 4 msec 2 37.140.232.1 8 msec 0 msec 0 msec 3 vl315.mag01.ams03.atlas.cogentco.com (149.6.128.189) 4 msec 0 msec 0 msec 4 te0-3-0-4.mpd21.ams03.atlas.cogentco.com (130.117.48.61) 4 msec 0 msec 4 msec 5 te0-4-0-2.ccr21.lpl01.atlas.cogentco.com (154.54.58.62) 24 msec te0-0-0-7.ccr21.lpl01.atlas.cogentco.com (154.54.37.74) 12 msec te0-4-0-2.ccr21.lpl01.atlas.cogentco.com (154.54.58.62) 8 msec 6 te0-3-0-3.ccr21.bos01.atlas.cogentco.com (154.54.31.229) 76 msec te0-1-0-3.ccr21.bos01.atlas.cogentco.com (154.54.31.181) 92 msec te0-4-0-2.ccr21.bos01.atlas.cogentco.com (154.54.44.197) 76 msec 7 te0-3-0-2.mpd21.jfk02.atlas.cogentco.com (154.54.44.13) 80 msec te0-0-0-11.mpd21.jfk02.atlas.cogentco.com (154.54.44.17) 80 msec te0-0-0-18.mpd21.jfk02.atlas.cogentco.com (154.54.44.5) 80 msec 8 * * * 9 * * * 10 * te0-1-0-6.ccr22.dfw01.atlas.cogentco.com (154.54.25.205) 124 msec te0-0-0-5.ccr21.dfw01.atlas.cogentco.com (154.54.5.217) 148 msec 11 te0-1-0-1.mpd22.iah01.atlas.cogentco.com (154.54.5.130) 148 msec te0-0-0-2.mpd21.iah01.atlas.cogentco.com (154.54.5.134) 132 msec te0-0-0-2.ccr21.iah01.atlas.cogentco.com (154.54.25.218) 132 msec 12 te4-3.ccr01.sat01.atlas.cogentco.com (154.54.46.6) 212 msec te8-1.ccr01.sat01.atlas.cogentco.com (154.54.29.38) 256 msec 240 msec 13 te7-1.ccr01.phx02.atlas.cogentco.com (154.54.6.249) 156 msec te8-2.ccr01.phx02.atlas.cogentco.com (154.54.83.62) 152 msec 156 msec 14 te4-1.ccr01.phx05.atlas.cogentco.com (154.54.84.226) 160 msec te3-1.ccr01.phx05.atlas.cogentco.com (154.54.84.222) 156 msec te4-1.ccr01.phx05.atlas.cogentco.com (154.54.84.226) 156 msec 15 * * * 16 * * * 17 * * * 18 * * * 19 * * * 20 * * * 21 * * * 22 * * * 23 * * * ============= Best Regards Raman Sud Sr. VP of Engineering Mojohost Toll Free: 888-345-MOJO x808 Cell: +1 480.206.0211 PGP Key: http://pgp.mit.edu:11371/pks/lookup?op=get&search=0x26BDA015F2DCF135 THIS MESSAGE IS INTENDED ONLY FOR THE USE OF THE INDIVIDUAL OR ENTITY TO WHICH IT IS ADDRESSED, AND MAY CONTAIN INFORMATION THAT IS PRIVILEGED, CONFIDENTIAL AND EXEMPT FROM DISCLOSURE UNDER APPLICABLE LAW. IF THE READER OF THIS MESSAGE IS NOT THE INTENDED RECIPIENT, YOU ARE HEREBY NOTIFIED THAT ANY DISSEMINATION, DISTRIBUTION OR COPYING OF THIS COMMUNICATION IS STRICTLY PROHIBITED, AND THAT THIS EMAIL AND ALL COPIES AND ATTACHMENTS SHOULD BE IMMEDIATELY DESTROYED. ----- Original Message ----- From: "Frank Bulk" <frnkblk@iname.com> To: "Matthew Huff" <mhuff@ox.com>, Outages@outages.org Sent: Thursday, December 6, 2012 10:17:08 AM Subject: Re: [outages] Cogent outage ??? Where do you peer? Frank -----Original Message----- From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On Behalf Of Matthew Huff Sent: Thursday, December 06, 2012 11:12 AM To: 'Outages@outages.org' Subject: [outages] Cogent outage ??? About 10 minutes ago we stopped being able to pass traffic through cogent. I de-peered us from Cogent, and everything appears better. When I call cogent, all I get is a busy signal (must be a major outage). Anyone else seeing anything? ---- Matthew Huff | 1 Manhattanville Rd Director of Operations | Purchase, NY 10577 OTA Management LLC | Phone: 914-460-4039 aim: matthewbhuff | Fax: 914-460-4139 _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Seems like they re-routed the traffic now SW1.AMS1#traceroute 199.30.108.0 Type escape sequence to abort. Tracing the route to bb1.phx1.umedia.com (199.30.108.0) 1 37.140.232.17 0 msec 0 msec 0 msec 2 37.140.232.1 0 msec 0 msec 0 msec 3 vl315.mag01.ams03.atlas.cogentco.com (149.6.128.189) 0 msec 4 msec 0 msec 4 te0-3-0-4.mpd21.ams03.atlas.cogentco.com (130.117.48.61) 4 msec 4 msec 0 msec 5 te0-1-0-2.ccr21.lpl01.atlas.cogentco.com (154.54.37.78) 12 msec te0-1-0-7.ccr22.lpl01.atlas.cogentco.com (154.54.37.114) 12 msec te0-1-0-7.ccr21.lpl01.atlas.cogentco.com (154.54.37.82) 12 msec 6 te0-4-0-4.ccr21.ymq02.atlas.cogentco.com (154.54.44.201) 80 msec te0-1-0-4.ccr22.ymq02.atlas.cogentco.com (66.28.4.21) 80 msec 84 msec 7 te0-2-0-7.ccr22.yyz02.atlas.cogentco.com (154.54.42.229) 88 msec te0-3-0-1.ccr22.yyz02.atlas.cogentco.com (154.54.44.110) 92 msec te0-3-0-5.ccr21.yyz02.atlas.cogentco.com (154.54.43.221) 92 msec 8 te0-3-0-3.ccr21.ord01.atlas.cogentco.com (66.28.4.213) 116 msec te0-1-0-3.mpd21.ord01.atlas.cogentco.com (154.54.42.9) 104 msec te0-2-0-6.mpd22.ord01.atlas.cogentco.com (66.28.4.57) 104 msec 9 te0-3-0-0.ccr22.mci01.atlas.cogentco.com (154.54.6.201) 116 msec te0-1-0-2.mpd22.mci01.atlas.cogentco.com (154.54.3.202) 112 msec te0-1-0-3.ccr21.mci01.atlas.cogentco.com (154.54.25.81) 116 msec 10 te0-3-0-6.ccr21.dfw01.atlas.cogentco.com (154.54.46.182) 124 msec te0-4-0-5.ccr21.dfw01.atlas.cogentco.com (154.54.46.198) 128 msec 128 msec 11 te0-2-0-2.ccr22.iah01.atlas.cogentco.com (154.54.25.214) 132 msec te0-0-0-2.mpd21.iah01.atlas.cogentco.com (154.54.5.134) 128 msec te0-2-0-2.ccr21.iah01.atlas.cogentco.com (154.54.1.74) 128 msec 12 te3-1.ccr01.sat01.atlas.cogentco.com (154.54.27.114) 136 msec te4-3.ccr01.sat01.atlas.cogentco.com (154.54.46.6) 136 msec te8-1.ccr01.sat01.atlas.cogentco.com (154.54.29.38) 136 msec 13 te8-1.ccr01.phx02.atlas.cogentco.com (154.54.80.209) 156 msec 152 msec te7-1.ccr01.phx02.atlas.cogentco.com (154.54.6.249) 152 msec 14 te4-1.ccr01.phx05.atlas.cogentco.com (154.54.84.226) 156 msec 156 msec te3-1.ccr01.phx05.atlas.cogentco.com (154.54.84.222) 156 msec 15 * * * 16 * * ============= Best Regards Raman Sud Sr. VP of Engineering Mojohost Toll Free: 888-345-MOJO x808 Cell: +1 480.206.0211 PGP Key: http://pgp.mit.edu:11371/pks/lookup?op=get&search=0x26BDA015F2DCF135 THIS MESSAGE IS INTENDED ONLY FOR THE USE OF THE INDIVIDUAL OR ENTITY TO WHICH IT IS ADDRESSED, AND MAY CONTAIN INFORMATION THAT IS PRIVILEGED, CONFIDENTIAL AND EXEMPT FROM DISCLOSURE UNDER APPLICABLE LAW. IF THE READER OF THIS MESSAGE IS NOT THE INTENDED RECIPIENT, YOU ARE HEREBY NOTIFIED THAT ANY DISSEMINATION, DISTRIBUTION OR COPYING OF THIS COMMUNICATION IS STRICTLY PROHIBITED, AND THAT THIS EMAIL AND ALL COPIES AND ATTACHMENTS SHOULD BE IMMEDIATELY DESTROYED. ----- Original Message ----- From: "Raman Sud" <raman@mojohost.com> To: "Frank Bulk" <frnkblk@iname.com> Cc: Outages@outages.org Sent: Thursday, December 6, 2012 10:20:26 AM Subject: Re: [outages] Cogent outage ??? I am seeing issues between JFK and Dallas Had to reroute traffic over L3 SW1.AMS1#traceroute 199.30.108.0 Type escape sequence to abort. Tracing the route to bb1.phx1.umedia.com (199.30.108.0) 1 37.140.232.17 0 msec 8 msec 4 msec 2 37.140.232.1 8 msec 0 msec 0 msec 3 vl315.mag01.ams03.atlas.cogentco.com (149.6.128.189) 4 msec 0 msec 0 msec 4 te0-3-0-4.mpd21.ams03.atlas.cogentco.com (130.117.48.61) 4 msec 0 msec 4 msec 5 te0-4-0-2.ccr21.lpl01.atlas.cogentco.com (154.54.58.62) 24 msec te0-0-0-7.ccr21.lpl01.atlas.cogentco.com (154.54.37.74) 12 msec te0-4-0-2.ccr21.lpl01.atlas.cogentco.com (154.54.58.62) 8 msec 6 te0-3-0-3.ccr21.bos01.atlas.cogentco.com (154.54.31.229) 76 msec te0-1-0-3.ccr21.bos01.atlas.cogentco.com (154.54.31.181) 92 msec te0-4-0-2.ccr21.bos01.atlas.cogentco.com (154.54.44.197) 76 msec 7 te0-3-0-2.mpd21.jfk02.atlas.cogentco.com (154.54.44.13) 80 msec te0-0-0-11.mpd21.jfk02.atlas.cogentco.com (154.54.44.17) 80 msec te0-0-0-18.mpd21.jfk02.atlas.cogentco.com (154.54.44.5) 80 msec 8 * * * 9 * * * 10 * te0-1-0-6.ccr22.dfw01.atlas.cogentco.com (154.54.25.205) 124 msec te0-0-0-5.ccr21.dfw01.atlas.cogentco.com (154.54.5.217) 148 msec 11 te0-1-0-1.mpd22.iah01.atlas.cogentco.com (154.54.5.130) 148 msec te0-0-0-2.mpd21.iah01.atlas.cogentco.com (154.54.5.134) 132 msec te0-0-0-2.ccr21.iah01.atlas.cogentco.com (154.54.25.218) 132 msec 12 te4-3.ccr01.sat01.atlas.cogentco.com (154.54.46.6) 212 msec te8-1.ccr01.sat01.atlas.cogentco.com (154.54.29.38) 256 msec 240 msec 13 te7-1.ccr01.phx02.atlas.cogentco.com (154.54.6.249) 156 msec te8-2.ccr01.phx02.atlas.cogentco.com (154.54.83.62) 152 msec 156 msec 14 te4-1.ccr01.phx05.atlas.cogentco.com (154.54.84.226) 160 msec te3-1.ccr01.phx05.atlas.cogentco.com (154.54.84.222) 156 msec te4-1.ccr01.phx05.atlas.cogentco.com (154.54.84.226) 156 msec 15 * * * 16 * * * 17 * * * 18 * * * 19 * * * 20 * * * 21 * * * 22 * * * 23 * * * ============= Best Regards Raman Sud Sr. VP of Engineering Mojohost Toll Free: 888-345-MOJO x808 Cell: +1 480.206.0211 PGP Key: http://pgp.mit.edu:11371/pks/lookup?op=get&search=0x26BDA015F2DCF135 THIS MESSAGE IS INTENDED ONLY FOR THE USE OF THE INDIVIDUAL OR ENTITY TO WHICH IT IS ADDRESSED, AND MAY CONTAIN INFORMATION THAT IS PRIVILEGED, CONFIDENTIAL AND EXEMPT FROM DISCLOSURE UNDER APPLICABLE LAW. IF THE READER OF THIS MESSAGE IS NOT THE INTENDED RECIPIENT, YOU ARE HEREBY NOTIFIED THAT ANY DISSEMINATION, DISTRIBUTION OR COPYING OF THIS COMMUNICATION IS STRICTLY PROHIBITED, AND THAT THIS EMAIL AND ALL COPIES AND ATTACHMENTS SHOULD BE IMMEDIATELY DESTROYED. ----- Original Message ----- From: "Frank Bulk" <frnkblk@iname.com> To: "Matthew Huff" <mhuff@ox.com>, Outages@outages.org Sent: Thursday, December 6, 2012 10:17:08 AM Subject: Re: [outages] Cogent outage ??? Where do you peer? Frank -----Original Message----- From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On Behalf Of Matthew Huff Sent: Thursday, December 06, 2012 11:12 AM To: 'Outages@outages.org' Subject: [outages] Cogent outage ??? About 10 minutes ago we stopped being able to pass traffic through cogent. I de-peered us from Cogent, and everything appears better. When I call cogent, all I get is a busy signal (must be a major outage). Anyone else seeing anything? ---- Matthew Huff | 1 Manhattanville Rd Director of Operations | Purchase, NY 10577 OTA Management LLC | Phone: 914-460-4039 aim: matthewbhuff | Fax: 914-460-4139 _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
On 12/06/2012 10:20 AM, Raman Sud wrote:
I am seeing issues between JFK and Dallas
Had to reroute traffic over L3
SW1.AMS1#traceroute 199.30.108.0
Seeing the same out of Denver - but looks like it's dying in Phoenix: [root@watcher ~]# mtr --report --report-cycles=10 --report-wide 199.30.108.0 HOST: watcher.axint.net Loss% Snt Last Avg Best Wrst StDev 1. ai-core.axint.net 0.0% 10 0.4 0.7 0.4 0.9 0.2 2. fa0-2.na01.b009854-0.den01.atlas.cogentco.com 0.0% 10 0.8 1.1 0.8 1.7 0.3 3. vl3509.mag02.den01.atlas.cogentco.com 0.0% 10 0.8 0.9 0.6 1.5 0.3 4. te7-4.ccr01.den01.atlas.cogentco.com 0.0% 10 1.0 28.9 0.7 128.3 50.8 5. te3-4.ccr01.elp01.atlas.cogentco.com 0.0% 10 14.4 28.3 13.9 97.3 29.8 6. te4-1.ccr01.phx02.atlas.cogentco.com 0.0% 10 22.6 55.7 22.4 159.7 52.0 7. te3-1.ccr01.phx05.atlas.cogentco.com 0.0% 10 23.6 23.9 23.5 25.3 0.5 8. ??? 100.0 10 0.0 0.0 0.0 0.0 0.0 Chris -- Chris Stone, MCSE Vice President, CTO AxisInternet, Inc. http://www.axint.net DSL, VoIP, hosting, email filtering, co-location, online backup & more **************************************************************** Authorized ScopTel IP PBX US Reseller - www.scopserv.com The most complete and comprehensive GUI on the market for telephony server management interface based on Voice over Internet Protocol (VoIP) ****************************************************************
In Westchester Co, NY (just north of NYC). There was an earlier report of an explosion at 75 Broad Street in NYC in the nanog mailing list. ---- Matthew Huff | 1 Manhattanville Rd Director of Operations | Purchase, NY 10577 OTA Management LLC | Phone: 914-460-4039 aim: matthewbhuff | Fax: 914-460-4139
-----Original Message----- From: Frank Bulk [mailto:frnkblk@iname.com] Sent: Thursday, December 06, 2012 12:17 PM To: Matthew Huff; Outages@outages.org Subject: RE: Cogent outage ???
Where do you peer?
Frank
-----Original Message----- From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On Behalf Of Matthew Huff Sent: Thursday, December 06, 2012 11:12 AM To: 'Outages@outages.org' Subject: [outages] Cogent outage ???
About 10 minutes ago we stopped being able to pass traffic through cogent. I de-peered us from Cogent, and everything appears better. When I call cogent, all I get is a busy signal (must be a major outage). Anyone else seeing anything?
---- Matthew Huff | 1 Manhattanville Rd Director of Operations | Purchase, NY 10577 OTA Management LLC | Phone: 914-460-4039 aim: matthewbhuff | Fax: 914-460-4139
Our L2 stuff in Atlanta and Dallas looks fine, but we don't have any L3 service from them. I also get a busy signal if I call their 877-7cogent number or their local DC number. ------------------ Aubrey Wells Director | Network Services VocalCloud 678.248.2637 support@vocalcloud.com www.vocalcloud.com On Thu, Dec 6, 2012 at 12:12 PM, Matthew Huff <mhuff@ox.com> wrote:
About 10 minutes ago we stopped being able to pass traffic through cogent. I de-peered us from Cogent, and everything appears better. When I call cogent, all I get is a busy signal (must be a major outage). Anyone else seeing anything?
---- Matthew Huff | 1 Manhattanville Rd Director of Operations | Purchase, NY 10577 OTA Management LLC | Phone: 914-460-4039 aim: matthewbhuff | Fax: 914-460-4139
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Most recent post on their status page( http://status.cogentco.com/) *Network Status Description* Welcome to the Cogent Communications status page. Customers located in the New York City area may be experiencing an outage. This is currently being investigated by the Cogent NOC. There is no Estimated time to repair. The master case is HD5267626. For customers hosted and provisioned through our New York hosting center at 33 Whitehall: Power is being supplied by generator and there is No Estimated Time of Repair for commercial power to be restored to the building. We will update our Network status page if there is any change. On Thu, Dec 6, 2012 at 12:17 PM, Aubrey Wells <aubrey@vocalcloud.com> wrote:
Our L2 stuff in Atlanta and Dallas looks fine, but we don't have any L3 service from them. I also get a busy signal if I call their 877-7cogent number or their local DC number.
------------------ Aubrey Wells Director | Network Services VocalCloud 678.248.2637 support@vocalcloud.com www.vocalcloud.com
On Thu, Dec 6, 2012 at 12:12 PM, Matthew Huff <mhuff@ox.com> wrote:
About 10 minutes ago we stopped being able to pass traffic through cogent. I de-peered us from Cogent, and everything appears better. When I call cogent, all I get is a busy signal (must be a major outage). Anyone else seeing anything?
---- Matthew Huff | 1 Manhattanville Rd Director of Operations | Purchase, NY 10577 OTA Management LLC | Phone: 914-460-4039 aim: matthewbhuff | Fax: 914-460-4139
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Same. Cogent link died, had BGP, but couldn't transport. Western Mass. ~Jeff Agle Network Engineer Farm Credit Financial Partners, Inc. Office: (413) 271-8787 Jeff.Agle@FinancialPartners.com -----Original Message----- From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On Behalf Of Matthew Huff Sent: Thursday, December 06, 2012 12:12 PM To: 'Outages@outages.org' Subject: [outages] Cogent outage ??? About 10 minutes ago we stopped being able to pass traffic through cogent. I de-peered us from Cogent, and everything appears better. When I call cogent, all I get is a busy signal (must be a major outage). Anyone else seeing anything? ---- Matthew Huff | 1 Manhattanville Rd Director of Operations | Purchase, NY 10577 OTA Management LLC | Phone: 914-460-4039 aim: matthewbhuff | Fax: 914-460-4139 ---------------------------------------------------------------------- CONFIDENTIALITY NOTICE: This e-mail transmission may contain confidential information. This information is solely for the use of the individual(s) or entity to whom or which it was intended. If not an intended recipient, any review, copying, printing, disclosure, distribution or any other use is strictly prohibited. If you have received this email in error, please immediately notify the sender by reply e-mail. Please delete this e-mail from your files if you are not the intended recipient. Thank you.
Seeing L2 fine, had to shut L3 peering. Points are Marietta and PHX On Thu, Dec 6, 2012 at 11:18 AM, Agle, Jeff <jeff.agle@financialpartners.com
wrote:
Same. Cogent link died, had BGP, but couldn't transport. Western Mass.
~Jeff Agle Network Engineer Farm Credit Financial Partners, Inc. Office: (413) 271-8787 Jeff.Agle@FinancialPartners.com
-----Original Message----- From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On Behalf Of Matthew Huff Sent: Thursday, December 06, 2012 12:12 PM To: 'Outages@outages.org' Subject: [outages] Cogent outage ???
About 10 minutes ago we stopped being able to pass traffic through cogent. I de-peered us from Cogent, and everything appears better. When I call cogent, all I get is a busy signal (must be a major outage). Anyone else seeing anything?
---- Matthew Huff | 1 Manhattanville Rd Director of Operations | Purchase, NY 10577 OTA Management LLC | Phone: 914-460-4039 aim: matthewbhuff | Fax: 914-460-4139
---------------------------------------------------------------------- CONFIDENTIALITY NOTICE: This e-mail transmission may contain confidential information. This information is solely for the use of the individual(s) or entity to whom or which it was intended. If not an intended recipient, any review, copying, printing, disclosure, distribution or any other use is strictly prohibited. If you have received this email in error, please immediately notify the sender by reply e-mail. Please delete this e-mail from your files if you are not the intended recipient. Thank you.
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
Hi, On Thu, Dec 06, 2012 at 12:18:31PM -0500, Agle, Jeff wrote:
Same. Cogent link died, had BGP, but couldn't transport. Western Mass.
I wish people would learn to build BGP networks that die properly - as in "if you can't forward my packets, don't claim you do". Damn MPLS+RR text book setups, plus lack of proper monitoring. *sigh* gert -- USENET is *not* the non-clickable part of WWW! //www.muc.de/~gert/ Gert Doering - Munich, Germany gert@greenie.muc.de fax: +49-89-35655025 gert@net.informatik.tu-muenchen.de
Yes, we are seeing this in NYC. From status.cogentco.com: *************************************************************************************************************** ** Cogent Network Status Report Last Updated Thurs Dec 06 17:00:49 2012 GMT ** *************************************************************************************************************** *************************************************************************************************************** Network Status: Normal DNS Status: Normal Network Status Description Welcome to the Cogent Communications status page. Customers located in the New York City area may be experiencing an outage. This is currently being investigated by the Cogent NOC. There is no Estimated time to repair. The master case is HD5267626. For customers hosted and provisioned through our New York hosting center at 33 Whitehall: Power is being supplied by generator and there is No Estimated Time of Repair for commercial power to be restored to the building. We will update our Network status page if there is any change. On Wednesday, December 5 - Limited access to the hosting center will be available from 12PM to 5PM To request site access you must send email to support@cogentco.com with a subject of "33 Whitehall Access Request". Building occupancy is limited to five individuals per escort interval, so only one representative per company at a time is allowed. This person will be allowed access for one hour. Customers will be assigned an access slot based on the order in which their email request was received. Customers will be notified via email of their assigned time. To request a server reboot you must send email to support@cogentco.com with a subject of "33 Whitehall Server Reboot Request" *************************************************************************************************************** Regards, Adam Durity Aetos Capital adurity@aetoscapital.com 650-234-1868 -----Original Message----- From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On Behalf Of Matthew Huff Sent: Thursday, December 6, 2012 9:12 AM To: 'Outages@outages.org' Subject: [outages] Cogent outage ??? About 10 minutes ago we stopped being able to pass traffic through cogent. I de-peered us from Cogent, and everything appears better. When I call cogent, all I get is a busy signal (must be a major outage). Anyone else seeing anything? ---- Matthew Huff | 1 Manhattanville Rd Director of Operations | Purchase, NY 10577 OTA Management LLC | Phone: 914-460-4039 aim: matthewbhuff | Fax: 914-460-4139 ________________________________ DISCLAIMER: This communication and any files transmitted with it are for informational purposes only and solely for the use of the individual and entity to whom they are addressed. This email may contain information that is privileged, confidential, legally privileged and/or exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution or the use of the information contained herein (including any reliance thereon) is strictly prohibited. If you received this email in error, please notify the sender immediately and delete it from your system.
Our gear in their Chicago POP is available, but I'm getting a busy signal as well when calling in. On Dec 6, 2012 11:16 AM, "Matthew Huff" <mhuff@ox.com> wrote:
About 10 minutes ago we stopped being able to pass traffic through cogent. I de-peered us from Cogent, and everything appears better. When I call cogent, all I get is a busy signal (must be a major outage). Anyone else seeing anything?
---- Matthew Huff | 1 Manhattanville Rd Director of Operations | Purchase, NY 10577 OTA Management LLC | Phone: 914-460-4039 aim: matthewbhuff | Fax: 914-460-4139
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
We had problems reaching addresses while going through te0-1-0-5.ccr22.atl01.atlas.cogentco.com about 10-20 minutes ago. I had a traceroute but I closed it :-) This is all the information I have today. It could have been congestion related (one of my network engineers explained this to me ;) rather than an equipment issue. The issue for us seems to have resolved itself and everything is back to normal. On Thu, Dec 6, 2012 at 12:12 PM, Matthew Huff <mhuff@ox.com> wrote:
About 10 minutes ago we stopped being able to pass traffic through cogent. I de-peered us from Cogent, and everything appears better. When I call cogent, all I get is a busy signal (must be a major outage). Anyone else seeing anything?
---- Matthew Huff | 1 Manhattanville Rd Director of Operations | Purchase, NY 10577 OTA Management LLC | Phone: 914-460-4039 aim: matthewbhuff | Fax: 914-460-4139
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
participants (14)
-
Adam Durity -
Agle, Jeff -
Aubrey Wells -
Avleen Vig -
Blake Dunlap -
Brandon Galbraith -
Chris Stone -
Frank Bulk -
Gert Doering -
Julian Y Koh -
Matthew Huff -
Mitch -
Raman Sud -
Ryan Rawdon