January 2013 - Outages - Outages Lists

Fwd: inetwork EVS Bandwidth Outage 1/18/2013
by Chris Stone 18 Jan '13

18 Jan '13

-------- Original Message -------- Subject: inetwork EVS Bandwidth Outage 1/18/2013 Date: Fri, 18 Jan 2013 12:50:25 -0500 (EST) From: Bandwidth <notification(a)bandwidth.com> Reply-To: notification(a)bandwidth.com *Bandwidth* *Impairment Notice*** Good Day, EVS Bandwidth experienced a network outage Friday, January 18, 2013. Our engineering teams are currently on-site working towards resolution. Network impairments continue to be cleared and are normalizing. We will continue to update on progress as it develops, in addition to maintaining an active watch over the network. If you are experiencing any call issues, please email carrieroutage(a)inetwork.com <mailto:support@inetwork.com> with current examples if available. Thank You, *Bandwidth Support* support(a)inetwork.com <mailto:support@inetwork.com> 855-864-7776 (855-VoIP-PRO)

1 0

Re: [outages] bandwidth.com voip outage
by Chris Stone 18 Jan '13

18 Jan '13

Update from bandwidth.com on our ticket and this issue: ===================================================================================== *Jan 18 11:15 am (EST):* Hi Chris, We are reviewing our network. Assigning ticket to our Tier2 Team for assistance and they will keep you updated as information becomes available. *Jan 18 11:18 am (EST):* We have confirmed with engineering that they are currently reviewing the network. I have no additional details at this time, however updates will be provided as they become available. *Jan 18 11:31 am (EST):* Good Day, Our engineering group has confirmed that service has been retored to the network and calls are normalizing. We are continuing to monitor progress, and will continue to provide updates as they become available. ===================================================================================== Our calls now seem to all be completing, but still seeing some packetloss and some of the audio is a bit choppy. Better, but service not fully restored at this point. Chris -- Chris Stone, MCSE Vice President, CTO AxisInternet, Inc. http://www.axint.net DSL, VoIP, hosting, email filtering, co-location, online backup & more Phone: +1 303 592 2947 x302 (office) +1 303 570 6947 (cell) **************************************************************** Authorized ScopTel IP PBX US Reseller - www.scopserv.com The most complete and comprehensive GUI on the market for telephony server management interface based on Voice over Internet Protocol (VoIP) ****************************************************************

1 0

bandwidth.com voip outage
by Chris Stone 18 Jan '13

18 Jan '13

Lots of packetloss to their servers and most, if not all calls fail. Just got off the phone with them - they are aware and working on the problem they say. They have not created a master ticket created as of yet. Chris -- Chris Stone, MCSE Vice President, CTO AxisInternet, Inc. http://www.axint.net DSL, VoIP, hosting, email filtering, co-location, online backup & more Phone: +1 303 592 2947 x302 (office) +1 303 570 6947 (cell) **************************************************************** Authorized ScopTel IP PBX US Reseller - www.scopserv.com The most complete and comprehensive GUI on the market for telephony server management interface based on Voice over Internet Protocol (VoIP) ****************************************************************

1 0

Charter Peering issues??(At least ICMP)
by Mitch 18 Jan '13

18 Jan '13

Anyone else having any issues reaching IP's on Charter communications? We have 3 sites down from our monitoring tool(peered via Level 3) and trace routes from level3's looking glass fail. Their looking glass also shows no routers to the /4 that our IP is part of. I've noticed issues from Centurylink's Looking glass as well on certain POP's and from Cogent as well. Traceroutes: I had to remove our office IP's for privacy but I think it gets the point across that we can't reach both ways. Oddly enough SSH seems to work sometimes so I wonder if its mostly an ICMP issue with occasional drops of other traffic. Office is in Ohio, IP/other route is in Tennessee but we also have it down in South and North Carolina. >From our office to the IP in question: Tracing route to 75-137-33-21.static.jcsn.tn.charter.com [75.137.33.21] over a maximum of 30 hops: 1 1 ms 1 ms 1 ms 172.23.0.1 2 34 ms 35 ms 35 ms <Scrubbed> 3 176 ms 203 ms 203 ms <Scrubbed> 4 22 ms 22 ms * vlan213.car1.Detroit1.Level3.net [4.53.76.5] 5 35 ms 35 ms 34 ms ae-8-8.ebr2.Chicago1.Level3.net[4.69.133.242] 6 37 ms 37 ms 37 ms ae-23-52.car3.chicago1.level3.net[4.69.138.37] 7 79 ms 44 ms 41 ms 4.68.127.138 8 84 ms 71 ms 71 ms nyk-bb1-link.telia.net [213.155.131.240] 9 94 ms 58 ms 58 ms ash-bb3-link.telia.net [213.155.134.149] 10 * * * Request timed out. 11 * * * Request timed out. 12 * * * Request timed out. 13 * * * Request timed out. 14 * * * Request timed out. 15 * * * Request timed out. 16 * * * Request timed out. 17 * * * Request timed out. 18 * * * Request timed out. 19 * * * Request timed out. 20 * * * Request timed out. 21 * * * Request timed out. 22 * * * Request timed out. 23 * * * Request timed out. 24 * * * Request timed out. 25 * * * Request timed out. 26 * * * Request timed out. 27 * * * Request timed out. 28 * * * Request timed out. 29 * * * Request timed out. 30 * * * Request timed out. Traceroutes from our router back to our office:(got into it via DSL link) 1 * * * 2 * * * 3 * * * 4 * * * Traceroutes from our router to 8.8.8.8: Tracing route to 8.8.8.8 from 75.137.33.22 over a maximum of 30 hops 1 5ms 3ms 5ms 75.137.33.21 2 14ms 9ms 12ms 10.214.212.1 3 8ms 10ms 12ms 96.34.70.4 4 11ms 27ms 11ms 96.34.71.178 5 18ms 26ms 20ms 96.34.79.74 6 18ms 22ms 18ms 96.34.79.62 7 35ms 32ms 31ms 96.34.78.18 8 44ms 35ms 35ms 96.34.2.70 9 33ms 31ms 34ms 96.34.3.17 10 31ms 33ms 33ms 96.34.150.46 11 32ms 31ms 32ms 72.14.239.100 12 33ms 34ms 32ms 66.249.94.24 13 34ms 34ms 49ms 209.85.248.57 14 * * * 15 34ms 32ms 31ms 8.8.8.8

2 1

[ADMIN] WIKI.OUTAGES.ORG Emergency Planned Maintenance (1/17/2013) from 20:00PST to 23:00PST
by virendra rode 17 Jan '13

17 Jan '13

Type of Work: Flushing old DB cache and restore. mysqldump to a backup server, delete cache files, re-import dump and restart mysql. Purpose: To address disk space issue. Start Date / Time: 1/17/2013 (Thursday) 20:00 PST End Date / Time: 1/17/2013 (Thursday) 23:00 PST Should additional time be required, notice will be provided and the work window will be expanded. Estimate Downtime: wiki.outages.org will be down 2 hours Service Impact: wiki.outages.org will not be available during this change window. Service will return back to normal after this upgrade. Steps for this Maintenance: ------------------------- 1. Backup DB (mysqldump) 2. Delete DB cache files. 3. Re-import dump file & restart mysql. 4. Lock down DB Regression Planning: There is a possibility that something unexpected may occur during the work process. Should any issues arise, all equipment will be placed back into standard operation and the work will be postponed until the issue is resolved. regards, /virendra

1 0

[ADMIN] wiki.outages.org
by virendra rode 17 Jan '13

17 Jan '13

wiki.outages.org is currently down as / partition was used up to 100% causing apache and mysql to choke. scloud@Nanog01r:~$ Filesystem Size Used Avail Use% Mounted on /dev/mapper/scloud--template-root 177G 168G 0 100% / udev 2.0G 4.0K 2.0G 1% /dev tmpfs 792M 252K 791M 1% /run none 5.0M 0 5.0M 0% /run/lock none 2.0G 0 2.0G 0% /run/shm /dev/sda1 228M 47M 169M 22% /boot overflow 1.0M 16K 1008K 2% /tmp However we may know what may have caused it and in order to address this issue we attempted to re-size / partition but we were unable to make the VMware see the new size as the dynamic resizing is not supported at the moment. Having said that, we have since created a new VM and looking at migrating data over including DNS change. Until this migration takes place, wiki.outages.org will be unaccessible. I apologize for any inconvenience. regards, /virendra

1 1

Webex dialing numbers down?
by Tony 16 Jan '13

16 Jan '13

Anyone hear anything about webex having issues. A couple of our conf calls went down and now all the call numbers are ringing fast busy

2 2

SEA-ME-WE3 undersea cable cut
by staticsafe 16 Jan '13

16 Jan '13

I'm getting reports of SEA-ME-WE3 being cut. Sources: http://www.telecom-media.co.nz/releases_detail.asp?id=3872&page=index https://www.computerworld.com.au/article/446522/sea-me-we_3_undersea_cable_… http://www.zdnet.com/au/perth-singapore-subsea-cable-cut-7000009803/ -- staticsafe O< ascii ribbon campaign - stop html mail - www.asciiribbon.org Please don't top post - http://goo.gl/YrmAb

8 12

Web UI
by cyates＠valueclick.com 16 Jan '13

16 Jan '13

Sender: Chris Yates Email: cyates(a)valueclick.com Phone: 818-575-4581 Message: Hello, First off, thanks for an amazing utility. This is in no way a gripe or complaint because what you are doing is awesome. I did want to let you know that we were considering displaying both the outages.org map and list views of open alerts on our NOC dashboard (TV), but really weren't able to because we wanted to be able to save things like filters for types of outages and date ranges after refresh. It appears that anytime the page is refreshed the view goes back to a 6 month unfiltered view. I'm sure you can imagine all sorts of awesomeness (Landscape vs portrait orientation etc etc etc) that would be nice, so my only suggestion is to consider unmanned-dashboard type of viewing functionality. (I feel funny even making a suggestion for something that I don't pay for and already is so awesome, so please take it for what it's worth) Thanks, Chris ~~~~~~~~~~~~~~~~~~~~~~ This message was sent from your website at http://tracker.outages.org/

1 0

Power Outage/Fiber Splice In Hawaii?
by Roberts, Brent 16 Jan '13

16 Jan '13

Looks like there is either a rather Large Power outage or Fiber Cut In Hawaii at the moment. I personally have multiple circuits down and Random packet loss to Different Carriers on the Islands. Anyone else seeing anything similar? B. ________________________________ Disclaimer: This e-mail has been sent by Epicor, formerly Progressive Solutions Inc. or one of its affiliates ("Epicor"). The contents of this e-mail and any attachments are intended for the named addressee only and may be confidential. Unless you are the named addressee or are authorized to receive the e-mail of the named addressee you may not disclose, use or copy the contents of the e-mail. If you received this e-mail in error, please contact the sender immediately and then delete the e-mail. Epicor does not accept responsibility for this message and any views or opinions contained in this e-mail are solely those of the author unless expressly stated otherwise.

5 4