Wikipedia is currently inaccessible for me, and for http://www.downforeveryoneorjustme.com. Port 80 is not accepting connections, and port 443 is returning a '502 Bad Gateway' error. dwhite@quark:~$ wget www.wikipedia.org --2012-08-02 16:17:31-- http://www.wikipedia.org/ Resolving www.wikipedia.org (www.wikipedia.org)... 208.80.154.225, 2620:0:861:ed1a::1 Connecting to www.wikipedia.org (www.wikipedia.org)|208.80.154.225|:80... failed: Connection refused. Connecting to www.wikipedia.org (www.wikipedia.org)|2620:0:861:ed1a::1|:80... failed: Network is unreachable. -- Dan White
Works just fine from PHX http://www.downforeveryoneorjustme.com/www.wikipedia.org Raman Sud Director, Network Infrastructure PhoenixNAP LLC 3402 East University Drive Phoenix AZ 85034 Phone: 480.449.8834 Fax: 480.682.3236 http://www.phoenixnap.com PGP Key: http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xBB765DA2 THIS MESSAGE IS INTENDED ONLY FOR THE USE OF THE INDIVIDUAL OR ENTITY TO WHICH IT IS ADDRESSED, AND MAY CONTAIN INFORMATION THAT IS PRIVILEGED, CONFIDENTIAL AND EXEMPT FROM DISCLOSURE UNDER APPLICABLE LAW. IF THE READER OF THIS MESSAGE IS NOT THE INTENDED RECIPIENT, YOU ARE HEREBY NOTIFIED THAT ANY DISSEMINATION, DISTRIBUTION OR COPYING OF THIS COMMUNICATION IS STRICTLY PROHIBITED, AND THAT THIS EMAIL AND ALL COPIES AND ATTACHMENTS SHOULD BE IMMEDIATELY DESTROYED. -----Original Message----- From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On Behalf Of Dan White Sent: Thursday, August 02, 2012 2:18 PM To: outages@outages.org Subject: [outages] Wikipedia Wikipedia is currently inaccessible for me, and for http://www.downforeveryoneorjustme.com. Port 80 is not accepting connections, and port 443 is returning a '502 Bad Gateway' error. dwhite@quark:~$ wget www.wikipedia.org --2012-08-02 16:17:31-- http://www.wikipedia.org/ Resolving www.wikipedia.org (www.wikipedia.org)... 208.80.154.225, 2620:0:861:ed1a::1 Connecting to www.wikipedia.org (www.wikipedia.org)|208.80.154.225|:80... failed: Connection refused. Connecting to www.wikipedia.org (www.wikipedia.org)|2620:0:861:ed1a::1|:80... failed: Network is unreachable. -- Dan White _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
I reported it on their internal/external tech list, I was seeing the outage for about 5-8 min and then it is back working in the last 5-ish. No clue yet what it was, nobody responded to the report yet. Not a hallucination, but transient. On Thu, Aug 2, 2012 at 2:19 PM, Raman Sud <ramans@cwie.net> wrote:
Works just fine from PHX
http://www.downforeveryoneorjustme.com/www.wikipedia.org
Raman Sud Director, Network Infrastructure
PhoenixNAP LLC 3402 East University Drive Phoenix AZ 85034 Phone: 480.449.8834 Fax: 480.682.3236 http://www.phoenixnap.com PGP Key: http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xBB765DA2
THIS MESSAGE IS INTENDED ONLY FOR THE USE OF THE INDIVIDUAL OR ENTITY TO WHICH IT IS ADDRESSED, AND MAY CONTAIN INFORMATION THAT IS PRIVILEGED, CONFIDENTIAL AND EXEMPT FROM DISCLOSURE UNDER APPLICABLE LAW. IF THE READER OF THIS MESSAGE IS NOT THE INTENDED RECIPIENT, YOU ARE HEREBY NOTIFIED THAT ANY DISSEMINATION, DISTRIBUTION OR COPYING OF THIS COMMUNICATION IS STRICTLY PROHIBITED, AND THAT THIS EMAIL AND ALL COPIES AND ATTACHMENTS SHOULD BE IMMEDIATELY DESTROYED.
-----Original Message----- From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On Behalf Of Dan White Sent: Thursday, August 02, 2012 2:18 PM To: outages@outages.org Subject: [outages] Wikipedia
Wikipedia is currently inaccessible for me, and for http://www.downforeveryoneorjustme.com.
Port 80 is not accepting connections, and port 443 is returning a '502 Bad Gateway' error.
dwhite@quark:~$ wget www.wikipedia.org --2012-08-02 16:17:31-- http://www.wikipedia.org/ Resolving www.wikipedia.org (www.wikipedia.org)... 208.80.154.225, 2620:0:861:ed1a::1 Connecting to www.wikipedia.org (www.wikipedia.org)|208.80.154.225|:80... failed: Connection refused. Connecting to www.wikipedia.org (www.wikipedia.org)|2620:0:861:ed1a::1|:80... failed: Network is unreachable.
-- Dan White _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
-- -george william herbert george.herbert@gmail.com
Initial answer on the mailing list, from an IRC discussion, is bad squid configuration pushed, rapidly unpushed. On Thu, Aug 2, 2012 at 2:28 PM, George Herbert <george.herbert@gmail.com> wrote:
I reported it on their internal/external tech list, I was seeing the outage for about 5-8 min and then it is back working in the last 5-ish.
No clue yet what it was, nobody responded to the report yet. Not a hallucination, but transient.
On Thu, Aug 2, 2012 at 2:19 PM, Raman Sud <ramans@cwie.net> wrote:
Works just fine from PHX
http://www.downforeveryoneorjustme.com/www.wikipedia.org
Raman Sud Director, Network Infrastructure
PhoenixNAP LLC 3402 East University Drive Phoenix AZ 85034 Phone: 480.449.8834 Fax: 480.682.3236 http://www.phoenixnap.com PGP Key: http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xBB765DA2
THIS MESSAGE IS INTENDED ONLY FOR THE USE OF THE INDIVIDUAL OR ENTITY TO WHICH IT IS ADDRESSED, AND MAY CONTAIN INFORMATION THAT IS PRIVILEGED, CONFIDENTIAL AND EXEMPT FROM DISCLOSURE UNDER APPLICABLE LAW. IF THE READER OF THIS MESSAGE IS NOT THE INTENDED RECIPIENT, YOU ARE HEREBY NOTIFIED THAT ANY DISSEMINATION, DISTRIBUTION OR COPYING OF THIS COMMUNICATION IS STRICTLY PROHIBITED, AND THAT THIS EMAIL AND ALL COPIES AND ATTACHMENTS SHOULD BE IMMEDIATELY DESTROYED.
-----Original Message----- From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On Behalf Of Dan White Sent: Thursday, August 02, 2012 2:18 PM To: outages@outages.org Subject: [outages] Wikipedia
Wikipedia is currently inaccessible for me, and for http://www.downforeveryoneorjustme.com.
Port 80 is not accepting connections, and port 443 is returning a '502 Bad Gateway' error.
dwhite@quark:~$ wget www.wikipedia.org --2012-08-02 16:17:31-- http://www.wikipedia.org/ Resolving www.wikipedia.org (www.wikipedia.org)... 208.80.154.225, 2620:0:861:ed1a::1 Connecting to www.wikipedia.org (www.wikipedia.org)|208.80.154.225|:80... failed: Connection refused. Connecting to www.wikipedia.org (www.wikipedia.org)|2620:0:861:ed1a::1|:80... failed: Network is unreachable.
-- Dan White _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
_______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
-- -george william herbert george.herbert@gmail.com
-- -george william herbert george.herbert@gmail.com
On Thu, 2 Aug 2012, George Herbert wrote:
I reported it on their internal/external tech list, I was seeing the outage for about 5-8 min and then it is back working in the last 5-ish.
Several times over the last few years I've seen WP outages which turned out to be bad config pushed in to production and then quickly reverted. A few were patches to the Mediawiki software, for example. I guess they don't have a preprod/UAT environment :) While I can understand them being able to simulate the scale, a small UAT environment to test config rationality wouldn't go astray. I hear Wikipedia has a monitoring system. It involves alerts issued by millions of people around the world :) Cheers, Rob -- Email: robert@timetraveller.org Linux counter ID #16440 IRC: Solver (OFTC & Freenode) Web: http://www.practicalsysadmin.com Director, Software in the Public Interest (http://spi-inc.org/) Information is a gas
On Fri, Aug 3, 2012 at 12:51 AM, Robert Brockway <robert@timetraveller.org> wrote:
On Thu, 2 Aug 2012, George Herbert wrote:
I reported it on their internal/external tech list, I was seeing the outage for about 5-8 min and then it is back working in the last 5-ish.
Several times over the last few years I've seen WP outages which turned out to be bad config pushed in to production and then quickly reverted. A few were patches to the Mediawiki software, for example.
I guess they don't have a preprod/UAT environment :) While I can understand them being able to simulate the scale, a small UAT environment to test config rationality wouldn't go astray.
I hear Wikipedia has a monitoring system. It involves alerts issued by millions of people around the world :)
Cheers,
Rob
I know some of the ops folks and have talked about ops stability on and off with the deputy director and VP of technology of the Wikimedia Foundation. I haven't professionally consulted per se, but have some info about the ops. They do have a preprod environment, but there are limitations to it, and the systems management process is not perfect. They have been focused over the last couple of years on stability and disaster recovery, but with the user growth they see and budget envelope, it's hard to make huge leaps ahead on stability while growing. Frankly, most of the large commercial environments I have seen were run worse, all things considered... -- -george william herbert george.herbert@gmail.com
Hi George, Okay we resolved the iperf download issue, we will be ready to go shortly Ken Kenneth Waugh | NOC Engineer | Sentry Managed Services | CCNP Office 301-623-1926 8am to 5pm US ET or kwaugh@presidio.com Presidio Networked Solutions | www.presidio.com -----Original Message----- From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On Behalf Of George Herbert Sent: Friday, August 03, 2012 2:58 PM To: Robert Brockway Cc: outages@outages.org Subject: Re: [outages] Wikipedia On Fri, Aug 3, 2012 at 12:51 AM, Robert Brockway <robert@timetraveller.org> wrote:
On Thu, 2 Aug 2012, George Herbert wrote:
I reported it on their internal/external tech list, I was seeing the outage for about 5-8 min and then it is back working in the last 5-ish.
Several times over the last few years I've seen WP outages which turned out to be bad config pushed in to production and then quickly reverted. A few were patches to the Mediawiki software, for example.
I guess they don't have a preprod/UAT environment :) While I can understand them being able to simulate the scale, a small UAT environment to test config rationality wouldn't go astray.
I hear Wikipedia has a monitoring system. It involves alerts issued by millions of people around the world :)
Cheers,
Rob
I know some of the ops folks and have talked about ops stability on and off with the deputy director and VP of technology of the Wikimedia Foundation. I haven't professionally consulted per se, but have some info about the ops. They do have a preprod environment, but there are limitations to it, and the systems management process is not perfect. They have been focused over the last couple of years on stability and disaster recovery, but with the user growth they see and budget envelope, it's hard to make huge leaps ahead on stability while growing. Frankly, most of the large commercial environments I have seen were run worse, all things considered... -- -george william herbert george.herbert@gmail.com _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages This message w/attachments (message) is intended solely for the use of the intended recipient(s) and may contain information that is privileged, confidential or proprietary. If you are not an intended recipient, please notify the sender, and then please delete and destroy all copies and attachments. Please be advised that any review or dissemination of, or the taking of any action in reliance on, the information contained in or attached to this message is prohibited.
Disregard previous email. It was not meant to go to this distribution list. Ken -----Original Message----- From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On Behalf Of Waugh, Kenneth Sent: Friday, August 03, 2012 3:30 PM To: George Herbert; Robert Brockway Cc: outages@outages.org Subject: Re: [outages] Wikipedia Hi George, Okay we resolved the iperf download issue, we will be ready to go shortly Ken Kenneth Waugh | NOC Engineer | Sentry Managed Services | CCNP Office 301-623-1926 8am to 5pm US ET or kwaugh@presidio.com Presidio Networked Solutions | www.presidio.com -----Original Message----- From: outages-bounces@outages.org [mailto:outages-bounces@outages.org] On Behalf Of George Herbert Sent: Friday, August 03, 2012 2:58 PM To: Robert Brockway Cc: outages@outages.org Subject: Re: [outages] Wikipedia On Fri, Aug 3, 2012 at 12:51 AM, Robert Brockway <robert@timetraveller.org> wrote:
On Thu, 2 Aug 2012, George Herbert wrote:
I reported it on their internal/external tech list, I was seeing the outage for about 5-8 min and then it is back working in the last 5-ish.
Several times over the last few years I've seen WP outages which turned out to be bad config pushed in to production and then quickly reverted. A few were patches to the Mediawiki software, for example.
I guess they don't have a preprod/UAT environment :) While I can understand them being able to simulate the scale, a small UAT environment to test config rationality wouldn't go astray.
I hear Wikipedia has a monitoring system. It involves alerts issued by millions of people around the world :)
Cheers,
Rob
I know some of the ops folks and have talked about ops stability on and off with the deputy director and VP of technology of the Wikimedia Foundation. I haven't professionally consulted per se, but have some info about the ops. They do have a preprod environment, but there are limitations to it, and the systems management process is not perfect. They have been focused over the last couple of years on stability and disaster recovery, but with the user growth they see and budget envelope, it's hard to make huge leaps ahead on stability while growing. Frankly, most of the large commercial environments I have seen were run worse, all things considered... -- -george william herbert george.herbert@gmail.com _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages This message w/attachments (message) is intended solely for the use of the intended recipient(s) and may contain information that is privileged, confidential or proprietary. If you are not an intended recipient, please notify the sender, and then please delete and destroy all copies and attachments. Please be advised that any review or dissemination of, or the taking of any action in reliance on, the information contained in or attached to this message is prohibited. _______________________________________________ Outages mailing list Outages@outages.org https://puck.nether.net/mailman/listinfo/outages
It's back up for me. On 08/02/12 16:18 -0500, Dan White wrote:
Wikipedia is currently inaccessible for me, and for http://www.downforeveryoneorjustme.com.
Port 80 is not accepting connections, and port 443 is returning a '502 Bad Gateway' error.
dwhite@quark:~$ wget www.wikipedia.org --2012-08-02 16:17:31-- http://www.wikipedia.org/ Resolving www.wikipedia.org (www.wikipedia.org)... 208.80.154.225, 2620:0:861:ed1a::1 Connecting to www.wikipedia.org (www.wikipedia.org)|208.80.154.225|:80... failed: Connection refused. Connecting to www.wikipedia.org (www.wikipedia.org)|2620:0:861:ed1a::1|:80... failed: Network is unreachable.
-- Dan White
participants (5)
-
Dan White -
George Herbert -
Raman Sud -
Robert Brockway -
Waugh, Kenneth