I feel my point was completely missed, regardless of the cause or fix: this change has a performance impact of roughly 28x. And since I have new questions, reopening.

It doesn't seem to be limited to broken citations, {{Special:WhatlinksHere/{{FULLPAGENAME}}}} also seems enough to reproduce this.

GDNSD supports HTTP health checks (or custom ones as well). Provided the DNS TTL is no longer than a few minutes, 'automatic failover' is possible.

The tasks regarding loss of PSU redundancy on cp303[2689] are normal priority, does this one need to be high priority?

(correct datacenter)

The same issue is occurring again at a high rate, and is causing problems. Also, there are users reporting they are silently being logged out of the site. I'm not sure how much they are related to each other, but I spotted this in the debug logs:

[session] Session "[50]CentralAuthSessionProvider<-:2:Southparkfan>[REDACTED]": Metadata merge failed: [Exception MediaWiki\Session\MetadataMergeException(
 /srv/mediawiki/w/includes/session/SessionProvider.php:195) Key "CentralAuthSource" changed]
#0 /srv/mediawiki/w/includes/session/SessionManager.php(629): MediaWiki\Session\SessionProvider->mergeMetadata(array, array)
#1 /srv/mediawiki/w/includes/session/SessionManager.php(498): MediaWiki\Session\SessionManager->loadSessionInfoFromStore(MediaWiki\Session\SessionInfo, WebRequest)
#2 /srv/mediawiki/w/includes/session/SessionManager.php(182): MediaWiki\Session\SessionManager->getSessionInfoForRequest(WebRequest)
#3 /srv/mediawiki/w/includes/WebRequest.php(700): MediaWiki\Session\SessionManager->getSessionForRequest(WebRequest)
#4 /srv/mediawiki/w/includes/session/SessionManager.php(121): WebRequest->getSession()
#5 /srv/mediawiki/w/includes/Setup.php(747): MediaWiki\Session\SessionManager::getGlobalSession()
#6 /srv/mediawiki/w/includes/WebStart.php(137): require_once(string)
#7 /srv/mediawiki/w/index.php(40): require(string)
#8 {main}

Is this the same as the recent behavior seen on various API appservers? https://ganglia.wikimedia.org/latest/?c=API%20application%20servers%20eqiad&m=cpu_report&r=hour&s=by%20name&hc=4&mc=2

I wonder why @RobH and @Cmjohnson are talking about 4TB disks. The current problems are caused by 4k (6TB) disks, and the accessoires link given by Cmjohnson mentions a non-4k 6TB disk.

I wonder why @RobH and @Cmjohnson are talking about 4TB disks. The current problems are caused by 4k (6TB) disks, and the accessoires link given by Cmjohnson mentions a non-4k 6TB disk.

Do we prefer a fallback that cannot be impacted by a Wikimedia outage of any kind? Conpherence is an option, but it is not off-site; a network outage affecting Wikimedia will render Conpherence useless.

In T141482#2509001, @Tgr wrote:

Possibly your farm is not sharing sessions, which is a requirement for CentralAuth. Session storage configuration has changed for 1.27, you might want to review that.

It could be possible that my configuration is not up-to-date. Like Wikimedia, we use redis for sessions (production 1.26 config):

$wgObjectCaches['redis'] = array(
	'class' => 'RedisBagOStuff',
	'servers' => array( '<IP>' ),
	'password' => $wmgRedisPassword,
);
$wgMainCacheType = 'redis';
$wgSessionCacheType = 'redis';
$wgSessionsInObjectCache = true;
$wgMessageCacheType = CACHE_NONE;
$wgParserCacheType = CACHE_DB;
$wgLanguageConverterCacheType = CACHE_DB;

Yes, it does.

Step 1 seems to be re-introducing the texvc package (whose deletion actually was unrelated when I created the issue, it just made this task more or less invalid, because the issue was kinda gone - since nobody (except users who still have that package installed) would be hit by it anymore). That is a blocker for this task.

@Andrew labvirt1012 lacks hyperthreading. Can you enable that?

mw1017 and mw1099 are the eqiad MW debug hosts, so those should not be in conftool-data.
mw1161-mw1169 were previous appservers, but have been converted to jobrunners in December, per rOPUPba0a47b56ded3dc748c436c2940f114389b312f2. Jobrunners are not present in conftool-data, so I guess everything is as expected?

@Kghbln yes, I am aware of that, and I spoke to @MoritzMuehlenhoff, and he said that in hindsight the mediawiki-math-texvc should never have been deleted. Perhaps it will be packaged again - but I'm not sure if Debian's policies allow that, since it would mean that it has to be introduced in the jessie (stable) suite again. Otherwise we'd still need to wait until Debian 9 has been released.

@Joe mw1306 seems to have the same IP as mw1091 (although that one shows up as mw1091 in Ganglia, whereas mw1090 shows up as mw1305...). So I think you need to fix that one as well.

Just tested this, this doesn't happen in Firefox 47.0 on Windows 8.1. Perhaps an upgrade to 47.0 might help, although I doubt it because I can't see in the 47.0 release notes anything related to this.

Yep, that change (and the maintenance script) worked perfect. Everything is displaying correctly now.

cp1044 has been decomissioned per T133614

In T130130#2353909, @awight wrote:

That would be great! The only additional benefit we might get from HHVM
would be to cut PHP cpu load by 50%, but that can be an optimization for
later. This card is concerned with the breaking language changes, so
Jessie is a perfect solution.

As of PHP 5.5 (Jessie has 5.6), there is an opcache extension included (only needs to be enabled in config), which caches compiled bytecode in the memory. Together with the other performance improvements since 5.4 (assuming paymentswiki runs on 5.3 now), CPU usage / response time should already decrease without using HHVM (so you get some more headroom before HHVM is really needed).

Jessie has PHP 5.6 actually.

I'm sure Legoktm is right :-)! But I failed to make it work, so someone else should take it over honestly...

It would be cool if this bug could be fixed (whatever the solution is).

@Paladox I've already upgraded to HHVM 3.12, which seems to work.

@Joe did you already install one of those servers? noticed https://ganglia.wikimedia.org/latest/?c=Application%20servers%20eqiad&h=mw1305.eqiad.wmnet&m=cpu_report&r=hour&s=by%20name&hc=4&mc=2 (I can't view which specs the new servers have, but this 'mw1305' seems to have the same cpu/ram specs as the old appservers?)

@Cmjohnson yeah, perhaps I have been a bit too fast by already doing the DNS part (despite that's the only thing I can do it seems) :-)

Just to learn how the process works, I've submitted a patch for the DNS adjustments. I noticed db1058 is referenced in the dhcpd and manifests/role/coredb.pp files in puppet but I have no idea how the latter one works, so I'll leave the puppet work to someone else.

Can this patch be backported to REL1_26, REL1_25 and REL1_23 too? The current security patch is only available in master, and these three branches still receive security support.

Yeah, this doesn't seem to be a Varnish problem:

@ori that is T124163 actually.

@Andrew mw1138 is not depooled (anymore), its CPU and network graphs show it is serving traffic. Looking at http://ganglia.wikimedia.org/latest/graph.php?r=day&z=xlarge&h=mw1132.eqiad.wmnet&m=cpu_report&s=by+name&mc=2&g=network_report&c=API+application+servers+eqiad it was idle for a few hours, but then it somehow got repooled..

I have a setup where I tell Varnish to redirect all /.well-known/acme-challenge traffic to one backend server (practically any server with a webserver should work), so I can use this (acme-tiny).

array(23) {
  ["OAuth"]=>
  string(41) "MediaWiki\Extensions\OAuth\SpecialMWOAuth"
  ["OAuthManageMyGrants"]=>
  string(55) "MediaWiki\Extensions\OAuth\SpecialMWOAuthManageMyGrants"
  ["OAuthListConsumers"]=>
  string(54) "MediaWiki\Extensions\OAuth\SpecialMWOAuthListConsumers"
}

I added this var_dump after the require of OAuth, so it is not like any of my extensions should be messing with the array..

@Tgr: I applied this patch:

diff --git a/frontend/MWOAuthUI.setup.php b/frontend/MWOAuthUI.setup.php
index d3aa768..6abe29e 100644
--- a/frontend/MWOAuthUI.setup.php
+++ b/frontend/MWOAuthUI.setup.php
@@ -49,7 +49,10 @@ class MWOAuthUISetup {
                global $wgSpecialPages;
                global $wgLogTypes, $wgLogNames, $wgLogHeaders, $wgLogActionsHandlers;

Cmjohnson noted there have been multiple machines lately with CPU temperature issues, where re-applying thermal paste has been an effective fix.

1. I understand your point, but it seems very likely to me that everyone knows when to (or to not) use them. This is not https://www.reddit.com/r/memes. Is there any proof that the application is currently harmful (i.e. frequently used in the wrong way)?
2. Depends on upstream, we can't unfortunately do anything about that I guess.
3. If you really think that this is a valid reason to remove the Macro application, then you should change this task to 'disable file uploads' (or open a new task with that subject). This is something that affects all image uploads, it isn't a problem that only affects Macro.

@Cmjohnson shouldn't T126350 be closed as a duplicate of this one instead of vice versa?

@Paladox looks like you haven't configured $wgExtDistAPIConfig:
./includes/providers/ExtDistProvider.php:

	/**
	 * Get the provider configured for the given type
	 *
	 * @param string $type
	 * @return ExtDistProvider
	 */
	public static function getProviderFor( $type ) {
		global $wgExtDistAPIConfig;
		if ( !$wgExtDistAPIConfig ) {
			return null;
		}
		return self::factory(
			$wgExtDistAPIConfig + array( 'repoType' => $type )
		);
	}

Aha, understood. Is there any documentation for using the PHP method?

How do I whitelist it?

@Krenair where does that come from (run curl with -v, and then Server: header)?

Guess what I've found in the logs:

Sun Mar 13 15:26:38 UTC 2016    mw1     metawiki        User::addToDatabase     185.52.1.77     1205    Lock wait timeout exceeded; try restarting transaction (185.52.1.77)    INSERT IGNORE INTO `user` (user_id,user_name,user_password,user_newpassword,user_newpass_time,user_email,user_email_authenticated,user_real_name,user_token,user_registration,user_editcount,user_touched) VALUES (NULL,'Z','','',NULL,'',NULL,'','[REDACTED]','20160313152546','0','20160313152552')
Sun Mar 13 15:38:22 UTC 2016    mw1     poserdazfreebieswiki    User::addGroup  185.52.1.77     1205    Lock wait timeout exceeded; try restarting transaction (185.52.1.77)    INSERT IGNORE INTO `user_groups` (ug_user,ug_group) VALUES ('13','autopatrolled')

(the User::addGroup() might need a separate ticket, though) This bug causes T115198 ('Could not find local user data') for me (and probably for WMF (production) too, although you should look at your log files which I have no access to), and a lot of users are complaining about all kinds of MWExceptions upon login and such.

Still occuring in REL1_26 (reported a while ago in T115198#1948270, ran checkLocalUser.php --delete for those entries but the error is now back again, so ran checkLocalUser.php --delete on all wikis again). Can a fix be ported to that branch too?

Duplicate of T50501?

Not very important, but is it possible to enable OPcache in PHP? Even if you only allow 64MB of cache usage, this should make the wiki a lot faster.

I know that you guys have way more knowledge of (upgrading) Varnish than I do (and Wikimedia's needs are totally different compared to mine), but I already use Varnish 4.0 (which isn't that much different to 4.1) for a production wiki farm - and some stuff about my setup is available at https://meta.miraheze.org/wiki/Tech:Varnish (#Configuration for config). Feel free to reuse some of the docs or config if you can find something interesting there.

I use the latest version:

root@mw1:/srv/mediawiki/w/extensions/Flow/maintenance# php FlowUpdateWorkflowPageId.php --wiki permanentfuturelabwiki --force
Updated 0  workflows