@demon: special request this week, could rMWb43ac35351e7: Grant 'editcontentmodel' permission to 'user' group / https://gerrit.wikimedia.org/r/#/c/309061/ be reverted in the wmf.19 branch (but not master)? I'd like to keep that patch on beta, but it's not ready to go out with the train and requires special wmf-config handling.

I thought it would be nicer to revert in master and fiddles with the rights in the -labs.php files. Chad pointed that is way easier to just revert in wmf.19. So lets do that :]

Can you prepare a revert patch for wmf.19, if you add it to the European SWAT on Wednesday I will be happy to handle it.

Already handling the revert: https://gerrit.wikimedia.org/r/#/c/310440/

Should be done shortly.

[19:14:44] <logmsgbot> !log hashar@tin rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis to 1.28.0-wmf.19

Easy. Nothing is happening.

Change 310617 had a related patch set uploaded (by Hashar):
Revert "group1 wikis to 1.28.0-wmf.19"

https://gerrit.wikimedia.org/r/310617

Change 310617 merged by jenkins-bot:
Revert "group1 wikis to 1.28.0-wmf.19"

https://gerrit.wikimedia.org/r/310617

Reverted due to T145673

Mentioned in SAL (#wikimedia-operations) [2016-09-14T20:28:39Z] <hashar@tin> Synchronized php-1.28.0-wmf.19/extensions/CentralAuth/includes/LocalRenameJob/LocalRenameJob.php: Fix LocalRenameJob transaction owner to match JobRunner T143328 T145596 (duration: 00m 48s)

[19:09:21] <logmsgbot> !log hashar@tin rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis to 1.28.0-wmf.19

Change 310948 had a related patch set uploaded (by Hashar):
All wiki but wikidatawiki to php-1.28.0-wmf.19

https://gerrit.wikimedia.org/r/310948

Change 310948 merged by jenkins-bot:
All wiki but wikidatawiki to php-1.28.0-wmf.19

https://gerrit.wikimedia.org/r/310948

Change 310953 had a related patch set uploaded (by Hashar):
Fix wikidata to .18 (previous was testwikidata)

https://gerrit.wikimedia.org/r/310953

Change 310953 merged by jenkins-bot:
Fix wikidata to .18 (previous was testwikidata)

https://gerrit.wikimedia.org/r/310953

Rollbacked due to account creation being broken T145839 Investigation is on T145819.

I briefly moved group0 to wmf.19 and saw a large spike in the overall fatalmonitor error-rate. I realized that the appservers that were throwing errors were missing /srv/mediawiki/php-1.28.0-wmf.19/cache/l10n/*.cdb files. I ran a full scap to rebuild the l10n cache for the servers that were missing it.

While I was running the scap to restore l10n, T146099: mw-1.28.0-wmf.18 load-time regression was discovered. I've halted wmf.19 rollout until that task has been investigated.

From T146099

load times increased by approx 600ms:

That got introduced by wmf.18 and left unnoticed for almost 10 days :-( I have filled a task to get alarms on that T146125: MediaWiki load time regression should trigger an alarm / page people