Enable MCR migration stage "write both, read new" on live systems
Closed, ResolvedPublic

Description

Before fully switching to the new schema, we should exercise the new retrieval code. This means setting

$wgMultiContentRevisionSchemaMigrationStage = SCHEMA_COMPAT_WRITE_BOTH | SCHEMA_COMPAT_READ_NEW;

This keeps writing to the old schema as well, so we can still easily back out of the migration.

NOTE: we'll want to have this enabled on the live systems for a while before releasing it as the default for 3rd party installs with MW-1.32-release. But we'll want to have this as the default on master / for CI before enabling it on the live systems, see T198561.
NOTE: This enables the usage of extra slots on the live systems! If we are forced to roll this back, we will lose access to the content of the new, extra slots. Rolling forward again would restore that access for old revisions, but extra cleanup would be needed to then again associate that content with newer revisions, made while the change was rolled back. Users should avoid large-scale edits that would be broken by a rollback until we're reasonably sure a rollback isn't going to be needed.
WARNING: This is blocked on operations being switched back to eqiad per Datacenter-Switchover-2018

Related Objects

daniel created this task.Jun 27 2018, 11:24 AM
Restricted Application added a project: Wikidata. · View Herald TranscriptJun 27 2018, 11:24 AM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
daniel updated the task description. (Show Details)Jul 1 2018, 12:43 PM
daniel added subscribers: CCicalese_WMF, Fjalapeno.
daniel updated the task description. (Show Details)Jul 2 2018, 2:29 PM
Anomie updated the task description. (Show Details)Jul 2 2018, 3:47 PM
Abit added a subscriber: Keegan.Jul 2 2018, 3:59 PM

fyi @Keegan, "Users should avoid large-scale edits that would be broken by a rollback until we're reasonably sure a rollback isn't going to be needed."

daniel updated the task description. (Show Details)Jul 4 2018, 12:21 PM
daniel updated the task description. (Show Details)Jul 8 2018, 8:17 PM
Anomie added a comment.Sep 4 2018, 3:39 PM

This task is currently blocked on some bugfixes. If it gets unblocked during the datacenter switchover (Sept 12 - Oct 12, I believe) we need to sync with the DBAs beforehand to be sure they have capacity to keep an eye out for DB overload in the read-new queries (and/or that the changed queries won't screw up other measurements they're trying to do).

daniel updated the task description. (Show Details)Sep 10 2018, 6:39 PM
daniel added a project: MW-1.32-release.
daniel added a comment.EditedSep 11 2018, 10:32 AM

We should probably not do all wikis immediately. Perhaps we can do something like this:

  • testwiki (T198309). watch for bugs.
  • mediawikiwiki and metawiki. watch for bugs.
  • commonswiki and dewiki. watch for performance issues on s4 and s5. Coordinate with DBAs first!
  • all the rest. Coordinate with DBAs first!
daniel updated the task description. (Show Details)Sep 17 2018, 6:08 PM
Addshore moved this task from incoming to monitoring on the Wikidata board.Sep 19 2018, 7:08 AM
Jdforrester-WMF moved this task from To Do to Monitoring on the SDC Engineering board.

Change 463764 had a related patch set uploaded (by Anomie; owner: Anomie):
[operations/mediawiki-config@master] Set wgMultiContentRevisionSchemaMigrationStage read-new on mediawikiwiki

https://gerrit.wikimedia.org/r/463764

Change 463764 merged by jenkins-bot:
[operations/mediawiki-config@master] Set wgMultiContentRevisionSchemaMigrationStage read-new on mediawikiwiki

https://gerrit.wikimedia.org/r/463764

Mentioned in SAL (#wikimedia-operations) [2018-10-01T14:14:00Z] <anomie@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Setting MCR migration stage to write-both/read-new on mediawikiwiki (T198308) (duration: 00m 56s)

Change 465194 had a related patch set uploaded (by Gergő Tisza; owner: Gergő Tisza):
[operations/mediawiki-config@master] Enable MCR read-new mode on some small wikis

https://gerrit.wikimedia.org/r/465194

Change 465194 merged by jenkins-bot:
[operations/mediawiki-config@master] Enable MCR read-new mode on some small wikis

https://gerrit.wikimedia.org/r/465194

Mentioned in SAL (#wikimedia-operations) [2018-10-08T19:00:10Z] <tgr@deploy1001> Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:465194|Enable MCR read-new mode on some small wikis (T198308)]] (duration: 00m 56s)

Change 467340 had a related patch set uploaded (by Anomie; owner: Anomie):
[operations/mediawiki-config@master] Enable MCR read-new on Commons

https://gerrit.wikimedia.org/r/467340

Change 467340 merged by jenkins-bot:
[operations/mediawiki-config@master] Enable MCR read-new on Commons

https://gerrit.wikimedia.org/r/467340

Mentioned in SAL (#wikimedia-operations) [2018-10-15T13:35:43Z] <anomie@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Setting MCR migration stage to write-both/read-new on Commons (T198308) (duration: 00m 49s)

Note: @jcrespo asked us to hold off on deploying to wikidatawiki until the s8 data issue is completely fixed. He requests we ping him whenever we're ready to do that wiki. He estimates it should be done by tomorrow or at the very latest next week, which should be fine since we're not planning further deployments here until after TechConf.

Note: @jcrespo asked us to hold off on deploying to wikidatawiki until the s8 data issue is completely fixed. He requests we ping him whenever we're ready to do that wiki. He estimates it should be done by tomorrow or at the very latest next week, which should be fine since we're not planning further deployments here until after TechConf.

He says we're ok to deploy to wikidatawiki when we're ready now.

I had a cursory eye on stats and logs for commons over the week, and didn't see anything alarming. So I suppose we can enable this on more wikis.

Do we want to make more smaller steps, or go all-in now?

Change 471005 had a related patch set uploaded (by Daniel Kinzler; owner: Daniel Kinzler):
[operations/mediawiki-config@master] Enable reading from the new MCR schema everywhere.

https://gerrit.wikimedia.org/r/471005

Change 471005 merged by jenkins-bot:
[operations/mediawiki-config@master] Enable reading from the new MCR schema everywhere.

https://gerrit.wikimedia.org/r/471005

Mentioned in SAL (#wikimedia-operations) [2018-11-05T16:06:21Z] <anomie@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Setting MCR to read-new on all wikis (T198308) (duration: 00m 55s)

Anomie closed this task as Resolved.Nov 5 2018, 5:33 PM