We attempted to fix this with a change to the config file: https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/570892
But this fix had to be reverted after applying it to production: https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/570901
We suspect it had to be reverted due to it causing problems and not just unfortunate co-incidence. It only caused a small number of errors: https://logstash.wikimedia.org/app/kibana#/dashboard/0a9ecdc0-b6dc-11e8-9d8f-dbc23b470465?_g=h@961394c&_a=h@a3fe00c
However we fired about 40 icinga alerts of app servers see this copy of the log from #wikimedia-operations {P10365}
And according to reports took the wikis down.
Below is a table of what we presume the config was actually set to at different times due to this typo:
Looking at the value for $wgWBClientSettings['tmpItemTermsMigrationStage']
| | Date | 3rd Feb (all on wmf.16) | 2020-02-04 22:35 (group0 on .18; group1, group2 on .16) | 2020-02-05 20:44:00 (group0, group1 on .18; group2 on .16) | 2020-02-06 20:25 (all on wmf.18) | 2020-02-06 20:44:00 (group0, group1 on .18; group2 on .16) | 2020-02-07 14:38:00 (group0, group1 on .18; group2 on .16) + typo fix |
|-----------------|---------------------|-----------------------------------|---------------------------------------------------------|------------------------------------------------------------|----------------------------------|------------------------------------------------------------|----------------------------------------------------------------------------------------------|
| Source of value | wikidatawiki | WikibaseLib.default.php | WikibaseLib.default.php | WikibaseLib.default.php | WikibaseLib.default.php | WikibaseLib.default.php | wmf-config wikidatawiki |
| Actual Value | | [ 'max' => MIGRATION_WRITE_BOTH ] | [ 'max' => MIGRATION_WRITE_BOTH ] | [ 'max' => MIGRATION_WRITE_NEW ] {icon exclamation-circle color=yellow } | [ 'max' => MIGRATION_WRITE_NEW ] | [ 'max' => MIGRATION_WRITE_NEW ] {icon exclamation-circle color=yellow } | [ 100000 => MIGRATION_WRITE_NEW, 74000000 => MIGRATION_WRITE_BOTH, 'max' => MIGRATION_OLD, ] |
| Source of value | testwikidatawiki | WikibaseLib.default.php | WikibaseLib.default.php | WikibaseLib.default.php | WikibaseLib.default.php | WikibaseLib.default.php | wmf-config testwikidatawiki |
| Actual Value | | [ 'max' => MIGRATION_WRITE_BOTH ] | [ 'max' => MIGRATION_WRITE_NEW ] | [ 'max' => MIGRATION_WRITE_NEW ] {icon exclamation-circle color=yellow } | [ 'max' => MIGRATION_WRITE_NEW ] {icon exclamation-circle color=yellow }| [ 'max' => MIGRATION_WRITE_NEW ] {icon exclamation-circle color=yellow } | [ 'max' => MIGRATION_WRITE_NEW ] {icon exclamation-circle color=yellow } |
| Source of value | random group 0 wiki | WikibaseLib.default.php | WikibaseLib.default.php | WikibaseLib.default.php | WikibaseLib.default.php | WikibaseLib.default.php | wmf-config default |
| Actual Value | | [ 'max' => MIGRATION_WRITE_BOTH ] | [ 'max' => MIGRATION_WRITE_NEW ] {icon exclamation-circle color=yellow } | [ 'max' => MIGRATION_WRITE_NEW ] {icon exclamation-circle color=yellow } | [ 'max' => MIGRATION_WRITE_NEW ] {icon exclamation-circle color=yellow } | [ 'max' => MIGRATION_WRITE_NEW ] {icon exclamation-circle color=yellow } | [ 8000000 => MIGRATION_WRITE_NEW, 'max' => MIGRATION_OLD, ] |
| Source of value | random group 1 wiki | WikibaseLib.default.php | WikibaseLib.default.php | WikibaseLib.default.php | WikibaseLib.default.php | WikibaseLib.default.php | wmf-config default |
| Actual Value | | [ 'max' => MIGRATION_WRITE_BOTH ] | [ 'max' => MIGRATION_WRITE_BOTH ] | [ 'max' => MIGRATION_WRITE_NEW ] {icon exclamation-circle color=yellow } | [ 'max' => MIGRATION_WRITE_NEW ] {icon exclamation-circle color=yellow }| [ 'max' => MIGRATION_WRITE_NEW ] {icon exclamation-circle color=yellow } | [ 8000000 => MIGRATION_WRITE_NEW, 'max' => MIGRATION_OLD, ] |
| Source of value | random group 2 wiki | WikibaseLib.default.php | WikibaseLib.default.php | WikibaseLib.default.php | WikibaseLib.default.php | WikibaseLib.default.php | wmf-config default |
| Actual Value | | [ 'max' => MIGRATION_WRITE_BOTH ] | [ 'max' => MIGRATION_WRITE_BOTH ] | [ 'max' => MIGRATION_WRITE_BOTH ] | [ 'max' => MIGRATION_WRITE_NEW ]{icon exclamation-circle color=yellow }{icon exclamation-circle color=red } | [ 'max' => MIGRATION_WRITE_BOTH ] | [ 8000000 => MIGRATION_WRITE_NEW, 'max' => MIGRATION_OLD, ] {icon exclamation-circle color=orange } |
{icon exclamation-circle color=yellow } = reading from potentially unmigrated rows
{icon exclamation-circle color=orange } = heavy load on new termstore; suspected to have partially brought down wikis
{icon exclamation-circle color=red } = very heavy load on new termstore suspected to have brought down the wikis