Page MenuHomePhabricator
Create Task
Maniphest T225057

Switch `tmpItemTermsMigrationStages` to MIGRATION_WRITE_NEW
Closed, DuplicatePublic
Actions

Details

Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
• alaa_wmde set Due Date to Jul 2 2019, 10:00 PM.Jun 5 2019, 6:55 AM
Restricted Application changed the subtype of this task from "Task" to "Deadline". · View Herald TranscriptJun 5 2019, 6:55 AM
• alaa_wmde renamed this task from Switch `tmpItemTermsMigrationStage` to MIGRATION_WRITE_NEW to Switch `tmpItemTermsTwoMillionMigrationStage` to MIGRATION_WRITE_NEW.Jun 5 2019, 8:00 AM
• alaa_wmde renamed this task from Switch `tmpItemTermsTwoMillionMigrationStage` to MIGRATION_WRITE_NEW to Switch `tmpItemTermsMigrationStages` to MIGRATION_WRITE_NEW.Jun 13 2019, 5:58 PM
• alaa_wmde changed Due Date from Jul 2 2019, 10:00 PM to Jul 9 2019, 10:00 PM.Jun 13 2019, 6:00 PM
Addshore moved this task from incoming to in progress on the Wikidata board.Jun 21 2019, 11:37 PM
gerritbot added a comment.Jul 31 2019, 10:40 AM

Change 526643 had a related patch set uploaded (by Ladsgroup; owner: Ladsgroup):
[operations/mediawiki-config@master] labs: Set tmpItemTermsMigrationStages to MIGRATION_WRITE_BOTH for half of Wikidata

https://gerrit.wikimedia.org/r/526643

gerritbot added a project: Patch-For-Review.Jul 31 2019, 10:40 AM
gerritbot added a comment.Jul 31 2019, 10:43 AM

Change 526643 merged by jenkins-bot:
[operations/mediawiki-config@master] labs: Set tmpItemTermsMigrationStages to MIGRATION_WRITE_BOTH for half of Wikidata

https://gerrit.wikimedia.org/r/526643

Ladsgroup subscribed.Jul 31 2019, 11:06 AM
In T225057#5379150, @gerritbot wrote:

Change 526643 had a related patch set uploaded (by Ladsgroup; owner: Ladsgroup):
[operations/mediawiki-config@master] labs: Set tmpItemTermsMigrationStages to MIGRATION_WRITE_BOTH for half of Wikidata

https://gerrit.wikimedia.org/r/526643

That supposed to go with T225055: Switch `tmpItemTermsMigrationStages` to MIGRATION_WRITE_BOTH instead of this ticket. Sorry

Maintenance_bot removed a project: Patch-For-Review.Jul 31 2019, 11:10 AM
Ladsgroup claimed this task.Nov 27 2019, 1:47 PM
Ladsgroup edited projects, added Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)); removed Wikidata wb_terms Trailblazing.
Restricted Application added a project: User-Ladsgroup. · View Herald TranscriptNov 27 2019, 1:47 PM
gerritbot added a comment.Nov 27 2019, 1:47 PM

Change 553342 had a related patch set uploaded (by Ladsgroup; owner: Ladsgroup):
[operations/mediawiki-config@master] Set all of testwikidatawiki to read from the new term store for items

https://gerrit.wikimedia.org/r/553342

gerritbot added a project: Patch-For-Review.Nov 27 2019, 1:47 PM
gerritbot added a comment.Nov 27 2019, 1:54 PM

Change 553342 merged by jenkins-bot:
[operations/mediawiki-config@master] Set all of testwikidatawiki to read from the new term store for items

https://gerrit.wikimedia.org/r/553342

Stashbot added a comment.Nov 27 2019, 1:57 PM

Mentioned in SAL (#wikimedia-operations) [2019-11-27T13:57:57Z] <ladsgroup@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Set all of testwikidatawiki to read from the new term store for items (T225057) (duration: 00m 56s)

Ladsgroup moved this task from To Do (prioritised from top to bottom) to Stalled/Waiting on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.Nov 27 2019, 2:00 PM

Test Wikidata is on read new now, we have time to check until next week that we enable reading new for wikidata (up to Q1000)

Maintenance_bot removed a project: Patch-For-Review.Nov 27 2019, 2:10 PM
Maintenance_bot moved this task from Incoming to In progress on the User-Ladsgroup board.Nov 27 2019, 2:15 PM
gerritbot added a comment.Dec 2 2019, 12:23 PM

Change 554049 had a related patch set uploaded (by Ladsgroup; owner: Ladsgroup):
[operations/mediawiki-config@master] Set read new for term store for items of wikidata up to Q1000

https://gerrit.wikimedia.org/r/554049

gerritbot added a project: Patch-For-Review.Dec 2 2019, 12:23 PM

Change 554049 merged by jenkins-bot:
[operations/mediawiki-config@master] Set read new for term store for items of wikidata up to Q1000

https://gerrit.wikimedia.org/r/554049

Stashbot added a comment.Dec 2 2019, 12:31 PM

Mentioned in SAL (#wikimedia-operations) [2019-12-02T12:31:46Z] <ladsgroup@deploy1001> Synchronized wmf-config/InitialiseSettings.php: [[gerrit:554049|Set read new for term store for items of wikidata up to Q1000 (T225057)]] (duration: 01m 00s)

Maintenance_bot removed a project: Patch-For-Review.Dec 2 2019, 1:10 PM
gerritbot added a comment.Dec 2 2019, 9:57 PM

Change 554164 had a related patch set uploaded (by Ladsgroup; owner: Ladsgroup):
[operations/mediawiki-config@master] Set item term store for read new up to Q1000 for clients

https://gerrit.wikimedia.org/r/554164

gerritbot added a project: Patch-For-Review.Dec 2 2019, 9:57 PM
Ladsgroup moved this task from Stalled/Waiting to Peer Review on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.Dec 2 2019, 9:58 PM
gerritbot added a comment.Dec 3 2019, 12:04 PM

Change 554164 merged by jenkins-bot:
[operations/mediawiki-config@master] Set item term store for read new up to Q1000 for clients

https://gerrit.wikimedia.org/r/554164

Stashbot added a comment.Dec 3 2019, 12:09 PM

Mentioned in SAL (#wikimedia-operations) [2019-12-03T12:09:31Z] <ladsgroup@deploy1001> Synchronized wmf-config/InitialiseSettings.php: [[gerrit:554164|Set read new for term store for items for client wikis up to Q1000 (T225057)]] (duration: 01m 00s)

Maintenance_bot removed a project: Patch-For-Review.Dec 3 2019, 12:10 PM
• alaa_wmde moved this task from Peer Review to Stalled/Waiting on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.Dec 4 2019, 11:03 AM

We moved to new store to up to Q1000. We wait for a while and move to a higher limit later

Addshore changed the task status from Open to Stalled.Jan 7 2020, 10:57 AM
Addshore subscribed.

Stalled, waiting for T225055 and other moving parts

Addshore removed Due Date.Jan 14 2020, 10:02 AM
Addshore triaged this task as Medium priority.Jan 14 2020, 4:20 PM
Addshore changed the task status from Stalled to Open.Jan 15 2020, 5:16 PM
Addshore removed Ladsgroup as the assignee of this task.
Addshore moved this task from Stalled/Waiting to Doing on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.
Addshore moved this task from Doing to To Do (prioritised from top to bottom) on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.
Addshore added a comment.Jan 15 2020, 5:18 PM

Took your face off for now, but feel free to grba it back when ready

gerritbot added a comment.Jan 15 2020, 5:42 PM

Change 565074 had a related patch set uploaded (by Ladsgroup; owner: Ladsgroup):
[operations/mediawiki-config@master] Set read for items in Wikidata for new term store up to Q8M

https://gerrit.wikimedia.org/r/565074

gerritbot added a project: Patch-For-Review.Jan 15 2020, 5:42 PM
Ladsgroup claimed this task.Jan 15 2020, 5:43 PM
Ladsgroup moved this task from To Do (prioritised from top to bottom) to Peer Review on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.

:D

gerritbot added a comment.Jan 16 2020, 12:06 PM

Change 565074 merged by jenkins-bot:
[operations/mediawiki-config@master] Set read for items in Wikidata for new term store up to Q8M

https://gerrit.wikimedia.org/r/565074

Maintenance_bot removed a project: Patch-For-Review.Jan 16 2020, 12:10 PM
Stashbot added a comment.Jan 16 2020, 12:10 PM

Mentioned in SAL (#wikimedia-operations) [2020-01-16T12:10:57Z] <ladsgroup@deploy1001> Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:565074|Set read for items in Wikidata for new term store up to Q8M (T225057)]] (duration: 01m 07s)

Ladsgroup moved this task from Peer Review to Doing on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.Jan 16 2020, 12:28 PM
Addshore mentioned this in T243705: Lexemes: in English, "unknown language" appears instead of "English" in search, and "Q1860" appears instead of "English" on lexeme pages.Jan 27 2020, 11:45 AM
Addshore mentioned this in T244529: mw.wikibase.getLabelByLang not return item label for some items, because MIGRATION_WRITE_NEW is being deployed with train for wmf.18.Feb 7 2020, 4:10 PM
Addshore mentioned this in T219123: Migrate to and read from new store for item terms.Feb 7 2020, 5:44 PM
In T244529#5859927, @Addshore wrote:
  • We should not read terms from the new term store for item ids higher than where the migration script has run to (currently ~10,000,000), as past that point is not populated fully, hence nothing being able to lookup labels and falling back to item ids.
    • This started happening on wikis with the wmf.18 train rollout which included a patch changing the defaults in Wikibase to read from the new term store for all items https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/Wikibase/+/566513/
    • This happened due to the oversight in configuration name which was changed in https://gerrit.wikimedia.org/r/570892, caused an outage, and then was reverted (NOT YET FIXED)
    • This fix meant that all wikis started reading from the new termstore for items under Q10,000,000 when previously the defaults were used meaning group0&1 reading from the new term store for all items and group2 was reading from the new termstore for no items.

Current state:

  • group0 and group1 sites will retrieve no or incorrect data when looking up most item labels and descriptions. Lots of these lookups will end up causing unexpected alterations in parsed and cached pages.
  • group2 is currently on .16 so still reading the wikibase default setting from .16, which is, do not read from the new terms storage for an item ids.

UBNs:

This will leave the train able to continue and all client wikis reading from the old term store (as everything was on monday of this week before .18 rolled out).

Folowups:

  • Alter mediawiki-config wmgWikibaseTmpItemTermsMigrationStage values in mediawiki-config to match what is currently being used in production for clients (read from old everywhere)
  • Fix the incorrectly named config var in mediawiki-config
  • Move forward with changes reading from the new term store for items on clients (see T225057: Switch `tmpItemTermsMigrationStages` to MIGRATION_WRITE_NEW) in a controlled manner.
gerritbot added a comment.Feb 17 2020, 10:38 AM

Change 572628 had a related patch set uploaded (by Ladsgroup; owner: Ladsgroup):
[operations/mediawiki-config@master] Start reading for the new term store for clients up to Q1000

https://gerrit.wikimedia.org/r/572628

gerritbot added a project: Patch-For-Review.Feb 17 2020, 10:38 AM
gerritbot added a comment.Feb 18 2020, 12:10 PM

Change 572628 merged by jenkins-bot:
[operations/mediawiki-config@master] Start reading for the new term store for clients up to Q1000

https://gerrit.wikimedia.org/r/572628

Stashbot added a comment.Feb 18 2020, 12:14 PM

Mentioned in SAL (#wikimedia-operations) [2020-02-18T12:14:38Z] <ladsgroup@deploy1001> Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:572628|Start reading for the new term store for clients up to Q1000 (T225057)]] (duration: 01m 05s)

gerritbot added a comment.Feb 18 2020, 12:29 PM

Change 572853 had a related patch set uploaded (by Ladsgroup; owner: Ladsgroup):
[operations/mediawiki-config@master] Increase the read for clients on the new term store up to Q100K

https://gerrit.wikimedia.org/r/572853

gerritbot added a comment.Feb 19 2020, 10:04 AM

Change 573236 had a related patch set uploaded (by Ladsgroup; owner: Ladsgroup):
[operations/mediawiki-config@master] Add 1000 more items to read from the new store in clients

https://gerrit.wikimedia.org/r/573236

gerritbot added a comment.Feb 19 2020, 12:16 PM

Change 573236 merged by jenkins-bot:
[operations/mediawiki-config@master] Add 1000 more items to read from the new store in clients

https://gerrit.wikimedia.org/r/573236

Stashbot added a comment.Feb 19 2020, 12:22 PM

Mentioned in SAL (#wikimedia-operations) [2020-02-19T12:22:09Z] <ladsgroup@deploy1001> Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:573236|Start reading for the new term store for clients up to Q2000 (T225057)]] (duration: 01m 06s)

Stashbot added a comment.Feb 19 2020, 12:25 PM

Mentioned in SAL (#wikimedia-operations) [2020-02-19T12:25:09Z] <ladsgroup@deploy1001> Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:573236|Start reading for the new term store for clients up to Q2000 (T225057)]], take II, the cache issue (duration: 01m 04s)

gerritbot added a comment.Feb 19 2020, 5:17 PM

Change 573340 had a related patch set uploaded (by Addshore; owner: Addshore):
[operations/mediawiki-config@master] From 2k->4k for items to read from the new store in clients

https://gerrit.wikimedia.org/r/573340

gerritbot added a comment.Feb 19 2020, 5:21 PM

Change 573340 merged by jenkins-bot:
[operations/mediawiki-config@master] From 2k->4k for items to read from the new store in clients

https://gerrit.wikimedia.org/r/573340

Stashbot added a comment.Feb 19 2020, 5:26 PM

Mentioned in SAL (#wikimedia-operations) [2020-02-19T17:26:29Z] <addshore@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Start reading for the new term store for clients up to Q4000 (T225057) (duration: 01m 01s)

Stashbot added a comment.Feb 19 2020, 5:39 PM

Mentioned in SAL (#wikimedia-operations) [2020-02-19T17:39:18Z] <addshore@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Start reading for the new term store for clients up to Q4000 (T225057) (just incase of cache issue) (duration: 01m 04s)

gerritbot added a comment.Feb 20 2020, 6:50 AM

Change 573442 had a related patch set uploaded (by Addshore; owner: Addshore):
[operations/mediawiki-config@master] From 4k->6k for items to read from the new store in clients

https://gerrit.wikimedia.org/r/573442

gerritbot added a comment.Feb 20 2020, 6:58 AM

Change 573442 merged by jenkins-bot:
[operations/mediawiki-config@master] From 4k->6k for items to read from the new store in clients

https://gerrit.wikimedia.org/r/573442

Stashbot added a comment.Feb 20 2020, 7:00 AM

Mentioned in SAL (#wikimedia-operations) [2020-02-20T07:00:04Z] <addshore@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Start reading for the new term store for clients up to Q6000 (T225057) (duration: 01m 06s)

Stashbot added a comment.Feb 20 2020, 7:01 AM

Mentioned in SAL (#wikimedia-operations) [2020-02-20T07:01:30Z] <addshore@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Start reading for the new term store for clients up to Q6000 (T225057) - extra sync for cache issue (duration: 01m 04s)

gerritbot added a comment.Feb 20 2020, 7:05 AM

Change 573446 had a related patch set uploaded (by Addshore; owner: Addshore):
[operations/mediawiki-config@master] From 6k->8k for items to read from the new store in clients

https://gerrit.wikimedia.org/r/573446

gerritbot added a comment.Feb 20 2020, 7:14 AM

Change 573446 merged by jenkins-bot:
[operations/mediawiki-config@master] From 6k->8k for items to read from the new store in clients

https://gerrit.wikimedia.org/r/573446

Stashbot added a comment.Feb 20 2020, 7:15 AM

Mentioned in SAL (#wikimedia-operations) [2020-02-20T07:15:53Z] <addshore@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Start reading for the new term store for clients up to Q8000 (T225057) (duration: 01m 03s)

Stashbot added a comment.Feb 20 2020, 7:17 AM

Mentioned in SAL (#wikimedia-operations) [2020-02-20T07:17:03Z] <addshore@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Start reading for the new term store for clients up to Q8000 (T225057) - in case of cache issue (duration: 01m 03s)

gerritbot added a comment.Feb 20 2020, 7:20 AM

Change 573448 had a related patch set uploaded (by Addshore; owner: Addshore):
[operations/mediawiki-config@master] From 8k->10k for items to read from the new store in clients

https://gerrit.wikimedia.org/r/573448

gerritbot added a comment.Feb 20 2020, 7:24 AM

Change 573448 merged by jenkins-bot:
[operations/mediawiki-config@master] From 8k->10k for items to read from the new store in clients

https://gerrit.wikimedia.org/r/573448

Stashbot added a comment.Feb 20 2020, 7:25 AM

Mentioned in SAL (#wikimedia-operations) [2020-02-20T07:25:43Z] <addshore@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Start reading for the new term store for clients up to Q10k (was Q8k) (T225057) (duration: 01m 03s)

Stashbot added a comment.Feb 20 2020, 7:26 AM

Mentioned in SAL (#wikimedia-operations) [2020-02-20T07:26:54Z] <addshore@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Start reading for the new term store for clients up to Q10k (was Q8k) (T225057) - in case of cache issue (duration: 01m 01s)

gerritbot added a comment.Feb 20 2020, 7:34 AM

Change 573452 had a related patch set uploaded (by Addshore; owner: Addshore):
[operations/mediawiki-config@master] From 10k->15k for items to read from the new store in clients

https://gerrit.wikimedia.org/r/573452

gerritbot added a comment.Feb 20 2020, 7:45 AM

Change 573452 merged by jenkins-bot:
[operations/mediawiki-config@master] From 10k->15k for items to read from the new store in clients

https://gerrit.wikimedia.org/r/573452

Stashbot added a comment.Feb 20 2020, 7:46 AM

Mentioned in SAL (#wikimedia-operations) [2020-02-20T07:46:34Z] <addshore@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Start reading for the new term store for clients up to Q15k (was Q10k) (T225057) (duration: 01m 03s)

Stashbot added a comment.Feb 20 2020, 7:47 AM

Mentioned in SAL (#wikimedia-operations) [2020-02-20T07:47:48Z] <addshore@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Start reading for the new term store for clients up to Q15k (was Q10k) (T225057) - in case of cache issues (duration: 01m 03s)

Addshore mentioned this in T245740: LUA getLabelWithLang calls result in many db queries to terms storage with spikey and unpredictable patterns.Feb 20 2020, 2:13 PM
Addshore closed this task as a duplicate of T219123: Migrate to and read from new store for item terms.Feb 24 2020, 1:50 PM

We decided to just track the read migration under T219123 instead of in these 2 seperate confusing tickets.

gerritbot added a comment.Feb 25 2020, 10:37 PM

Change 572853 abandoned by Ladsgroup:
Increase the read for clients on the new term store up to Q100K

Reason:
Already on Q2M now :]

https://gerrit.wikimedia.org/r/572853

Maintenance_bot removed a project: Patch-For-Review.Feb 25 2020, 11:11 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL