Page MenuHomePhabricator
Create Task
Maniphest T225055

Switch `tmpItemTermsMigrationStages` to MIGRATION_WRITE_BOTH
Closed, DuplicatePublic
Actions

Description

would watch the logs for errors for couple of hours before moving on to running the rebuild script

Details

Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
• alaa_wmde set Due Date to Jun 24 2019, 10:00 PM.Jun 5 2019, 6:50 AM
• alaa_wmde updated the task description. (Show Details)
Restricted Application changed the subtype of this task from "Task" to "Deadline". · View Herald TranscriptJun 5 2019, 6:50 AM
• alaa_wmde renamed this task from Switch `tmpItemTermsMigrationStage` to MIGRATION_WRITE_BOTH to Switch `tmpItemTermsTwoMillionMigrationStage` to MIGRATION_WRITE_BOTH.Jun 5 2019, 8:00 AM
• alaa_wmde renamed this task from Switch `tmpItemTermsTwoMillionMigrationStage` to MIGRATION_WRITE_BOTH to Switch `tmpItemTermsMigrationStages` to MIGRATION_WRITE_BOTH.Jun 13 2019, 5:58 PM
• alaa_wmde changed Due Date from Jun 24 2019, 10:00 PM to Jul 2 2019, 10:00 PM.Jun 13 2019, 6:00 PM
gerritbot added a comment.Jun 19 2019, 10:05 AM

Change 517820 had a related patch set uploaded (by Alaa Sarhan; owner: Alaa Sarhan):
[operations/mediawiki-config@master] Switch property terms migration to WRITE_BOTH on test wikidata

https://gerrit.wikimedia.org/r/517820

gerritbot added a project: Patch-For-Review.Jun 19 2019, 10:05 AM
• alaa_wmde added a comment.EditedJun 19 2019, 10:06 AM
In T225055#5268070, @gerritbot wrote:

Change 517820 had a related patch set uploaded (by Alaa Sarhan; owner: Alaa Sarhan):
[operations/mediawiki-config@master] Switch property terms migration to WRITE_BOTH on test wikidata

https://gerrit.wikimedia.org/r/517820

ignore: wrong association by mistake

• alaa_wmde removed a project: Patch-For-Review.Jun 19 2019, 12:19 PM
Addshore moved this task from incoming to in progress on the Wikidata board.Jun 21 2019, 11:37 PM
Ladsgroup added a project: Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)).Jul 31 2019, 10:32 AM
Ladsgroup claimed this task.Jul 31 2019, 11:04 AM
Restricted Application added a project: User-Ladsgroup. · View Herald TranscriptJul 31 2019, 11:04 AM
gerritbot added a comment.Jul 31 2019, 11:04 AM

Change 526646 had a related patch set uploaded (by Ladsgroup; owner: Ladsgroup):
[operations/mediawiki-config@master] Fix typo in name of config

https://gerrit.wikimedia.org/r/526646

gerritbot added a project: Patch-For-Review.Jul 31 2019, 11:04 AM
Ladsgroup mentioned this in T225057: Switch `tmpItemTermsMigrationStages` to MIGRATION_WRITE_NEW.Jul 31 2019, 11:06 AM
gerritbot added a comment.Jul 31 2019, 11:37 AM

Change 526646 merged by jenkins-bot:
[operations/mediawiki-config@master] Fix typo in name of config

https://gerrit.wikimedia.org/r/526646

Stashbot added a comment.Jul 31 2019, 11:40 AM

Mentioned in SAL (#wikimedia-operations) [2019-07-31T11:40:17Z] <ladsgroup@deploy1001> Synchronized wmf-config/Wikibase.php: SWAT: [[gerrit:526646|Fix typo in name of config (T225055) (duration: 00m 47s)

gerritbot added a comment.Jul 31 2019, 12:36 PM

Change 526659 had a related patch set uploaded (by Ladsgroup; owner: Ladsgroup):
[mediawiki/extensions/Wikibase@master] Order keys in ByIdDispatchingItemTermStore

https://gerrit.wikimedia.org/r/526659

Ladsgroup moved this task from To Do (prioritised from top to bottom) to Peer Review on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.Jul 31 2019, 12:36 PM
Ladsgroup moved this task from Peer Review to Doing on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.Jul 31 2019, 2:41 PM
gerritbot added a comment.Jul 31 2019, 11:39 PM

Change 526659 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Order keys in ByIdDispatchingItemTermStore

https://gerrit.wikimedia.org/r/526659

ReleaseTaggerBot added a project: MW-1.34-notes (1.34.0-wmf.17; 2019-08-06).Aug 1 2019, 12:00 AM
gerritbot added a comment.Aug 5 2019, 10:50 AM

Change 528103 had a related patch set uploaded (by Ladsgroup; owner: Ladsgroup):
[operations/mediawiki-config@master] labs: Set half of wikidata to read from the new term store

https://gerrit.wikimedia.org/r/528103

gerritbot added a comment.Aug 5 2019, 12:34 PM

Change 528103 merged by jenkins-bot:
[operations/mediawiki-config@master] labs: Set half of wikidata to read from the new term store

https://gerrit.wikimedia.org/r/528103

• alaa_wmde moved this task from Doing to To Do (prioritised from top to bottom) on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.Aug 19 2019, 10:32 AM
WMDE-leszek subscribed.Aug 28 2019, 11:05 AM

Is this waiting for some other task to be done? Sorry for possibly asking about obvious stuff, I've lost orientation in the state of the migration.

Ladsgroup added a comment.Aug 28 2019, 11:07 AM

It's sorta got blocked on T229816: RebuildItemTerms (and probably RebuildPropertyTerms) explodes when there's a (double) redirect (most of the task description got removed making it a little bit hard to understand IMO). But since it got resolved yesterday and today I tried it in beta cluster and it worked. I'm moving forward with this in test wikidata first though.

Ladsgroup added a comment.Aug 28 2019, 11:08 AM
In T225055#5445236, @Ladsgroup wrote:

It's sorta got blocked on T229816: RebuildItemTerms (and probably RebuildPropertyTerms) explodes when there's a (double) redirect (most of the task description got removed making it a little bit hard to understand IMO). But since it got resolved yesterday and today I tried it in beta cluster and it worked. I'm moving forward with this in test wikidata first though.

Plus the fact that switching property terms exploded production and we needed to make sure things are alright. That got resolved in Monday.

gerritbot added a comment.Aug 28 2019, 11:33 AM

Change 532983 had a related patch set uploaded (by Ladsgroup; owner: Ladsgroup):
[operations/mediawiki-config@master] Enable WRITE_BOTH for items term store for testwikidatawiki

https://gerrit.wikimedia.org/r/532983

Ladsgroup moved this task from To Do (prioritised from top to bottom) to Doing on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.Aug 28 2019, 11:38 AM
gerritbot added a comment.Aug 28 2019, 11:46 AM

Change 532983 merged by jenkins-bot:
[operations/mediawiki-config@master] Enable WRITE_BOTH for items term store for testwikidatawiki

https://gerrit.wikimedia.org/r/532983

Stashbot added a comment.Aug 28 2019, 11:48 AM

Mentioned in SAL (#wikimedia-operations) [2019-08-28T11:48:27Z] <ladsgroup@deploy1001> Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:532983|Enable WRITE_BOTH for items term store for testwikidatawiki (T225055)]] (duration: 00m 54s)

WMDE-leszek removed Due Date.Aug 30 2019, 10:25 AM
gerritbot added a comment.Sep 2 2019, 9:43 AM

Change 533882 had a related patch set uploaded (by Ladsgroup; owner: Ladsgroup):
[operations/mediawiki-config@master] Enable WRITE_BOTH for items term store for wikidata

https://gerrit.wikimedia.org/r/533882

Ladsgroup moved this task from Doing to Peer Review on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.Sep 2 2019, 9:46 AM
gerritbot added a comment.Sep 3 2019, 11:05 AM

Change 533882 merged by jenkins-bot:
[operations/mediawiki-config@master] Enable WRITE_BOTH for items term store for wikidata

https://gerrit.wikimedia.org/r/533882

Stashbot added a comment.Sep 3 2019, 11:07 AM

Mentioned in SAL (#wikimedia-operations) [2019-09-03T11:07:27Z] <ladsgroup@deploy1001> Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:533882|Enable WRITE_BOTH for items term store for wikidatawiki (T225055)]] (duration: 00m 55s)

gerritbot added a comment.Sep 3 2019, 3:37 PM

Change 534183 had a related patch set uploaded (by Ladsgroup; owner: Ladsgroup):
[operations/mediawiki-config@master] Set item terms migration stage for Wikidata on WRITE_BOTH up to Q2m

https://gerrit.wikimedia.org/r/534183

gerritbot added a comment.Sep 4 2019, 11:33 AM

Change 534183 merged by jenkins-bot:
[operations/mediawiki-config@master] Set item terms migration stage for Wikidata on WRITE_BOTH up to Q2m

https://gerrit.wikimedia.org/r/534183

Stashbot added a comment.Sep 4 2019, 11:37 AM

Mentioned in SAL (#wikimedia-operations) [2019-09-04T11:37:47Z] <ladsgroup@deploy1001> Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:534183|Set item terms migration stage for Wikidata on WRITE_BOTH up to Q2m (T225055)]] (duration: 00m 55s)

Ladsgroup moved this task from Peer Review to Doing on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.Sep 4 2019, 2:30 PM
Ladsgroup added a comment.Sep 9 2019, 11:43 AM
In T225055#5464571, @Stashbot wrote:

Mentioned in SAL (#wikimedia-operations) [2019-09-04T11:37:47Z] <ladsgroup@deploy1001> Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:534183|Set item terms migration stage for Wikidata on WRITE_BOTH up to Q2m (T225055)]] (duration: 00m 55s)

This is done. Tomorrow we push for the rest.

gerritbot added a comment.Sep 10 2019, 9:05 AM

Change 535521 had a related patch set uploaded (by Ladsgroup; owner: Ladsgroup):
[operations/mediawiki-config@master] Set items term store on write both for all of Wikidata

https://gerrit.wikimedia.org/r/535521

Ladsgroup moved this task from Doing to Peer Review on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.Sep 10 2019, 9:06 AM
gerritbot added a comment.Sep 10 2019, 11:02 AM

Change 535521 merged by jenkins-bot:
[operations/mediawiki-config@master] Set items term store on write both for all of Wikidata

https://gerrit.wikimedia.org/r/535521

Stashbot added a comment.Sep 10 2019, 11:06 AM

Mentioned in SAL (#wikimedia-operations) [2019-09-10T11:06:02Z] <urbanecm@deploy1001> Synchronized wmf-config/InitialiseSettings.php: SWAT: 6afe963: Set items term store on write both for all of Wikidata (T225055) (duration: 00m 55s)

Ladsgroup moved this task from Peer Review to Test (Verification) on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.Sep 10 2019, 11:10 AM
• alaa_wmde moved this task from Test (Verification) to To Do (prioritised from top to bottom) on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.Sep 11 2019, 10:05 AM
gerritbot added a comment.Sep 11 2019, 10:47 AM

Change 535815 had a related patch set uploaded (by Ladsgroup; owner: Ladsgroup):
[operations/mediawiki-config@master] Set item terms on write both up to Q10mio

https://gerrit.wikimedia.org/r/535815

gerritbot added a comment.Sep 11 2019, 11:12 AM

Change 535815 merged by jenkins-bot:
[operations/mediawiki-config@master] Set item terms on write both up to Q10mio

https://gerrit.wikimedia.org/r/535815

Stashbot added a comment.Sep 11 2019, 11:15 AM

Mentioned in SAL (#wikimedia-operations) [2019-09-11T11:15:28Z] <ladsgroup@deploy1001> Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:535815|Set item terms on write both up to Q10mio (T225055)]] (duration: 01m 03s)

gerritbot added a comment.Sep 12 2019, 11:56 AM

Change 536167 had a related patch set uploaded (by Ladsgroup; owner: Ladsgroup):
[operations/mediawiki-config@master] Set item terms on write both up to Q20Mio

https://gerrit.wikimedia.org/r/536167

gerritbot added a comment.Sep 12 2019, 12:01 PM

Change 536167 merged by jenkins-bot:
[operations/mediawiki-config@master] Set item terms on write both up to Q20Mio

https://gerrit.wikimedia.org/r/536167

Stashbot added a comment.Sep 12 2019, 12:03 PM

Mentioned in SAL (#wikimedia-operations) [2019-09-12T12:03:25Z] <ladsgroup@deploy1001> Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:536167|Set item terms on write both up to Q20mio (T225055)]] (duration: 01m 31s)

gerritbot added a comment.Sep 23 2019, 9:12 AM

Change 538577 had a related patch set uploaded (by Ladsgroup; owner: Ladsgroup):
[operations/mediawiki-config@master] Set item terms on write both up to Q40Mio

https://gerrit.wikimedia.org/r/538577

gerritbot added a comment.Sep 23 2019, 11:21 AM

Change 538577 merged by jenkins-bot:
[operations/mediawiki-config@master] Set item terms on write both up to Q40Mio

https://gerrit.wikimedia.org/r/538577

Stashbot added a comment.Sep 23 2019, 11:23 AM

Mentioned in SAL (#wikimedia-operations) [2019-09-23T11:23:32Z] <awight@deploy1001> Synchronized wmf-config/VariantSettings.php: SWAT: [[gerrit:538577|Set item terms on write both up to Q40Mio (T225055)]] (duration: 00m 55s)

gerritbot added a comment.Oct 7 2019, 1:21 PM

Change 541249 had a related patch set uploaded (by Ladsgroup; owner: Ladsgroup):
[operations/mediawiki-config@master] Set all of wikidata to write both for item term store

https://gerrit.wikimedia.org/r/541249

gerritbot added a comment.Oct 7 2019, 1:23 PM

Change 541249 merged by jenkins-bot:
[operations/mediawiki-config@master] Set all of wikidata to write both for item term store

https://gerrit.wikimedia.org/r/541249

Stashbot added a comment.Oct 7 2019, 1:27 PM

Mentioned in SAL (#wikimedia-operations) [2019-10-07T13:27:21Z] <ladsgroup@deploy1001> Synchronized wmf-config/InitialiseSettings.php: [[gerrit:541249|Set all of wikidata to write both for item term store (T225055)]] (duration: 00m 54s)

Ladsgroup moved this task from To Do (prioritised from top to bottom) to Test (Verification) on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.Oct 7 2019, 1:42 PM
Maintenance_bot moved this task from Incoming to In progress on the User-Ladsgroup board.Oct 23 2019, 10:28 AM
Addshore moved this task from Test (Verification) to Doing on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.Oct 29 2019, 1:44 PM
Addshore subscribed.

Moving back to doing as @Ladsgroup said so

Ladsgroup moved this task from Doing to Stalled/Waiting on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.Oct 30 2019, 10:32 AM

Waiting to fix deadlocks and then we can move forward

gerritbot added a comment.Nov 5 2019, 7:31 AM

Change 548584 had a related patch set uploaded (by Ladsgroup; owner: Ladsgroup):
[operations/mediawiki-config@master] Set all of wikidata for write both for term store

https://gerrit.wikimedia.org/r/548584

Ladsgroup moved this task from Stalled/Waiting to Peer Review on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.Nov 5 2019, 7:31 AM
gerritbot added a comment.Nov 12 2019, 11:13 AM

Change 548584 merged by jenkins-bot:
[operations/mediawiki-config@master] Set all of wikidata for write both for term store

https://gerrit.wikimedia.org/r/548584

Stashbot added a comment.Nov 12 2019, 11:20 AM

Mentioned in SAL (#wikimedia-operations) [2019-11-12T11:20:52Z] <ladsgroup@deploy1001> Synchronized wmf-config/InitialiseSettings.php: [[gerrit:548584|Set all of wikidata for write both for term store (T225055)]] (duration: 00m 52s)

Ladsgroup moved this task from Peer Review to Stalled/Waiting on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.Nov 12 2019, 4:25 PM
Addshore added a comment.Nov 26 2019, 2:04 PM

Currently at 74000000 => MIGRATION_WRITE_BOTH,

DannyS712 mentioned this in T237319: ATS serving 502 errors due to malformed responses from wikibase (HTTP 304s with message body content).Nov 26 2019, 9:09 PM
Addshore added a comment.Jan 7 2020, 10:56 AM

@Ladsgroup is there any reason we can't be at write both for all now?

Ladsgroup added a comment.Jan 7 2020, 11:00 AM
In T225055#5781172, @Addshore wrote:

@Ladsgroup is there any reason we can't be at write both for all now?

We are still running the migration script (to fill the holes) and with migration running at the same time, the load would be too much. We definitely should start look into to getting the rest done but not right now I assume

Addshore triaged this task as Medium priority.Jan 14 2020, 4:19 PM
Addshore moved this task from Stalled/Waiting to Doing on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.Jan 15 2020, 5:17 PM
gerritbot added a comment.Feb 13 2020, 12:23 PM

Change 571961 had a related patch set uploaded (by Ladsgroup; owner: Ladsgroup):
[operations/mediawiki-config@master] Read and write more in the new term store

https://gerrit.wikimedia.org/r/571961

gerritbot added a comment.Feb 13 2020, 12:25 PM

Change 571961 merged by jenkins-bot:
[operations/mediawiki-config@master] Read and write more in the new term store

https://gerrit.wikimedia.org/r/571961

Stashbot mentioned this in T219123: Migrate to and read from new store for item terms.Feb 13 2020, 12:30 PM

Mentioned in SAL (#wikimedia-operations) [2020-02-13T12:30:26Z] <ladsgroup@deploy1001> Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:571961|Read and write more in the new term store]], take II, the cache issue (T219123 T225055) (duration: 01m 03s)

Addshore closed this task as a duplicate of T219123: Migrate to and read from new store for item terms.Feb 24 2020, 1:51 PM

We decided to just track the read migration under T219123 instead of in these 2 seperate confusing tickets.

Lucas_Werkmeister_WMDE mentioned this in T263730: Wikidata term store contains rows for deleted items.Sep 24 2020, 10:25 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL