Page MenuHomePhabricator
Create Task
Maniphest T225054

Switch `tmpPropertyTermsMigrationStage` to MIGRATION_NEW
Closed, ResolvedPublic
Actions

Description

If all goes well by the due date here, we should be confident that property terms can be now switch completely (reading and writing) to new schema. Wait couple of days after switching to MIGRATION_WRITE_NEW before doing this

Details

SubjectRepoBranchLines +/-
Stop writing to the old term store for propertiesoperations/mediawiki-configmaster+1 -2
Stop writing to wb_terms for properties in Test Wikidataoperations/mediawiki-configmaster+1 -0
labs: Set migration stage for properties to newoperations/mediawiki-configmaster+2 -2
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedAddshoreT208425 [EPIC] Kill the wb_terms table
 ResolvedArielGlennT226167 audit public tables and make sure we dump them all
 ResolvedAddshoreT219175 [Mega] - Migrate data from wb_terms to new schema
 ResolvedAddshoreT219120 [Checkpoint 1] Create Schema, Migration plan and Doctrine DBAL connection
Resolved• alaa_wmdeT219142 Send the schema to DBAs and follow up with their review
 Resolved• alaa_wmdeT219143 Write DDL sql files for creating the schema
 Resolved• alaa_wmdeT220210 Create a script to generate lots of Items/Properties with lots of Terms
 ResolvedNoneT220230 Tailor a migration plan
 DeclinedNoneT220231 Migration Plan 2
 Declined• alaa_wmdeT219145 Migration Plan 1
 ResolvedLadsgroupT220480 Migration Plan 3
 ResolvedAddshoreT219121 [Checkpoint 2] Refactor Write & Migration code for Properties
 ResolvedLadsgroupT219294 Create interface for writing and deleting entity terms
 ResolvedJeroenDeDauwT219295 Create service for writing property terms
 DeclinedNoneT220169 Optimize term updating
 ResolvedNoneT219297 Adjust current write logic to write property terms to new schema
 ResolvedNoneT219894 Create maintenance script for migrating property terms to new schema
 ResolvedLadsgroupT219296 Update TermSqlStore to comply with EntityTermStoreWriter interface
 ResolvedNoneT220150 [Story] Clean up unused records
 InvalidNoneT221699 Introduce appropriate abstraction for clean up invocation in wikibase-term-store
 DeclinedNoneT221700 implement clean up logic in doctrine-term-store
 ResolvedLucas_Werkmeister_WMDET221701 implement clean up logic in MW DBAL store
 InvalidNoneT221703 add a job to wikibase to invoke clean up logic
 DeclinedNoneT221704 Update wikibase logic to queue a deferred update for cleanup after updating/delete terms in terms store
 ResolvedLadsgroupT221615 Make a test instance in cloud VPS to test new term store
 DeclinedNoneT219805 Investigate Doctrine DBAL usage possibility
 ResolvedJeroenDeDauwT220155 Implement Doctrine DBAL Driver Connection wrapper for MW db connection
 DeclinedNoneT221568 Improve MW connection access
 Resolved• alaa_wmdeT224269 Handle race-conditions between storage and cleanup logic
 ResolvedLucas_Werkmeister_WMDET225050 Update DatabaseItemTermStore and DatabasePropertyTermStore to use callback logic in DatabaseTermIdsAcquirer
 ResolvedLucas_Werkmeister_WMDET224504 Implement database-based PropertyTermStore
 ResolvedLucas_Werkmeister_WMDET224541 Update Wikibase entity saving logic to use PropertyTermStore
 ResolvedAddshoreT219122 [Checkpoint 4] Refactor Write & Migration code for Items
 ResolvedNoneT221564 Create maintenance script for populating new item term persistence
 InvalidNoneT224542 Update Wikibase entity saving logic to use ItemTermStore
 Resolved• alaa_wmdeT224527 Implement database-based ItemTermStore
 ResolvedLadsgroupT219123 Migrate to and read from new store for item terms
 InvalidNoneT221715 Implement ItemTermStore in mediawiki dbal lib
 DuplicateLadsgroupT225055 Switch `tmpItemTermsMigrationStages` to MIGRATION_WRITE_BOTH
 ResolvedLadsgroupT225056 Run Item Terms Rebuild script
 DuplicateLadsgroupT225057 Switch `tmpItemTermsMigrationStages` to MIGRATION_WRITE_NEW
 ResolvedLadsgroupT234329 Investigate, fix and reimport failing item terms into new store
 ResolvedLadsgroupT232393 Find out why rebuilding some items in new term store failed
 ResolvedLadsgroupT234948 New Wikibase deadlocks on Wikidata wiki since 2019-10-08T00:00:02: Wikibase\Lib\Store\Sql\Terms\{closure} Deadlock found when trying to get lock; try restarting transaction
 ResolvedAddshoreT235265 Slow query on SpecialMostLinked creating lag on wikidata slaves
 ResolvedAddshoreT239470 Check the success of the initial terms migration (does it have holes)
 ResolvedAddshoreT239471 Sqoop wikidata terms tables into hadoop
 ResolvedLadsgroupT240235 Alter rebuildItemTerms.php to allow passing of a a list of int item ids to process
 ResolvedAddshoreT245740 LUA getLabelWithLang calls result in many db queries to terms storage with spikey and unpredictable patterns
 ResolvedLadsgroupT246005 Wikibase client description API module results in 15k selected rows with new term storage
 ResolvedLadsgroupT246159 Setting on read new for items causes increased worst response time and number of open connections
 ResolvedLadsgroupT219301 Migrate to and read from new store for property terms
 ResolvedLadsgroupT225054 Switch `tmpPropertyTermsMigrationStage` to MIGRATION_NEW

Event Timeline

• alaa_wmde created this task.Jun 5 2019, 6:43 AM
• alaa_wmde updated the task description. (Show Details)Jun 13 2019, 5:57 PM
• alaa_wmde added a comment.Jun 20 2019, 3:13 PM

Thinking again about this, we probably should not switch this to WRITE_NEW until we can switch tmpItemermsMigrationStage to WRITE_NEW as well.. just in case we have to switch back to WRITE_OLD or WRITE_BOTH temporarily to fix any issues we face while running item terms migration.

• alaa_wmde updated the task description. (Show Details)Jun 20 2019, 3:14 PM
Addshore moved this task from incoming to in progress on the Wikidata board.Jun 21 2019, 11:37 PM
Addshore triaged this task as Medium priority.Jan 15 2020, 5:19 PM
Addshore added a project: Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)).
Ladsgroup claimed this task.Jan 15 2020, 5:19 PM
Restricted Application added a project: User-Ladsgroup. · View Herald TranscriptJan 15 2020, 5:19 PM
gerritbot added a comment.Jan 15 2020, 5:21 PM

Change 565064 had a related patch set uploaded (by Ladsgroup; owner: Ladsgroup):
[operations/mediawiki-config@master] labs: Set migration stage for properties to new

https://gerrit.wikimedia.org/r/565064

gerritbot added a project: Patch-For-Review.Jan 15 2020, 5:21 PM
Addshore moved this task from To Do (prioritised from top to bottom) to Doing on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.Jan 15 2020, 5:21 PM
gerritbot added a comment.Jan 15 2020, 5:35 PM

Change 565064 merged by jenkins-bot:
[operations/mediawiki-config@master] labs: Set migration stage for properties to new

https://gerrit.wikimedia.org/r/565064

gerritbot added a comment.Jan 15 2020, 5:38 PM

Change 565073 had a related patch set uploaded (by Ladsgroup; owner: Ladsgroup):
[operations/mediawiki-config@master] Stop writing to wb_terms for properties in Test Wikidata

https://gerrit.wikimedia.org/r/565073

Ladsgroup moved this task from Doing to Peer Review on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.Jan 15 2020, 5:38 PM
Maintenance_bot moved this task from Incoming to In progress on the User-Ladsgroup board.Jan 15 2020, 6:15 PM
Stashbot mentioned this in T219301: Migrate to and read from new store for property terms.Jan 16 2020, 9:31 AM

Mentioned in SAL (#wikimedia-releng) [2020-01-16T09:31:55Z] <Amir1> "delete from wikidatawiki.wb_terms where term_full_entity_id like 'P%'" (T219301 T225054)

gerritbot added a comment.Jan 16 2020, 12:12 PM

Change 565073 merged by jenkins-bot:
[operations/mediawiki-config@master] Stop writing to wb_terms for properties in Test Wikidata

https://gerrit.wikimedia.org/r/565073

Stashbot added a comment.Jan 16 2020, 12:15 PM

Mentioned in SAL (#wikimedia-operations) [2020-01-16T12:15:48Z] <ladsgroup@deploy1001> Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:565073|Stop writing to wb_terms for properties in Test Wikidata (T225054)]] (duration: 01m 04s)

Stashbot added a comment.Jan 16 2020, 12:19 PM

Mentioned in SAL (#wikimedia-operations) [2020-01-16T12:19:32Z] <Amir1> "delete from testwikidatawiki.wb_terms where term_full_entity_id like 'P%'" (T219301 T225054)

Ladsgroup added a comment.Jan 16 2020, 12:26 PM
In T225054#5808841, @Stashbot wrote:

Mentioned in SAL (#wikimedia-operations) [2020-01-16T12:19:32Z] <Amir1> "delete from testwikidatawiki.wb_terms where term_full_entity_id like 'P%'" (T219301 T225054)

This dropped 253K rows from wb_terms in testwikidatawiki

Ladsgroup moved this task from Peer Review to Doing on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.Jan 16 2020, 12:28 PM
Stashbot added a comment.Jan 16 2020, 12:30 PM

Mentioned in SAL (#wikimedia-operations) [2020-01-16T12:30:54Z] <ladsgroup@deploy1001> Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:565073|Stop writing to wb_terms for properties in Test Wikidata (T225054)]] (duration: 01m 05s)

Maintenance_bot removed a project: Patch-For-Review.Jan 16 2020, 1:10 PM
Addshore removed Ladsgroup as the assignee of this task.Feb 3 2020, 7:36 PM
Addshore moved this task from Doing to Stalled/Waiting on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.
Addshore added a subscriber: Ladsgroup.
Addshore subscribed.

Blocked on a fix for T244115: Investigate & Fix holes for aliases in new term tables (take 3)

Addshore changed the task status from Open to Stalled.Feb 3 2020, 7:37 PM
Ladsgroup claimed this task.Mar 5 2020, 11:32 AM
Ladsgroup moved this task from Stalled/Waiting to Peer Review on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.
gerritbot added a comment.Mar 5 2020, 11:32 AM

Change 577218 had a related patch set uploaded (by Ladsgroup; owner: Ladsgroup):
[operations/mediawiki-config@master] Stop writing to the old term store for properties

https://gerrit.wikimedia.org/r/577218

gerritbot added a project: Patch-For-Review.Mar 5 2020, 11:32 AM
gerritbot added a comment.Mar 5 2020, 11:37 AM

Change 577218 merged by jenkins-bot:
[operations/mediawiki-config@master] Stop writing to the old term store for properties

https://gerrit.wikimedia.org/r/577218

Stashbot added a comment.Mar 5 2020, 11:42 AM

Mentioned in SAL (#wikimedia-operations) [2020-03-05T11:42:17Z] <ladsgroup@deploy1001> Synchronized wmf-config/InitialiseSettings.php: [[gerrit:577218|Stop writing to the old term store for properties (T219301 T225054)]] (duration: 01m 04s)

Stashbot added a comment.Mar 5 2020, 11:43 AM

Mentioned in SAL (#wikimedia-operations) [2020-03-05T11:43:35Z] <ladsgroup@deploy1001> Synchronized wmf-config/InitialiseSettings.php: [[gerrit:577218|Stop writing to the old term store for properties (T219301 T225054)]], take II (duration: 01m 04s)

Stashbot added a comment.Mar 5 2020, 11:45 AM

Mentioned in SAL (#wikimedia-operations) [2020-03-05T11:45:35Z] <Amir1> deleting property terms from wb_terms in wikidatawiki (T225054)

Stashbot added a comment.Mar 5 2020, 12:06 PM

Mentioned in SAL (#wikimedia-operations) [2020-03-05T12:06:42Z] <Amir1> the property terms removal is finished. 312K rows deleted (T225054)

Ladsgroup changed the task status from Stalled to Open.Mar 5 2020, 12:07 PM
Ladsgroup moved this task from Peer Review to Test (Verification) on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.
Maintenance_bot removed a project: Patch-For-Review.Mar 5 2020, 12:11 PM
Addshore closed this task as Resolved.Mar 6 2020, 2:00 PM
Maintenance_bot moved this task from In progress to Done on the User-Ladsgroup board.Mar 6 2020, 2:15 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL