Page MenuHomePhabricator

Migrate to and read from new store for property terms
Closed, ResolvedPublic

Description

  • first run on test wikidata
  • then on production

The migration will go through 4 stages:

  • stage 1: data is moved from old to the new table
  • stage 2: reading still happens from the old table and writing happens on both (we write twice) + run migration scripts
  • stage 3: writing to both + reading only from the new one. In case of problems, we will still have the two tables synced.
  • stage 4: switch everything to the new table. The old table will stay there, idle, with the last state but the data won't change again, since the writing will only happen on the new table now.

The scripts already exist.

We will have to test after every stage whether everything/wikidata works as expected.

Couple of thousands of properties with their aliases.

Related Objects

StatusSubtypeAssignedTask
ResolvedAddshore
ResolvedArielGlenn
ResolvedAddshore
ResolvedAddshore
Resolved alaa_wmde
Resolved alaa_wmde
Resolved alaa_wmde
ResolvedNone
DeclinedNone
Declined alaa_wmde
ResolvedLadsgroup
ResolvedAddshore
ResolvedLadsgroup
ResolvedJeroenDeDauw
DeclinedNone
ResolvedNone
ResolvedNone
ResolvedLadsgroup
ResolvedNone
InvalidNone
DeclinedNone
ResolvedLucas_Werkmeister_WMDE
InvalidNone
DeclinedNone
ResolvedLadsgroup
DeclinedNone
ResolvedJeroenDeDauw
DeclinedNone
Resolved alaa_wmde
ResolvedLucas_Werkmeister_WMDE
ResolvedLucas_Werkmeister_WMDE
ResolvedLucas_Werkmeister_WMDE
ResolvedAddshore
ResolvedNone
InvalidNone
Resolved alaa_wmde
ResolvedLadsgroup
InvalidNone
DuplicateLadsgroup
ResolvedLadsgroup
DuplicateLadsgroup
ResolvedLadsgroup
ResolvedLadsgroup
ResolvedLadsgroup
ResolvedAddshore
ResolvedAddshore
ResolvedAddshore
ResolvedLadsgroup
ResolvedAddshore
ResolvedLadsgroup
ResolvedLadsgroup
ResolvedLadsgroup
ResolvedJun 17 2019 alaa_wmde
Resolved alaa_wmde
ResolvedJun 18 2019Ladsgroup
ResolvedNone
ResolvedJun 23 2019 alaa_wmde
ResolvedJun 25 2019Ladsgroup
Resolved alaa_wmde
Resolved alaa_wmde
ResolvedBUG REPORT alaa_wmde
ResolvedLadsgroup

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 26 2019, 4:42 PM
alaa_wmde renamed this task from [Checkpoint 3] Run migration of property terms to Run migration of property terms.Jun 17 2019, 10:51 AM
alaa_wmde moved this task from Incoming to Ready to estimate on the Wikidata-Campsite board.
alaa_wmde set Due Date to Jun 18 2019, 10:00 PM.
Restricted Application changed the subtype of this task from "Task" to "Deadline". · View Herald TranscriptJun 17 2019, 10:52 AM
darthmon_wmde updated the task description. (Show Details)
alaa_wmde changed Due Date from Jun 18 2019, 10:00 PM to Jun 23 2019, 10:00 PM.Jun 20 2019, 3:05 PM
Addshore moved this task from incoming to in progress on the Wikidata board.Jun 21 2019, 11:28 PM
Restricted Application changed the subtype of this task from "Deadline" to "Task". · View Herald TranscriptJun 26 2019, 6:26 AM
alaa_wmde renamed this task from Run migration of property terms to Migrate property terms into new store.Oct 1 2019, 2:13 PM

Aaah, except this ticket also includes switching to read from the new store etc. my bad..

Addshore renamed this task from Migrate property terms into new store to Migrate to and read from new store for property terms.Dec 19 2019, 5:10 PM

Mentioned in SAL (#wikimedia-operations) [2020-01-06T03:53:50Z] <Amir1> ladsgroup@mwmaint1002:~$ mwscript extensions/Wikibase/repo/maintenance/rebuildItemTerms.php --wiki=wikidatawiki --batch-size=100 --sleep=2 --file=/tmp/100k.lines (T219301)

Mentioned in SAL (#wikimedia-operations) [2020-01-06T07:37:26Z] <Amir1> ladsgroup@mwmaint1002:~$ mwscript extensions/Wikibase/repo/maintenance/rebuildItemTerms.php --wiki=wikidatawiki --batch-size=100 --sleep=2 --file=/tmp/1mio.lines (T219301)

Addshore triaged this task as Medium priority.Jan 14 2020, 4:17 PM

As of 1.35.0-wmf.15 we believe we can stop writing to properties now.
Some notes about why we are ready for wikidata org can be found in P10163 (we cant change the default for wikibase until a few more things happen.

This will need a re announcement to stop writing to wb_terms for properties.

Restricted Application added a project: User-Ladsgroup. · View Herald TranscriptJan 15 2020, 5:22 PM

Mentioned in SAL (#wikimedia-releng) [2020-01-16T09:31:55Z] <Amir1> "delete from wikidatawiki.wb_terms where term_full_entity_id like 'P%'" (T219301 T225054)

Mentioned in SAL (#wikimedia-operations) [2020-01-16T12:19:32Z] <Amir1> "delete from testwikidatawiki.wb_terms where term_full_entity_id like 'P%'" (T219301 T225054)

Addshore claimed this task.Jan 31 2020, 9:13 AM
Restricted Application added a project: User-Addshore. · View Herald TranscriptJan 31 2020, 9:13 AM

T243944 identified more holes, and also more holes in properties, so this migration stage will also have to be run once more / have things manually fixed
I'll do that as @Ladsgroup is going on vacation!

Mentioned in SAL (#wikimedia-operations) [2020-01-31T09:18:16Z] <addshore> addshore@mwmaint1002:~$ mwscript extensions/Wikibase/repo/maintenance/rebuildPropertyTerms.php --wiki=wikidatawiki --sleep 4 --batch-size=25 # In a screen for T219301

The property tables are once again complete...

mysql:research@dbstore1005.eqiad.wmnet [wikidatawiki]> SELECT   wbpt_property_id as id,   wby_name as type,   wbxl_language as language,   wbx_text as text FROM wbt_property_terms LEFT JOIN wbt_term_in_ lang ON wbpt_term_in_lang_id = wbtl_id LEFT JOIN wbt_type ON wbtl_type_id = wby_id LEFT JOIN wbt_text_in_lang ON wbtl_text_in_lang_id = wbxl_id LEFT JOIN wbt_text ON wbxl_text_id = wbx_id WHERE wbx_text  IS NULL;
Empty set (1.13 sec)
mysql:research@dbstore1005.eqiad.wmnet [wikidatawiki]> SELECT   wbpt_property_id as id,   wby_name as type,   wbxl_language as language,   wbx_text as text FROM wbt_property_terms LEFT JOIN wbt_term_in_ lang ON wbpt_term_in_lang_id = wbtl_id LEFT JOIN wbt_type ON wbtl_type_id = wby_id LEFT JOIN wbt_text_in_lang ON wbtl_text_in_lang_id = wbxl_id LEFT JOIN wbt_text ON wbxl_text_id = wbx_id WHERE wbx_text
  IS NULL;
+------+-------+----------+------+
| id   | type  | language | text |
+------+-------+----------+------+
| 7850 | alias | NULL     | NULL |
| 7786 | alias | NULL     | NULL |
| 7884 | alias | NULL     | NULL |
| 7885 | alias | NULL     | NULL |
| 1416 | alias | NULL     | NULL |
|  932 | alias | NULL     | NULL |
|  846 | alias | NULL     | NULL |
| 2326 | alias | NULL     | NULL |
|  590 | alias | NULL     | NULL |
| 1566 | alias | NULL     | NULL |
|  356 | alias | NULL     | NULL |
+------+-------+----------+------+
11 rows in set (6.33 sec)

Checking this again today it appears there might still be one situation that is causing issue that we should investigate.

Addshore claimed this task.Mar 2 2020, 10:44 AM

Change 577218 had a related patch set uploaded (by Ladsgroup; owner: Ladsgroup):
[operations/mediawiki-config@master] Stop writing to the old term store for properties

https://gerrit.wikimedia.org/r/577218

Change 577218 merged by jenkins-bot:
[operations/mediawiki-config@master] Stop writing to the old term store for properties

https://gerrit.wikimedia.org/r/577218

Mentioned in SAL (#wikimedia-operations) [2020-03-05T11:42:17Z] <ladsgroup@deploy1001> Synchronized wmf-config/InitialiseSettings.php: [[gerrit:577218|Stop writing to the old term store for properties (T219301 T225054)]] (duration: 01m 04s)

Mentioned in SAL (#wikimedia-operations) [2020-03-05T11:43:35Z] <ladsgroup@deploy1001> Synchronized wmf-config/InitialiseSettings.php: [[gerrit:577218|Stop writing to the old term store for properties (T219301 T225054)]], take II (duration: 01m 04s)