Page MenuHomePhabricator

Migrate to and read from new store for property terms
Open, MediumPublic

Description

  • first run on test wikidata
  • then on production

The migration will go through 4 stages:

  • stage 1: data is moved from old to the new table
  • stage 2: reading still happens from the old table and writing happens on both (we write twice) + run migration scripts
  • stage 3: writing to both + reading only from the new one. In case of problems, we will still have the two tables synced.
  • stage 4: switch everything to the new table. The old table will stay there, idle, with the last state but the data won't change again, since the writing will only happen on the new table now.

The scripts already exist.

We will have to test after every stage whether everything/wikidata works as expected.

Couple of thousands of properties with their aliases.

Related Objects

StatusSubtypeAssignedTask
OpenNone
OpenArielGlenn
OpenNone
ResolvedAddshore
Resolved alaa_wmde
Resolved alaa_wmde
Resolved alaa_wmde
ResolvedNone
DeclinedNone
Declined alaa_wmde
ResolvedLadsgroup
ResolvedAddshore
ResolvedLadsgroup
ResolvedJeroenDeDauw
DeclinedNone
ResolvedNone
ResolvedNone
ResolvedLadsgroup
ResolvedNone
InvalidNone
DeclinedNone
ResolvedLucas_Werkmeister_WMDE
InvalidNone
DeclinedNone
ResolvedLadsgroup
DeclinedNone
ResolvedJeroenDeDauw
DeclinedNone
Resolved alaa_wmde
ResolvedLucas_Werkmeister_WMDE
ResolvedLucas_Werkmeister_WMDE
ResolvedLucas_Werkmeister_WMDE
ResolvedAddshore
ResolvedNone
InvalidNone
Resolved alaa_wmde
OpenLadsgroup
InvalidNone
OpenLadsgroup
ResolvedLadsgroup
OpenLadsgroup
ResolvedLadsgroup
ResolvedLadsgroup
ResolvedLadsgroup
OpenNone
ResolvedAddshore
ResolvedAddshore
ResolvedLadsgroup
OpenLadsgroup
ResolvedJun 17 2019 alaa_wmde
Resolved alaa_wmde
ResolvedJun 18 2019Ladsgroup
ResolvedNone
ResolvedJun 23 2019 alaa_wmde
ResolvedJun 25 2019Ladsgroup
Resolved alaa_wmde
Resolved alaa_wmde
ResolvedBUG REPORT alaa_wmde
OpenLadsgroup

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 26 2019, 4:42 PM
alaa_wmde renamed this task from [Checkpoint 3] Run migration of property terms to Run migration of property terms.Jun 17 2019, 10:51 AM
alaa_wmde moved this task from Incoming to Ready to estimate on the Wikidata-Campsite board.
alaa_wmde set Due Date to Jun 18 2019, 10:00 PM.
Restricted Application changed the subtype of this task from "Task" to "Deadline". · View Herald TranscriptJun 17 2019, 10:52 AM
darthmon_wmde updated the task description. (Show Details)
alaa_wmde changed Due Date from Jun 18 2019, 10:00 PM to Jun 23 2019, 10:00 PM.Jun 20 2019, 3:05 PM
Addshore moved this task from incoming to in progress on the Wikidata board.Jun 21 2019, 11:28 PM
Restricted Application changed the subtype of this task from "Deadline" to "Task". · View Herald TranscriptJun 26 2019, 6:26 AM
alaa_wmde renamed this task from Run migration of property terms to Migrate property terms into new store.Oct 1 2019, 2:13 PM

Aaah, except this ticket also includes switching to read from the new store etc. my bad..

Addshore renamed this task from Migrate property terms into new store to Migrate to and read from new store for property terms.Dec 19 2019, 5:10 PM

Mentioned in SAL (#wikimedia-operations) [2020-01-06T03:53:50Z] <Amir1> ladsgroup@mwmaint1002:~$ mwscript extensions/Wikibase/repo/maintenance/rebuildItemTerms.php --wiki=wikidatawiki --batch-size=100 --sleep=2 --file=/tmp/100k.lines (T219301)

Mentioned in SAL (#wikimedia-operations) [2020-01-06T07:37:26Z] <Amir1> ladsgroup@mwmaint1002:~$ mwscript extensions/Wikibase/repo/maintenance/rebuildItemTerms.php --wiki=wikidatawiki --batch-size=100 --sleep=2 --file=/tmp/1mio.lines (T219301)

Addshore triaged this task as Medium priority.Tue, Jan 14, 4:17 PM

As of 1.35.0-wmf.15 we believe we can stop writing to properties now.
Some notes about why we are ready for wikidata org can be found in P10163 (we cant change the default for wikibase until a few more things happen.

This will need a re announcement to stop writing to wb_terms for properties.

Restricted Application added a project: User-Ladsgroup. · View Herald TranscriptWed, Jan 15, 5:22 PM

Mentioned in SAL (#wikimedia-releng) [2020-01-16T09:31:55Z] <Amir1> "delete from wikidatawiki.wb_terms where term_full_entity_id like 'P%'" (T219301 T225054)

Mentioned in SAL (#wikimedia-operations) [2020-01-16T12:19:32Z] <Amir1> "delete from testwikidatawiki.wb_terms where term_full_entity_id like 'P%'" (T219301 T225054)