Page MenuHomePhabricator

Migrate to and read from new store for item terms
Open, MediumPublic

Details

Related Gerrit Patches:
operations/puppet : productionmediawiki: Drop the rebuildItemTerms cron
operations/mediawiki-config : masterSet read new for item term store up to Q100K
operations/puppet : productionmediawiki: Stop rebuildItemTermIndex job

Related Objects

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 25 2019, 9:19 AM
alaa_wmde renamed this task from [Checkpoint 4] Migrate item terms & Refactor code to [Checkpoint 4] Migrate item terms.Mar 25 2019, 12:28 PM
alaa_wmde renamed this task from [Checkpoint 4] Migrate item terms to [Checkpoint 4] Migrate item terms & Prepare report, announcements and next step.
Addshore moved this task from incoming to in progress on the Wikidata board.Mar 25 2019, 4:00 PM
alaa_wmde renamed this task from [Checkpoint 4] Migrate item terms & Prepare report, announcements and next step to [Checkpoint 5] Migrate item terms.Mar 26 2019, 4:43 PM
alaa_wmde renamed this task from [Checkpoint 5] Migrate item terms to [Checkpoint 6] Migrate item terms.Apr 23 2019, 9:08 PM
alaa_wmde renamed this task from [Checkpoint 6] Migrate item terms to [Checkpoint 6] Run Migration of item terms.Jun 5 2019, 6:44 AM
alaa_wmde moved this task from Incoming to Ready to estimate on the Wikidata-Campsite board.
alaa_wmde set Due Date to Jul 2 2019, 10:00 PM.
Restricted Application changed the subtype of this task from "Task" to "Deadline". · View Herald TranscriptJun 17 2019, 10:53 AM
alaa_wmde renamed this task from [Checkpoint 6] Run Migration of item terms to Run Migration of item terms.Jun 18 2019, 12:32 PM
Restricted Application changed the subtype of this task from "Deadline" to "Task". · View Herald TranscriptJul 5 2019, 1:02 PM

Moved back as it hasn't been worked on for a while, and the priority now is to make sure we switch properties to new store successfully in production.

Restricted Application added a project: User-Ladsgroup. · View Herald TranscriptSep 2 2019, 11:34 AM

We just passed Q43Mio now.

This comment was removed by Addshore.
Addshore renamed this task from Run Migration of item terms to Migrate to and read from new store for item terms.Dec 19 2019, 5:11 PM

We are done:

ladsgroup@mwmaint1002:~$ tail /var/log/wikidata/wikidata-rebuildItemTerms.log
Rebuilding Q97070475 till Q97070624
Rebuilding Q97070625 till Q97070774
Rebuilding Q97070775 till Q97070924
Rebuilding Q97070925 till Q97071074
Rebuilding Q97071075 till Q97071224
Rebuilding Q97071225 till Q97071374
Rebuilding Q97071375 till Q97071524
Rebuilding Q97071525 till Q97071674
Rebuilding Q97071675 till Q97071824
Rebuilding Q97071825 till Q97071974

Change 562080 had a related patch set uploaded (by Ladsgroup; owner: Ladsgroup):
[operations/puppet@production] mediawiki: Stop rebuildItemTermIndex job

https://gerrit.wikimedia.org/r/562080

Change 562081 had a related patch set uploaded (by Ladsgroup; owner: Ladsgroup):
[operations/mediawiki-config@master] Set read new for item term store up to Q100K

https://gerrit.wikimedia.org/r/562081

Change 562080 merged by Ema:
[operations/puppet@production] mediawiki: Stop rebuildItemTermIndex job

https://gerrit.wikimedia.org/r/562080

Change 562081 merged by jenkins-bot:
[operations/mediawiki-config@master] Set read new for item term store up to Q100K

https://gerrit.wikimedia.org/r/562081

Mentioned in SAL (#wikimedia-operations) [2020-01-06T12:15:02Z] <ladsgroup@deploy1001> Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:562081|Set read new for item term store up to Q100K (T219123)]] (duration: 00m 55s)

Setting to read new for up to Q100k caused this:

Reads are on the old table are dropping.

Mentioned in SAL (#wikimedia-operations) [2020-01-06T21:28:08Z] <Amir1> starting rebuild of holes in new term store from Q1Mio to Q10Mio using screen in mwmaint1002 (T219123)

Change 562487 had a related patch set uploaded (by Ladsgroup; owner: Ladsgroup):
[operations/puppet@production] mediawiki: Drop the rebuildItemTerms cron

https://gerrit.wikimedia.org/r/562487

Change 562487 merged by Marostegui:
[operations/puppet@production] mediawiki: Drop the rebuildItemTerms cron

https://gerrit.wikimedia.org/r/562487

Addshore triaged this task as Medium priority.Tue, Jan 14, 4:09 PM

From @Ladsgroup in mattermost:

Up to Q15M is already fixed. We can turn on the reading slowly on, I was thinking of it today

From @Ladsgroup

The filling holes is now up to Q17Mio
17.6Mio

Currently at Q25.3 million ish

Currently at Q34 million ish.

Addshore claimed this task.Tue, Jan 28, 1:45 PM
Addshore added a subscriber: Ladsgroup.

I'm stealing this for the next week.

Restricted Application added a project: User-Addshore. · View Herald TranscriptTue, Jan 28, 1:45 PM

Mentioned in SAL (#wikimedia-operations) [2020-01-28T15:06:51Z] <addshore> Start addshore@mwmaint1002:~$ ./T219123.sh # Taking over from @Ladsgroup for T219123

Mentioned in SAL (#wikimedia-operations) [2020-01-28T15:06:51Z] <addshore> Start addshore@mwmaint1002:~$ ./T219123.sh # Taking over from @Ladsgroup for T219123