Description

This should be done using the same process as T239470: Check the success of the initial terms migration (does it have holes)

Checklist:

Make sure current state of tables (after the time specified) are sqooped into hadoop
Use the notebook https://analytics.wikimedia.org/published/datasets/one-off/wikidata/addshore/T239470-notebook.html and queires there to generate a new list and publish to data sets.
T219123: Migrate to and read from new store for item terms will then use this list in the general population of the tables

		Status	Subtype	Assigned	Task
		Resolved		Addshore	T208425 [EPIC] Kill the wb_terms table
		Resolved		Ladsgroup	T241211 Identify holes in the items terms tables for prod as of 17:00 on the 19th December for rebuilding

Event Timeline

Addshore created this task.Dec 20 2019, 8:33 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptDec 20 2019, 8:33 AM

Addshore added a parent task: T208425: [EPIC] Kill the wb_terms table.Dec 20 2019, 8:33 AM

Addshore triaged this task as Medium priority.Dec 20 2019, 8:35 AM

Addshore mentioned this in T208425: [EPIC] Kill the wb_terms table.

Addshore moved this task from Incoming to Wikidata-Campsite-Iteration-∞ (On Hold) on the Wikidata-Campsite board.

Addshore edited projects, added Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)); removed Wikidata-Campsite.

3:15 PM <joal> Amir1, addshore - Should be good with 'snapshot=2019-12-20'

I'm doing it.

Ladsgroup claimed this task.Dec 20 2019, 10:59 PM

Ladsgroup moved this task from To Do (prioritised from top to bottom) to Doing on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.

Restricted Application added a project: User-Ladsgroup. · View Herald TranscriptDec 20 2019, 10:59 PM

Finally I made it to work, waiting until it finishes.

list of them: https://people.wikimedia.org/~ladsgroup/T241211-20191220-item-terms-migration-holes.gz (gz file, 115MB)

So the maintenance script is on this row right now:

Rebuilding Q63394343 till Q63394492

It means anything after this doesn't need rebuild. I plan to fix the holes million by million using a screen.

Given that there was an almost a day between the time the bug got fixed and when the sqoop happened, the maintenance script got to fix around Q62Mio and still there's 17% inconsistency there. It's way better than ~60% though.

In T241211#5758950, @Ladsgroup wrote:

I plan to fix the holes million by million using a screen.

Awesome, keep T219123 updated! :)

Identify holes in the items terms tables for prod as of 17:00 on the 19th December for rebuilding
Closed, ResolvedPublic
Actions

Description

Related Objects
Search...

Event Timeline

Identify holes in the items terms tables for prod as of 17:00 on the 19th December for rebuildingClosed, ResolvedPublicActions

Description

Related ObjectsSearch...

Event Timeline

Identify holes in the items terms tables for prod as of 17:00 on the 19th December for rebuilding
Closed, ResolvedPublic
Actions

Related Objects
Search...