⚓ T239471 Sqoop wikidata terms tables into hadoop

Subject	Repo	Branch	Lines +/-
sqoop, wb_terms, use term_full_entity_id not term_entity_id	analytics/refinery	master	+3 -3
hive tables for wikibase term secondary storage	analytics/refinery	master	+211 -0
sqoop, add wikidata terms related tables	analytics/refinery	master	+132 -0

Status	Assigned	Task
Resolved	Addshore	T208425 [EPIC] Kill the wb_terms table
Resolved	ArielGlenn	T226167 audit public tables and make sure we dump them all
Resolved	Addshore	T219175 [Mega] - Migrate data from wb_terms to new schema
Resolved	Ladsgroup	T219123 Migrate to and read from new store for item terms
Resolved	Addshore	T239470 Check the success of the initial terms migration (does it have holes)
Resolved	Addshore	T239471 Sqoop wikidata terms tables into hadoop

Addshore created this task.Nov 29 2019, 12:47 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptNov 29 2019, 12:47 PM

Addshore added a parent task: T239470: Check the success of the initial terms migration (does it have holes).Nov 29 2019, 12:47 PM

Change 553698 had a related patch set uploaded (by Addshore; owner: Addshore):
[analytics/refinery@master] sqoop, add wikidata terms related tables

https://gerrit.wikimedia.org/r/553698

gerritbot added a project: Patch-For-Review.Nov 29 2019, 12:48 PM

Addshore claimed this task.Nov 29 2019, 12:48 PM

Addshore added a project: Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)).

Addshore moved this task from To Do (prioritised from top to bottom) to Doing on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.

Restricted Application added a project: User-Addshore. · View Herald TranscriptNov 29 2019, 12:48 PM

Addshore updated the task description. (Show Details)Nov 29 2019, 12:51 PM

Maintenance_bot moved this task from incoming to in progress on the Wikidata board.Nov 29 2019, 1:15 PM

Change 553727 had a related patch set uploaded (by Addshore; owner: Addshore):
[analytics/refinery@master] hive tables for wikibase term secondary storage

https://gerrit.wikimedia.org/r/553727

Should be all done, and tables are created in joal db

@JAllemandou I'll move this to waiting on our board for now.
I guess we should probably merge it all?

Addshore moved this task from Unsorted 💣 to Back Burner 🏛️ on the User-Addshore board.Nov 29 2019, 3:49 PM

JAllemandou added a project: Analytics-Kanban.Nov 29 2019, 4:57 PM

JAllemandou moved this task from Next Up to In Code Review on the Analytics-Kanban board.

GoranSMilovanovic subscribed.Nov 29 2019, 4:58 PM

GoranSMilovanovic awarded a token.Nov 29 2019, 5:00 PM

Change 553698 merged by Joal:
[analytics/refinery@master] sqoop, add wikidata terms related tables

https://gerrit.wikimedia.org/r/553698

Change 553727 merged by Joal:
[analytics/refinery@master] hive tables for wikibase term secondary storage

https://gerrit.wikimedia.org/r/553727

Maintenance_bot removed a project: Patch-For-Review.Dec 2 2019, 9:10 AM

JAllemandou moved this task from In Code Review to Ready to Deploy on the Analytics-Kanban board.Dec 2 2019, 9:21 AM

Ping for @JAllemandou

4:09 PM <addshore> hiya joal, just checking regarding https://gerrit.wikimedia.org/r/#/c/analytics/refinery/+/553698/6..7/python/refinery/sqoop.py
4:09 PM <addshore> should that change have been included in the 2019-10 snapshot? / did it get re run?
4:10 PM <addshore> I see lots of nulls in that field in hadoop which is unexpected

JAllemandou moved this task from Ready to Deploy to Done on the Analytics-Kanban board.Dec 3 2019, 2:53 PM

These things now exist in wmf_raw.wikibase_wbt_item_terms for example =]
And the issue mentioned in T239471#5705470 is resolved.

Addshore reopened this task as Open.Dec 3 2019, 4:35 PM

Addshore moved this task from Done to Doing on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.

Change 554329 had a related patch set uploaded (by Addshore; owner: Addshore):
[analytics/refinery@master] sqoop, wb_terms, use term_full_entity_id not term_entity_id

https://gerrit.wikimedia.org/r/554329

gerritbot added a project: Patch-For-Review.Dec 3 2019, 4:41 PM

Addshore moved this task from Done to In Code Review on the Analytics-Kanban board.Dec 3 2019, 5:32 PM

Change 554329 merged by Joal:
[analytics/refinery@master] sqoop, wb_terms, use term_full_entity_id not term_entity_id

https://gerrit.wikimedia.org/r/554329

Maintenance_bot removed a project: Patch-For-Review.Dec 3 2019, 6:10 PM

JAllemandou moved this task from In Code Review to Done on the Analytics-Kanban board.Dec 4 2019, 7:19 PM

Addshore closed this task as Resolved.Dec 5 2019, 10:02 AM

Addshore moved this task from Doing to Done on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.