Page MenuHomePhabricator
Create Task
Maniphest T368103

Data QA of wikilambda_zobject_join table
Closed, ResolvedPublic
Actions

Description

Description

This new wikilambda_zobject_join WikiLambda table (created by T357552) will be sqoop'ed into the Data Lake, once the data transfer patches are deployed, and will be used for certain Wikifunctions "inventory metrics", such as T355637 and T355638. The location of the table is given in this patch

We will need to QA the aggregate data once we start receiving data to confirm data is logged as expected once in production.

The 3 data transfer tasks (declaring the table in the Analytics environment, adding the table to sqoop config; triggering the sqoop job every 24 hours) are all children of T363439, as is this task.

Related Objects
Search...

Event Timeline

DMartin-WMF created this task.Jun 20 2024, 7:39 PM
DMartin-WMF added a parent task: T363439: [24Q4] WikiLambda Metrics: Create DB table wikilambda_zobject_join & make it available in Analytics Data Lake.
DMartin-WMF added subtasks: T367847: WikiLambda: On production, create wikilambda_zobject_join table and initialize its contents, T363436: WikiLambda metrics: Create wikilambda_zobject_join table in Hive, T363434: WikiLambda metrics: Add wikilambda_zobject_join query to sqoop, T363435: WikiLambda metrics: Create puppet scripting for wikilambda_zobject_join.Jun 20 2024, 7:44 PM
Jdforrester-WMF moved this task from To Triage to 25Q1 (Jul–Sep) on the Abstract Wikipedia team board.Jun 26 2024, 4:54 PM
Jdforrester-WMF edited projects, added: Abstract Wikipedia team (25Q1 (Jul–Sep)); removed: Abstract Wikipedia team.
Jdforrester-WMF triaged this task as High priority.Jun 26 2024, 5:00 PM
Jdforrester-WMF moved this task from Incoming to Essential work on the Abstract Wikipedia team (25Q1 (Jul–Sep)) board.
Jdforrester-WMF changed the status of subtask T367847: WikiLambda: On production, create wikilambda_zobject_join table and initialize its contents from Open to In Progress.Jun 27 2024, 9:11 AM
DMartin-WMF added a subtask: T357552: WikiLambda DB: Extend DB content to support metrics & search needs.Jul 2 2024, 4:51 AM
Jdforrester-WMF closed subtask T367847: WikiLambda: On production, create wikilambda_zobject_join table and initialize its contents as Resolved.Jul 2 2024, 5:37 PM
Jdforrester-WMF closed subtask T357552: WikiLambda DB: Extend DB content to support metrics & search needs as Resolved.
Jdforrester-WMF closed subtask T363436: WikiLambda metrics: Create wikilambda_zobject_join table in Hive as Resolved.Jul 10 2024, 7:13 AM
Jdforrester-WMF closed subtask T363434: WikiLambda metrics: Add wikilambda_zobject_join query to sqoop as Resolved.
Jdforrester-WMF closed subtask T363435: WikiLambda metrics: Create puppet scripting for wikilambda_zobject_join as Resolved.
DMartin-WMF claimed this task.Jul 12 2024, 7:58 PM
DMartin-WMF added a comment.Jul 17 2024, 5:36 PM

The table contents look good and the table grows a reasonable amount every 24 hours as expected. The number of functions recorded in the table is consistently very close to the number of functions listed at https://www.wikifunctions.org/wiki/Special:ListObjectsByType/Z8 - but there usually seems to be few more functions in the table than on the wiki listing, which is a bit puzzling.

DMartin-WMF closed this task as Resolved.Jul 18 2024, 3:24 PM

That small discrepancy is gone today, and when it did occur there are several possible explanations, so closing this. Will continue to keep an eye on it.

Jdforrester-WMF moved this task from Essential work to Needs Sign-off on the Abstract Wikipedia team (25Q1 (Jul–Sep)) board.Aug 19 2024, 7:10 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits