Maniphest T363434

WikiLambda metrics: Add wikilambda_zobject_join query to sqoop
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	DMartin-WMF
	Apr 25 2024, 5:03 AM

Project Tags

Referenced Files

None

Subscribers

Description

Description

Add a query to sqoop.py (analytics/refinery repo) for periodic extraction of data from the table wikilambda_zobject_join (or whatever table name results from the work on T357552), and its transmission to the data lake. See T341724 for previous similar work.

Completion checklist

Before closing this task, review one by one the checklist available here: https://www.mediawiki.org/wiki/Abstract_Wikipedia_team/Definition_of_Done#Front-end_Task/Bug_Completion_Checklist

Details

Related Changes in Gerrit:

	Subject	Repo	Branch	Lines +/-
	Add missing commas to the sqoop query for wikilambda_zobject_join	analytics/refinery	master	+2 -2
	Add wikilambda_zobject_join query to sqoop	analytics/refinery	master	+25 -0

Customize query in gerrit

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Open		None	T356027 WikiLambda metrics: ideas for expansion or improvement
		Open		None	T355637 WikiLambda metrics: collect type usage for run-function requests
		Resolved		DMartin-WMF	T355638 WikiLambda metrics: track types usage for inputs/outputs of functions
		Open		None	T355923 WikiLambda metrics: Track inventory & uses of "type support" functions
		Resolved		DMartin-WMF	T363439 [24Q4] WikiLambda Metrics: Create DB table wikilambda_zobject_join & make it available in Analytics Data Lake
		Resolved		DMartin-WMF	T368103 Data QA of wikilambda_zobject_join table
		Resolved		DMartin-WMF	T363434 WikiLambda metrics: Add wikilambda_zobject_join query to sqoop
		Resolved		Jdforrester-WMF	T367847 WikiLambda: On production, create wikilambda_zobject_join table and initialize its contents
		Resolved		DMartin-WMF	T357552 WikiLambda DB: Extend DB content to support metrics & search needs

Event Timeline

DMartin-WMF created this task.Apr 25 2024, 5:03 AM

DMartin-WMF mentioned this in T363435: WikiLambda metrics: Create puppet scripting for wikilambda_zobject_join.Apr 25 2024, 5:13 AM

DMartin-WMF added a parent task: T363439: [24Q4] WikiLambda Metrics: Create DB table wikilambda_zobject_join & make it available in Analytics Data Lake.Apr 25 2024, 5:37 AM

DMartin-WMF edited projects, added: Abstract Wikipedia team (24Q4 (Apr–Jun)); removed: Abstract Wikipedia team.Apr 25 2024, 3:56 PM

DMartin-WMF triaged this task as Medium priority.Apr 25 2024, 3:59 PM

DMartin-WMF moved this task from Incoming to Ready, Features on the Abstract Wikipedia team (24Q4 (Apr–Jun)) board.

Change #1041815 had a related patch set uploaded (by David Martin; author: David Martin):

[analytics/refinery@master] Add wikilambda_zobject_join query to sqoop

https://gerrit.wikimedia.org/r/1041815

gerritbot added a project: Patch-For-Review.Jun 12 2024, 1:58 AM

Jdforrester-WMF changed the task status from Open to In Progress.Jun 12 2024, 4:43 PM

Jdforrester-WMF moved this task from Ready, Features to In Progress on the Abstract Wikipedia team (24Q4 (Apr–Jun)) board.

Jdforrester-WMF assigned this task to DMartin-WMF.Jun 14 2024, 3:10 PM

Jdforrester-WMF moved this task from In Progress to Ready to deploy on the Abstract Wikipedia team (24Q4 (Apr–Jun)) board.Jun 20 2024, 6:55 PM

DMartin-WMF added a parent task: T368103: Data QA of wikilambda_zobject_join table.Jun 20 2024, 7:44 PM

DMartin-WMF removed a parent task: T363439: [24Q4] WikiLambda Metrics: Create DB table wikilambda_zobject_join & make it available in Analytics Data Lake.Jun 20 2024, 7:47 PM

DMartin-WMF added a subtask: T367847: WikiLambda: On production, create wikilambda_zobject_join table and initialize its contents.Jul 2 2024, 4:57 AM

Jdforrester-WMF closed subtask T367847: WikiLambda: On production, create wikilambda_zobject_join table and initialize its contents as Resolved.Jul 2 2024, 5:37 PM

Change #1041815 merged by Mforns:

[analytics/refinery@master] Add wikilambda_zobject_join query to sqoop

https://gerrit.wikimedia.org/r/1041815

Jdforrester-WMF closed this task as Resolved.Jul 10 2024, 7:14 AM

Jdforrester-WMF moved this task from Ready to deploy to Needs Sign-off on the Abstract Wikipedia team (24Q4 (Apr–Jun)) board.

Maintenance_bot removed a project: Patch-For-Review.Jul 10 2024, 7:30 AM

Change #1053364 had a related patch set uploaded (by Btullis; author: Btullis):

[analytics/refinery@master] Add missing commas to the sqoop query for wikilambda_zobject_join

https://gerrit.wikimedia.org/r/1053364

gerritbot added a project: Patch-For-Review.Jul 10 2024, 6:03 PM

Change #1053364 merged by Milimetric:

[analytics/refinery@master] Add missing commas to the sqoop query for wikilambda_zobject_join

https://gerrit.wikimedia.org/r/1053364