Maniphest T341724

WikiLambda metrics: Add wikilambda_zobject_function_join query to sqoop
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	DMartin-WMF
	Jul 12 2023, 8:21 PM

Tags

Referenced Files

None

Subscribers

Description

Description

Add a query to sqoop.py (analytics/refinery repo) for periodic extraction of data from the table wikilambda_zobject_function_join, to be sent to the data lake. An example query is given by this patch.

The table is declared here.

Consider whether to extract everything, or possibly omit some columns from the extraction.

Completion checklist

Before closing this task, review one by one the checklist available here: https://www.mediawiki.org/wiki/Abstract_Wikipedia_team/Definition_of_Done#Front-end_Task/Bug_Completion_Checklist

Details

	Subject	Repo	Branch	Lines +/-
	Add wikilambda_zobject_labels & wikilambda_zobject_function_join queries to sqoop	analytics/refinery	master	+46 -0

Customize query in gerrit

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Open	BUG REPORT	None	T344206 Special:MathStatus exception error on Wikifunctions
		Resolved		Jdforrester-WMF	T342865 Post-creation work for wikifunctionswiki
		Resolved		Jdforrester-WMF	T323399 Add Wikifunctions as a sibling project to the Portals once it launches
		Resolved		cmassaro	T342820 Migrate wikifunctions.org from locked-down to limited mode, letting users edit wikitext pages and some
		Resolved		DMartin-WMF	T341748 Wikifunctions metrics: Track inventory of functions, implementations, tests
		Resolved		DMartin-WMF	T341733 WikiLambda metrics: Track how many tests & implementations exist
		Open		None	T341734 WikiLambda metrics: Track how many tests & implementations are connected to their functions
		Resolved		DMartin-WMF	T341724 WikiLambda metrics: Add wikilambda_zobject_function_join query to sqoop

Event Timeline

DMartin-WMF created this task.Jul 12 2023, 8:21 PM

DMartin-WMF updated the task description. (Show Details)Jul 12 2023, 8:25 PM

DMartin-WMF mentioned this in T341728: WikiLambda metrics: Create wikilambda_zobject_function_join table in Hive.Jul 12 2023, 8:39 PM

DMartin-WMF mentioned this in T341733: WikiLambda metrics: Track how many tests & implementations exist.Jul 12 2023, 8:58 PM

DMartin-WMF mentioned this in T341734: WikiLambda metrics: Track how many tests & implementations are connected to their functions.Jul 12 2023, 9:01 PM

DMartin-WMF added a parent task: T341733: WikiLambda metrics: Track how many tests & implementations exist.Jul 12 2023, 9:04 PM

DMartin-WMF added a parent task: T341730: WikiLambda: Use wikilambda_zobject_function_join to record whether implementations/tests are connected.Jul 12 2023, 9:08 PM

DMartin-WMF removed a parent task: T341730: WikiLambda: Use wikilambda_zobject_function_join to record whether implementations/tests are connected.Jul 12 2023, 9:11 PM

DMartin-WMF added a parent task: T341734: WikiLambda metrics: Track how many tests & implementations are connected to their functions.

DMartin-WMF mentioned this in T341730: WikiLambda: Use wikilambda_zobject_function_join to record whether implementations/tests are connected.Jul 12 2023, 10:31 PM

DMartin-WMF added a parent task: T341748: Wikifunctions metrics: Track inventory of functions, implementations, tests.Jul 12 2023, 11:15 PM

Jdforrester-WMF moved this task from To triage to Phase λ – Launch on the Abstract Wikipedia team board.Jul 13 2023, 4:49 PM

Jdforrester-WMF edited projects, added Abstract Wikipedia team (Phase λ – Launch); removed Abstract Wikipedia team.

Jdforrester-WMF moved this task from Incoming to Engineering tasks on the Abstract Wikipedia team (Phase λ – Launch) board.Jul 13 2023, 4:51 PM

Change 937568 had a related patch set uploaded (by David Martin; author: David Martin):

[analytics/refinery@master] Add wikilambda_zobject_labels & wikilambda_zobject_function_join queries to sqoop

https://gerrit.wikimedia.org/r/937568

gerritbot added a project: Patch-For-Review.Jul 14 2023, 2:26 AM

DMartin-WMF added a project: Data Products.Jul 14 2023, 6:14 PM

Jdforrester-WMF changed the task status from Open to In Progress.Jul 17 2023, 7:02 PM

Jdforrester-WMF moved this task from Engineering tasks to In progress on the Abstract Wikipedia team (Phase λ – Launch) board.

DMartin-WMF claimed this task.Jul 17 2023, 8:53 PM

Change 937568 merged by Milimetric:

[analytics/refinery@master] Add wikilambda_zobject_labels & wikilambda_zobject_function_join queries to sqoop

https://gerrit.wikimedia.org/r/937568

Maintenance_bot removed a project: Patch-For-Review.Jul 18 2023, 6:12 PM

Jdforrester-WMF moved this task from In progress to QA needed on the Abstract Wikipedia team (Phase λ – Launch) board.Jul 19 2023, 12:01 AM

DMartin-WMF mentioned this in T342199: WikiLambda metrics: Create puppet scripting for Wikifunctions tables.Jul 19 2023, 2:27 AM

DVrandecic closed this task as Resolved.Jul 19 2023, 4:03 AM

DMartin-WMF mentioned this in T363434: WikiLambda metrics: Add wikilambda_zobject_join query to sqoop.Thu, Apr 25, 5:03 AM