Refactor Edit Analytics according to the new AQSAssist
Open, HighPublic2 Estimated Story Points
Actions

Assigned To

Authored By

	JEbe-WMF
	Mar 20 2024, 2:13 PM

Description

Goal

Extract to aqsassist some repeated and common code that is present in all the services and refactor them to avoid code duplication as we already did with the functions we created to validate the parameters in the handlers.

About the Druid common code refactoring we will take the following into consideration:

The way to parameterize Druid queries that was implemented in AQS 1.0 (https://github.com/wikimedia/analytics-aqs/blob/master/sys/mediawiki-history-schemas.yaml) to parametrize Druid queries using a separate file
We'll start refactoring editor-analytics because its code is already done and we'll have the opportunity of using its unit and integration tests to test the refactored service.

Tasks

Refactor
QA test
Deploy

Details

	Subject	Repo	Branch	Lines +/-
	Refactor Edit Analytics with AQS-Assist	generated-data-platform/aqs/edit-analytics	main	+411 -473

Customize query in gerrit

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Open		JEbe-WMF	T343907 AQS 2.0: aqsassist - Extract some repeated common code from services to this module
		Open		EChukwukere-WMF	T360527 Refactor Edit Analytics according to the new AQSAssist

Event Timeline

JEbe-WMF created this task.Mar 20 2024, 2:13 PM

Restricted Application removed a project: Patch-For-Review. · View Herald TranscriptMar 20 2024, 2:13 PM

JEbe-WMF moved this task from Sprint Backlog to In Process on the Data Products (Data Products Sprint 10) board.Mar 20 2024, 2:14 PM

WDoranWMF moved this task from In Process to Protal to Sprint 11 on the Data Products (Data Products Sprint 10) board.Mar 21 2024, 2:03 PM

WDoranWMF edited projects, added Data Products (Data Products Sprint 11); removed Data Products (Data Products Sprint 10).

WDoranWMF moved this task from Sprint Backlog to In Process on the Data Products (Data Products Sprint 11) board.

JEbe-WMF set the point value for this task to 2.Mar 21 2024, 6:17 PM

Change #1013514 had a related patch set uploaded (by Jennifer Ebe; author: Jennifer Ebe):

[generated-data-platform/aqs/edit-analytics@main] Refactor Edit Analytics with AQS-Assist

https://gerrit.wikimedia.org/r/1013514

gerritbot added a project: Patch-For-Review.Mar 22 2024, 10:29 AM

JEbe-WMF moved this task from In Process to Code Review / Tech Input on the Data Products (Data Products Sprint 11) board.Mar 22 2024, 10:33 AM

VirginiaPoundstone triaged this task as Low priority.Mar 29 2024, 4:26 PM

SGupta-WMF claimed this task.Apr 1 2024, 2:25 AM

SGupta-WMF reassigned this task from SGupta-WMF to JEbe-WMF.Apr 4 2024, 11:06 AM

JEbe-WMF moved this task from Code Review / Tech Input to Testing on the Data Products (Data Products Sprint 11) board.Apr 17 2024, 10:01 AM

JEbe-WMF edited projects, added Data Products (Data Products Sprint 12); removed Data Products (Data Products Sprint 11).Apr 17 2024, 2:10 PM

JEbe-WMF moved this task from Sprint Backlog to Code Review / Tech Input on the Data Products (Data Products Sprint 12) board.

JEbe-WMF moved this task from Code Review / Tech Input to Testing on the Data Products (Data Products Sprint 12) board.Apr 17 2024, 2:13 PM

JEbe-WMF reassigned this task from JEbe-WMF to EChukwukere-WMF.Apr 17 2024, 2:15 PM

Change #1013514 merged by jenkins-bot:

[generated-data-platform/aqs/edit-analytics@main] Refactor Edit Analytics with AQS-Assist

https://gerrit.wikimedia.org/r/1013514

The changes here have been merged to prod. This has been tested through regression run for Edit analytics and no issues found. All tests passed

EChukwukere-WMF moved this task from Testing to To Deploy on the Data Products (Data Products Sprint 12) board.Apr 24 2024, 3:02 PM

EChukwukere-WMF reassigned this task from EChukwukere-WMF to JEbe-WMF.Apr 24 2024, 3:05 PM

xcollazo moved this task from To Deploy to Testing on the Data Products (Data Products Sprint 12) board.Apr 24 2024, 4:27 PM

Pppery removed a project: Patch-For-Review.Apr 30 2024, 4:12 AM

Test status: QA PASS

PROD url to test QA env with, aggregate : 'https://wikimedia.org/api/rest_v1/metrics/edits/aggregate/sw.wikipedia/all-editor-types/all-page-types/daily/20200101/20200131'

PROD url to test QA env with, edits_edited_pages_top_by_edits: 'https://wikimedia.org/api/rest_v1/metrics/edited-pages/top-by-edits/sw.wikipedia/all-editor-types/all-page-types/2018/03/01'

Local env version, aggregate: 'http://localhost:8095/metrics/edits/aggregate/sw.wikipedia/all-editor-types/all-page-types/daily/20200101/20200131'

Local env version, edits_edited_pages_top_by_edits : 'http://localhost:8095/metrics/edited-pages/top-by-edits/sw.wikipedia/all-editor-types/all-page-types/2018/03/01'

data from prod and local were semantically identical.

No regression issues and ready for deployment.

EChukwukere-WMF moved this task from Testing to To Deploy on the Data Products (Data Products Sprint 12) board.Apr 30 2024, 5:14 PM

VirginiaPoundstone edited projects, added Data Products (Data Products Sprint 13); removed Data Products (Data Products Sprint 12), good first task, AQS2.0.May 1 2024, 10:13 PM

VirginiaPoundstone moved this task from Sprint Backlog to To Deploy on the Data Products (Data Products Sprint 13) board.May 1 2024, 10:14 PM

Sfaci reassigned this task from JEbe-WMF to EChukwukere-WMF.Wed, May 29, 12:21 PM

Sfaci edited projects, added Data Products (Data Products Sprint 14), AQS2.0; removed Data Products (Data Products Sprint 13).

Sfaci moved this task from Sprint Backlog to Testing on the Data Products (Data Products Sprint 14) board.Wed, May 29, 12:25 PM

VirginiaPoundstone raised the priority of this task from Low to High.Wed, May 29, 4:32 PM

Test status in PROD : QA PASS

There were slight changes made to validation errors that I needed minro test update

11 features passed, 0 failed, 17 skipped
340 scenarios passed, 0 failed, 890 skipped
3269 steps passed, 0 failed, 7382 skipped, 0 undefined
Took 0m31.646s

EChukwukere-WMF updated the task description. (Show Details)Wed, May 29, 5:37 PM

EChukwukere-WMF moved this task from Testing to To Deploy on the Data Products (Data Products Sprint 14) board.

EChukwukere-WMF moved this task from To Deploy to Done on the Data Products (Data Products Sprint 14) board.Wed, May 29, 7:39 PM