⚓ T354692 [Data Quality] Implement basic data quality metrics for MW history

Subject	Repo	Branch	Lines +/-
Update column definition for uniqueness check.	analytics/refinery/source	master	+11 -2
Update column definition for uniqueness check.	analytics/refinery/source	master	+11 -2
Update column definition for uniqueness check.	analytics/refinery/source	master	+34 -25
Mediawiki History Data Quality Metrics	analytics/refinery/source	master	+373 -85

	Title	Reference	Author	Source Branch	Dest Branch
	Use refinery job jar v0.2.43 containing fix for MediawwikiHistory duplicate checker	repos/data-engineering/airflow-dags!744	ebysans	mw_dq	main
	Add Dag to Compute Mediawiki History Data Quality	repos/data-engineering/airflow-dags!629	ebysans	Mediawiki_history_dq	main

		Status	Subtype	Assigned	Task
		Open		Snwachukwu	T354692 [Data Quality] Implement basic data quality metrics for MW history
		Open		Snwachukwu	T362803 Update MW history data quality job to use Deequ Anomaly detection Capability

Ahoelzl created this task.Jan 9 2024, 6:37 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 9 2024, 6:37 PM

Ahoelzl assigned this task to Antoine_Quhen.Jan 10 2024, 5:29 PM

Ahoelzl removed Antoine_Quhen as the assignee of this task.Jan 11 2024, 1:59 AM

Ahoelzl added a subscriber: Antoine_Quhen.

Ahoelzl assigned this task to tchin.Jan 11 2024, 2:08 AM

lbowmaker set the point value for this task to 5.Jan 17 2024, 12:49 PM

Ahoelzl reassigned this task from tchin to Snwachukwu.Jan 18 2024, 10:56 PM

Ahoelzl added a subscriber: tchin.

lbowmaker edited projects, added Data-Engineering (Sprint 8); removed Data-Engineering (Sprint 7).Jan 30 2024, 12:22 PM

Ahoelzl updated the task description. (Show Details)Feb 2 2024, 10:35 PM

Ahoelzl changed the point value for this task from 5 to 8.

Ahoelzl added a subscriber: gmodena.

Snwachukwu moved this task from Next Up to In progress on the Data-Engineering (Sprint 8) board.Feb 4 2024, 12:15 AM

@JAllemandou mentioned existing MW checks that should be migrated.

Indeed! here is the code:
https://github.com/wikimedia/analytics-refinery-source/blob/master/refinery-job/src/main/scala/org/wikimedia/analytics/refinery/job/mediawikihistory/MediawikiHistoryChecker.scala

It looks at the top 50 wikis by size, and validates cross-dimensions values, as well as growth expectations.

lbowmaker mentioned this in T342911: Data Quality Issue: Wikitext History Job fail / rerun in Airflow.Feb 6 2024, 1:53 PM

Mayakp.wiki subscribed.Feb 8 2024, 8:05 PM

lbowmaker edited projects, added Data-Engineering (Sprint 9); removed Data-Engineering (Sprint 8).Feb 16 2024, 4:17 PM

lbowmaker moved this task from Next Up to In progress on the Data-Engineering (Sprint 9) board.

Mayakp.wiki mentioned this in T357833: [Data Quality] Implement basic data quality metrics for Unique Devices datasets.Feb 16 2024, 10:46 PM

Change 1008934 had a related patch set uploaded (by Snwachukwu; author: Snwachukwu):

[analytics/refinery/source@master] Mediawiki History Data Quality Metrics

https://gerrit.wikimedia.org/r/1008934

gerritbot added a project: Patch-For-Review.Mar 5 2024, 6:12 PM

Snwachukwu moved this task from In progress to In Review on the Data-Engineering (Sprint 9) board.Mar 6 2024, 9:00 PM

ebysans opened https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/629

Add Dag to Compute Mediawiki History Data Quality

lbowmaker mentioned this in T361016: [Data Quality] Migrate MWHistoryChecker to DeeQu checks.Mar 26 2024, 1:53 PM

In the code implementation for change in size detection, I have compared previous snapshot and current snapshot by reading previous snapshot into a dataframe.
One TODO will be to use the AWS Deequ Anomaly detectiom and filesystem repository capability to implement this check.

Change #1008934 merged by jenkins-bot:

[analytics/refinery/source@master] Mediawiki History Data Quality Metrics

https://gerrit.wikimedia.org/r/1008934

lbowmaker edited projects, added Data-Engineering (Q4 2024 April 1st - June 30th); removed Data-Engineering (Sprint 9).Apr 1 2024, 12:07 PM

lbowmaker moved this task from Next Up to In Review on the Data-Engineering (Q4 2024 April 1st - June 30th) board.

ebysans merged https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/629

Add Dag to Compute Mediawiki History Data Quality

Maintenance_bot removed a project: Patch-For-Review.Apr 2 2024, 1:30 PM

Snwachukwu moved this task from In Review to Ready to Deploy on the Data-Engineering (Q4 2024 April 1st - June 30th) board.Apr 2 2024, 1:32 PM

Snwachukwu moved this task from Ready to Deploy to Done on the Data-Engineering (Q4 2024 April 1st - June 30th) board.Apr 16 2024, 1:35 PM

mpopov mentioned this in T365203: [Data Quality] Implement wiki completeness check for MediaWiki History.May 16 2024, 9:10 PM

The originally defined set of columns for uniques check result in almost 45% redundancy:
https://superset.wikimedia.org/superset/explore/p/dk7bE7BQjzV/

A new set is proposed for more meaningful tracking:

wiki_db
event_entity
event_type
timestamp
event_user_text_historical
user_text_historical
page_id
revision_id

Ottomata subscribed.Thu, Jun 13, 4:20 PM

Change #1049561 had a related patch set uploaded (by Snwachukwu; author: Snwachukwu):