Page MenuHomePhabricator
Create Task
Maniphest T368754

Production PySpark job that can run consistency checks for wmf_dumps.wikitext_raw
Closed, ResolvedPublic5 Estimated Story Points
Actions

Description

See T368753 for details.

Related Objects
Search...

Event Timeline

xcollazo created this task.Jun 28 2024, 4:14 PM
xcollazo renamed this task from a PySpark job that can run similar checks as in T367570, but that is parameterized properly. to Production PySpark job that can run similar checks as in T367570.Jun 28 2024, 4:17 PM
xcollazo updated the task description. (Show Details)
xcollazo changed the task status from Open to In Progress.Jul 1 2024, 3:46 PM
xcollazo claimed this task.
xcollazo triaged this task as High priority.
xcollazo set the point value for this task to 5.
xcollazo moved this task from Sprint Backlog to In Process on the Dumps 2.0 (Kanban Board) board.Jul 22 2024, 8:28 PM
CodeReviewBot added a comment.Jul 22 2024, 8:32 PM

xcollazo updated https://gitlab.wikimedia.org/repos/data-engineering/dumps/mediawiki-content-dump/-/merge_requests/30

Draft: Job to emit reconciliation events

xcollazo moved this task from In Process to Code Review / Tech Input on the Dumps 2.0 (Kanban Board) board.Jul 24 2024, 6:28 PM
CodeReviewBot added a comment.Jul 31 2024, 6:06 PM

xcollazo merged https://gitlab.wikimedia.org/repos/data-engineering/dumps/mediawiki-content-dump/-/merge_requests/30

Job to do consistency check

xcollazo moved this task from Code Review / Tech Input to To Deploy on the Dumps 2.0 (Kanban Board) board.Jul 31 2024, 6:07 PM
xcollazo moved this task from To Deploy to Done on the Dumps 2.0 (Kanban Board) board.Aug 12 2024, 6:20 PM
xcollazo closed this task as Resolved.Aug 16 2024, 8:01 PM
xcollazo renamed this task from Production PySpark job that can run similar checks as in T367570 to Production PySpark job that can run consistency checks for wmf_dumps.wikitext_raw.Aug 16 2024, 8:03 PM
xcollazo mentioned this in T368753: Implement production mechanism that emits (wiki_db, revision_id) pairs for missing or inaccurate rows.Aug 16 2024, 8:06 PM
xcollazo mentioned this in T372677: Figure a performant way to read all data from revision table via Spark.Aug 16 2024, 9:20 PM
xcollazo mentioned this in T377852: Tune Reconciliation mechanism to do historic runs (all revisions, all wikis).Tue, Oct 22, 2:37 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits