Page MenuHomePhabricator
Create Task
Maniphest T305475

Milestone: Ingest and Transform Input Data
Open, Needs TriagePublic
Actions

Description

Goal: Upload and Ingest the Input data and implement the defined transformations

Key Tasks for Milestone Completion:

  • Manually upload the input datasets into the data lake
  • Implement transformations in a data pipeline
  • Have the datasets validated.

Related Objects
Search...

StatusSubtypeAssignedTask
 OpenNoneT305468 Deploy the GDI Equity Landscape Dashboard
OpenNoneT305475 Milestone: Ingest and Transform Input Data
 Resolved• ntsakoT306617 Readership Output Rank Metrics
 Resolved• ntsakoT306624 Population output rank metrics
 Resolved• ntsakoT306618 Editorship Output Rank Metrics
 Resolved• ntsakoT306619 Affiliates output rank metrics
 Resolved• ntsakoT306620 Grants output metrics
 InvalidJAnstee_WMFT306621 Programs Metrics Transform
 Resolved• ntsakoT306622 Overall Engagement output rank metric
 OpenNoneT306625 Extract + Transformation Raw Data into Input Metrics
 Resolved• ntsakoT309273 Readership input metrics
 Resolved• ntsakoT309274 Editorship Input Metrics
 Resolved• ntsakoT309275 Affiliates input metric
 Resolved• ntsakoT309276 Grants input metric
 ResolvedKCVelaga_WMFT309277 Programs input metric (not until 2022 data update)
 Invalid• ntsakoT309278 Overall Engagement input metric
 Resolved• ntsakoT309279 Population input metrics
 Resolved• ntsakoT309282 World Bank Data
 Resolved• ntsakoT309283 Wiki DB Map
 Resolved• ntsakoT310712 Load country data
 ResolvedMayakp.wikiT318850 Provide recommendations for Regional data
ResolvedJAnstee_WMFT320985 Split country data into regional classification and main country data
 Resolved• ntsakoT324681 Add country_meta_data
 Resolved• ntsakoT310713 Get PySpark to with Airflow
 Resolved• ntsakoT324968 Access input metrics
 Resolved• ntsakoT329185 Access output metrics
 Resolved• ntsakoT330295 2022 - Affiliates input metrics
Resolved• ntsakoT330897 2022 - Social Progress index
 Resolved• ntsakoT313709 UDF to calculate the average of two or more column values
 Resolved• ntsakoT315886 Load language data
 ResolvedKCVelaga_WMFT336047 Fix average_monthly_editors metric in Equity Landscape

Event Timeline

• EChetty created this task.Apr 5 2022, 2:51 PM
• ntsako claimed this task.Apr 6 2022, 2:48 PM
• ntsako added a project: Equity-Landscape.
JAnstee_WMF moved this task from Backlog to Up Next on the Equity-Landscape board.Apr 6 2022, 5:07 PM
KCVelaga_WMF added a subtask: T306624: Population output rank metrics.Apr 21 2022, 2:04 PM
KCVelaga_WMF added a subtask: T306618: Editorship Output Rank Metrics.
KCVelaga_WMF added a subtask: T306619: Affiliates output rank metrics.
KCVelaga_WMF added a subtask: T306620: Grants output metrics.
KCVelaga_WMF added a subtask: T306621: Programs Metrics Transform.
KCVelaga_WMF added a subtask: T306622: Overall Engagement output rank metric.Apr 21 2022, 2:06 PM
• EChetty moved this task from Incoming (new tickets) to DaaS Work on the Data-Engineering board.Apr 24 2022, 7:01 AM
KCVelaga_WMF closed subtask T306617: Readership Output Rank Metrics as Resolved.Jun 16 2022, 1:02 PM
KCVelaga_WMF closed subtask T306618: Editorship Output Rank Metrics as Resolved.Jun 17 2022, 1:12 PM
• Iflorez subscribed.Jun 22 2022, 6:51 PM
KCVelaga_WMF closed subtask T306621: Programs Metrics Transform as Resolved.Jul 21 2022, 9:22 AM
KCVelaga_WMF closed subtask T306624: Population output rank metrics as Resolved.
KCVelaga_WMF closed subtask T306622: Overall Engagement output rank metric as Resolved.
KCVelaga_WMF closed subtask T306620: Grants output metrics as Resolved.
KCVelaga_WMF closed subtask T306619: Affiliates output rank metrics as Resolved.
• ntsako moved this task from Up Next to In Development on the Equity-Landscape board.Aug 2 2022, 3:27 PM
• ntsako closed subtask T313709: UDF to calculate the average of two or more column values as Resolved.Sep 14 2022, 9:03 AM
KCVelaga_WMF reopened subtask T306617: Readership Output Rank Metrics as Open.Sep 15 2022, 10:19 AM
KCVelaga_WMF reopened subtask T306618: Editorship Output Rank Metrics as Open.Sep 15 2022, 10:41 AM
JAnstee_WMF changed the status of subtask T306618: Editorship Output Rank Metrics from Open to In Progress.Oct 6 2022, 3:21 PM
JAnstee_WMF changed the status of subtask T306617: Readership Output Rank Metrics from Open to In Progress.
KCVelaga_WMF reopened subtask T306620: Grants output metrics as Open.Oct 17 2022, 4:08 PM
KCVelaga_WMF reopened subtask T306621: Programs Metrics Transform as Open.Nov 14 2022, 10:27 AM
KCVelaga_WMF reopened subtask T306619: Affiliates output rank metrics as Open.
KCVelaga_WMF reopened subtask T306624: Population output rank metrics as Open.
KCVelaga_WMF reopened subtask T306622: Overall Engagement output rank metric as Open.
JAnstee_WMF changed the status of subtask T306620: Grants output metrics from Open to In Progress.Nov 14 2022, 3:58 PM
JAnstee_WMF changed the status of subtask T306621: Programs Metrics Transform from Open to In Progress.
• ntsako moved this task from In Development to In QA Review on the Equity-Landscape board.Jan 17 2023, 2:03 PM
KCVelaga_WMF closed subtask T306621: Programs Metrics Transform as Invalid.Jan 19 2023, 10:32 AM
• ntsako closed subtask T306618: Editorship Output Rank Metrics as Resolved.Jan 23 2023, 10:04 AM
• ntsako closed subtask T306617: Readership Output Rank Metrics as Resolved.
• ntsako closed subtask T306620: Grants output metrics as Resolved.Feb 2 2023, 12:46 PM
• ntsako closed subtask T306622: Overall Engagement output rank metric as Resolved.
• ntsako closed subtask T306619: Affiliates output rank metrics as Resolved.
• ntsako moved this task from In QA Review to In Development on the Equity-Landscape board.Feb 16 2023, 11:47 AM
• ntsako moved this task from In Development to Up Next on the Equity-Landscape board.May 31 2023, 10:46 AM
• ntsako removed a project: Data-Engineering.Jun 21 2023, 2:10 PM
• ntsako closed subtask T306624: Population output rank metrics as Resolved.Jul 18 2023, 4:51 PM
• ntsako closed subtask T315886: Load language data as Resolved.Jul 18 2023, 5:09 PM
JAnstee_WMF moved this task from Up Next to Pending Deploy on the Equity-Landscape board.Aug 25 2023, 6:03 PM
JAnstee_WMF removed • ntsako as the assignee of this task.Jan 2 2024, 5:01 PM
JAnstee_WMF added a subscriber: • ntsako.
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits