Page MenuHomePhabricator
Create Task
Maniphest T357316

Develop pipelines for research datasets - Q3/Q4
Closed, ResolvedPublic
Actions

Description

This task is to track the data pipelines research engineering will productionize in Q3.

  • article topic model dataset (T351118)
  • list building embedding dataset (T351674)

For reference, T348819 was the task for Q2.

Related Objects
Search...

Event Timeline

fkaelin created this task.Feb 12 2024, 3:14 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 12 2024, 3:14 PM
fkaelin added subscribers: • XiaoXiao-WMF, • MunizaA.Feb 12 2024, 3:23 PM
• XiaoXiao-WMF added a comment.Apr 15 2024, 1:08 PM

update from @MunizaA - the pipeline is ready (research-datasets)
next step - airflow integration

• XiaoXiao-WMF reassigned this task from fkaelin to • MunizaA.Apr 15 2024, 1:08 PM
• XiaoXiao-WMF moved this task from FY2023-24-Research-January-March to FY2023-24-Research-April-June on the Research board.
• XiaoXiao-WMF edited projects, added: Research (FY2023-24-Research-April-June); removed: Research (FY2023-24-Research-January-March).
• XiaoXiao-WMF renamed this task from Develop pipelines for research datasets - Q3 to Develop pipelines for research datasets - Q3/Q4.Apr 15 2024, 1:48 PM
• XiaoXiao-WMF added a project: Research-engineering.Jun 10 2024, 3:02 PM
• XiaoXiao-WMF added a comment.Jun 10 2024, 3:07 PM

airflow done, will merge this week

fkaelin added a comment.Jun 17 2024, 1:37 PM

Weekly update

  • pipeline are merged
  • airflow dags are deployed, final testing in progress
• XiaoXiao-WMF added a comment.Jun 28 2024, 6:17 PM

@MunizaA any comments please put in the ticket and please close this out

• XiaoXiao-WMF moved this task from FY2023-24-Research-April-June to FY2024-25-Research-July-September on the Research board.Jun 28 2024, 6:17 PM
• XiaoXiao-WMF edited projects, added: Research (FY2024-25-Research-July-September); removed: Research (FY2023-24-Research-April-June).
• MunizaA updated the task description. (Show Details)Jun 28 2024, 6:22 PM
• MunizaA added a comment.Jun 28 2024, 6:44 PM

The pipeline for generating article topics and embeddings is in research-datasets now and the Airflow DAG has been merged and deployed. The DAG might require a few minor changes requested in T351118: [Research Engineering Request] Produce regular snapshots of all Wikipedia article topics but since this is being tracked there, I'll close this one out.

• MunizaA closed this task as Resolved.Jun 28 2024, 6:56 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits