Page MenuHomePhabricator
Create Task
Maniphest T311745

[EPIC] Section topics data pipeline
Open, Needs TriagePublic
Actions

Description

The section topics data pipeline takes as input wikitext and Wikidata item page links, and outputs a dataset of wiki pages, their section titles, and the set of Wikidata identifiers describing section blue links.

This epic keeps track of all the relevant tasks.

Related Objects
Search...

StatusSubtypeAssignedTask
 OpenNoneT275882 [EPIC] Structured sections with topical metadata
 OpenNoneT311745 [EPIC] Section topics data pipeline
 ResolvedmfossatiT311734 Design technical architecture for section topics pipeline
 ResolvedmfossatiT312900 [M] Design database model for section topics pipeline
 InvalidmfossatiT311750 Combine section extraction with blue links relevance score
 ResolvedmfossatiT311751 Convert section topics notebook code into idiomatic data pipeline code
 ResolvedmfossatiT311824 [M] Create the section topics Airflow DAG
 ResolvedmfossatiT311860 Set up the section topics data pipeline Spark code base
 ResolvedmfossatiT313901 [SPIKE] Mint section identifiers
 ResolvedmfossatiT314863 [SPIKE] Section topics article-level relevance score
 ResolvedmfossatiT314865 Include section zero in the data pipeline
 ResolvedmfossatiT315021 [M] Document technical architecture for section topics onwiki
 ResolvedmfossatiT315207 [SPIKE] Investigate bad section parsing
 ResolvedmfossatiT315851 Finalize code base
 ResolvedmfossatiT316925 [L] Crystallize article-level section topics relevance score
 ResolvedmfossatiT318324 Implement section-level topic relevance score
 ResolvedmfossatiT318092 [M] Exclude certain sections from having topics in the section topics pipeline
 ResolvedmfossatiT318348 [SPIKE] Section-level topic relevance score
 ResolvedmfossatiT318431 [M] Handle redirects
 OpenNoneT320949 [SPIKE] Research which sections and articles on Wikipedia should not have topics generated from bluelinks
 ResolvedxcollazoT323107 [M] Upgrade code base to Spark 3
 ResolvedmfossatiT323489 Pre-compute bad section parsing for ptwiki
 ResolvedmfossatiT323505 [L] Exclude sections-tables from having section topics
 ResolvedmfossatiT323504 Exclude additional section titles
 ResolvedmfossatiT323523 [L] Sanitize blue links that yield null topics
 ResolvedmatthiasmullieT323036 [L] Exclude media topics from section topics dataset
ResolvedxcollazoT323597 [M] Exclude date format topics from section topics pipeline
 In ProgressmfossatiT330848 [XL] Exclude sections with non-standard tables and lists

Event Timeline

mfossati created this task.Jun 30 2022, 4:25 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 30 2022, 4:25 PM
mfossati added a subtask: T311734: Design technical architecture for section topics pipeline.Jun 30 2022, 4:25 PM
CBogen added a project: Structured-Data-Backlog (Current Work).Jun 30 2022, 4:27 PM
CBogen moved this task from Incoming to Epics on the Structured-Data-Backlog (Current Work) board.
mfossati added projects: Research-Backlog, Generated Data Platform.Jun 30 2022, 4:28 PM
CBogen added a parent task: T275882: [EPIC] Structured sections with topical metadata.Jun 30 2022, 4:28 PM
mfossati added a subtask: T311750: Combine section extraction with blue links relevance score.Jun 30 2022, 4:45 PM
mfossati added a subtask: T311751: Convert section topics notebook code into idiomatic data pipeline code.Jun 30 2022, 4:51 PM
mfossati changed the status of subtask T311750: Combine section extraction with blue links relevance score from Open to In Progress.Jul 1 2022, 8:42 AM
mfossati mentioned this in T311814: [EPIC] Section-level image suggestions data pipeline.Jul 1 2022, 9:30 AM
mfossati mentioned this in T311818: Convert section-level image suggestions notebook code into idiomatic data pipeline code.Jul 1 2022, 9:43 AM
mfossati mentioned this in T311824: [M] Create the section topics Airflow DAG.Jul 1 2022, 10:49 AM
mfossati added a subtask: T311824: [M] Create the section topics Airflow DAG.
mfossati added a subtask: T311860: Set up the section topics data pipeline Spark code base.Jul 1 2022, 3:59 PM
mfossati mentioned this in T312037: Create new GitLab project group: structured-data.Jul 20 2022, 11:02 AM
xcollazo added a subscriber: xcollazo.Jul 22 2022, 8:29 PM
mfossati added a subtask: T313901: [SPIKE] Mint section identifiers.Jul 27 2022, 10:33 AM
mfossati changed the status of subtask T313901: [SPIKE] Mint section identifiers from Open to In Progress.Aug 8 2022, 2:16 PM
mfossati added a subtask: T314863: [SPIKE] Section topics article-level relevance score.Aug 9 2022, 1:30 PM
mfossati added a subtask: T314865: Include section zero in the data pipeline.Aug 9 2022, 1:58 PM
mfossati closed subtask T311734: Design technical architecture for section topics pipeline as Resolved.Aug 11 2022, 3:39 PM
CBogen added a subtask: T315021: [M] Document technical architecture for section topics onwiki.Aug 11 2022, 3:56 PM
mfossati changed the status of subtask T311751: Convert section topics notebook code into idiomatic data pipeline code from Open to In Progress.Aug 12 2022, 8:35 AM
mfossati closed subtask T311750: Combine section extraction with blue links relevance score as Invalid.Aug 12 2022, 8:42 AM
mfossati added a subtask: T315207: [SPIKE] Investigate bad section parsing.Aug 15 2022, 8:51 AM
mfossati closed subtask T313901: [SPIKE] Mint section identifiers as Resolved.Aug 17 2022, 9:38 AM
mfossati closed subtask T314865: Include section zero in the data pipeline as Resolved.Aug 17 2022, 10:15 AM
mfossati closed subtask T311860: Set up the section topics data pipeline Spark code base as Resolved.Aug 19 2022, 2:03 PM
mfossati closed subtask T311751: Convert section topics notebook code into idiomatic data pipeline code as Resolved.Aug 19 2022, 3:34 PM
mfossati added a subtask: T315851: Finalize code base.Aug 22 2022, 8:51 AM
mfossati changed the status of subtask T315851: Finalize code base from Open to In Progress.
mfossati mentioned this in T307325: Section Identification & Section Topics.Aug 25 2022, 8:12 AM
mfossati closed subtask T315851: Finalize code base as Resolved.Aug 25 2022, 3:13 PM
mfossati changed the status of subtask T314863: [SPIKE] Section topics article-level relevance score from Open to In Progress.Aug 26 2022, 12:41 PM
lbowmaker edited projects, added Data Pipelines; removed Generated Data Platform.Aug 26 2022, 2:23 PM
santhosh added a subscriber: santhosh.Aug 29 2022, 5:27 AM
mfossati added a subtask: T316925: [L] Crystallize article-level section topics relevance score.Sep 2 2022, 11:02 AM
• EChetty moved this task from Backlog to Epics on the Data Pipelines board.Sep 6 2022, 10:10 AM
AUgolnikova-WMF added a subtask: T318092: [M] Exclude certain sections from having topics in the section topics pipeline.Sep 19 2022, 3:53 PM
mfossati added a subtask: T318348: [SPIKE] Section-level topic relevance score.Sep 22 2022, 2:49 PM
mfossati added a subtask: T318324: Implement section-level topic relevance score.
mfossati added a subtask: T318431: [M] Handle redirects.Sep 23 2022, 4:48 PM
mfossati changed the status of subtask T318092: [M] Exclude certain sections from having topics in the section topics pipeline from Open to In Progress.Oct 7 2022, 10:20 AM
AUgolnikova-WMF added a subtask: T320949: [SPIKE] Research which sections and articles on Wikipedia should not have topics generated from bluelinks.Oct 17 2022, 11:58 AM
mfossati closed subtask T318092: [M] Exclude certain sections from having topics in the section topics pipeline as Resolved.Oct 19 2022, 10:00 AM
mfossati closed subtask T314863: [SPIKE] Section topics article-level relevance score as Resolved.
mfossati changed the status of subtask T316925: [L] Crystallize article-level section topics relevance score from Open to In Progress.
mfossati changed the status of subtask T318348: [SPIKE] Section-level topic relevance score from Open to In Progress.Oct 24 2022, 1:15 PM
mfossati changed the status of subtask T315207: [SPIKE] Investigate bad section parsing from Open to In Progress.Oct 31 2022, 3:49 PM
mfossati added a subtask: T323107: [M] Upgrade code base to Spark 3.Nov 15 2022, 12:02 PM
mfossati changed the status of subtask T318431: [M] Handle redirects from Open to In Progress.Nov 17 2022, 9:54 AM
mfossati added a subtask: T323489: Pre-compute bad section parsing for ptwiki.Nov 21 2022, 12:10 PM
AUgolnikova-WMF added a subtask: T323505: [L] Exclude sections-tables from having section topics.Nov 21 2022, 3:46 PM
mfossati added a subtask: T323504: Exclude additional section titles.Nov 21 2022, 5:46 PM
mfossati added a subtask: T323523: [L] Sanitize blue links that yield null topics.Nov 21 2022, 6:06 PM
mfossati closed subtask T315207: [SPIKE] Investigate bad section parsing as Resolved.Nov 21 2022, 6:12 PM
mfossati added a subtask: T323036: [L] Exclude media topics from section topics dataset .
mfossati changed the status of subtask T323489: Pre-compute bad section parsing for ptwiki from Open to In Progress.Nov 22 2022, 10:20 AM
AUgolnikova-WMF added a subtask: T323597: [M] Exclude date format topics from section topics pipeline.Nov 22 2022, 2:38 PM
mfossati closed subtask T318431: [M] Handle redirects as Resolved.Nov 23 2022, 3:54 PM
mfossati changed the status of subtask T323504: Exclude additional section titles from Open to In Progress.Nov 28 2022, 5:24 PM
mfossati closed subtask T323489: Pre-compute bad section parsing for ptwiki as Resolved.Dec 5 2022, 9:35 AM
xcollazo changed the status of subtask T323107: [M] Upgrade code base to Spark 3 from Open to In Progress.Dec 6 2022, 4:12 PM
mfossati closed subtask T323504: Exclude additional section titles as Resolved.Dec 6 2022, 4:36 PM
mfossati changed the status of subtask T323523: [L] Sanitize blue links that yield null topics from Open to In Progress.Dec 6 2022, 4:40 PM
mfossati closed subtask T318348: [SPIKE] Section-level topic relevance score as Resolved.Dec 13 2022, 12:52 PM
mfossati closed subtask T318324: Implement section-level topic relevance score as Resolved.Dec 13 2022, 12:57 PM
mfossati closed subtask T316925: [L] Crystallize article-level section topics relevance score as Resolved.Dec 16 2022, 6:31 PM
mfossati changed the status of subtask T323505: [L] Exclude sections-tables from having section topics from Open to In Progress.Jan 16 2023, 11:05 AM
mfossati closed subtask T323523: [L] Sanitize blue links that yield null topics as Resolved.Jan 19 2023, 2:39 PM
mfossati closed subtask T323107: [M] Upgrade code base to Spark 3 as Resolved.Jan 19 2023, 2:47 PM
mfossati changed the status of subtask T311824: [M] Create the section topics Airflow DAG from Open to In Progress.Jan 23 2023, 10:51 AM
mfossati closed subtask T311824: [M] Create the section topics Airflow DAG as Resolved.Jan 27 2023, 3:04 PM
mfossati closed subtask T323505: [L] Exclude sections-tables from having section topics as Resolved.Jan 30 2023, 12:06 PM
mfossati changed the status of subtask T315021: [M] Document technical architecture for section topics onwiki from Open to In Progress.Feb 1 2023, 10:17 AM
Sannita closed subtask T315021: [M] Document technical architecture for section topics onwiki as Resolved.Feb 3 2023, 7:40 PM
xcollazo closed subtask T323597: [M] Exclude date format topics from section topics pipeline as Resolved.Feb 13 2023, 9:43 PM
mfossati closed subtask T323036: [L] Exclude media topics from section topics dataset as Resolved.Feb 16 2023, 2:54 PM
mfossati reopened subtask T323505: [L] Exclude sections-tables from having section topics as Open.Wed, Mar 1, 9:38 AM
mfossati added a subtask: T330848: [XL] Exclude sections with non-standard tables and lists.Wed, Mar 1, 10:40 AM
mfossati changed the status of subtask T323505: [L] Exclude sections-tables from having section topics from Open to In Progress.Thu, Mar 2, 4:07 PM
mfossati changed the status of subtask T330848: [XL] Exclude sections with non-standard tables and lists from Open to In Progress.Fri, Mar 10, 3:39 PM
Cparle closed subtask T323505: [L] Exclude sections-tables from having section topics as Resolved.Tue, Mar 14, 12:22 PM
Content licensed under Creative Commons Attribution-ShareAlike 3.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL