Maniphest T311745

[EPIC] Section topics data pipeline
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	• mfossati
	Jun 30 2022, 4:25 PM

Tags

Referenced Files

None

Subscribers

Description

The section topics data pipeline takes as input wikitext and Wikidata item page links, and outputs a dataset of wiki pages, their section titles, and the set of Wikidata identifiers describing section blue links.

This epic keeps track of all the relevant tasks.

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Resolved		CBogen	T275882 [EPIC] Structured sections with topical metadata
		Resolved		CBogen	T311745 [EPIC] Section topics data pipeline
		Resolved		• mfossati	T311734 Design technical architecture for section topics pipeline
		Resolved		• mfossati	T312900 [M] Design database model for section topics pipeline
		Invalid		• mfossati	T311750 Combine section extraction with blue links relevance score
		Resolved		• mfossati	T311751 Convert section topics notebook code into idiomatic data pipeline code
		Resolved		• mfossati	T311824 [M] Create the section topics Airflow DAG
		Resolved		• mfossati	T311860 Set up the section topics data pipeline Spark code base
		Resolved		• mfossati	T313901 [SPIKE] Mint section identifiers
		Resolved		• mfossati	T314863 [SPIKE] Section topics article-level relevance score
		Resolved		• mfossati	T314865 Include section zero in the data pipeline
		Resolved		• mfossati	T315021 [M] Document technical architecture for section topics onwiki
		Resolved		• mfossati	T315207 [SPIKE] Investigate bad section parsing
		Resolved		• mfossati	T315851 Finalize code base
		Resolved		• mfossati	T316925 [L] Crystallize article-level section topics relevance score
		Resolved		• mfossati	T318324 Implement section-level topic relevance score
		Resolved		• mfossati	T318092 [M] Exclude certain sections from having topics in the section topics pipeline
		Resolved		• mfossati	T318348 [SPIKE] Section-level topic relevance score
		Resolved		• mfossati	T318431 [M] Handle redirects
		Open		None	T320949 [SPIKE] Research which sections and articles on Wikipedia should not have topics generated from bluelinks
		Resolved		xcollazo	T323107 [M] Upgrade code base to Spark 3
		Resolved		• mfossati	T323489 Pre-compute bad section parsing for ptwiki
		Resolved		• mfossati	T323505 [L] Exclude sections-tables from having section topics
		Resolved		• mfossati	T323504 Exclude additional section titles
		Resolved		• mfossati	T323523 [L] Sanitize blue links that yield null topics
		Resolved		matthiasmullie	T323036 [L] Exclude media topics from section topics dataset
		Resolved		xcollazo	T323597 [M] Exclude date format topics from section topics pipeline
		Resolved		• mfossati	T330848 [XL] Exclude sections with non-standard tables and lists
		Open		None	T337925 [L] Publish full image suggestions (and intermediate) dataset

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

• mfossati added a subtask: T311751: Convert section topics notebook code into idiomatic data pipeline code.Jun 30 2022, 4:51 PM

• mfossati changed the status of subtask T311750: Combine section extraction with blue links relevance score from Open to In Progress.Jul 1 2022, 8:42 AM

• mfossati mentioned this in T311814: [EPIC] Section-level image suggestions data pipeline.Jul 1 2022, 9:30 AM

• mfossati mentioned this in T311818: Convert section-level image suggestions notebook code into idiomatic data pipeline code.Jul 1 2022, 9:43 AM

• mfossati mentioned this in T311824: [M] Create the section topics Airflow DAG.Jul 1 2022, 10:49 AM

• mfossati added a subtask: T311824: [M] Create the section topics Airflow DAG.

• mfossati added a subtask: T311860: Set up the section topics data pipeline Spark code base.Jul 1 2022, 3:59 PM

• mfossati mentioned this in T312037: Create new GitLab project group: structured-data.Jul 20 2022, 11:02 AM

xcollazo subscribed.Jul 22 2022, 8:29 PM

• mfossati added a subtask: T313901: [SPIKE] Mint section identifiers.Jul 27 2022, 10:33 AM

• mfossati changed the status of subtask T313901: [SPIKE] Mint section identifiers from Open to In Progress.Aug 8 2022, 2:16 PM

• mfossati added a subtask: T314863: [SPIKE] Section topics article-level relevance score.Aug 9 2022, 1:30 PM

• mfossati added a subtask: T314865: Include section zero in the data pipeline.Aug 9 2022, 1:58 PM

• mfossati closed subtask T311734: Design technical architecture for section topics pipeline as Resolved.Aug 11 2022, 3:39 PM

CBogen added a subtask: T315021: [M] Document technical architecture for section topics onwiki.Aug 11 2022, 3:56 PM

• mfossati changed the status of subtask T311751: Convert section topics notebook code into idiomatic data pipeline code from Open to In Progress.Aug 12 2022, 8:35 AM

• mfossati closed subtask T311750: Combine section extraction with blue links relevance score as Invalid.Aug 12 2022, 8:42 AM

• mfossati added a subtask: T315207: [SPIKE] Investigate bad section parsing.Aug 15 2022, 8:51 AM

• mfossati closed subtask T313901: [SPIKE] Mint section identifiers as Resolved.Aug 17 2022, 9:38 AM

• mfossati closed subtask T314865: Include section zero in the data pipeline as Resolved.Aug 17 2022, 10:15 AM

• mfossati closed subtask T311860: Set up the section topics data pipeline Spark code base as Resolved.Aug 19 2022, 2:03 PM

• mfossati closed subtask T311751: Convert section topics notebook code into idiomatic data pipeline code as Resolved.Aug 19 2022, 3:34 PM

• mfossati added a subtask: T315851: Finalize code base.Aug 22 2022, 8:51 AM

• mfossati changed the status of subtask T315851: Finalize code base from Open to In Progress.

• mfossati mentioned this in T307325: Section Identification & Section Topics.Aug 25 2022, 8:12 AM

• mfossati closed subtask T315851: Finalize code base as Resolved.Aug 25 2022, 3:13 PM

• mfossati changed the status of subtask T314863: [SPIKE] Section topics article-level relevance score from Open to In Progress.Aug 26 2022, 12:41 PM

• lbowmaker edited projects, added Data Pipelines; removed Generated Data Platform.Aug 26 2022, 2:23 PM

santhosh subscribed.Aug 29 2022, 5:27 AM

• mfossati added a subtask: T316925: [L] Crystallize article-level section topics relevance score.Sep 2 2022, 11:02 AM

• EChetty moved this task from Backlog to Epics on the Data Pipelines board.Sep 6 2022, 10:10 AM

• AUgolnikova-WMF added a subtask: T318092: [M] Exclude certain sections from having topics in the section topics pipeline.Sep 19 2022, 3:53 PM

• mfossati added a subtask: T318348: [SPIKE] Section-level topic relevance score.Sep 22 2022, 2:49 PM

• mfossati added a subtask: T318324: Implement section-level topic relevance score.

• mfossati added a subtask: T318431: [M] Handle redirects.Sep 23 2022, 4:48 PM

• mfossati changed the status of subtask T318092: [M] Exclude certain sections from having topics in the section topics pipeline from Open to In Progress.Oct 7 2022, 10:20 AM

• AUgolnikova-WMF added a subtask: T320949: [SPIKE] Research which sections and articles on Wikipedia should not have topics generated from bluelinks.Oct 17 2022, 11:58 AM

• mfossati closed subtask T318092: [M] Exclude certain sections from having topics in the section topics pipeline as Resolved.Oct 19 2022, 10:00 AM

• mfossati closed subtask T314863: [SPIKE] Section topics article-level relevance score as Resolved.

• mfossati changed the status of subtask T316925: [L] Crystallize article-level section topics relevance score from Open to In Progress.

• mfossati changed the status of subtask T318348: [SPIKE] Section-level topic relevance score from Open to In Progress.Oct 24 2022, 1:15 PM

• mfossati changed the status of subtask T315207: [SPIKE] Investigate bad section parsing from Open to In Progress.Oct 31 2022, 3:49 PM

• mfossati added a subtask: T323107: [M] Upgrade code base to Spark 3.Nov 15 2022, 12:02 PM

• mfossati changed the status of subtask T318431: [M] Handle redirects from Open to In Progress.Nov 17 2022, 9:54 AM

• mfossati added a subtask: T323489: Pre-compute bad section parsing for ptwiki.Nov 21 2022, 12:10 PM

• AUgolnikova-WMF added a subtask: T323505: [L] Exclude sections-tables from having section topics.Nov 21 2022, 3:46 PM

• mfossati added a subtask: T323504: Exclude additional section titles.Nov 21 2022, 5:46 PM

• mfossati added a subtask: T323523: [L] Sanitize blue links that yield null topics.Nov 21 2022, 6:06 PM

• mfossati closed subtask T315207: [SPIKE] Investigate bad section parsing as Resolved.Nov 21 2022, 6:12 PM

• mfossati added a subtask: T323036: [L] Exclude media topics from section topics dataset .

• mfossati changed the status of subtask T323489: Pre-compute bad section parsing for ptwiki from Open to In Progress.Nov 22 2022, 10:20 AM

• AUgolnikova-WMF added a subtask: T323597: [M] Exclude date format topics from section topics pipeline.Nov 22 2022, 2:38 PM

• mfossati closed subtask T318431: [M] Handle redirects as Resolved.Nov 23 2022, 3:54 PM

• mfossati changed the status of subtask T323504: Exclude additional section titles from Open to In Progress.Nov 28 2022, 5:24 PM

• mfossati closed subtask T323489: Pre-compute bad section parsing for ptwiki as Resolved.Dec 5 2022, 9:35 AM

xcollazo changed the status of subtask T323107: [M] Upgrade code base to Spark 3 from Open to In Progress.Dec 6 2022, 4:12 PM

• mfossati closed subtask T323504: Exclude additional section titles as Resolved.Dec 6 2022, 4:36 PM

• mfossati changed the status of subtask T323523: [L] Sanitize blue links that yield null topics from Open to In Progress.Dec 6 2022, 4:40 PM

• mfossati closed subtask T318348: [SPIKE] Section-level topic relevance score as Resolved.Dec 13 2022, 12:52 PM

• mfossati closed subtask T318324: Implement section-level topic relevance score as Resolved.Dec 13 2022, 12:57 PM

• mfossati closed subtask T316925: [L] Crystallize article-level section topics relevance score as Resolved.Dec 16 2022, 6:31 PM

• mfossati changed the status of subtask T323505: [L] Exclude sections-tables from having section topics from Open to In Progress.Jan 16 2023, 11:05 AM

• mfossati closed subtask T323523: [L] Sanitize blue links that yield null topics as Resolved.Jan 19 2023, 2:39 PM

• mfossati closed subtask T323107: [M] Upgrade code base to Spark 3 as Resolved.Jan 19 2023, 2:47 PM

• mfossati changed the status of subtask T311824: [M] Create the section topics Airflow DAG from Open to In Progress.Jan 23 2023, 10:51 AM

• mfossati closed subtask T311824: [M] Create the section topics Airflow DAG as Resolved.Jan 27 2023, 3:04 PM

• mfossati closed subtask T323505: [L] Exclude sections-tables from having section topics as Resolved.Jan 30 2023, 12:06 PM

• mfossati changed the status of subtask T315021: [M] Document technical architecture for section topics onwiki from Open to In Progress.Feb 1 2023, 10:17 AM

Sannita closed subtask T315021: [M] Document technical architecture for section topics onwiki as Resolved.Feb 3 2023, 7:40 PM

xcollazo closed subtask T323597: [M] Exclude date format topics from section topics pipeline as Resolved.Feb 13 2023, 9:43 PM

• mfossati closed subtask T323036: [L] Exclude media topics from section topics dataset as Resolved.Feb 16 2023, 2:54 PM

• mfossati reopened subtask T323505: [L] Exclude sections-tables from having section topics as Open.Mar 1 2023, 9:38 AM

• mfossati added a subtask: T330848: [XL] Exclude sections with non-standard tables and lists.Mar 1 2023, 10:40 AM

• mfossati changed the status of subtask T323505: [L] Exclude sections-tables from having section topics from Open to In Progress.Mar 2 2023, 4:07 PM

• mfossati changed the status of subtask T330848: [XL] Exclude sections with non-standard tables and lists from Open to In Progress.Mar 10 2023, 3:39 PM

Cparle closed subtask T323505: [L] Exclude sections-tables from having section topics as Resolved.Mar 14 2023, 12:22 PM

• mfossati closed subtask T330848: [XL] Exclude sections with non-standard tables and lists as Resolved.Apr 13 2023, 2:05 PM

CBogen closed this task as Resolved.May 8 2023, 4:21 PM

CBogen claimed this task.

• mfossati added a subtask: T337925: [L] Publish full image suggestions (and intermediate) dataset.Jun 13 2023, 2:31 PM

• mfossati added a subtask: T339205: Optimize Spark's query plan.Jun 15 2023, 9:46 AM

• mfossati added a subtask: T339222: Optimize detection of standard lists and tables.Jun 15 2023, 10:05 AM

• mfossati added a subtask: T339197: Handle media links at parsing time.

• mfossati added a subtask: T333699: [XL] Unify files that are duplicated between section-topics, section-image-recs and image-suggestions.Jun 15 2023, 10:15 AM

• mfossati added a subtask: T331522: [XL] Let section alignment consume section topics output.

• mfossati added a subtask: T331968: [XL] Let the model that learns section alignments consume section topics output.Jun 15 2023, 10:19 AM

• mfossati added a subtask: T339120: [XL] Deduplicate code in section-topics, section-image-recs and image-suggestions.

• mfossati added a subtask: T339129: [Spike] Periodically regenerate various variable data sets/files.Jun 15 2023, 10:21 AM

• mfossati added a subtask: T339146: [M] Fix detect_html_tables.py execution error on incomplete article data.

• AUgolnikova-WMF mentioned this in T340437: [EPIC] Image suggestions data pipelines maintenance .Jun 26 2023, 11:42 AM

• AUgolnikova-WMF removed a subtask: T339146: [M] Fix detect_html_tables.py execution error on incomplete article data.Jun 26 2023, 11:50 AM

• AUgolnikova-WMF removed a subtask: T339129: [Spike] Periodically regenerate various variable data sets/files.

• AUgolnikova-WMF removed a subtask: T339120: [XL] Deduplicate code in section-topics, section-image-recs and image-suggestions.

• AUgolnikova-WMF removed a subtask: T331968: [XL] Let the model that learns section alignments consume section topics output.

• AUgolnikova-WMF removed a subtask: T331522: [XL] Let section alignment consume section topics output.

• AUgolnikova-WMF removed a subtask: T333699: [XL] Unify files that are duplicated between section-topics, section-image-recs and image-suggestions.

• AUgolnikova-WMF removed a subtask: T339222: Optimize detection of standard lists and tables.Jun 26 2023, 11:55 AM

• AUgolnikova-WMF removed a subtask: T339205: Optimize Spark's query plan.

• AUgolnikova-WMF removed a subtask: T339197: Handle media links at parsing time.

Aklapper changed the status of subtask T337925: [L] Publish full image suggestions (and intermediate) dataset from In Progress to Open.Apr 11 2025, 10:13 PM