Page MenuHomePhabricator
Create Task
Maniphest T322905

[EPIC] Upgrade Search Platform spark jobs to spark 3
Closed, ResolvedPublic
Actions

Description

Support for Spark 2 will end on March 31st, 2023. We need to upgrade all of our jobs to Spark 3.

There is a tracking spreadsheet at https://docs.google.com/spreadsheets/d/1j1HzjebGU61mDRRMS8Lyx7DpYiqQrxaM09HL5QRPRfc/edit#gid=0

AC:

  • all Search Platform Spark jobs are running on Spark 3

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedGehelT322905 [EPIC] Upgrade Search Platform spark jobs to spark 3
 ResolvedGehelT323042 [ENG SPIKE] Clarify what needs to be done to migrate Search Platform spark jobs to spark 3
 ResolvedpfischerT327381 Migrate RDF Tooling to Spark 3
 ResolvedGehelT318414 [Tracking] Migrate Search Airflow jobs to Airflow 2 and use shared supporting code from the data engineering Airflow
 ResolvedbkingT327970 Create airflow v2 instance and supporting repos for search platform
 ResolvedEBernhardsonT329239 migrate mjolnir application and dag to airflow v2 and spark3
 ResolvedEBernhardsonT329870 Migrate drop_old_data_daily.py from airflow 1 to airflow 2
 ResolvedEBernhardsonT329871 Migrate export_queries_to_relforge.py from airflow 1 to airflow 2
 ResolvedEBernhardsonT329872 Migrate glent_weekly.py from airflow 1 to airflow 2
 ResolvedEBernhardsonT329873 Migrate import_cirrus_indexes.py from airflow 1 to airflow 2
 ResolvedEBernhardsonT329874 Migrate import_ttl.py from airflow 1 to airflow 2
 ResolvedEBernhardsonT329875 Migrate incoming_links.py from airflow 1 to airflow 2
 ResolvedpfischerT329876 Migrate ores_predictions.py from airflow 1 to airflow 2
 ResolvedEBernhardsonT329877 Migrate popularity_score.py from airflow 1 to airflow 2
 ResolvedEBernhardsonT329878 Migrate query_clicks.py from airflow 1 to airflow 2
 ResolveddcausseT329879 Migrate rdf_streaming_updater_reconcile.py from airflow 1 to airflow 2
 ResolvedEBernhardsonT329880 Migrate search_satisfaction.py from airflow 1 to airflow 2
 ResolvedpfischerT329881 Migrate transfer_to_es.py from airflow 1 to airflow 2
 ResolvedEBernhardsonT330446 Migrate fulltext_head_queries.py from airflow 1 to airflow 2
 ResolvedEBernhardsonT330447 Migrate mediawiki_revision_recommendation_create.py from airflow 1 to airflow 2
 ResolvedEBernhardsonT330448 Migrate process_sparql_query.py from airflow 1 to airflow 2
 DuplicateNoneT330450 Migrate rdf_streaming_updater_reconcile.py from airflow 1 to airflow 2
 ResolvedEBernhardsonT330451 Migrate subgraph_and_query_mapping.py from airflow 1 to airflow 2
 ResolvedEBernhardsonT330452 Migrate subgraph_and_query_metrics.py from airflow 1 to airflow 2
 ResolvedBTullisT333697 decom an-airflow1001

Event Timeline

Gehel created this task.Nov 11 2022, 8:17 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptNov 11 2022, 8:17 AM
• MPhamWMF triaged this task as High priority.Nov 14 2022, 4:27 PM
TJones renamed this task from Upgrade Search Platform spark jobs to spark 3 to [EPIC] Upgrade Search Platform spark jobs to spark 3.Nov 14 2022, 4:28 PM
TJones added a project: Epic.
• MPhamWMF moved this task from needs triage to ML & Data Pipeline on the Discovery-Search board.Nov 14 2022, 4:29 PM
Gehel closed subtask T323042: [ENG SPIKE] Clarify what needs to be done to migrate Search Platform spark jobs to spark 3 as Resolved.Jan 13 2023, 9:52 AM
Gehel moved this task from ML & Data Pipeline to Current work on the Discovery-Search board.Jan 16 2023, 4:10 PM
Gehel edited projects, added Discovery-Search (Current work); removed Discovery-Search.
Gehel moved this task from Incoming to Epics on the Discovery-Search (Current work) board.
pfischer added a subtask: T327381: Migrate RDF Tooling to Spark 3.Jan 19 2023, 8:52 AM
pfischer changed the status of subtask T327381: Migrate RDF Tooling to Spark 3 from Open to In Progress.
Gehel added a subtask: T318414: [Tracking] Migrate Search Airflow jobs to Airflow 2 and use shared supporting code from the data engineering Airflow.Jan 23 2023, 4:41 PM
Gehel closed subtask T327381: Migrate RDF Tooling to Spark 3 as Resolved.Feb 24 2023, 2:07 PM
Gehel closed this task as Resolved.Mar 31 2023, 8:04 AM
Gehel claimed this task.
Gehel closed subtask T318414: [Tracking] Migrate Search Airflow jobs to Airflow 2 and use shared supporting code from the data engineering Airflow as Resolved.
Gehel moved this task from Epics to Needs Reporting on the Discovery-Search (Current work) board.
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits