Airflow Hackathon (May 2022)
Closed, ResolvedPublic
Actions

Assigned To

None

Authored By

	mforns
	May 3 2022, 7:55 PM

Description

This is the parent task of Data Engineering Airflow hackathon (May 2022).
Its child tasks are the potential things to work on during the hackathon.

Details

Due Date: May 31 2022, 9:00 PM

Related Objects
Search...

Status	Subtype	Assigned	Task
Resolved		None	T291464 Upgrade analytics-hadoop to Spark 3 + scala 2.12
Duplicate		None	T291465 Analytics-test-hadoop Spark3 package upgrade
Duplicate		None	T291466 Analytics-hadoop Spark3 package upgrade (production)
Resolved		JAllemandou	T306955 Spark3 migration - Currently existing airflow jobs
Resolved		Ottomata	T291386 Upgrade Refinery Jobs to Spark 3
Resolved		None	T307500 Airflow Hackathon (May 2022)
Resolved		Ahoelzl	T307505 Refine jobs should be scheduled by Airflow
Duplicate		None	T312785 Change the way Refine handles its status (currently flags in partitions)
Resolved		Antoine_Quhen	T356192 [Refine refactoring] Refactor and migrate navigationtiming to Airflow
Resolved		JAllemandou	T356363 [Refine Refactoring] Refactor refinery code for compatibility with Airflow integration
Resolved		tchin	T355542 [Dataset Config Store] - Define config API for navigationtiming and implement local development instance
Resolved		Antoine_Quhen	T356362 [Refine Refactoring] [Spike] Define a concept and provide a PoC for dynamic DAG execution in Airflow
Resolved		Antoine_Quhen	T356762 [Refine refactoring] Refine jobs should be scheduled by Airflow: implementation
Resolved		gmodena	T361853 [Datasets Config][Spike] Understand and document the details and conflicts between Datasets Config, Refine refactor, Dynamic EventStreamConfig, and Metrics Platform Instrumentation Configurator
Resolved	Spike	gmodena	T361017 [SPIKE] Can we express Event Platform configs in Datasets Config?
Resolved		Ottomata	T366627 [MPIC] Analyse risk of potential performance issues with static approach to stream configuration
Resolved		Antoine_Quhen	T365223 Fix generation of _IMPORTED flags by Gobblin
Resolved		• Stevemunene	T365449 Upgrade Airflow to 2.9.3
			Restricted Task
Resolved		Antoine_Quhen	T357430 Airflow mapped tasks UI & metrics
Resolved		Antoine_Quhen	T365563 Timeout hive-metastore locks
Resolved		tchin	T367134 [Refine Refactoring] Changes to EventStreamConfig needed for scheduling Refine via airflow
Open		None	T371803 Refine optimizations on output and parallelization
Resolved		Ottomata	T255818 Refine drops $schema field values
Open		None	T259924 HiveExtensions.convertToSchema does not properly convert arrays of structs
Resolved		tchin	T366487 Event Platform schemas should not support type changes to structs as array element or map value types
Resolved		Antoine_Quhen	T361502 [Refine Refactoring] Define and implement a automated testing / comparison tool for config store configured datasets
Resolved		Antoine_Quhen	T369845 [Refine Refactoring] Refine jobs should be scheduled by Airflow: deployment
In Progress		Antoine_Quhen	T392668 Refine to Hive with Airflow – Kubernetes Resource Optimization
Resolved		JAllemandou	T370665 Refine to Hive with Airflow – Handle Late-Arrived Events
Resolved		Ottomata	T376026 Update event-producing tools to overwrite `meta.dt`
Resolved		Antoine_Quhen	T392696 Refine to Hive with Airflow – Switch Over day scripts
Resolved		Antoine_Quhen	T392697 Refine to Hive with Airflow – Update Refine Documentation on Wikitech
Resolved		Antoine_Quhen	T392698 Refine to Hive with Airflow – Post-Migration Cleanup
Open		Antoine_Quhen	T375064 Move more of refine_hive_hourly dag logic into RefineConfiguration
Resolved		Antoine_Quhen	T402186 Improve handling of new stream onboarding in Refine
Duplicate		None	T377739 [Refine Refactoring] Refine Data Quality - late events, RefineMonitor refactor, etc.
Open		Antoine_Quhen	T381072 [Refine Simplification] Remove Schema Merging in Refine Process by Enforcing Backward Compatibility
Resolved		Antoine_Quhen	T381073 [Airflow Optimization] Reduce Overhead in Refine DAG by Precomputing Parameters
Resolved		Antoine_Quhen	T381074 [Refine DAG Improvement] Add Parameter to Reduce Spark Driver Logs in Skein Log Collection
Open		None	T307508 Migrate 1+ Druid load jobs
Open		None	T307540 [Airflow Migration] Migrate reportupdater jobs
Resolved		awight	T333537 Deprecate WMDE Technical Wishes reportupdater jobs
Resolved		amastilovic	T354552 [Maintenance] Migrate ReportUpdater browser queries to Airflow
Resolved		amastilovic	T356424 [Maintenance] Migrate cx ReportUpdater job
Resolved		amastilovic	T357372 [Maintenance] Migrate pingback to Airflow
Resolved		JAllemandou	T357419 Turn off ReportUpdater jobs no longer used
Resolved		JAllemandou	T358210 Delete reportupdater jobs data/puppet-code
Resolved		amastilovic	T357938 [Maintenance] Migrate wmcs to Airflow
Open		mforns	T364579 Deprecate reportupdater documentation
Resolved		mforns	T305841 Migrate unique devices jobs
Resolved		mforns	T319213 Backfill unique devices jobs [Tracking Task]
Resolved		JAllemandou	T307569 [Oozie Migration] Migrate edit hourly job
Resolved		Snwachukwu	T305844 Migrate the projectview jobs
Resolved		Snwachukwu	T309023 Update and add copies of projectview hql script to analytics/refinery/hql path
Resolved		• JEbe-WMF	T305842 Migrate the referrer job