Dataset Config Store
Open, Needs TriagePublic
Actions

Assigned To

None

Authored By

	lbowmaker
	Jan 8 2024, 8:03 PM

Description

This Epic covers the work to implement a dataset config store so that the users of the data platform can store technical configuration in one place. Sub tickets will cover investigating existing solutions, similar work from other teams and the work required to implement or adopt a solution.

This is the hypothesis from annual planning.

SDS 2.6.3 If we decide on a technical solution for a dataset configuration store, implement it for one existing data pipeline then we can use the configuration tool to automatically generate simple data pipelines to support and increase the speed to production of new metrics

Technical Decision Record: https://docs.google.com/document/d/1tuoRviz3kNgUNOnSjtP5Pr6ikAiZWOdWxUytrDd1ZKs/edit#heading=h.3k1uzt7e33l4

High level deck: https://docs.google.com/presentation/d/15gGj41kCC7DiDKMtvPWaRp-jGoCO4ifuw9AhgqdBEzM/edit#slide=id.g15105b408d_0_287

Related Objects
Search...

Status	Subtype	Assigned	Task
Resolved		WDoranWMF	T360647 [Sprint 11 GOAL] SDS 2.5.5 Deliver working prototype for MP Instrumentation Configuration
Open		None	T360736 [Epic] Build the mechanism to adapt and deliver the output of the MPIC application
Resolved		phuedx	T360738 Update EventStreamConfigs extension to use MPMW hook
Open		Antoine_Quhen	T356762 [Refine refactoring] Extract refine schema management into a dedicated tool
Open		None	T354557 Dataset Config Store
Resolved		JAllemandou	T354558 [Dataset Config Store] [SPIKE] Investigate existing backend solutions
Resolved		tchin	T355542 [Dataset Config Store] - Define config API for navigationtiming and implement local development instance
Open		None	T355911 Replace use of the aqs.config table with Dataset Config Store
Open		tchin	T357434 [Dataset Config Store] Deploy poc to dse-k8s
Open		tchin	T357468 [Dataset Config Store] Setup initial CI checks
Open	Spike	None	T360896 [SPIKE] [Dataset Config Store] - Design how config store feeds DataHub
Open		gmodena	T361853 [Datasets Config][Spike] Understand and document the details and conflicts between Datasets Config, Refine refactor, Dynamic EventStreamConfig, and Metrics Platform Instrumentation Configurator
Open	Spike	None	T361017 [SPIKE] Can we express Event Platform configs in Datasets Config?

Event Timeline

lbowmaker created this task.Jan 8 2024, 8:03 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 8 2024, 8:03 PM

lbowmaker moved this task from Incoming (new tickets) to Dataset Config Store on the Data-Engineering board.Jan 8 2024, 8:04 PM

lbowmaker added a subscriber: mforns.Jan 22 2024, 5:57 PM

mforns mentioned this in T352948: Create Airflow job to populate a new Cassandra table with latest MediaWiki history snapshot configuration.Jan 22 2024, 10:18 PM

lbowmaker closed subtask T354558: [Dataset Config Store] [SPIKE] Investigate existing backend solutions as Resolved.Jan 30 2024, 12:19 PM

tchin subscribed.Jan 31 2024, 6:14 PM

lbowmaker mentioned this in T327816: [Airflow] Gather dataset information from DataHub.Feb 9 2024, 3:21 PM

lbowmaker closed subtask T355542: [Dataset Config Store] - Define config API for navigationtiming and implement local development instance as Resolved.Feb 16 2024, 3:21 PM

Worth investigating? https://datacontract.com/

Have we looked around to see if there are existing 'dataset' config formats/specs we can already use?

Have we looked around to see if there are existing 'dataset' config formats/specs we can already use?

I have not investigated this road at all.
https://datacontract.com/ is cool, I think we need some time to investigate the cost of integrating it to our need before deciding to use that (or something else).

Ottomata added a subtask: T361853: [Datasets Config][Spike] Understand and document the details and conflicts between Datasets Config, Refine refactor, Dynamic EventStreamConfig, and Metrics Platform Instrumentation Configurator.Thu, Apr 4, 6:17 PM

Ottomata added a subtask: T361017: [SPIKE] Can we express Event Platform configs in Datasets Config?.Thu, Apr 4, 7:17 PM

Dataset Config StoreOpen, Needs TriagePublicActions

Description

Related ObjectsSearch...

Event Timeline

Dataset Config Store
Open, Needs TriagePublic
Actions

Related Objects
Search...