Maniphest T204950

Public Edit Data Lake: Mediawiki history snapshots available in SQL data store to cloud (labs) users
Open, LowPublic0 Estimated Story Points
Actions

Assigned To

None

Authored By

	Nuria
	Sep 20 2018, 1:44 PM

Description

As part of the Wikistats 2 project we have developed the Edit Data Lake (see https://wikitech.wikimedia.org/wiki/Analytics/Data_Lake/Edits). The Edit Data Lake is a denormalized data store that is the best dataset we have had to date to answer questions about content and contributors. At this time this data is only available for the WMF in the private hadoop cluster.
This is the parent task for all the work to make the Data Lake data available on our public cloud infrastructure for our community at large; the more accessible that data is, the more impact it can have.

Related Objects
Search...

Status	Assigned	Task
Open	None	T204950 Public Edit Data Lake: Mediawiki history snapshots available in SQL data store to cloud (labs) users
Declined	Ottomata	T204951 Presto cluster online and usable with test data pushed from analytics prod infrastructure accessible by Cloud (labs) users
Resolved	None	T199674 eqiad \| (3) Labs Data Lake hardware
		Unknown Object (Task)
Resolved	Cmjohnson	T207192 rack/setup/install an-worker10[78-95].eqiad.wmnet
Resolved	Ottomata	T207194 rack/setup/install cloudvirtan100[1-5].eqiad.wmnet
Resolved	Ottomata	T203115 Create .deb package for Presto
Declined	None	T207205 Set up 3 Ganeti VMs for datalake cloud analytics Hadoop cluster
Resolved	Ottomata	T207321 Figure out networking details for new cloud-analytics-eqiad Hadoop/Presto cluster
Resolved	JAllemandou	T208614 Presto on Cloud Platform Design Document
Resolved	Andrew	T211599 Set up 5 decidated VMs on the cloudvirtan hardware in the cloud-analytics project
Declined	Andrew	T211634 Increase quota in cloud-analytics project for zookeeper nodes
Declined	Ottomata	T211640 Grafana, icinga, prometheus in cloud-analytics project
Duplicate	None	T214043 Make edit data lake data available as a snapshot on dump hosts
Resolved	Cmjohnson	T225128 Move cloudvirtan* hardware out of CloudVPS back into production Analytics VLAN.

Event Timeline

Nuria created this task.Sep 20 2018, 1:44 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSep 20 2018, 1:44 PM

Nuria renamed this task from Edit Data Lake available in labs: Mediawiki history snapshots available in SQL data store to Public Edit Data Lake: Mediawiki history snapshots available in SQL data store to clod (labs) users.Sep 20 2018, 1:45 PM

Nuria added a subtask: T204537: Document on wikitech presto choice for data lake .

Nuria removed a subtask: T204537: Document on wikitech presto choice for data lake .

Nuria updated the task description. (Show Details)Sep 20 2018, 1:53 PM

Nuria updated the task description. (Show Details)

Milimetric renamed this task from Public Edit Data Lake: Mediawiki history snapshots available in SQL data store to clod (labs) users to Public Edit Data Lake: Mediawiki history snapshots available in SQL data store to cloud (labs) users.Sep 24 2018, 3:49 PM

Milimetric triaged this task as Medium priority.

Milimetric removed a project: Analytics-Kanban.

Milimetric moved this task from Incoming to Smart Tools for Better Data on the Analytics board.

• Tbayer added a subscriber: • Tbayer.Sep 27 2018, 1:09 AM

Milimetric raised the priority of this task from Medium to High.Oct 18 2018, 5:36 PM

Milimetric added a project: Analytics-Kanban.

Milimetric set the point value for this task to 0.

Milimetric removed a project: Analytics.Oct 19 2018, 4:15 PM

Jdforrester-WMF added a subscriber: Jdforrester-WMF.Nov 7 2018, 5:48 PM

nshahquinn-wmf added a subscriber: nshahquinn-wmf.Feb 22 2019, 1:39 PM

Milimetric moved this task from Parent Tasks to Paused on the Analytics-Kanban board.Mar 4 2019, 4:58 PM

Milimetric added a project: Cloud-Services.Mar 4 2019, 5:08 PM

Milimetric merged a task: T169572: Provide mediawiki history data to Cloud Services users.

Milimetric added subscribers: MusikAnimal, zhuyifei1999.

Ottomata added a subtask: T225128: Move cloudvirtan* hardware out of CloudVPS back into production Analytics VLAN..Jun 27 2019, 5:26 PM

Nuria edited projects, added Analytics; removed Analytics-Kanban.Jul 22 2019, 8:49 PM

elukey closed subtask T225128: Move cloudvirtan* hardware out of CloudVPS back into production Analytics VLAN. as Resolved.Sep 27 2019, 2:17 PM

Nuria edited projects, added Analytics-Kanban; removed Analytics.Nov 19 2019, 12:25 AM

Nuria moved this task from Paused to Parent Tasks on the Analytics-Kanban board.

De-prioritizing until cloud infrastructure can support monitoring similar to what we can do in production.

Milimetric moved this task from Smart Tools for Better Data to Deprioritized on the Analytics board.Jan 2 2020, 6:21 PM

This needs further consideration.

Milimetric closed subtask T204951: Presto cluster online and usable with test data pushed from analytics prod infrastructure accessible by Cloud (labs) users as Declined.Aug 31 2020, 5:03 PM

nshahquinn-wmf removed a subtask: T204953: Data quality issues in the mediawiki_history data stream.Oct 30 2020, 10:32 AM

nskaggs added a subscriber: nskaggs.Nov 18 2020, 8:24 PM

• Jhernandez added a subscriber: • Jhernandez.Nov 19 2020, 11:51 AM

taavi edited projects, added Data-Services; removed Cloud-Services.Oct 9 2021, 5:17 PM

bd808 moved this task from Backlog to Datasets on the Data-Services board.Apr 12 2022, 11:24 PM

ttaylor added a subscriber: ttaylor.Dec 1 2022, 10:25 PM

Aklapper added a project: Data-Engineering-Icebox.Feb 10 2023, 5:44 PM

Public Edit Data Lake: Mediawiki history snapshots available in SQL data store to cloud (labs) usersOpen, LowPublic0 Estimated Story PointsActions

Description

Related ObjectsSearch...

Event Timeline

Public Edit Data Lake: Mediawiki history snapshots available in SQL data store to cloud (labs) users
Open, LowPublic0 Estimated Story Points
Actions

Related Objects
Search...