Create a dashboard from the fsImage Dataset extracted from the HDFS FsImage
Closed, ResolvedPublic3 Estimated Story Points
Actions

Assigned To

Authored By

	• EChetty
	Oct 19 2022, 10:29 AM

Description

Primary Task

Create a Superset dashboard ~~& jupyter notebook~~ to extract information from the datasets

The dashboard/notebook should clearly:

Identify inefficient file storage (large number of small files)
Get trends of the footprint of our dataset
~~Monitor cluster capacity, corruption~~

Details

Related Changes in Gerrit:

	Subject	Repo	Branch	Lines +/-
	Declare the HDFS usage dataset in hive metastore	analytics/refinery	master	+48 -0

Customize query in gerrit

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Resolved		Antoine_Quhen	T261283 Productionize HDFS fsimage data analysis job
		Resolved		Antoine_Quhen	T321169 Create a dashboard from the fsImage Dataset extracted from the HDFS FsImage

Event Timeline

• EChetty created this task.Oct 19 2022, 10:29 AM

Restricted Application removed a project: Patch-For-Review. · View Herald TranscriptOct 19 2022, 10:29 AM

• EChetty set the point value for this task to 3.Oct 19 2022, 10:29 AM

• EChetty moved this task from Ready to Next Up on the Data Pipelines (Sprint 03) board.

Antoine_Quhen renamed this task from Extract the analysis and make it available on superset. to Create a dashboard from the Dataset extracted from the HDFS FsImage dataset.Nov 2 2022, 5:13 PM

Antoine_Quhen renamed this task from Create a dashboard from the Dataset extracted from the HDFS FsImage dataset to Create a dashboard from the Dataset extracted from the HDFS FsImage.

• EChetty renamed this task from Create a dashboard from the Dataset extracted from the HDFS FsImage to Create a dashboard from the fsImage Dataset extracted from the HDFS FsImage.Nov 2 2022, 5:23 PM

Antoine_Quhen moved this task from Next Up to In Progress on the Data Pipelines (Sprint 03) board.Nov 3 2022, 9:16 AM

Change 853303 had a related patch set uploaded (by Aqu; author: Aqu):

[analytics/refinery@master] Declare HDFS fsimage dataset in hive metastore

https://gerrit.wikimedia.org/r/853303

gerritbot added a project: Patch-For-Review.Nov 4 2022, 2:43 PM

JArguello-WMF moved this task from In Progress to In Review on the Data Pipelines (Sprint 03) board.Nov 4 2022, 4:50 PM

• EChetty moved this task from Sprint 03 to Sprint 04 on the Data Pipelines board.Nov 7 2022, 10:20 AM

• EChetty edited projects, added Data Pipelines (Sprint 04); removed Data Pipelines (Sprint 03).

• EChetty moved this task from Ready to In Review on the Data Pipelines (Sprint 04) board.

• EChetty removed a project: Data-Engineering-Planning.Nov 7 2022, 11:12 AM

Antoine_Quhen added a comment.Nov 14 2022, 1:45 PM

This comment was removed by Antoine_Quhen.

The dashboard needs to be set up on production after the deployment of the dataset.

Draft is here: https://superset.wikimedia.org/r/2073

Antoine_Quhen updated the task description. (Show Details)Nov 14 2022, 1:51 PM

We could also create a dashboard to know where we have Parquet mostly, Avro mostly, etc,

• EChetty moved this task from Sprint 04 to Sprint 05-06 on the Data Pipelines board.Nov 28 2022, 11:06 AM

• EChetty edited projects, added Data Pipelines (Sprint 05-06); removed Data Pipelines (Sprint 04).

• EChetty moved this task from Ready to In Review on the Data Pipelines (Sprint 05-06) board.

HouseBlaster moved this task from Unsorted to Add / Create on the Technical-Debt board.Dec 3 2022, 7:43 PM

JArguello-WMF moved this task from In Review to Ready to Deploy on the Data Pipelines (Sprint 05-06) board.Dec 6 2022, 5:03 PM

Change 853303 merged by Aqu:

[analytics/refinery@master] Declare the HDFS usage dataset in hive metastore

https://gerrit.wikimedia.org/r/853303

Antoine_Quhen moved this task from Ready to Deploy to Done on the Data Pipelines (Sprint 05-06) board.Dec 9 2022, 2:44 PM

Here is a dashboard with 2 treemaps, 1 for the aggregated files sizes, 1 for the aggregated files counts: https://superset.wikimedia.org/superset/dashboard/409/

Milimetric awarded a token.Jan 4 2023, 4:50 PM

Antoine_Quhen updated the task description. (Show Details)Jan 4 2023, 8:26 PM

@Antoine_Quhen the dashboard doesn't show any results currently.

• EChetty closed this task as Resolved.Jan 17 2023, 11:38 AM

Create a dashboard from the fsImage Dataset extracted from the HDFS FsImageClosed, ResolvedPublic3 Estimated Story PointsActions