Page MenuHomePhabricator
Create Task
Maniphest T299910

Data Catalog MVP
Closed, ResolvedPublic
Actions

Description

Goal: :
Validate if having a Data Catalog will improve the discoverability and usability of our data.

Task One Liner:
Deploy a single data catalog solution hosting the Hive Metastore & Event Platform to create an active inventory of the data assets.

Functional Requirements
[Primary] Searching and filtering options to allow users to quickly find relevant sets of data for analytics or data engineering requirements.

Technical Requirements:
[Primary] Have the complete Hive Metastore imported into the Data Catalog

Product One Pager:
Here

Details

SubjectRepoBranchLines +/-
Disable analytics for datahuboperations/deployment-chartsmaster+1 -0
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedBTullisT299910 Data Catalog MVP
 Resolved• razziT301382 Set up opensearch cluster for datahub
 Resolved• razziT301383 eqiad: 3 VMs requested for datahub opensearch cluster
 ResolvedBTullisT301458 Define LVS load-balancing for OpenSearch cluster
 ResolvedBTullisT302818 Complete monitoring setup of datahubsearch nodes
 DeclinedNoneT301386 Proof-of-concept Karapace as Confluent schema registry replacement
 ResolvedBTullisT301453 Create DataHub containers with deployment pipeline
 ResolvedBTullisT301454 Define the Helm charts and helmfile deployments for Datahub
 ResolvedJMeybohmT303049 New Service Request: DataHub
 ResolvedBTullisT301460 Update DNS for the DataHub MVP services
 ResolvedBTullisT301886 Upload required datahub dependencies to Archiva
 ResolvedBTullisT301459 Configure MariaDB database for DataHub on an-coord1001
 ResolvedBTullisT301462 Configure LDAP authentication for the DataHub frontend
 Resolved• razziT301562 Set up karapace instance for datahub
 ResolvedakosiarisT301563 eqiad: 1 VM requested for karapace
 Resolved• razziT301565 Create debian package of karapace
 ResolvedBTullisT303381 Review and improve the build process for DataHub containers
 ResolvedStevemuneneT305874 Switch DataHub authentication to OIDC
 ResolvedBTullisT306019 Update DataHub to latest version for MVP
 ResolvedBTullisT302730 Validate that we can use ES as a Graph Backend
 ResolvedBTullisT306448 Configure Datahub Authentication and Authorization for the MVP
 OpenNoneT306896 Spike: Integrate Spark with DataHub
 ResolvedBTullisT306903 Integrate Superset with DataHub
 ResolvedMilimetricT299897 Connect MVP to Hive metastore [Mile Stone 4]
 DuplicateNoneT306977 Integrate Airflow with DataHub
 DeclinedNoneT343236 Get datahub-staging.wikimedia.org working with the staging deployment of datahub

Event Timeline

• EChetty created this task.Jan 24 2022, 12:44 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 24 2022, 12:44 PM
• EChetty added projects: Epic, Data-Engineering.Jan 24 2022, 1:05 PM
• EChetty updated the task description. (Show Details)
odimitrijevic moved this task from Incoming (new tickets) to Security & Governance on the Data-Engineering board.Feb 1 2022, 11:12 PM
BTullis claimed this task.Feb 11 2022, 2:47 PM
BTullis merged a task: T301385: Deploy DataHub in MVP phase.
BTullis moved this task from Backlog to In Progress on the Data-Catalog board.
BTullis added subscribers: BTullis, odimitrijevic, • razzi, Milimetric.

Here is the design document for this MVP deployment.

BTullis added a subtask: T301382: Set up opensearch cluster for datahub.Feb 11 2022, 2:50 PM
BTullis added a subtask: T301386: Proof-of-concept Karapace as Confluent schema registry replacement.
BTullis added a subtask: T301453: Create DataHub containers with deployment pipeline.
BTullis added a subtask: T301454: Define the Helm charts and helmfile deployments for Datahub.
BTullis added a subtask: T301458: Define LVS load-balancing for OpenSearch cluster.
BTullis added a subtask: T301459: Configure MariaDB database for DataHub on an-coord1001.Feb 11 2022, 2:52 PM
BTullis added a subtask: T301460: Update DNS for the DataHub MVP services.
BTullis added a subtask: T301462: Configure LDAP authentication for the DataHub frontend.
Milimetric removed a project: Data-Catalog.Feb 14 2022, 5:06 PM
Milimetric added a project: Data-Catalog.Feb 14 2022, 5:09 PM
Milimetric moved this task from In Progress to Epics on the Data-Catalog board.
BTullis closed subtask T301453: Create DataHub containers with deployment pipeline as Resolved.Feb 23 2022, 5:32 PM
BTullis triaged this task as High priority.Mar 2 2022, 10:39 AM
BTullis added a subtask: T303381: Review and improve the build process for DataHub containers.Mar 9 2022, 12:18 PM
BTullis closed subtask T301382: Set up opensearch cluster for datahub as Resolved.Mar 9 2022, 6:24 PM
BTullis closed subtask T301458: Define LVS load-balancing for OpenSearch cluster as Resolved.Mar 17 2022, 10:59 AM
BTullis closed subtask T301459: Configure MariaDB database for DataHub on an-coord1001 as Resolved.Mar 17 2022, 11:05 AM
BTullis closed subtask T301454: Define the Helm charts and helmfile deployments for Datahub as Resolved.Apr 13 2022, 10:36 AM
JArguello-WMF closed subtask T301386: Proof-of-concept Karapace as Confluent schema registry replacement as Declined.Apr 13 2022, 3:16 PM
BTullis added a subtask: T302730: Validate that we can use ES as a Graph Backend.Apr 13 2022, 4:25 PM
BTullis closed subtask T302730: Validate that we can use ES as a Graph Backend as Resolved.
BTullis closed subtask T301462: Configure LDAP authentication for the DataHub frontend as Resolved.Apr 14 2022, 5:02 PM
BTullis added a subtask: T306448: Configure Datahub Authentication and Authorization for the MVP.Apr 20 2022, 2:46 PM
gerritbot added a comment.Apr 20 2022, 3:47 PM

Change 784726 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/deployment-charts@master] Disable analytics for datahub

https://gerrit.wikimedia.org/r/784726

gerritbot added a project: Patch-For-Review.Apr 20 2022, 3:47 PM
gerritbot added a comment.Apr 20 2022, 4:03 PM

Change 784726 merged by jenkins-bot:

[operations/deployment-charts@master] Disable analytics for datahub

https://gerrit.wikimedia.org/r/784726

Maintenance_bot removed a project: Patch-For-Review.Apr 20 2022, 4:31 PM
BTullis closed subtask T306019: Update DataHub to latest version for MVP as Resolved.Apr 20 2022, 5:38 PM
BTullis closed subtask T306448: Configure Datahub Authentication and Authorization for the MVP as Resolved.
BTullis closed subtask T301460: Update DNS for the DataHub MVP services as Resolved.Apr 26 2022, 1:51 PM
BTullis added a subtask: T306896: Spike: Integrate Spark with DataHub.Apr 26 2022, 1:57 PM
BTullis closed subtask T301562: Set up karapace instance for datahub as Resolved.Apr 26 2022, 2:02 PM
BTullis added a subtask: T299897: Connect MVP to Hive metastore [Mile Stone 4].Apr 26 2022, 3:12 PM
BTullis mentioned this in T293647: Data Catalog POC.Apr 29 2022, 9:56 AM
Milimetric reopened subtask T306903: Integrate Superset with DataHub as Open.Jul 7 2022, 5:02 PM
BTullis closed subtask T306903: Integrate Superset with DataHub as Resolved.Jul 15 2022, 4:36 PM
JArguello-WMF closed subtask T299897: Connect MVP to Hive metastore [Mile Stone 4] as Resolved.Jul 25 2022, 9:37 AM
JArguello-WMF removed a project: Data-Catalog.Jun 29 2023, 3:41 PM
JArguello-WMF moved this task from Security & Governance to Data Products & Metrics on the Data-Engineering board.Jun 29 2023, 11:43 PM
BTullis closed subtask T303381: Review and improve the build process for DataHub containers as Resolved.Jul 11 2023, 8:01 PM
BTullis added a project: Data-Platform-SRE.Jul 15 2023, 12:19 AM
BTullis moved this task from Incoming to Epics on the Data-Platform-SRE board.Jul 19 2023, 5:39 PM
BTullis closed subtask T343236: Get datahub-staging.wikimedia.org working with the staging deployment of datahub as Declined.Aug 15 2023, 4:03 PM
Stevemunene closed subtask T305874: Switch DataHub authentication to OIDC as Resolved.Sep 22 2023, 11:17 AM
BTullis mentioned this in T306896: Spike: Integrate Spark with DataHub.Sep 29 2023, 3:13 PM
BTullis closed this task as Resolved.Sep 29 2023, 3:32 PM
BTullis moved this task from Epics to Done on the Data-Platform-SRE board.

I'm going to be bold and resolve this epic.
We have completed all of the tasks that were originally defined to get DataHub to an MVP stage.
There are improvements to be done to make it a production quality service, but these can be handle under a different ticket.

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits