Page MenuHomePhabricator
Create Task
Maniphest T310079

Upgrade DataHub V0.8.38
Closed, ResolvedPublic
Actions

Assigned To
BTullis
Authored By
odimitrijevic
Jun 7 2022, 2:48 PM
Tags
Referenced Files
Restricted File
Jun 17 2022, 5:20 PM
Restricted File
Jun 17 2022, 5:20 PM
Restricted File
Jun 16 2022, 3:33 PM
Subscribers
Aklapper
odimitrijevic

Description

Version 0.8.36 of DataHub provides a UI for editing the Data Glossary, which is a feature we would really like to try.

They skipped version 0.8.37 due to an error on their part, then they have followed it up quickly with 0.8.38

Version 0.8.38 includes more Glossary related features:

  • Link Glossary Terms to one another via the UI - You can now link Glossary Terms from the UI with the following relationships:
    • Contains - relate Terms when one is a superset of another, eg. Address contains Zip Code
    • Inherits - relate Terms with one is a subtype of another, eg. Email inherits PII

Details

SubjectRepoBranchLines +/-
Update the container used for datahub deploymentsoperations/deployment-chartsmaster+1 -1
Update the blubber configuration for the datahub-frontendanalytics/datahubwmf+15 -5
Disable native authentication for DataHuboperations/deployment-chartsmaster+3 -1
Disable the telemetry for DataHuboperations/deployment-chartsmaster+3 -1
Update the container image used by DataHub 0.8.38operations/deployment-chartsmaster+1 -1
Update the name of the binary used to launch datahub-frontendanalytics/datahubwmf+1 -1
Update the version of the datahub containers that is deployedoperations/deployment-chartsmaster+11 -11
Release v0.8.38 of DataHub using WMF customizationanalytics/datahubwmf+2 -0
Customize query in gerrit

Related Objects

Event Timeline

odimitrijevic created this task.Jun 7 2022, 2:48 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 7 2022, 2:48 PM
• EChetty assigned this task to BTullis.Jun 7 2022, 2:56 PM
• EChetty triaged this task as High priority.
• EChetty moved this task from Next Up to In Progress on the Data-Catalog board.
• EChetty moved this task from In Progress to Next Up on the Data-Catalog board.
• EChetty moved this task from Next Up to In Progress on the Data-Catalog board.
BTullis renamed this task from Upgrade DataHub V0.8.36 to Upgrade DataHub V0.8.38.Jun 10 2022, 3:42 PM
BTullis updated the task description. (Show Details)
BTullis added a comment.Jun 10 2022, 4:03 PM

I have started work on this and I am following the guidelines here: https://wikitech.wikimedia.org/wiki/Analytics/Systems/DataHub/Upgrading

Get update from upstream

git remote update linkedin-github

Push master branch to Gerrit's master banch

git push origin linkedin-github/master:master

Push tags to Gerrit

git push origin --tags

Rebase our changes against v0.8.38

git rebase -i v0.8.38

Fixed one small merge conflict in docker/mysql-setup/init.sh

vi  docker/mysql-setup/init.sh
git add docker/
git rebase --continue

Push the updated wmf branch.

git push --force-with-lease
gerritbot added a comment.Jun 10 2022, 4:04 PM

Change 804611 had a related patch set uploaded (by Btullis; author: Btullis):

[analytics/datahub@wmf] Release v0.8.38 of DataHub using WMF customization

https://gerrit.wikimedia.org/r/804611

gerritbot added a project: Patch-For-Review.Jun 10 2022, 4:04 PM
BTullis added a comment.Jun 10 2022, 4:05 PM

I have now made a small change to the README-WMF.md file so that it triggers the build pipeline:
https://gerrit.wikimedia.org/r/c/analytics/datahub/+/804611

gerritbot added a comment.Jun 14 2022, 2:53 PM

Change 804611 merged by jenkins-bot:

[analytics/datahub@wmf] Release v0.8.38 of DataHub using WMF customization

https://gerrit.wikimedia.org/r/804611

BTullis mentioned this in R3101:b50096faf734: Release v0.8.38 of DataHub using WMF customization.Jun 14 2022, 2:54 PM
Maintenance_bot removed a project: Patch-For-Review.Jun 14 2022, 3:30 PM
gerritbot added a comment.Jun 15 2022, 2:08 PM

Change 805826 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/deployment-charts@master] Update the version of the datahub containers that is deployed

https://gerrit.wikimedia.org/r/805826

gerritbot added a project: Patch-For-Review.Jun 15 2022, 2:08 PM
gerritbot added a comment.Jun 15 2022, 2:20 PM

Change 805826 merged by jenkins-bot:

[operations/deployment-charts@master] Update the version of the datahub containers that is deployed

https://gerrit.wikimedia.org/r/805826

Maintenance_bot removed a project: Patch-For-Review.Jun 15 2022, 2:31 PM
gerritbot added a comment.Jun 16 2022, 2:52 PM

Change 805839 merged by jenkins-bot:

[analytics/datahub@wmf] Update the name of the binary used to launch datahub-frontend

https://gerrit.wikimedia.org/r/805839

BTullis mentioned this in R3101:47e62ef66755: Update the name of the binary used to launch datahub-frontend.Jun 16 2022, 2:54 PM
gerritbot added a comment.Jun 16 2022, 3:16 PM

Change 806232 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/deployment-charts@master] Update the container image used by DataHub 0.8.38

https://gerrit.wikimedia.org/r/806232

gerritbot added a project: Patch-For-Review.Jun 16 2022, 3:16 PM
gerritbot added a comment.Jun 16 2022, 3:25 PM

Change 806232 merged by jenkins-bot:

[operations/deployment-charts@master] Update the container image used by DataHub 0.8.38

https://gerrit.wikimedia.org/r/806232

BTullis added a comment.Jun 16 2022, 3:33 PM

Unfortunately, we're getting an error after deployment of 0.8.38.
{F35247060}
I will investigate now.

gerritbot added a comment.Jun 17 2022, 9:40 AM

Change 806381 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/deployment-charts@master] Disable the telemetry for DataHub

https://gerrit.wikimedia.org/r/806381

gerritbot added a comment.Jun 17 2022, 9:50 AM

Change 806381 merged by jenkins-bot:

[operations/deployment-charts@master] Disable the telemetry for DataHub

https://gerrit.wikimedia.org/r/806381

gerritbot added a comment.Jun 17 2022, 11:23 AM

Change 806396 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/deployment-charts@master] Disable native authentication for DataHub

https://gerrit.wikimedia.org/r/806396

gerritbot added a comment.Jun 17 2022, 11:31 AM

Change 806396 merged by jenkins-bot:

[operations/deployment-charts@master] Disable native authentication for DataHub

https://gerrit.wikimedia.org/r/806396

gerritbot added a comment.Jun 17 2022, 2:26 PM

Change 806427 had a related patch set uploaded (by Btullis; author: Btullis):

[analytics/datahub@wmf] Update the blubber configuration for the datahub-frontend

https://gerrit.wikimedia.org/r/806427

gerritbot added a comment.Jun 17 2022, 3:52 PM

Change 806427 merged by jenkins-bot:

[analytics/datahub@wmf] Update the blubber configuration for the datahub-frontend

https://gerrit.wikimedia.org/r/806427

BTullis mentioned this in R3101:8e8079b2eb49: Update the blubber configuration for the datahub-frontend.Jun 17 2022, 3:52 PM
gerritbot added a comment.Jun 17 2022, 4:03 PM

Change 806435 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/deployment-charts@master] Update the container used for datahub deployments

https://gerrit.wikimedia.org/r/806435

gerritbot added a comment.Jun 17 2022, 4:31 PM

Change 806435 merged by jenkins-bot:

[operations/deployment-charts@master] Update the container used for datahub deployments

https://gerrit.wikimedia.org/r/806435

BTullis added a comment.Jun 17 2022, 5:20 PM

This is now working. The issue was caused by a small change to their build process, which we hadn't copied to the blubber pipeline.
The result was that when we built the datahub-frontend container it was missing the web-react components.

Nice things about this release:

  1. Analytics is working now: https://datahub.wikimedia.org/analytics

{F35249401}

  1. The Glossary is now editable in the UI - This is accessible from the Govern menu at the top right.

{F35249403}

BTullis moved this task from In Progress to Done on the Data-Catalog board.Jun 17 2022, 5:20 PM
BTullis edited projects, added Data-Engineering-Kanban, Data-Engineering; removed Patch-For-Review.
BTullis moved this task from Incoming (new tickets) to Security & Governance on the Data-Engineering board.
BTullis moved this task from Next Up to Done on the Data-Engineering-Kanban board.
BTullis closed this task as Resolved.Jun 27 2022, 1:44 PM
BTullis mentioned this in R3101:cefeece1f42f: Release v0.8.38 of DataHub using WMF customization.Aug 26 2022, 11:55 AM
BTullis mentioned this in R3101:a76974138ecc: Update the name of the binary used to launch datahub-frontend.
BTullis mentioned this in R3101:48d31fbaecd2: Update the blubber configuration for the datahub-frontend.
BTullis mentioned this in R3101:f00876a8d49c: Release v0.8.38 of DataHub using WMF customization.Oct 31 2022, 5:23 PM
BTullis mentioned this in R3101:e35e7df2e300: Update the name of the binary used to launch datahub-frontend.
BTullis mentioned this in R3101:d0d269555ee0: Update the blubber configuration for the datahub-frontend.
Stevemunene mentioned this in R3101:8ec14704be83: Release v0.8.38 of DataHub using WMF customization.Mar 8 2023, 9:54 AM
Stevemunene mentioned this in R3101:b5bf56af8cc8: Update the name of the binary used to launch datahub-frontend.
Stevemunene mentioned this in R3101:9ea9045dfe04: Update the blubber configuration for the datahub-frontend.
Stevemunene mentioned this in R3101:2e085a27e64d: Release v0.8.38 of DataHub using WMF customization.Mar 8 2023, 11:27 AM
Stevemunene mentioned this in R3101:5c8ba1f0c1c0: Update the name of the binary used to launch datahub-frontend.
Stevemunene mentioned this in R3101:0e98b5f6530e: Update the blubber configuration for the datahub-frontend.
BTullis mentioned this in R3101:b938336e3c94: Release v0.8.38 of DataHub using WMF customization.Jul 5 2023, 11:13 AM
BTullis mentioned this in R3101:6874fa0bf8c4: Update the name of the binary used to launch datahub-frontend.
BTullis mentioned this in R3101:639b9e88977e: Update the blubber configuration for the datahub-frontend.
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL