Configure MariaDB database for DataHub on an-coord1001
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	BTullis
	Feb 10 2022, 12:14 PM

Description

We need a database for the DataHub MVP so it makes sense to use the existing MariaDB instance on an-coord1001.

The initial creation of the database user and schema was done for the prototype here: T299703#7659244

There are two database init scripts provided, one for mysql and one for mariadb:

mysql based setup: https://github.com/linkedin/datahub/blob/master/docker/mysql-setup/init.sql
mariadb based setup: https://github.com/linkedin/datahub/blob/master/docker/mariadb/init.sql

We can use either one. The only difference is the mysql version creates a metadata_index table, which according to DataHub slack is no longer used. (TODO: send a PR removing it from the script)

Details

Subject	Repo	Branch	Lines +/-
Allow kikikube staging pods to access the analytics-meta test instance	operations/puppet	production	+1 -1
Use test coordinator for staging datahub deploy	operations/puppet	production	+3 -3
Allow access to MariaDB analytics-meta from Kubernetes pods	operations/puppet	production	+2 -2

Customize query in gerrit

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Resolved		BTullis	T299910 Data Catalog MVP
		Resolved		BTullis	T301459 Configure MariaDB database for DataHub on an-coord1001

Event Timeline

BTullis created this task.Feb 10 2022, 12:14 PM

BTullis edited parent tasks, added: T299910: Data Catalog MVP; removed: T301385: Deploy DataHub in MVP phase.Feb 11 2022, 2:52 PM

Milimetric moved this task from Backlog to Next Up on the Data-Catalog board.Feb 14 2022, 4:52 PM

odimitrijevic moved this task from Incoming (new tickets) to Security & Governance on the Data-Engineering board.Feb 14 2022, 4:53 PM

Milimetric updated the task description. (Show Details)Feb 17 2022, 4:26 PM

Milimetric updated the task description. (Show Details)Feb 22 2022, 6:42 PM

BTullis claimed this task.Mar 11 2022, 3:19 PM

BTullis triaged this task as High priority.

BTullis moved this task from Next Up to In Progress on the Data-Catalog board.

BTullis moved this task from Next Up to In Progress on the Data-Engineering-Kanban board.

Change 769993 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/puppet@production] Allow access to MariaDB analytics-meta from Kubernetes pods

https://gerrit.wikimedia.org/r/769993

gerritbot added a project: Patch-For-Review.Mar 11 2022, 3:24 PM

I have created a CR that allows the wikikube and staging Kubernetes clusters access to the MariaDB port running on an-coord100[1-2].
https://gerrit.wikimedia.org/r/c/operations/puppet/+/769993

I will create the datahub database and its user account manually on an-coord1001.
However I will try using the mysql-setup-job job to create the schema, if possible. This way we will have a mechanism to apply schema updates automatically whenever the source file changes.

Change 769993 merged by Btullis:

[operations/puppet@production] Allow access to MariaDB analytics-meta from Kubernetes pods

https://gerrit.wikimedia.org/r/769993

Maintenance_bot removed a project: Patch-For-Review.Mar 11 2022, 4:10 PM

I have created the user manually on an-coord1001.

The password is in pwstore.

@BTullis you could instead put the password in puppet private repo, and then use that to render it in the configs you need. IIRC You can also use this in puppet to render secrets into values files that are only available on the deployment host, and will be available in your helm templates.

@BTullis you could instead put the password in puppet private repo, and then use that to render it in the configs you need.

Thanks, yes I do intend to use this method and when it comes to it I will transfer the password from pwstore to puppet (to avoid duplication).

• razzi moved this task from In Progress to Done on the Data-Catalog board.Mar 16 2022, 4:22 PM

I have now moved this password to the private repo, where it is in: hieradata/role/common/deployment_server/kubernetes.yaml
The entry in pwstore has been deleted.

BTullis closed this task as Resolved.Mar 17 2022, 11:05 AM

Change 774458 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/puppet@production] Use test coordinator for staging datahub deploy

https://gerrit.wikimedia.org/r/774458

gerritbot added a project: Patch-For-Review.Mar 28 2022, 11:51 AM

Change 774458 merged by Btullis:

[operations/puppet@production] Use test coordinator for staging datahub deploy

https://gerrit.wikimedia.org/r/774458

Maintenance_bot removed a project: Patch-For-Review.Mar 29 2022, 2:10 PM

Change 777329 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/puppet@production] Allow kikikube staging pods to access the analytics-meta test instance

https://gerrit.wikimedia.org/r/777329

gerritbot added a project: Patch-For-Review.Apr 5 2022, 9:20 AM

Change 777329 merged by Btullis: