Page MenuHomePhabricator

Move db1110 to m3
Closed, ResolvedPublic

Description

Let's move db1110 temporarily to m3 for T335080: phabricator->phorge migration - database handling

Event Timeline

Change 913029 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] db1118: Will be moved to m3

https://gerrit.wikimedia.org/r/913029

Change 913029 merged by Marostegui:

[operations/puppet@production] db1118: Will be moved to m3

https://gerrit.wikimedia.org/r/913029

Marostegui renamed this task from Move db1118 to m3 to Move db1110 to m3.May 3 2023, 7:42 AM
Marostegui updated the task description. (Show Details)

Change 914703 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] db1118,db1110: Update notes

https://gerrit.wikimedia.org/r/914703

Change 914703 merged by Marostegui:

[operations/puppet@production] db1118,db1110: Update notes

https://gerrit.wikimedia.org/r/914703

Mentioned in SAL (#wikimedia-operations) [2023-05-03T09:44:14Z] <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 4 days, 4:00:00 on db1110.eqiad.wmnet with reason: Moving to m3 T335092

Mentioned in SAL (#wikimedia-operations) [2023-05-03T09:44:27Z] <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4 days, 4:00:00 on db1110.eqiad.wmnet with reason: Moving to m3 T335092

Mentioned in SAL (#wikimedia-operations) [2023-05-03T09:53:37Z] <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1217.eqiad.wmnet with reason: Cloning db1110 from db1217:3323 T335092

Mentioned in SAL (#wikimedia-operations) [2023-05-03T09:53:50Z] <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1217.eqiad.wmnet with reason: Cloning db1110 from db1217:3323 T335092

Change 914727 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] mariadb: Move db1110 to m3

https://gerrit.wikimedia.org/r/914727

Change 914727 merged by Marostegui:

[operations/puppet@production] mariadb: Move db1110 to m3

https://gerrit.wikimedia.org/r/914727

db1110 is now replicating in m3

db1110 is now replicating in m3

Hi Manuel, we finally wanted to use this but turns out db1110 was already physically decom'ed on May 9th, just a few days after this?

Is there another host we can use instead?

As I mentioned on IRC, I probably forgot and decommissioned it.
I will get another host ready.

Change 932356 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] db1118: Disable notifications

https://gerrit.wikimedia.org/r/932356

Change 932356 merged by Marostegui:

[operations/puppet@production] db1118: Disable notifications

https://gerrit.wikimedia.org/r/932356

Change 932686 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] mariadb: Move db1118 to m3

https://gerrit.wikimedia.org/r/932686

Change 932686 merged by Marostegui:

[operations/puppet@production] mariadb: Move db1118 to m3

https://gerrit.wikimedia.org/r/932686

@Dzahn you can use db1118 now.
It is up and running with all the phabricator databases (not sure if you'll need some firewall changes from your side).

Thank you, Manuel :) that's great!

Hey @Marostegui I can confirm no firewall changes are needed. I can talk to db1118 from our test VM.

That being said, I do get "Access denied" from mysql itself when using the same credentials that we use in production when connecting to m3-slave.

What is the expectation around that? Should the grants have been copied with the databases?

Since there are several users for several databases.. not sure which one I should expect to work if any. We only really need the "phuser" to work though.

Hey @Marostegui I can confirm no firewall changes are needed. I can talk to db1118 from our test VM.

That being said, I do get "Access denied" from mysql itself when using the same credentials that we use in production when connecting to m3-slave.

What is the expectation around that? Should the grants have been copied with the databases?

Since there are several users for several databases.. not sure which one I should expect to work if any. We only really need the "phuser" to work though.

Should be fixed now

@Dzahn although I am thinking that maybe you are still connecting through the proxy? WHich isn't going to work for this host. What is the IP of the host you are connecting from?
I might need to add those too.

@Marostegui I am directly connecting to db1118.eqiad.wmnet, not a proxy. I tried both default port and port 3323 like in prod.

I am connecting from 10.64.48.35 (phab-test1001.eqiad.wmnet)

mysql -u phuser -h db1118.eqiad.wmnet -p phabricator_project
mysql -u phuser -h db1118.eqiad.wmnet -p phabricator_project -P 3323

on default port, after pasting prod password for "phuser": "Access denied"

on port 3323, after pasting prod password for "phuser": <times out.. nothing>

For the record, it is the default port, 3306

Should be all fixed for phuser now

I confirm I can now connect from our test VM (phab-test1001) to db1118 with the phuser credentials.

Also I copied the mysqldump files you created for us on request. Thank you!