Page MenuHomePhabricator
Create Task
Maniphest T334455

Failover all db1115 services to db1215
Closed, ResolvedPublic
Actions

Description

db1215 will replace db1115 once it is ready.
There's currently lots of things relaying on db1115 that we need to failover:

Details

Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
gerritbot added a comment.Apr 12 2023, 6:01 AM

Change 908157 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] kormat/bashrc.wmf: Change alias location

https://gerrit.wikimedia.org/r/908157

gerritbot added a project: Patch-For-Review.Apr 12 2023, 6:01 AM

Change 908157 merged by Marostegui:

[operations/puppet@production] kormat/bashrc.wmf: Change alias location

https://gerrit.wikimedia.org/r/908157

Marostegui updated the task description. (Show Details)Apr 12 2023, 6:02 AM
Maintenance_bot removed a project: Patch-For-Review.Apr 12 2023, 6:10 AM
Ladsgroup updated the task description. (Show Details)Apr 12 2023, 8:04 AM
Ladsgroup updated the task description. (Show Details)Apr 12 2023, 10:26 AM
Marostegui mentioned this in T334564: Switchover x1 master (db1179 -> db1220).Apr 12 2023, 10:28 AM
gerritbot added a comment.Apr 17 2023, 5:57 PM

Change 909324 had a related patch set uploaded (by Jcrespo; author: Jcrespo):

[operations/puppet@production] .bashrc: Change alias location

https://gerrit.wikimedia.org/r/909324

gerritbot added a project: Patch-For-Review.Apr 17 2023, 5:57 PM
jcrespo updated the task description. (Show Details)Apr 17 2023, 6:01 PM
Marostegui added a comment.Apr 19 2023, 8:31 AM

I am going to create a specific task for the switchover itself and then use this as "services" migrations. It is a complex process and I want to have it split for my own sanity.
Though, given that tomorrow is the designed last day for maintenance before the DC switch back to eqiad, I am not going to attempt this today as many things can break.

Marostegui mentioned this in T335014: Switchover db1115 -> db1215.Apr 19 2023, 8:46 AM
Marostegui updated the task description. (Show Details)Apr 19 2023, 9:28 AM
Marostegui updated the task description. (Show Details)
gerritbot added a comment.Apr 19 2023, 9:29 AM

Change 909964 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/software@master] section: Update zarcillo location

https://gerrit.wikimedia.org/r/909964

gerritbot added a comment.Apr 19 2023, 9:30 AM

Change 909964 merged by Marostegui:

[operations/software@master] section: Update zarcillo location

https://gerrit.wikimedia.org/r/909964

Marostegui updated the task description. (Show Details)Apr 19 2023, 9:30 AM
Marostegui updated the task description. (Show Details)
gerritbot added a comment.Apr 19 2023, 9:31 AM

Change 909965 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/software@master] host-to-instance: Change zarcillo location

https://gerrit.wikimedia.org/r/909965

Marostegui updated the task description. (Show Details)Apr 19 2023, 9:32 AM
gerritbot added a comment.Apr 19 2023, 9:32 AM

Change 909965 merged by jenkins-bot:

[operations/software@master] host-to-instance: Change zarcillo location

https://gerrit.wikimedia.org/r/909965

Marostegui updated the task description. (Show Details)Apr 19 2023, 9:33 AM
gerritbot added a comment.Apr 19 2023, 9:38 AM

Change 909966 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/software@master] check-master-heartbeat.sh: Change zarcillo location

https://gerrit.wikimedia.org/r/909966

Marostegui updated the task description. (Show Details)Apr 19 2023, 9:39 AM
gerritbot added a comment.Apr 19 2023, 9:42 AM

Change 909966 merged by Marostegui:

[operations/software@master] check-master-heartbeat.sh: Change zarcillo location

https://gerrit.wikimedia.org/r/909966

gerritbot added a comment.Apr 19 2023, 9:44 AM

Change 909967 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] common.yaml: Add db1215 to mysql clients

https://gerrit.wikimedia.org/r/909967

gerritbot added a comment.Apr 19 2023, 9:46 AM

Change 909967 merged by Marostegui:

[operations/puppet@production] common.yaml: Add db1215 to mysql clients

https://gerrit.wikimedia.org/r/909967

gerritbot added a comment.Apr 19 2023, 10:10 AM

Change 909972 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] prometheus.yaml: Change zarcillo location

https://gerrit.wikimedia.org/r/909972

Marostegui updated the task description. (Show Details)Apr 19 2023, 10:11 AM
gerritbot added a comment.Apr 19 2023, 10:29 AM

Change 909972 merged by Marostegui:

[operations/puppet@production] prometheus.yaml: Change zarcillo location

https://gerrit.wikimedia.org/r/909972

Marostegui updated the task description. (Show Details)Apr 19 2023, 10:39 AM
gerritbot added a comment.Apr 19 2023, 9:33 PM

Change 910076 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] prometheus: Change zarcillo location

https://gerrit.wikimedia.org/r/910076

gerritbot added a comment.Apr 19 2023, 9:35 PM

Change 910076 merged by Cwhite:

[operations/puppet@production] prometheus: Change zarcillo location

https://gerrit.wikimedia.org/r/910076

Marostegui updated the task description. (Show Details)Apr 20 2023, 5:07 AM
gerritbot added a comment.Apr 20 2023, 7:32 AM

Change 910414 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/software@master] change_mw_mysql_pass.sh: Change zarcillo host

https://gerrit.wikimedia.org/r/910414

Marostegui updated the task description. (Show Details)Apr 20 2023, 7:33 AM
gerritbot added a comment.Apr 20 2023, 7:34 AM

Change 910414 merged by jenkins-bot:

[operations/software@master] change_mw_mysql_pass.sh: Change zarcillo host

https://gerrit.wikimedia.org/r/910414

gerritbot added a comment.Apr 25 2023, 6:27 AM

Change 911693 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/software@master] switchover-tmpl.py: Replace zarcillo host

https://gerrit.wikimedia.org/r/911693

Marostegui updated the task description. (Show Details)Apr 25 2023, 6:28 AM
gerritbot added a comment.Apr 25 2023, 6:28 AM

Change 911693 merged by jenkins-bot:

[operations/software@master] switchover-tmpl.py: Replace zarcillo host

https://gerrit.wikimedia.org/r/911693

Marostegui updated the task description. (Show Details)Apr 25 2023, 6:30 AM
• I subscribed.May 1 2023, 4:30 AM

This ID is so easy to remember that the task goes so smoothly

Peachey88 subscribed.May 1 2023, 6:02 AM
In T334455#8815619, @I wrote:

This ID is so easy to remember that the task goes so smoothly

Please don't post unhelpful and off topic comments on phabricator tasks.

gerritbot added a comment.May 8 2023, 11:29 AM

Change 917313 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] orchestrator: Change database

https://gerrit.wikimedia.org/r/917313

Marostegui updated the task description. (Show Details)May 8 2023, 11:30 AM
gerritbot added a comment.May 8 2023, 11:31 AM

Change 917320 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/software/wmfmariadbpy@master] switchover.py: Replace zarcillo host

https://gerrit.wikimedia.org/r/917320

Marostegui updated the task description. (Show Details)May 8 2023, 11:32 AM
Ladsgroup closed subtask T336166: Tests failing for wmfmariadbpy/cli_admin/switchover.py as Resolved.May 8 2023, 12:55 PM
gerritbot added a comment.May 9 2023, 5:30 AM

Change 917313 merged by Marostegui:

[operations/puppet@production] orchestrator: Change database

https://gerrit.wikimedia.org/r/917313

gerritbot added a comment.May 9 2023, 5:33 AM

Change 917320 merged by Marostegui:

[operations/software/wmfmariadbpy@master] switchover.py: Replace zarcillo host

https://gerrit.wikimedia.org/r/917320

Marostegui mentioned this in rOSMD6d2f77bc653c: switchover.py: Replace zarcillo host.May 9 2023, 5:33 AM
gerritbot added a comment.May 9 2023, 5:35 AM

Change 917434 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/software@master] report_users.sh: Change zarcillo host

https://gerrit.wikimedia.org/r/917434

Marostegui updated the task description. (Show Details)May 9 2023, 5:35 AM
Marostegui updated the task description. (Show Details)
gerritbot added a comment.May 9 2023, 5:37 AM

Change 909324 merged by Marostegui:

[operations/puppet@production] .bashrc: Change alias location

https://gerrit.wikimedia.org/r/909324

gerritbot added a comment.May 9 2023, 5:39 AM

Change 917434 merged by Marostegui:

[operations/software@master] report_users.sh: Change zarcillo host

https://gerrit.wikimedia.org/r/917434

Marostegui updated the task description. (Show Details)May 9 2023, 5:39 AM
Marostegui closed subtask T335014: Switchover db1115 -> db1215 as Resolved.
Maintenance_bot removed a project: Patch-For-Review.May 9 2023, 6:10 AM
Marostegui added a comment.May 9 2023, 6:13 AM

I think everything is failed over. What I am going to do in around 1h, is to stop mariadb on db1115 and see if something breaks.

gerritbot added a comment.May 9 2023, 6:15 AM

Change 917587 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] db1115: Disable notifications

https://gerrit.wikimedia.org/r/917587

gerritbot added a project: Patch-For-Review.May 9 2023, 6:15 AM

Change 917587 merged by Marostegui:

[operations/puppet@production] db1115: Disable notifications

https://gerrit.wikimedia.org/r/917587

Maintenance_bot removed a project: Patch-For-Review.May 9 2023, 6:30 AM
Stashbot added a comment.May 9 2023, 8:40 AM

Mentioned in SAL (#wikimedia-operations) [2023-05-09T08:40:01Z] <marostegui> Stop mariadb on db1115 (old zarcillo master) T334455

gerritbot added a comment.May 9 2023, 9:49 AM

Change 917833 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/software/wmfdb@master] db_mysql.py: Replace zarcillo master

https://gerrit.wikimedia.org/r/917833

gerritbot added a project: Patch-For-Review.May 9 2023, 9:49 AM
Marostegui updated the task description. (Show Details)May 9 2023, 9:50 AM
Ladsgroup closed subtask T336240: Tests failing for wmfdb/cli_admin/db_mysql.py as Resolved.May 9 2023, 11:01 AM
gerritbot added a comment.May 9 2023, 11:02 AM

Change 917833 merged by jenkins-bot:

[operations/software/wmfdb@master] db_mysql.py: Replace zarcillo master

https://gerrit.wikimedia.org/r/917833

Marostegui updated the task description. (Show Details)May 9 2023, 11:02 AM
Ladsgroup subscribed.May 9 2023, 11:03 AM

I don't know how wmfdb is deployed. Do we need a release and packaging of it too?

Ladsgroup added a comment.May 9 2023, 11:04 AM

And https://wikitech.wikimedia.org/w/index.php?limit=50&offset=0&profile=default&search=db1115&title=Special:Search&ns0=1&ns12=1&ns116=1&ns498=1

Marostegui added a subscriber: MoritzMuehlenhoff.May 9 2023, 11:07 AM

To be honest, I don't know, there's not much doc: https://wikitech.wikimedia.org/wiki/Wmfdb
From this task https://phabricator.wikimedia.org/T304915 I understand it is "just" creating the debs and installing them

root@cumin1001:/usr# dpkg -l | grep wmfdb
ii  python3-wmfdb                        0.1.2+deb11u1                  amd64        Libraries for interacting with WMF's mariadb deployments
ii  wmfdb-admin                          0.1.2+deb11u1                  amd64        Utilities for maintaining WMF's mariadb deployments

On the debian directory of the wmfdb repo there're debian package files:

compat				changelog			source
control				python3-wmfdb.install		wmfdb-admin.install
copyright			rules				wmfdb-admin.lintian-overrides

I am not sure if @MoritzMuehlenhoff was involved and might recall something, or could help us figure this out.

Marostegui added a comment.May 9 2023, 11:08 AM
In T334455#8836413, @Ladsgroup wrote:

And https://wikitech.wikimedia.org/w/index.php?limit=50&offset=0&profile=default&search=db1115&title=Special:Search&ns0=1&ns12=1&ns116=1&ns498=1

Will take care of this, thanks!

Maintenance_bot removed a project: Patch-For-Review.May 9 2023, 11:10 AM
Marostegui mentioned this in T336253: decommission db1115.eqiad.wmnet.May 9 2023, 11:44 AM
Marostegui mentioned this in rOSWD524b56bee7bb: db_mysql.py: Replace zarcillo master.May 9 2023, 1:26 PM
MoritzMuehlenhoff added a comment.May 10 2023, 3:10 PM
In T334455#8836415, @Marostegui wrote:

I am not sure if @MoritzMuehlenhoff was involved and might recall something, or could help us figure this out.

I wasn't involved in this, but I can take care of updating the deb, just let me know. It also currently seems to lack puppet integration, at least I don't see anything in puppet.git which installs it on the cumin hosts.

Marostegui added a comment.May 16 2023, 7:02 AM

@Ladsgroup I want to close this task, you are following up the new version of wmfmariadbpy with T336174
Do you want me to create a specific task for packaging wmfdb?

Ladsgroup added a comment.May 16 2023, 9:38 AM

Sorry it looks like I dropped the ball on this. I'm not lazy. just wmfdb has so many changes since the last release and I want to package and release when you're around in case things go sideways. It's my top prio today and I don't care either way regarding the ticket.

gerritbot added a comment.May 16 2023, 9:58 AM

Change 920214 had a related patch set uploaded (by Ladsgroup; author: Amir Sarabadani):

[operations/software/wmfdb@master] Prepare for v0.1.3 release

https://gerrit.wikimedia.org/r/920214

gerritbot added a project: Patch-For-Review.May 16 2023, 9:58 AM
gerritbot added a comment.May 16 2023, 7:08 PM

Change 920214 merged by jenkins-bot:

[operations/software/wmfdb@master] Prepare for v0.1.3 release

https://gerrit.wikimedia.org/r/920214

Ladsgroup mentioned this in rOSWDda3bd2619193: Prepare for v0.1.3 release.May 16 2023, 7:09 PM
Maintenance_bot removed a project: Patch-For-Review.May 16 2023, 7:11 PM
Ladsgroup added a comment.May 16 2023, 7:44 PM
ladsgroup@build2001:~/wmfdb$ ls  -la /var/cache/pbuilder/result/bullseye-amd64/*wmfdb*
-rw-r--r-- 1 ladsgroup wikidev  6540 May 16 19:14 /var/cache/pbuilder/result/bullseye-amd64/python3-wmfdb_0.1.3_amd64.deb
-rw-r--r-- 1 ladsgroup wikidev  6311 May 16 19:14 /var/cache/pbuilder/result/bullseye-amd64/wmfdb_0.1.3_amd64.buildinfo
-rw-r--r-- 1 ladsgroup wikidev  1888 May 16 19:14 /var/cache/pbuilder/result/bullseye-amd64/wmfdb_0.1.3_amd64.changes
-rw-r--r-- 1 ladsgroup wikidev   702 May 16 19:14 /var/cache/pbuilder/result/bullseye-amd64/wmfdb_0.1.3.dsc
-rw-r--r-- 1 ladsgroup wikidev  1176 May 16 19:14 /var/cache/pbuilder/result/bullseye-amd64/wmfdb_0.1.3_source.changes
-rw-r--r-- 1 ladsgroup wikidev 19456 May 16 19:14 /var/cache/pbuilder/result/bullseye-amd64/wmfdb_0.1.3.tar.xz
-rw-r--r-- 1 ladsgroup wikidev  4412 May 16 19:14 /var/cache/pbuilder/result/bullseye-amd64/wmfdb-admin_0.1.3_amd64.deb
Ladsgroup added a comment.May 16 2023, 7:55 PM

Pushed to apt. Gonna try it out in cumin2002 tomorrow

Marostegui added a comment.May 17 2023, 7:15 AM
In T334455#8857006, @Ladsgroup wrote:

Pushed to apt. Gonna try it out in cumin2002 tomorrow

<3

Ladsgroup closed subtask T336174: Release a new version of wmfmariadbpy as Resolved.May 17 2023, 4:02 PM
Ladsgroup added a comment.May 17 2023, 4:09 PM
ladsgroup@cumin1001:~$ sudo debdeploy deploy -u 2023-05-17-wmfdb.yaml -s cumin-all
Rolling out wmfdb:
Non-daemon update, no service restart needed

python3-wmfdb was updated: 0.1.2+deb11u1 -> 0.1.3
  cumin1001.eqiad.wmnet (1 hosts)

wmfdb-admin was updated: 0.1.2+deb11u1 -> 0.1.3
  cumin1001.eqiad.wmnet (1 hosts)

These hosts are already up-to-date:
  cumin2002.codfw.wmnet (1 hosts)

The package to be updated isn't installed on these hosts:
  cloudcumin2001.codfw.wmnet,cloudcumin1001.eqiad.wmnet,cuminunpriv100
1.eqiad.wmnet (3 hosts)

ladsgroup@cumin1001:~$

It should be done now.

Ladsgroup closed this task as Resolved.May 17 2023, 6:28 PM
Maintenance_bot moved this task from In progress to Done on the DBA board.May 17 2023, 6:29 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits