| Marostegui | |
| Jun 25 2024, 5:57 AM |
| F56203655: image.png | |
| Jul 3 2024, 4:31 PM |
cumin hosts have a .my.cnf that is configured so we can access clouddb* host just reading it while using db-mysql:
[clientlabsdb] user = root password = XXX
With an-redacteddb1001 we need to tell db-mysql to also read that entry.
@BTullis can you send a patch so that can be fixed, as otherwise an-redactddb1001 can not be accessed from cumin hosts with the current tooling, I believe dbutil.py is what we need to start changing - @Ladsgroup can you confirm?
| Title | Reference | Author | Source Branch | Dest Branch | |
|---|---|---|---|---|---|
| Prepare version 0.1.5 release | repos/sre/wmfdb!17 | btullis | release_v_0_1_5 | main | |
| Configure mysql client preferences for an-redacteddb* hosts | repos/sre/wmfdb!16 | btullis | switch_redacteddb | main |
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Unknown Object (Task) | |||||
| Resolved | BTullis | T355571 Q#:rack/setup/install an-redacteddb1001 | |||
| Resolved | BTullis | T365453 Bring an-redacteddb1001 into service to replace clouddb1021 | |||
| Resolved | BTullis | T368354 Modify db-mysql to connect to an-redacteddb1001 from cumin hosts |
I believe dbutil.py is what we need to start changing - @Ladsgroup can you confirm?
Actually I think that file is not used anymore or only used in switchover and some other scripts. You need to change the scripts in wmfdb: https://gitlab.wikimedia.org/repos/sre/wmfdb
btullis opened https://gitlab.wikimedia.org/repos/sre/wmfdb/-/merge_requests/16
Configure mysql client preferences for an-redacteddb* hosts
Thanks both. I have created a merge request here: https://gitlab.wikimedia.org/repos/sre/wmfdb/-/merge_requests/16
See if you think it's OK.
btullis merged https://gitlab.wikimedia.org/repos/sre/wmfdb/-/merge_requests/16
Configure mysql client preferences for an-redacteddb* hosts
btullis opened https://gitlab.wikimedia.org/repos/sre/wmfdb/-/merge_requests/17
Prepare version 0.1.5 release
As I told @ABran-WMF a quick way to check if this was fixed is:
root@cumin1002:~# db-mysql an-redacteddb1001.eqiad.wmnet:3311 ERROR 1045 (28000): Access denied for user 'cumin2024'@'10.64.48.98' (using password: YES) root@cumin1002:~#
That means it is still using the wrong credentials cause it is not reading clientlabsdb from .my.cnf as if I use the right password from that file and bypass the wrapper it all works fine
root@cumin1002:~# mysql --ssl-verify-server-cert -uroot -p -h an-redacteddb1001.eqiad.wmnet -P 3311 Enter password: Welcome to the MariaDB monitor. Commands end with ; or \g. Your MariaDB connection id is 817724 Server version: 10.6.18-MariaDB MariaDB Server Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. root@an-redacteddb1001.eqiad.wmnet[(none)]> Ctrl-C -- exit!
btullis merged https://gitlab.wikimedia.org/repos/sre/wmfdb/-/merge_requests/17
Prepare version 0.1.5 release
I have merged https://gitlab.wikimedia.org/repos/sre/wmfdb/-/merge_requests/17 but I am afraid that I still have no idea how to create the deb package for this.
I see all of the most recent GitLab-CI pipelines for this repository here: https://gitlab.wikimedia.org/repos/sre/wmfdb/-/pipelines and they were created by @ABran-WMF against dgit/bookworm-wikimedia and dgit/bullseye-wikimedia branches, but I am hesitant to proceed without some instruction.
I have read this https://wikitech.wikimedia.org/wiki/Debian_packaging_with_dgit_and_CI but it's pretty impenetrable. Should I just pull those branches and rebase from main, then push them both?
Do I need to use dgit on my workstation to do it, or will standard git be enough?
Apologies for the delay in getting this working, but any help in how to create the packages would be appreciated.
@Marostegui yep no worries! @BTullis you can remove and create dgit/$distro-wikimedia branches no problem. Our current packages have their version known and we can always revert to that if needed to. The specific instructions are here. @MatthewVernon has produced an extensive doc which I tried to follow and proof check last time I went through it. Please let me know if there is a missing link or component in that chapter that would help. Otherwise I'll just add a header saying that its OK to have to delete/overwrite dgit/$branch!
I'm not sure if its the right pattern but I did something like you were suggesting:
with just changelog modifications between bullseye/buster releases, CI handles the rest AFAIR
Thanks. I think I have done that, so the CI jobs have run. Do I have to download these two artifacts.zip files to apt1002 and extract them?
https://gitlab.wikimedia.org/repos/sre/wmfdb/-/artifacts
It would be nice if we could use the built-in package repository feature: https://gitlab.wikimedia.org/repos/sre/wmfdb/-/packages instead of just zip files of artifacts. Or am I missing something about how it works?
Ah @BTullis I see the issue you face, I had the same one, sorry for not spotting it sooner!
https://wikitech.wikimedia.org/wiki/Debian_packaging#Upload_to_Wikimedia_Repo
You can follow that last part to pass the finish line!
OK, thanks. I've done that now. I had to go back and add the ~wmf1 suffix to the changelog on the bullseye-wikimedia version, to avoid the filename clash in reprepro.
However, those packages are available on the apt servers now.
btullis@apt1002:~$ sudo -i reprepro ls wmfdb-admin wmfdb-admin | 0.1.2+deb10u1 | buster-wikimedia | amd64 wmfdb-admin | 0.1.5~wmf1 | bullseye-wikimedia | amd64 wmfdb-admin | 0.1.5 | bookworm-wikimedia | amd64 btullis@apt1002:~$ sudo -i reprepro ls python3-wmfdb python3-wmfdb | 0.1.2+deb10u1 | buster-wikimedia | amd64 python3-wmfdb | 0.1.5~wmf1 | bullseye-wikimedia | amd64 python3-wmfdb | 0.1.5 | bookworm-wikimedia | amd64
I can prepare a debdeploy spec to roll them out now.
Are you happy for me to deploy them this way, or would you prefer me to try manually with a canary host @ABran-WMF ?
Amazing 🎉 lets maye try the first deployment from a canary cumin host so we're 100% sure that there is no breaking change.
Thanks.
Just checking, is wmfdb-admin installed by hand? I can't find any reference to either this package or python3-wmfdb in puppet, nor in any cookbook.
Is it only on the cumin hosts that they are deployed, or do any other roles have them as well? Apologies if I have missed something.
I'm affraid thats an answer I don't have @BTullis maybe @Marostegui or @Ladsgroup knows.
From what I've checked on cumin1002:
Log started: 2024-01-24 12:57:12 (Reading database ... 56449 files and directories currently installed.) Preparing to unpack .../wmfmariadbpy-admin_0.11.2_amd64.deb ... Unpacking wmfmariadbpy-admin (0.11.2) over (0.10) ... Preparing to unpack .../python3-wmfmariadbpy-remote_0.11.2_amd64.deb ... Unpacking python3-wmfmariadbpy-remote (0.11.2) over (0.10) ... Preparing to unpack .../python3-wmfmariadbpy_0.11.2_amd64.deb ... Unpacking python3-wmfmariadbpy (0.11.2) over (0.10) ... Setting up python3-wmfmariadbpy-remote (0.11.2) ... Setting up python3-wmfmariadbpy (0.11.2) ... Setting up wmfmariadbpy-admin (0.11.2) ... Log ended: 2024-01-24 12:57:14
$ grep -zi wmfdb-admin /var/log/apt/* /var/log/apt/term.log.6.gz 44:Selecting previously unselected package wmfdb-admin. 46:Preparing to unpack .../wmfdb-admin_0.1.4~wmf1_amd64.deb ... 47:Unpacking wmfdb-admin (0.1.4~wmf1) ... 48:Setting up wmfdb-admin (0.1.4~wmf1) ... /var/log/apt/history.log.6.gz 24:Commandline: apt install wmfdb-admin 26:Install: wmfdb-admin:amd64 (0.1.4~wmf1) /var/log/apt/eipp.log.xz 3522:Package: wmfdb-admin
it looks like it's been installed manually indeed. Maybe we should add it to cumin's packages?
It is installed manually and it should go to all the hosts which are root-clients (currently: if !($::fqdn in ['cumin1002.eqiad.wmnet', 'cumin2002.codfw.wmnet']) {
Those include
include profile::mariadb::wmfmariadbpy
Not wmfdb-admin because we were (and still are) in the middle of a migration when some changes happened to the team.
OK, thanks @Marostegui
I confirmed the hosts like this:
btullis@cumin1002:~$ sudo cumin P:mariadb::wmf_root_client 2 hosts will be targeted: cumin2002.codfw.wmnet,cumin1002.eqiad.wmnet
I installed the upgraded wmfdb-admin on cumin1002 like this:
btullis@cumin1002:~$ sudo apt install wmfdb-admin <snip snip> The following additional packages will be installed: python3-wmfdb The following packages will be upgraded: python3-wmfdb wmfdb-admin 2 upgraded, 0 newly installed, 0 to remove and 24 not upgraded. <snip snip> Preparing to unpack .../wmfdb-admin_0.1.5~wmf1_amd64.deb ... Unpacking wmfdb-admin (0.1.5~wmf1) over (0.1.4~wmf1) ... Preparing to unpack .../python3-wmfdb_0.1.5~wmf1_amd64.deb ... Unpacking python3-wmfdb (0.1.5~wmf1) over (0.1.4~wmf1) ... Setting up python3-wmfdb (0.1.5~wmf1) ... Setting up wmfdb-admin (0.1.5~wmf1) ...
I tested it with:
btullis@cumin1002:~$ sudo db-mysql an-redacteddb1001.eqiad.wmnet:3311 ERROR 1045 (28000): Access denied for user 'cumin2024'@'10.64.48.98' (using password: YES)
So it doesn't work :-(
I verified that it still works for clouddb1021 with...
btullis@cumin1002:~$ sudo db-mysql clouddb1021.eqiad.wmnet:3311 Welcome to the MariaDB monitor. Commands end with ; or \g. Your MariaDB connection id is 47581742 Server version: 10.6.14-MariaDB MariaDB Server Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. root@clouddb1021.eqiad.wmnet[(none)]> Bye
I'll see if I can find out why it's not working, then have another go.
Adding some debug logs, it's clear that it didn't add the --defaults-group-suffix=labsdb argument to the mysql command line.
btullis@cumin1002:~$ sudo db-mysql --log=debug an-redacteddb1001.eqiad.wmnet:3311 2024-07-04 15:28:21,549 2792095 [INFO] wmfdb.cli_admin.db_mysql:48 - Execing: ['mysql', '-han-redacteddb1001.eqiad.wmnet', '-P3311', '--ssl', '--ssl-ca=/etc/ssl/certs/wmf-ca-certificates.crt', '--ssl-verify-server-cert'] ERROR 1045 (28000): Access denied for user 'cumin2024'@'10.64.48.98' (using password: YES) btullis@cumin1002:~$ sudo db-mysql --log=debug clouddb1021.eqiad.wmnet:3311 2024-07-04 15:29:13,207 2793631 [INFO] wmfdb.cli_admin.db_mysql:48 - Execing: ['mysql', '--defaults-group-suffix=labsdb', '-hclouddb1021.eqiad.wmnet', '-P3311', '--ssl', '--ssl-ca=/etc/ssl/certs/wmf-ca-certificates.crt', '--ssl-verify-server-cert']
So it doesn't like my conditional here.
Oh, somehow it's a bad build. The mysql_cli.py file doesn't contain my changes.
btullis@cumin1002:~$ sed -n '60,62p' /usr/lib/python3/dist-packages/wmfdb/mysql_cli.py
if host.startswith("clouddb"):
# This has to appear before any other options.
args.append("--defaults-group-suffix=labsdb")My changes don't appear in either dgit/ branch, so I must have messed up the branch manipulation. I will try it again.
OK, I have fixed the build, deleted the botched packages from apt1002 and replaced them.
The I reinstalled the packages manually on cumin1002 with:
btullis@cumin1002:~$ sudo apt install --reinstall python3-wmfdb wmfdb-admin <snip snip> Preparing to unpack .../python3-wmfdb_0.1.5~wmf1_amd64.deb ... Unpacking python3-wmfdb (0.1.5~wmf1) over (0.1.5~wmf1) ... Preparing to unpack .../wmfdb-admin_0.1.5~wmf1_amd64.deb ... Unpacking wmfdb-admin (0.1.5~wmf1) over (0.1.5~wmf1) ...
Tested that it works:
btullis@cumin1002:~$ sudo db-mysql an-redacteddb1001.eqiad.wmnet:3311 Welcome to the MariaDB monitor. Commands end with ; or \g. Your MariaDB connection id is 1173440 Server version: 10.6.18-MariaDB MariaDB Server Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. root@an-redacteddb1001.eqiad.wmnet[(none)]>
@ABran-WMF I'll wait for you to double-check that there haven't been any regressions, then install it to cumin2002 as well when you're happy for me to proceed.
we've not seen any regression since you released the update, I think you're good to go!
The plan is instead that we will have a staging apt repository to which successfully-built packages will automatically get pushed (this is being worked on by collaboration-services, I think it's not quite ready yet).
OK, thanks all.
I've deployed the updated package to cumin2002 as well now.
btullis@cumin2002:~$ sudo db-mysql an-redacteddb1001.eqiad.wmnet:3311 Welcome to the MariaDB monitor. Commands end with ; or \g. Your MariaDB connection id is 1616808 Server version: 10.6.18-MariaDB MariaDB Server Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. root@an-redacteddb1001.eqiad.wmnet[(none)]> Ctrl-C -- exit! Aborted
Hopefully this is all complete now.
I'll be moving on to decommissioning clouddb1021, so we will find out if any stray processes still rely on that host.