Page MenuHomePhabricator

Migrate all db-* scripts to Bookworm
Closed, ResolvedPublic

Description

db-mysql was fixed and it can now be run from cumin1002, but we still have to fix all the other scripts before we can move entirely to cumin1002 (bookworm)

  • db-compare
  • db-move-replica
  • db-replication-tree
  • db-switchover
  • db-kill
  • db-stop-in-sync

Event Timeline

Marostegui moved this task from Triage to Ready on the DBA board.
ABran-WMF changed the task status from Open to In Progress.Jan 22 2024, 12:36 PM

then:

python3-wmfmariadbpy |       0.10 | http://apt.wikimedia.org/wikimedia bullseye-wikimedia/main amd64 Packages
wmfmariadbpy |       0.10 | http://apt.wikimedia.org/wikimedia bullseye-wikimedia/main Sources

now:

python3-wmfmariadbpy |  0.11~wmf1 | http://apt.wikimedia.org/wikimedia bullseye-wikimedia/main amd64 Packages
wmfmariadbpy |       0.10 | http://apt.wikimedia.org/wikimedia bullseye-wikimedia/main Sources

I'm not sure wmfmariadbpy has to be fixed as well

From cumin1001 I've seen that neither db-compare, db-move-replica nor db-compare work with dbstore1009:3316
This is an example that should work and it doesn't work:

root@cumin1001:/home/marostegui# db-compare frwiki user user_id dbstore1009.eqiad.wmnet:3316 db1173.eqiad.wmnet
Could not connect to dbstore1009.eqiad.wmnet

root@cumin1001:/home/marostegui# db-compare frwiki user user_id db1231.eqiad.wmnet db1173.eqiad.wmnet
Starting comparison between id 1 and 4879741
^CCtrl-c pressed ...

And for what is worth:

root@cumin1001:/home/marostegui# telnet dbstore1009 3316
Trying 10.64.135.22...
Connected to dbstore1009.eqiad.wmnet.
Escape character is '^]'.
Y
5.5.5-10.6.16-MariaDB:�1M<'J#lw2�2EB]-e9-rw#Bmysql_native_passwordConnection closed by foreign host.
root@cumin1001:/home/marostegui# db-mysql dbstore1009:3316 -e "select @@hostname"
+-------------+
| @@hostname  |
+-------------+
| dbstore1009 |
+-------------+
root@cumin1001:/home/marostegui#

it comes from the fact that we use pymysql version v0.9.3, SSL identity verification disabling was not implemented yet:

TypeError: __init__() got an unexpected keyword argument 'ssl_verify_cert'
TypeError: __init__() got an unexpected keyword argument 'ssl_verify_identity'

We could upgrade pymysql to fix the issue, the last version is v1.1.0

I'd say, whatever is easier/faster: either fix that or fix all the scripts to be run from cumin1002 so we can forget cumin1001 :)

I think we should upgrade pymysql regardless honestly.

After a quick talk with @Ladsgroup it seems that the best angle on this is to bring pymysql debian 12 package on debian 11

To have the reasoning written somewhere so it wouldn't be confusing to future selves:

  • Using deb package for pymysql is better to make sure it shows up in debmonitor and properly tracked for security updates
  • backporting the debian 12 pymysql to debian 11 would make the upgrade to bookworm later easier.

After a quick talk with @Ladsgroup it seems that the best angle on this is to bring pymysql debian 12 package on debian 11

I can create a backport if you want?

After a quick talk with @Ladsgroup it seems that the best angle on this is to bring pymysql debian 12 package on debian 11

I can create a backport if you want?

oh if you're have the time to do so, please and thank you!

After a quick talk with @Ladsgroup it seems that the best angle on this is to bring pymysql debian 12 package on debian 11

I can create a backport if you want?

oh if you're have the time to do so, please and thank you!

Sure thing, I'll take care of it tomorrow

Mentioned in SAL (#wikimedia-operations) [2024-01-24T10:36:40Z] <moritzm> upgrading cumin1002 to pymsql 1.0.2-2~wmf11u1 T355531

Following up on:

Sure thing, I'll take care of it tomorrow

Mentioned in SAL (#wikimedia-operations) [2024-01-24T10:36:40Z] <moritzm> upgrading cumin1002 to pymsql 1.0.2-2~wmf11u1 T355531

and this merge request: scripts are connecting properly to all instances

ABran-WMF updated the task description. (Show Details)

I think this is a little premature to close? cumin2002 still has the old wmfdb and we still need to roll out the backported pymysql on a wider scale as well

Mentioned in SAL (#wikimedia-operations) [2024-01-24T15:11:02Z] <moritzm> uploading pymsql 1.0.2-2~wmf11u1 to apt.wikimedia.org T355531

Mentioned in SAL (#wikimedia-operations) [2024-01-25T10:17:04Z] <moritzm> upgrading python-pymysql in S6 DB hosts to 1.0.2-2~wmf11u1 T355531

By the way, db-switchover worked fine from cumin1002 in today's master switchover.

Mentioned in SAL (#wikimedia-operations) [2024-01-29T09:13:23Z] <moritzm> upgrading python-pymysql in S7 DB hosts to 1.0.2-2~wmf11u1 T355531

Mentioned in SAL (#wikimedia-operations) [2024-01-30T08:29:36Z] <moritzm> upgrading python-pymysql on remaining DB hosts to 1.0.2-2~wmf11u1 T355531