pymysql.err.OperationalError: (2003, "Can't connect to MySQL server on 'db1164.eqiad.wmnet' ([SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1123))") on backup
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	jcrespo
	Nov 17 2023, 8:04 AM

Description

wmfbackups broke its statistics gathering with the following error:

pymysql.err.OperationalError: (2003, "Can't connect to MySQL server on 'db1164.eqiad.wmnet' ([SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1123))")

This is probably because of the puppet 7 migration, with handles tls slightly differently.

We have to update the package to move mysql connection configuration to puppet so we can tune the tls CA or verification options to make it work.

The backup itself succeeds (at least for snapshots, dumps are to be tested), it is the gathering of backup metadata into m1 what fails.

Details

	Subject	Repo	Branch	Lines +/-
	dbbackups: Update config for mysql backup monitoring	operations/puppet	production	+29 -19
	dbbackups: Update mysql CA for content and metadata backups	operations/puppet	production	+27 -9

Customize query in gerrit

	Title	Reference	Author	Source Branch	Dest Branch
	BackupStatistics: Change the statistics yaml file to a mysql config	repos/sre/wmfbackups!4	jynus	T351491	main

Customize query in GitLab

Related Objects

Mentioned In: T351617: Several backup alerts fired
T347740: wmfbackups packages for Debian Bookworm

Event Timeline

jcrespo created this task.Nov 17 2023, 8:04 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptNov 17 2023, 8:04 AM

jcrespo triaged this task as Unbreak Now! priority.Nov 17 2023, 8:05 AM

jynus opened https://gitlab.wikimedia.org/repos/sre/wmfbackups/-/merge_requests/4

BackupStatistics: Change the statistics yaml file to a mysql config

Change 975231 had a related patch set uploaded (by Jcrespo; author: Jcrespo):

[operations/puppet@production] dbbackups: Update mysql CA for content and metadata backups

https://gerrit.wikimedia.org/r/975231

jcrespo mentioned this in T347740: wmfbackups packages for Debian Bookworm.Nov 17 2023, 10:24 AM

jynus merged https://gitlab.wikimedia.org/repos/sre/wmfbackups/-/merge_requests/4

BackupStatistics: Change the statistics yaml file to a mysql config

Mentioned in SAL (#wikimedia-operations) [2023-11-17T13:48:08Z] <jynus> reenable puppet on dbprov2001 T351491

Change 975231 merged by Jcrespo:

[operations/puppet@production] dbbackups: Update mysql CA for content and metadata backups

https://gerrit.wikimedia.org/r/975231

Maintenance_bot removed a project: Patch-For-Review.Nov 17 2023, 2:10 PM

This should be now fixed with the new packages + puppet code (but we will see over the weekend).

Change 976158 had a related patch set uploaded (by Jcrespo; author: Jcrespo):

[operations/puppet@production] dbbackups: Update config for mysql backup monitoring

https://gerrit.wikimedia.org/r/976158

gerritbot added a project: Patch-For-Review.Nov 21 2023, 10:23 AM

Change 976158 merged by Jcrespo: