db1047 out of disk space, eventlogging_sync spam
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	fgiunchedi
	Dec 5 2016, 2:06 AM

Description

db1047 paged for disk space, on /var/log examination there's a lot of spam from eventlogging_sync.{log,err}. While the .log file was symlinked to /srv the .err file wasn't. I symlinked it to /srv as well.

db1047:/var/log$ ls -la eventlogging_sync.*
lrwxrwxrwx 1 root root      43 Dec  5 01:56 eventlogging_sync.err -> /srv/log/eventlogging/eventlogging_sync.err
lrwxrwxrwx 1 root root      43 Aug 26 20:53 eventlogging_sync.log -> /srv/log/eventlogging/eventlogging_sync.log

The error itself is about tls errors

ERROR 2026 (HY000): SSL connection error: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed
ERROR 2026 (HY000): SSL connection error: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed
ERROR 2026 (HY000): SSL connection error: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed
ERROR 2026 (HY000): SSL connection error: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed

dbstore1002 suffers the same problem but has a larger /

Details

	Subject	Repo	Branch	Lines +/-
	Renew expired TLS certificate for eventlogging hosts	operations/puppet	production	+1 -1
	eventlogging_sync: By pass ssl check on localhost	operations/puppet	production	+3 -3

Customize query in gerrit

Related Objects
Search...

Status	Assigned	Task
		Restricted Task
Open	None	T120532 Use user-specific passwords for accessing Analytics MariaDB replica databases
Resolved	aaron	T88445 MediaWiki active/active datacenter investigation and work (tracking)
Resolved	Krinkle	T270223 FY2021-2022: Enable basic Multi-DC operations for read traffic (tracking)
Resolved	tstarling	T134809 App servers <=> mariadb SSL/TLS for cross-datacenter writes
Resolved	LSobanski	T111653 Encrypt all the things
Resolved	jcrespo	T111654 Set up TLS for MariaDB replication
Resolved	jcrespo	T152188 Restart pending mysql hosts with old TLS cert
Resolved	• Marostegui	T152364 db1047 out of disk space, eventlogging_sync spam

Event Timeline

fgiunchedi created this task.Dec 5 2016, 2:06 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptDec 5 2016, 2:06 AM

Mentioned in SAL (#wikimedia-operations) [2016-12-05T02:12:38Z] <godog> add --skip-ssl to mysql commands on eventlogging_sync on db1047 - T152364

Mentioned in SAL (#wikimedia-operations) [2016-12-05T02:14:20Z] <godog> add --skip-ssl to mysql commands on eventlogging_sync on dbstore1002 - T152364

Change 325257 had a related patch set uploaded (by Marostegui):
eventlogging_sync: By pass ssl check on localhost

https://gerrit.wikimedia.org/r/325257

gerritbot added a project: Patch-For-Review.Dec 5 2016, 7:16 AM

Thanks for taking care of this. I have submitted a patch to skip this check: https://gerrit.wikimedia.org/r/325257
I am not completely aware of the whole history of this script and process, so I will wait for @jcrespo and his input on this.

Change 325257 merged by Jcrespo:
eventlogging_sync: By pass ssl check on localhost

https://gerrit.wikimedia.org/r/325257

This was caused by the cert expiration on all analytics hosts, making all mysql connections from other databases to fail. This was part of the mitigation of the ongoing issue on T152188. End users were not affected because there is a backup refilling process that took over, although this created lots of issues. We need to restart all eventlog hosts to revert the patch.

jcrespo added a parent task: T152188: Restart pending mysql hosts with old TLS cert.Dec 5 2016, 10:29 AM

Change 325273 had a related patch set uploaded (by Jcrespo):
Renew expired TLS certificate for eventlogging hosts

https://gerrit.wikimedia.org/r/325273