Page MenuHomePhabricator

db1171:s8 is having performance issues and lagging
Closed, ResolvedPublic

Description

Report run at 2024-09-12T01:00:02.500265 at backupmon1001.

1 backups were found in the last week that took over 12.0 hours:

* dump of s8 in eqiad started at 2024-09-10T01:22:43 and finished at 2024-09-11T10:33:47 (1 day, 9:11:04).

image.png (704×1 px, 91 KB)

Related Objects

Event Timeline

jcrespo triaged this task as High priority.

Change #1072515 had a related patch set uploaded (by Jcrespo; author: Jcrespo):

[operations/puppet@production] mariadb: Increase buffer pool for db1171:s8, which is lagging

https://gerrit.wikimedia.org/r/1072515

I think, not sure, is that because it just finished an alter table on revision table. T367856: Cleanup revision table schema

Yeah, that would explain it. So root cause found. I still want to merge the patch to optimize memory assignment (future schema changes will happen there).

Change #1072515 merged by Jcrespo:

[operations/puppet@production] mariadb: Increase buffer pool for db1171:s8, which is lagging

https://gerrit.wikimedia.org/r/1072515

[11:44] <jynus> I will restart now db1171:s7, and s8 on the same host in ~1h, when the dumps there finish
[11:44] <jynus> to apply the buffer pool change

Mentioned in SAL (#wikimedia-operations) [2024-09-12T11:46:59Z] <jynus> restarting db1171:s7 mysql process T374610

Mentioned in SAL (#wikimedia-operations) [2024-09-12T17:14:23Z] <jynus> restarting db1171:s8 mysql process T374610

After restart, the server looks way less io stressed.