Investigate if stopping mysql with buffer_pool dump between 10.4 versions is safe
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	Marostegui
	Mar 10 2022, 9:15 AM

Description

During the upgrades from 10.1 to 10.4 we saw a bunch of crashes. MariaDB thought they could be related to the innodb_buffer_pool_dump flag between restarts. We disabled it when upgrading.
However, now we are fully running 10.4 everywhere, we should test if this is safe to do again

Details

	Project	Branch	Lines +/-	Subject
	operations/cookbooks	master	+1 -1	mysql/upgrade.py: Allow buffer pool dumps
	operations/puppet	production	+1 -0	db1099: Disable notifications

Customize query in gerrit

Related Objects

Mentioned In: rCCKB9549be1b0bb7: mysql/upgrade.py: Allow buffer pool dumps
Mentioned Here: P22670 dbctl commit (dc=all): 'Depool db1099:3318 T303498'

Event Timeline

Marostegui created this task.Mar 10 2022, 9:15 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 10 2022, 9:15 AM

Marostegui triaged this task as High priority.Mar 10 2022, 9:15 AM

Marostegui moved this task from Triage to Ready on the DBA board.

LSobanski added a subscriber: LSobanski.Mar 10 2022, 9:19 AM

Volans added projects: Sustainability (Incident Followup), SRE-OnFire.Mar 10 2022, 10:17 AM

Marostegui moved this task from Ready to In progress on the DBA board.Mar 11 2022, 6:14 AM

I have done quite a bunch of testing and so far I have not been able to reproduce the crashes when doing 10.4->10.4 migrations. Going to try a few more things.

Mentioned in SAL (#wikimedia-operations) [2022-03-16T09:15:34Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1099:3318 T303498', diff saved to https://phabricator.wikimedia.org/P22670 and previous config saved to /var/cache/conftool/dbconfig/20220316-091533-marostegui.json

Change 771324 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] db1099: Disable notifications

https://gerrit.wikimedia.org/r/771324

Change 771324 merged by Marostegui:

[operations/puppet@production] db1099: Disable notifications

https://gerrit.wikimedia.org/r/771324

Maintenance_bot removed a project: Patch-For-Review.Mar 16 2022, 10:10 AM

I haven't been able to reproduce a crash again. Before closing this, I am going to upgrade a few more hosts as part of the s3 upgrade dumping the buffer pool, to see if we get some extra insights.

Change 775260 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/cookbooks@master] mysql/upgrade.py: Allow buffer pool dumps

https://gerrit.wikimedia.org/r/775260

gerritbot added a project: Patch-For-Review.Mar 30 2022, 9:12 AM

I haven't been able to replicate the crashes/see more errors when upgrading, so I am considering this fixed for now. We can reopen if we see something else coming.
Ready to close this ticket once the above patch is reviewed + merged.

Change 775260 merged by jenkins-bot:

[operations/cookbooks@master] mysql/upgrade.py: Allow buffer pool dumps

https://gerrit.wikimedia.org/r/775260

Maintenance_bot removed a project: Patch-For-Review.Mar 30 2022, 12:10 PM