Page MenuHomePhabricator
Create Task
Maniphest T281094

xtrabackup --prepare hits open_files_limit on buster
Closed, ResolvedPublic
Actions

Description

S3 and x1 backups get stuck on prepare phase due to the amount of files that are attempted to be open at the same time:

2021-04-26  8:50:16 0 [ERROR] InnoDB: Operating system error number 24 in a file operation.
2021-04-26  8:50:16 0 [ERROR] InnoDB: Error number 24 means 'Too many open files'

Adding "--open-files-limit=100000" fixes the issue.

This didn't happen on Stretch, so it must be a new behavior of innodb for 10.4 (or a bug). While s3 contains a lot of files, I would expect xtrabackup to run almost serially (unlike on normal database execution). Maybe a new optimization leads to file descriptor exahustion.

Details

SubjectRepoBranchLines +/-
Release new v0.5 versionoperations/software/wmfbackupsmaster+7 -1
Xtrabackup: Increase default open-files-limit to match productionoperations/software/wmfbackupsmaster+2 -0
Increase default memory usage of xtrabackup --prepare to 40GBoperations/software/wmfbackupsmaster+1 -1
Customize query in gerrit

Related Objects
Search...

Event Timeline

jcrespo created this task.Apr 26 2021, 8:56 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 26 2021, 8:56 AM
gerritbot added a comment.Apr 26 2021, 9:08 AM

Change 682536 had a related patch set uploaded (by Jcrespo; author: Jcrespo):

[operations/software/wmfbackups@master] Increase default memory usage of xtrabackup --prepare to 40GB

https://gerrit.wikimedia.org/r/682536

gerritbot added a project: Patch-For-Review.Apr 26 2021, 9:08 AM

Change 682537 had a related patch set uploaded (by Jcrespo; author: Jcrespo):

[operations/software/wmfbackups@master] Xtrabackup: Increase default open-files-limit to match production

https://gerrit.wikimedia.org/r/682537

jcrespo triaged this task as High priority.Apr 26 2021, 9:19 AM
jcrespo added a parent task: T280979: Upgrade pending stretch backup hosts to buster.
jcrespo moved this task from Triage to In Progress on the Data-Persistence-Backup board.Apr 26 2021, 10:20 AM
jcrespo added a subscriber: LSobanski.Apr 26 2021, 11:30 AM
gerritbot added a comment.Apr 27 2021, 8:03 AM

Change 682536 merged by Jcrespo:

[operations/software/wmfbackups@master] Increase default memory usage of xtrabackup --prepare to 40GB

https://gerrit.wikimedia.org/r/682536

gerritbot added a comment.Apr 27 2021, 8:07 AM

Change 682537 merged by Jcrespo:

[operations/software/wmfbackups@master] Xtrabackup: Increase default open-files-limit to match production

https://gerrit.wikimedia.org/r/682537

Maintenance_bot removed a project: Patch-For-Review.Apr 27 2021, 8:10 AM
jcrespo added a comment.Apr 27 2021, 8:46 AM

New package version has been locally installed on dbprov2003. Run looks fine so far:

[08:43:32]: DEBUG - ['xtrabackup', '--prepare', '--target-dir', '/srv/backups/snapshots/ongoing/snapshot.s3.2021-04-24--16-00-21', '--use-memory', '40G', '--open-files-limit', '200000']

But waiting to see it completes succesfully.

gerritbot added a comment.Apr 27 2021, 10:00 AM

Change 682916 had a related patch set uploaded (by Jcrespo; author: Jcrespo):

[operations/software/wmfbackups@master] Release new v0.5 version

https://gerrit.wikimedia.org/r/682916

gerritbot added a project: Patch-For-Review.Apr 27 2021, 10:00 AM
gerritbot added a comment.Apr 27 2021, 5:29 PM

Change 682916 merged by Jcrespo:

[operations/software/wmfbackups@master] Release new v0.5 version

https://gerrit.wikimedia.org/r/682916

jcrespo closed this task as Resolved.Apr 27 2021, 5:45 PM

This is now fixed, I have uploaded v0.5 packages fixing the issue, but I will only have updated for now buster dbprov hosts, as the others have 0 changes on the relevant package.

jcrespo added a comment.Apr 28 2021, 6:06 AM

2 successful backups since deploy, everything looking good.

jcrespo added a comment.Apr 28 2021, 6:11 AM

Not generating a 0.5 package for stretch as they are scheduled to disappear soon and not affected by the bug.

Screenshot from 2021-04-28 08-09-48.png (824×2 px, 120 KB)

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL