Page MenuHomePhabricator
Create Task
Maniphest T237233

vega, bromine (bugzilla-static) is trying to backup a directory that doesn't exist
Closed, ResolvedPublic
Actions

Description

30-Oct 04:45 bromine.eqiad.wmnet-fd JobId 158849:      Could not stat "/srv/org/wikimedia/bugzilla-backup": ERR=No such file or directory
04-Nov 05:50 vega.codfw.wmnet-fd JobId 159561:      Could not stat "/srv/org/wikimedia/bugzilla-backup": ERR=No such file or directory

Please check the status of such directory, and either fix puppet (e.g. by changing the working backup dataset) or clarify its status (e.g. host is being decommissioned, it is being set up, etc.), but make sure backup exists for the service. Guessing Daniel may know about this or put us in contact with the owner, otherwise please reassign back to me.

Details

SubjectRepoBranchLines +/-
backups: remove bugzilla backup directoryoperations/puppetproduction+0 -4
Customize query in gerrit

Related Objects
Search...

Event Timeline

jcrespo created this task.Nov 4 2019, 10:05 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptNov 4 2019, 10:05 AM
jcrespo renamed this task from Vega (bugzilla-static) is trying to backup a directory that doesn't exist to vega, bromine (bugzilla-static) is trying to backup a directory that doesn't exist.Nov 4 2019, 10:15 AM
jcrespo updated the task description. (Show Details)
jcrespo mentioned this in T236406: Switchover backup director service from helium to backup1001.Nov 4 2019, 10:55 AM
Dzahn triaged this task as High priority.Nov 4 2019, 5:34 PM
gerritbot added a comment.Nov 4 2019, 6:41 PM

Change 548440 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] backups: remove bugzilla backup directory

https://gerrit.wikimedia.org/r/548440

gerritbot added a project: Patch-For-Review.Nov 4 2019, 6:41 PM
Dzahn added a project: Wikimedia-Bugzilla.Nov 4 2019, 6:45 PM

Yea, this directory does not exist anymore on the servers. (I am not sure who deleted it when though)..but:

There is a sanitized DB dump at:

https://dumps.wikimedia.org/other/bugzilla/

And there is:

https://static-bugzilla.wikimedia.org/

gerritbot added a comment.Nov 4 2019, 6:49 PM

Change 548440 merged by Dzahn:
[operations/puppet@production] backups: remove bugzilla backup directory

https://gerrit.wikimedia.org/r/548440

Dzahn closed this task as Resolved.Nov 4 2019, 6:53 PM

Removed.

On backup1001:

Notice: /Stage[main]/Bacula::Director/File[/etc/bacula/conf.d/fileset-bugzilla-backup.conf]/ensure: removed
..
Notice: /Stage[main]/Bacula::Director/File[/etc/bacula/jobs.d/bromine.eqiad.wmnet-bugzilla-backup-Monthly-1st-Sun-production.conf]/ensure: removed

This should be fine now.

Maintenance_bot removed a project: Patch-For-Review.Nov 4 2019, 7:10 PM
jcrespo awarded a token.Nov 4 2019, 9:20 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits