Page MenuHomePhabricator

cloudweb2001-dev & labweb1001 backing up non-existing or empty directories
Closed, ResolvedPublic

Description

Both cloudweb2001-dev.wikimedia.org-Monthly-1st-Mon-production-a-backup and labweb1001.wikimedia.org-Monthly-1st-Sat-production-a-backup are failing due to being setup to backup /a/backup, which are either non-existing or practically empty on those hosts. Please review backup configuration to make sure backups are running on the right directory so file are safe. Most likely this is either a puppet missconfiguration or those hosts are being setup up or being decommisioned. Please let us know how to proceed.

Event Timeline

jcrespo created this task.Nov 4 2019, 10:46 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptNov 4 2019, 10:46 AM
Andrew added a comment.Nov 4 2019, 2:50 PM

I hope this is the last hidden reference to /a!

All of that moved to /srv long ago; I'm attaching a patch but I don't know much about bacula config so let me know if there are other pieces I need to update.

Change 548273 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] labweb/cloudweb: fix path for backup jobs

https://gerrit.wikimedia.org/r/548273

Change 548273 merged by Jcrespo:
[operations/puppet@production] labweb/cloudweb: fix path for backup jobs

https://gerrit.wikimedia.org/r/548273

jcrespo added a comment.EditedNov 5 2019, 7:44 AM

This will be fixed soon- I will resolve when I have confirmation the first backup ran correctly. Thanks for the quick patch and response!

@Andrew I have an additional question: I see there is 2 configured jobs, one on labweb1001.wikimedia.org-Monthly-1st-Sat-production-cloudweb-srv-backup and another on labweb1002.wikimedia.org-Monthly-1st-Wed-production-cloudweb-srv-backup (in addition, of course, to cloudweb2001-dev) this is not a big deal if the dataset is small. Also an expected occurrence if you are in the middle of replacing one server with the other. However, if these are redundant servers, please consider enabling backups only on 1? Please clarify, although that is independent of this task.

jcrespo reassigned this task from jcrespo to Andrew.Nov 5 2019, 7:44 AM
jcrespo triaged this task as High priority.
jcrespo closed this task as Resolved.Nov 5 2019, 10:39 AM