Page MenuHomePhabricator
Create Task
Maniphest T182327

Alert on age of backups on analytics1002
Closed, ResolvedPublic3 Estimated Story Points
Actions

Description

We keep various backups (hadoop namenode, analytics1003 mysql instance) on analytics1002. We should alert if backups get old, perhaps if there are no files newer than a few days old in each backup.

Details

SubjectRepoBranchLines +/-
Allow nagios to sudo to check analytics backup newest file ageoperations/puppetproduction+16 -2
Add nrpe check_newest_file_age; monitor some analytics file backupsoperations/puppetproduction+385 -0
Customize query in gerrit

Event Timeline

Ottomata created this task.Dec 7 2017, 2:36 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptDec 7 2017, 2:36 PM
elukey triaged this task as High priority.Dec 7 2017, 3:49 PM
elukey edited projects, added Analytics-Kanban; removed Analytics-Clusters, Analytics.
Ottomata claimed this task.Dec 19 2017, 8:48 PM
Ottomata moved this task from Next Up to In Progress on the Analytics-Kanban board.
gerritbot subscribed.Dec 19 2017, 8:49 PM

Change 399255 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Add nrpe check_newest_file_age; monitor some analytics file backups

https://gerrit.wikimedia.org/r/399255

gerritbot added a project: Patch-For-Review.Dec 19 2017, 8:49 PM
gerritbot added a comment.Dec 19 2017, 8:53 PM

Change 399255 merged by Ottomata:
[operations/puppet@production] Add nrpe check_newest_file_age; monitor some analytics file backups

https://gerrit.wikimedia.org/r/399255

gerritbot added a comment.Dec 19 2017, 9:08 PM

Change 399260 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Allow nagios to sudo to check analytics database backup newest file age

https://gerrit.wikimedia.org/r/399260

gerritbot added a comment.Dec 19 2017, 9:26 PM

Change 399260 merged by Ottomata:
[operations/puppet@production] Allow nagios to sudo to check analytics backup newest file age

https://gerrit.wikimedia.org/r/399260

Ottomata moved this task from In Progress to Done on the Analytics-Kanban board.Dec 19 2017, 9:32 PM
• Nuria closed this task as Resolved.Jan 3 2018, 5:00 PM
• Nuria set the point value for this task to 3.
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL