Page MenuHomePhabricator
Create Task
Maniphest T225265

Fix labstore checks on cloudstore1008/9
Closed, ResolvedPublic0 Estimated Story Points
Actions

Description

The labstore status checks are all showing UNKNOWN: No valid datapoints found for cloudstore1008/9 due to a configuration problem. This is likely a diamond vs. prometheus thing.

Details

SubjectRepoBranchLines +/-
cloudstore: allow rpc.mountd traffic between hostsoperations/puppetproduction+5 -0
cloudstore: add sudo config for the nagios useroperations/puppetproduction+5 -0
cloudstore: move secondary monitoring stuff into profile and fix itoperations/puppetproduction+23 -49
cloudstore: re-enable diamond collectors for monitoringoperations/puppetproduction+0 -1
Customize query in gerrit

Event Timeline

• Bstorm triaged this task as Medium priority.Jun 6 2019, 11:28 PM
• Bstorm created this task.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 6 2019, 11:28 PM
• Bstorm moved this task from Backlog to Shared Storage on the Data-Services board.Jun 6 2019, 11:28 PM
• Bstorm moved this task from Inbox to Doing on the cloud-services-team (Kanban) board.
gerritbot added a comment.Jun 13 2019, 10:29 PM

Change 516967 had a related patch set uploaded (by Bstorm; owner: Bstorm):
[operations/puppet@production] cloudstore: re-enable diamond collectors for monitoring

https://gerrit.wikimedia.org/r/516967

gerritbot added a project: Patch-For-Review.Jun 13 2019, 10:29 PM
gerritbot added a comment.Jun 13 2019, 10:58 PM

Change 516967 merged by Bstorm:
[operations/puppet@production] cloudstore: re-enable diamond collectors for monitoring

https://gerrit.wikimedia.org/r/516967

Maintenance_bot removed a project: Patch-For-Review.Jun 13 2019, 11:10 PM
• Bstorm added a comment.Jun 13 2019, 11:10 PM

Ok, so that fixed all the checks except the NFS service one. That might be a stretch incompatibility, depending.
Since that was the fix, I now am wondering what, if any, replacement there is for being able to page on a trend. I haven't seen anything like that running against prometheus here so far? I'll have to dig deeper.

• Bstorm added a comment.Jun 13 2019, 11:24 PM

Nope, the NFS one is purely an issue of firewall rules. I'll have to figure out exactly what to open up in order to make showmount -e work.

• Bstorm added a comment.Jun 17 2019, 4:55 PM

So this needs to connect to 111 over UDP from the public network, which is currently is not allowed to do.

gerritbot added a comment.Jun 17 2019, 5:55 PM

Change 517470 had a related patch set uploaded (by Bstorm; owner: Bstorm):
[operations/puppet@production] cloudstore: move secondary monitoring stuff into profile and fix it

https://gerrit.wikimedia.org/r/517470

gerritbot added a project: Patch-For-Review.Jun 17 2019, 5:55 PM
gerritbot added a comment.Jun 19 2019, 3:02 PM

Change 517470 merged by Bstorm:
[operations/puppet@production] cloudstore: move secondary monitoring stuff into profile and fix it

https://gerrit.wikimedia.org/r/517470

Maintenance_bot removed a project: Patch-For-Review.Jun 19 2019, 3:10 PM
gerritbot added a comment.Jun 20 2019, 4:44 PM

Change 518079 had a related patch set uploaded (by Bstorm; owner: Bstorm):
[operations/puppet@production] cloudstore: add sudo config for the nagios user

https://gerrit.wikimedia.org/r/518079

gerritbot added a project: Patch-For-Review.Jun 20 2019, 4:45 PM
gerritbot added a comment.Jun 20 2019, 5:47 PM

Change 518079 merged by Bstorm:
[operations/puppet@production] cloudstore: add sudo config for the nagios user

https://gerrit.wikimedia.org/r/518079

• Bstorm closed this task as Resolved.Jun 20 2019, 5:54 PM

And when that runs on puppet, I see we are green.

gerritbot added a comment.Jul 19 2019, 2:41 PM

Change 524528 had a related patch set uploaded (by Jhedden; owner: Jhedden):
[operations/puppet@production] cloudstore: allow rpc.mountd traffic between hosts

https://gerrit.wikimedia.org/r/524528

• JHedden subscribed.Jul 19 2019, 2:42 PM

That patch ^ fixes the NRPE error CHECK_NRPE STATE UNKNOWN: Socket timeout after 10 seconds when the host not supporting the VIP runs showmount.

gerritbot added a comment.Jul 19 2019, 2:57 PM

Change 524528 merged by Jhedden:
[operations/puppet@production] cloudstore: allow rpc.mountd traffic between hosts

https://gerrit.wikimedia.org/r/524528

Maintenance_bot removed a project: Patch-For-Review.Jul 19 2019, 3:10 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL