Page MenuHomePhabricator
Create Task
Maniphest T219953

labservices1001/1002 sometimes unresponsive
Closed, DeclinedPublic
Actions

Description

Twice today icinga has complained about not being able to reach labservices1001. The first time this happened there were similar alerts for labservices1002.

The host is very slow to respond to ssh and/or sudo. Also, appending to the syslog seems to have stopped working.

Details

SubjectRepoBranchLines +/-
pdns-recursor: reduce maximum number of file descriptorsoperations/puppetproduction+1 -1
Customize query in gerrit

Related Objects

Event Timeline

Andrew created this task.Apr 3 2019, 1:44 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 3 2019, 1:44 AM
Andrew added a comment.Apr 3 2019, 1:44 AM

This seems related, although we have already taken the measures recommended:

https://blog.powerdns.com/2014/02/06/related-to-recent-dos-attacks-recursor-configuration-file-guidance/

gerritbot added a comment.Apr 3 2019, 1:59 AM

Change 500880 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] pdns-recursor: reduce maximum number of file descriptors

https://gerrit.wikimedia.org/r/500880

gerritbot added a project: Patch-For-Review.Apr 3 2019, 1:59 AM
gerritbot added a comment.Apr 3 2019, 2:02 AM

Change 500880 merged by Andrew Bogott:
[operations/puppet@production] pdns-recursor: reduce maximum number of file descriptors

https://gerrit.wikimedia.org/r/500880

Andrew closed this task as Declined.Apr 26 2019, 5:44 PM

I think this is moot since we're shutting down these systems. T221857

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL