Page MenuHomePhabricator

Logstash on beta yields 500 due to NFS outage (can't open /data/project/logstash/.htpasswd)
Closed, ResolvedPublic

Description

@Krenair wrote:

Logstash-beta is broken, HTTP 500 ISE due to:

[Thu Jun 18 14:53:57 2015] [error] [client 10.68.16.65]
(2)No such file or directory: Could not open password file: /data/project/logstash/.htpasswd,
referer: https://logstash-beta.wmflabs.org/

The password should probably be on the beta cluster puppetmaster as a local hack in /var/lib/git/labs/private then populated via puppet.

Event Timeline

hashar raised the priority of this task from to Needs Triage.
hashar updated the task description. (Show Details)
hashar renamed this task from Logstash on beta yields 500 due to NFS outage to Logstash on beta yields 500 due to NFS outage (can't open /data/project/logstash/.htpasswd).Jun 18 2015, 3:18 PM
hashar set Security to None.

I have temporarily hacked around this by creating an /etc/logstash/htpasswd file, changing the path in /etc/apache2/sites-enabled/50-logstash-beta-wmflabs-org.conf and disabling puppet on deployment-logstash1.

I'll get a patch in to make the path change permanent and then we can probably close this.

Change 219184 had a related patch set uploaded (by BryanDavis):
beta: Change kibana password location

https://gerrit.wikimedia.org/r/219184

Patch submitted to gerrit, cherry-picked to deployment-salt and puppet re-enabled on deployment-logstash1. Things are in the hands of folks with +2 on operations/puppet now.

Change 219184 merged by Yuvipanda:
beta: Change kibana password location

https://gerrit.wikimedia.org/r/219184