Page MenuHomePhabricator

Logstash on beta yields 500 due to NFS outage (can't open /data/project/logstash/.htpasswd)
Closed, ResolvedPublic

Description

@Krenair wrote:

Logstash-beta is broken, HTTP 500 ISE due to:

[Thu Jun 18 14:53:57 2015] [error] [client 10.68.16.65]
(2)No such file or directory: Could not open password file: /data/project/logstash/.htpasswd,
referer: https://logstash-beta.wmflabs.org/

The password should probably be on the beta cluster puppetmaster as a local hack in /var/lib/git/labs/private then populated via puppet.

Details

Event Timeline

hashar raised the priority of this task from to Needs Triage.
hashar updated the task description. (Show Details)
hashar added subscribers: zhuyifei1999, Krenair, hashar and 2 others.
hashar renamed this task from Logstash on beta yields 500 due to NFS outage to Logstash on beta yields 500 due to NFS outage (can't open /data/project/logstash/.htpasswd).Jun 18 2015, 3:18 PM
hashar set Security to None.

I have temporarily hacked around this by creating an /etc/logstash/htpasswd file, changing the path in /etc/apache2/sites-enabled/50-logstash-beta-wmflabs-org.conf and disabling puppet on deployment-logstash1.

I'll get a patch in to make the path change permanent and then we can probably close this.

Change 219184 had a related patch set uploaded (by BryanDavis):
beta: Change kibana password location

https://gerrit.wikimedia.org/r/219184

Patch submitted to gerrit, cherry-picked to deployment-salt and puppet re-enabled on deployment-logstash1. Things are in the hands of folks with +2 on operations/puppet now.

Change 219184 merged by Yuvipanda:
beta: Change kibana password location

https://gerrit.wikimedia.org/r/219184