Page MenuHomePhabricator
Create Task
Maniphest T103804

deployment-logstash02 fails puppet: Apache2 can't start, mod_authz_groupfile not enabled on Jessie
Closed, ResolvedPublic
Actions

Description

deployment-logstash02.deployment-prep.eqiad.wmflabs fails puppet:

Exec[extend-vd-/var/lib/elasticsearch]/returns: executed successfully
Error: Could not start Service[apache2]: Execution of '/usr/sbin/service apache2 start' returned 1:
    Job for apache2.service failed. See 'systemctl status apache2.service' and 'journalctl -xn' for details.
    Wrapped exception:
        Execution of '/usr/sbin/service apache2 start' returned 1:
            Job for apache2.service failed. See 'systemctl status apache2.service' and 'journalctl -xn' for details.

Error: /Stage[main]/Apache/Service[apache2]/ensure: change from stopped to running failed:
    Could not start Service[apache2]: Execution of '/usr/sbin/service apache2 start' returned 1:
        Job for apache2.service failed. See 'systemctl status apache2.service' and 'journalctl -xn' for details.
# systemctl status apache2.service
● apache2.service - LSB: Apache2 web server
   Loaded: loaded (/etc/init.d/apache2)
   Active: failed (Result: exit-code) since Thu 2015-06-25 09:13:49 UTC; 2min 1s ago
  Process: 5500 ExecStart=/etc/init.d/apache2 start (code=exited, status=1/FAILURE)

Jun 25 09:13:49 deployment-logstash2 apache2[5500]: Starting web server: apache2 failed!
Jun 25 09:13:49 deployment-logstash2 apache2[5500]: The apache2 configtest failed. ... (warning).
Jun 25 09:13:49 deployment-logstash2 apache2[5500]: Output of config test was:
Jun 25 09:13:49 deployment-logstash2 apache2[5500]: AH00526: Syntax error on line 40 of /etc/apache2/sites-enabled/50-logstash-beta-wmflabs-org.conf:
Jun 25 09:13:49 deployment-logstash2 apache2[5500]: Invalid command 'AuthGroupFile', perhaps misspelled or defined by a module not included in the server configuration
Jun 25 09:13:49 deployment-logstash2 apache2[5500]: Action 'configtest' failed.
Jun 25 09:13:49 deployment-logstash2 apache2[5500]: The Apache error log may have more information.
Jun 25 09:13:49 deployment-logstash2 systemd[1]: apache2.service: control process exited, code=exited status=1
Jun 25 09:13:49 deployment-logstash2 systemd[1]: Failed to start LSB: Apache2 web server.
Jun 25 09:13:49 deployment-logstash2 systemd[1]: Unit apache2.service entered failed state.

So Invalid command 'AuthGroupFile' in /etc/apache2/sites-enabled/50-logstash-beta-wmflabs-org.conf

36:  <Location />
37:    AuthName "Logstash (ssh deployment-bastion.eqiad.wmflabs sudo cat /root/secrets.txt)"
38:    AuthType Basic
39:    AuthUserFile /etc/logstash/htpasswd
40:    AuthGroupFile /dev/null  # <-----------  FAILS!
41:    Require valid-user
42:  </Location>

Details

SubjectRepoBranchLines +/-
logstash: Enable user & group authz modules for Kibanaoperations/puppetproduction+7 -2
Customize query in gerrit

Related Objects

Event Timeline

hashar created this task.Jun 25 2015, 9:18 AM
hashar raised the priority of this task from to Needs Triage.
hashar updated the task description. (Show Details)
hashar added projects: Beta-Cluster-Infrastructure, Wikimedia-Logstash.
hashar subscribed.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 25 2015, 9:18 AM
hashar added a comment.Jun 25 2015, 9:23 AM

The instance runs Debian Jessie. It has apache 2.4

The AuthGroupFile seems to be provided by Apache module mod_authz_groupfile. And we have:

ls -1d /etc/apache2/mods-enabled/auth*
/etc/apache2/mods-enabled/auth_basic.load
/etc/apache2/mods-enabled/authn_core.load
/etc/apache2/mods-enabled/authn_file.load
/etc/apache2/mods-enabled/authnz_ldap.load
/etc/apache2/mods-enabled/authz_core.load
/etc/apache2/mods-enabled/authz_host.load
/etc/apache2/mods-enabled/authz_user.load

The module is available but not enabled:

 ls -1d /etc/apache2/*/authz_groupfile.load
/etc/apache2/mods-available/authz_groupfile.load

I guess it is a difference between Ubuntu and Debian and Apache 2 version.

hashar renamed this task from deployment-logstash02 fails puppet: Apache2 can't start to deployment-logstash02 fails puppet: Apache2 can't start, mod_authz_groupfile not enabled on Jessie.Jun 25 2015, 9:23 AM
hashar set Security to None.
hashar added a comment.Jun 25 2015, 9:27 AM

For reference deployment-logtash1 is a Precise instance with Apache 2.2.

$ grep -R AuthGroupFile /etc/apache2
./sites-enabled/50-logstash-beta-wmflabs-org.conf:    AuthGroupFile /dev/null
./sites-available/50-logstash-beta-wmflabs-org.conf:    AuthGroupFile /dev/null
./sites-available/logstash.beta.wmflabs.org:    AuthGroupFile /dev/null

And the module is enabled there:

$ ls -1d /etc/apache2/mods-*/*authz_groupfile*
/etc/apache2/mods-available/authz_groupfile.load
/etc/apache2/mods-enabled/authz_groupfile.load
hashar added a subscriber: bd808.Jul 6 2015, 7:31 PM

Hello @bd808 , seems our puppet manifests for logstash are not complete for Jessie. We fail to enable the Apache module authz_groupfile , though maybe it has been done manually on the Trusty instance.

Restricted Application added a subscriber: Luke081515. · View Herald TranscriptJul 6 2015, 7:31 PM
hashar triaged this task as Medium priority.Jul 6 2015, 7:32 PM
hashar triaged this task as Medium priority.
hashar moved this task from To Triage to Backlog on the Beta-Cluster-Infrastructure board.
hashar moved this task from To Triage to Backlog on the Beta-Cluster-Infrastructure board.
gerritbot subscribed.Jul 6 2015, 8:51 PM

Change 223172 had a related patch set uploaded (by BryanDavis):
logstash: Enable user & group authz modules for Kibana

https://gerrit.wikimedia.org/r/223172

gerritbot added a project: Patch-For-Review.Jul 6 2015, 8:51 PM
gerritbot added a comment.Jul 7 2015, 7:39 PM

Change 223172 merged by Yuvipanda:
logstash: Enable user & group authz modules for Kibana

https://gerrit.wikimedia.org/r/223172

yuvipanda mentioned this in rOPUPd67ceaf801cd: logstash: Enable user & group authz modules for Kibana.Jul 7 2015, 7:40 PM
bd808 closed this task as Resolved.Jul 7 2015, 7:41 PM
bd808 claimed this task.
bd808 moved this task from Backlog to Archive on the Wikimedia-Logstash board.Jul 28 2015, 4:49 PM
greg moved this task from Backlog to Done on the Beta-Cluster-Infrastructure board.Sep 28 2015, 3:49 PM
fgiunchedi added a project: observability.Aug 19 2019, 2:33 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL