Page MenuHomePhabricator

Puppet (silently) fails to setup apache on new trusty instances
Closed, ResolvedPublic

Description

Of the 5 new instances, one of them failed with this:

Mar  6 22:23:47 integration-slave1402 puppet-agent[18466]: (/Stage[main]/Mediawiki::Packages/Package[lilypond]/ensure) ensure changed 'purged' to 'present'
Mar  6 22:24:40 integration-slave1402 puppet-agent[18466]: (/Stage[main]/Mediawiki::Packages/Package[texlive-bibtex-extra]/ensure) ensure changed 'purged' to 'present'
Mar  6 22:24:47 integration-slave1402 puppet-agent[18466]: (/Stage[main]/Mediawiki::Packages/Package[mediawiki-math-texvc]/ensure) ensure changed 'purged' to 'present'
Mar  6 22:24:49 integration-slave1402 libapache2-mod-php5: php5_invoke: Enable module xmlrpc for apache2 SAPI
Mar  6 22:24:49 integration-slave1402 libapache2-mod-php5: php5_invoke: Enable module pdo for apache2 SAPI
Mar  6 22:24:49 integration-slave1402 libapache2-mod-php5: php5_invoke: Enable module memcached for apache2 SAPI
Mar  6 22:24:49 integration-slave1402 libapache2-mod-php5: php5_invoke: Enable module pdo_mysql for apache2 SAPI
Mar  6 22:24:49 integration-slave1402 libapache2-mod-php5: php5_invoke: Enable module fss for apache2 SAPI
Mar  6 22:24:49 integration-slave1402 libapache2-mod-php5: php5_invoke: Enable module geoip for apache2 SAPI
Mar  6 22:24:49 integration-slave1402 libapache2-mod-php5: php5_invoke: Enable module redis for apache2 SAPI
Mar  6 22:24:49 integration-slave1402 libapache2-mod-php5: php5_invoke: Enable module json for apache2 SAPI
Mar  6 22:24:49 integration-slave1402 libapache2-mod-php5: php5_invoke: Enable module mysqli for apache2 SAPI
Mar  6 22:24:49 integration-slave1402 libapache2-mod-php5: php5_invoke: Enable module mysql for apache2 SAPI
Mar  6 22:24:49 integration-slave1402 libapache2-mod-php5: php5_invoke: Enable module luasandbox for apache2 SAPI
Mar  6 22:24:49 integration-slave1402 libapache2-mod-php5: php5_invoke: Enable module opcache for apache2 SAPI
Mar  6 22:24:49 integration-slave1402 libapache2-mod-php5: php5_invoke: Enable module intl for apache2 SAPI
Mar  6 22:24:49 integration-slave1402 libapache2-mod-php5: php5_invoke: Enable module curl for apache2 SAPI
Mar  6 22:24:50 integration-slave1402 libapache2-mod-php5: apache2_invoke: Enable module php5
Mar  6 22:24:50 integration-slave1402 libapache2-mod-php5: apache2_reload: Your configuration is broken. Not restarting Apache 2
Mar  6 22:24:50 integration-slave1402 puppet-agent[18466]: (/Stage[main]/Mediawiki::Packages/Package[php-mail-mime]/ensure) ensure changed 'purged' to 'present'

After the puppet run:

$ curl 'http://localhost:9412/'
curl: (7) Failed to connect to localhost port 9412: Connection refused

Whereas on all other instances:

$ curl 'http://localhost:9412/'
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>403 Forbidden</title>
</head><body>
<h1>Forbidden</h1>
<p>You don't have permission to access /
on this server.</p>
</body></html>

Event Timeline

Krinkle raised the priority of this task from to Needs Triage.
Krinkle updated the task description. (Show Details)
Krinkle added a subscriber: Krinkle.
Krinkle updated the task description. (Show Details)
Krinkle set Security to None.

Re-running puppet two (!) more times eventually fixed this. Lowering priority since we're moving on regardless, but it's definitely a bug somewhere.

This still happens on the latest branch of operations/puppet:

Apr  1 07:39:11 integration-slave1410 puppet-agent[16320]: (/Stage[main]/Mediawiki::Packages/Package[texlive-bibtex-extra]/ensure) ensure changed 'purged' to 'present'
Apr  1 07:39:17 integration-slave1410 puppet-agent[16320]: (/Stage[main]/Mediawiki::Packages/Package[mediawiki-math-texvc]/ensure) ensure changed 'purged' to 'present'
Apr  1 07:39:21 integration-slave1410 libapache2-mod-php5: php5_invoke: Enable module xmlrpc for apache2 SAPI
Apr  1 07:39:21 integration-slave1410 libapache2-mod-php5: php5_invoke: Enable module pdo for apache2 SAPI
Apr  1 07:39:21 integration-slave1410 libapache2-mod-php5: php5_invoke: Enable module memcached for apache2 SAPI
Apr  1 07:39:21 integration-slave1410 libapache2-mod-php5: php5_invoke: Enable module pdo_mysql for apache2 SAPI
Apr  1 07:39:21 integration-slave1410 libapache2-mod-php5: php5_invoke: Enable module fss for apache2 SAPI
Apr  1 07:39:21 integration-slave1410 libapache2-mod-php5: php5_invoke: Enable module geoip for apache2 SAPI
Apr  1 07:39:21 integration-slave1410 libapache2-mod-php5: php5_invoke: Enable module redis for apache2 SAPI
Apr  1 07:39:21 integration-slave1410 libapache2-mod-php5: php5_invoke: Enable module json for apache2 SAPI
Apr  1 07:39:21 integration-slave1410 libapache2-mod-php5: php5_invoke: Enable module mysqli for apache2 SAPI
Apr  1 07:39:21 integration-slave1410 libapache2-mod-php5: php5_invoke: Enable module mysql for apache2 SAPI
Apr  1 07:39:21 integration-slave1410 libapache2-mod-php5: php5_invoke: Enable module luasandbox for apache2 SAPI
Apr  1 07:39:21 integration-slave1410 libapache2-mod-php5: php5_invoke: Enable module opcache for apache2 SAPI
Apr  1 07:39:21 integration-slave1410 libapache2-mod-php5: php5_invoke: Enable module intl for apache2 SAPI
Apr  1 07:39:21 integration-slave1410 libapache2-mod-php5: php5_invoke: Enable module curl for apache2 SAPI
Apr  1 07:39:21 integration-slave1410 libapache2-mod-php5: apache2_invoke: Enable module php5
Apr  1 07:39:21 integration-slave1410 libapache2-mod-php5: apache2_reload: Your configuration is broken. Not restarting Apache 2
Apr  1 07:39:21 integration-slave1410 puppet-agent[16320]: (/Stage[main]/Mediawiki::Packages/Package[php-mail-mime]/ensure) ensure changed 'purged' to 'present'
Apr  1 07:39:23 integration-slave1410 puppet-agent[16320]: (/Stage[main]/Mediawiki::Packages/Package[libtiff-tools]/ensure) ensure changed 'purged' to 'present'

I suspect an Apache configuration file that include some keywords from one of the module. Potentially Apache would be reloaded before the module actually got installed which would cause the configuration is broken error.

A way to verify the apache config is to run apachectl configtest which should spurts the appropriate error.

Later in the puppet run there is:

(/Stage[main]/Apache::Mod::Rewrite/Apache::Mod_conf[rewrite]/Exec[ensure_present_mod_rewrite]/returns) executed successfully
(/Stage[main]/Apache/Service[apache2]) Triggered 'refresh' from 22 events

So maybe that is just mod_rewrite being missed on the first refresh.

Krinkle renamed this task from Puppet (silently) fails to setup apache on some integration-slave14xx instances to Puppet (silently) fails to setup apache on new trusty instances.Apr 2 2015, 10:04 PM
hashar claimed this task.

I haven't hit that issue when building a new Trusty slave.