Page MenuHomePhabricator
Create Task
Maniphest T91832

Puppet (silently) fails to setup apache on new trusty instances
Closed, ResolvedPublic
Actions

Description

Of the 5 new instances, one of them failed with this:

Mar  6 22:23:47 integration-slave1402 puppet-agent[18466]: (/Stage[main]/Mediawiki::Packages/Package[lilypond]/ensure) ensure changed 'purged' to 'present'
Mar  6 22:24:40 integration-slave1402 puppet-agent[18466]: (/Stage[main]/Mediawiki::Packages/Package[texlive-bibtex-extra]/ensure) ensure changed 'purged' to 'present'
Mar  6 22:24:47 integration-slave1402 puppet-agent[18466]: (/Stage[main]/Mediawiki::Packages/Package[mediawiki-math-texvc]/ensure) ensure changed 'purged' to 'present'
Mar  6 22:24:49 integration-slave1402 libapache2-mod-php5: php5_invoke: Enable module xmlrpc for apache2 SAPI
Mar  6 22:24:49 integration-slave1402 libapache2-mod-php5: php5_invoke: Enable module pdo for apache2 SAPI
Mar  6 22:24:49 integration-slave1402 libapache2-mod-php5: php5_invoke: Enable module memcached for apache2 SAPI
Mar  6 22:24:49 integration-slave1402 libapache2-mod-php5: php5_invoke: Enable module pdo_mysql for apache2 SAPI
Mar  6 22:24:49 integration-slave1402 libapache2-mod-php5: php5_invoke: Enable module fss for apache2 SAPI
Mar  6 22:24:49 integration-slave1402 libapache2-mod-php5: php5_invoke: Enable module geoip for apache2 SAPI
Mar  6 22:24:49 integration-slave1402 libapache2-mod-php5: php5_invoke: Enable module redis for apache2 SAPI
Mar  6 22:24:49 integration-slave1402 libapache2-mod-php5: php5_invoke: Enable module json for apache2 SAPI
Mar  6 22:24:49 integration-slave1402 libapache2-mod-php5: php5_invoke: Enable module mysqli for apache2 SAPI
Mar  6 22:24:49 integration-slave1402 libapache2-mod-php5: php5_invoke: Enable module mysql for apache2 SAPI
Mar  6 22:24:49 integration-slave1402 libapache2-mod-php5: php5_invoke: Enable module luasandbox for apache2 SAPI
Mar  6 22:24:49 integration-slave1402 libapache2-mod-php5: php5_invoke: Enable module opcache for apache2 SAPI
Mar  6 22:24:49 integration-slave1402 libapache2-mod-php5: php5_invoke: Enable module intl for apache2 SAPI
Mar  6 22:24:49 integration-slave1402 libapache2-mod-php5: php5_invoke: Enable module curl for apache2 SAPI
Mar  6 22:24:50 integration-slave1402 libapache2-mod-php5: apache2_invoke: Enable module php5
Mar  6 22:24:50 integration-slave1402 libapache2-mod-php5: apache2_reload: Your configuration is broken. Not restarting Apache 2
Mar  6 22:24:50 integration-slave1402 puppet-agent[18466]: (/Stage[main]/Mediawiki::Packages/Package[php-mail-mime]/ensure) ensure changed 'purged' to 'present'

After the puppet run:

$ curl 'http://localhost:9412/'
curl: (7) Failed to connect to localhost port 9412: Connection refused

Whereas on all other instances:

$ curl 'http://localhost:9412/'
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>403 Forbidden</title>
</head><body>
<h1>Forbidden</h1>
<p>You don't have permission to access /
on this server.</p>
</body></html>

Related Objects
Search...

Event Timeline

Krinkle created this task.Mar 6 2015, 10:45 PM
Krinkle raised the priority of this task from to Needs Triage.
Krinkle updated the task description. (Show Details)
Krinkle added projects: Continuous-Integration-Infrastructure, Puppet, acl*sre-team.
Krinkle subscribed.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 6 2015, 10:45 PM
Krinkle triaged this task as Low priority.Mar 6 2015, 10:46 PM
Krinkle updated the task description. (Show Details)
Krinkle set Security to None.
Krinkle added a comment.Mar 6 2015, 10:48 PM

Re-running puppet two (!) more times eventually fixed this. Lowering priority since we're moving on regardless, but it's definitely a bug somewhere.

Krinkle added a parent task: T91524: Re-create ci slaves (March 2015).Mar 14 2015, 1:09 AM
Krinkle added a comment.Apr 1 2015, 7:43 AM

This still happens on the latest branch of operations/puppet:

Apr  1 07:39:11 integration-slave1410 puppet-agent[16320]: (/Stage[main]/Mediawiki::Packages/Package[texlive-bibtex-extra]/ensure) ensure changed 'purged' to 'present'
Apr  1 07:39:17 integration-slave1410 puppet-agent[16320]: (/Stage[main]/Mediawiki::Packages/Package[mediawiki-math-texvc]/ensure) ensure changed 'purged' to 'present'
Apr  1 07:39:21 integration-slave1410 libapache2-mod-php5: php5_invoke: Enable module xmlrpc for apache2 SAPI
Apr  1 07:39:21 integration-slave1410 libapache2-mod-php5: php5_invoke: Enable module pdo for apache2 SAPI
Apr  1 07:39:21 integration-slave1410 libapache2-mod-php5: php5_invoke: Enable module memcached for apache2 SAPI
Apr  1 07:39:21 integration-slave1410 libapache2-mod-php5: php5_invoke: Enable module pdo_mysql for apache2 SAPI
Apr  1 07:39:21 integration-slave1410 libapache2-mod-php5: php5_invoke: Enable module fss for apache2 SAPI
Apr  1 07:39:21 integration-slave1410 libapache2-mod-php5: php5_invoke: Enable module geoip for apache2 SAPI
Apr  1 07:39:21 integration-slave1410 libapache2-mod-php5: php5_invoke: Enable module redis for apache2 SAPI
Apr  1 07:39:21 integration-slave1410 libapache2-mod-php5: php5_invoke: Enable module json for apache2 SAPI
Apr  1 07:39:21 integration-slave1410 libapache2-mod-php5: php5_invoke: Enable module mysqli for apache2 SAPI
Apr  1 07:39:21 integration-slave1410 libapache2-mod-php5: php5_invoke: Enable module mysql for apache2 SAPI
Apr  1 07:39:21 integration-slave1410 libapache2-mod-php5: php5_invoke: Enable module luasandbox for apache2 SAPI
Apr  1 07:39:21 integration-slave1410 libapache2-mod-php5: php5_invoke: Enable module opcache for apache2 SAPI
Apr  1 07:39:21 integration-slave1410 libapache2-mod-php5: php5_invoke: Enable module intl for apache2 SAPI
Apr  1 07:39:21 integration-slave1410 libapache2-mod-php5: php5_invoke: Enable module curl for apache2 SAPI
Apr  1 07:39:21 integration-slave1410 libapache2-mod-php5: apache2_invoke: Enable module php5
Apr  1 07:39:21 integration-slave1410 libapache2-mod-php5: apache2_reload: Your configuration is broken. Not restarting Apache 2
Apr  1 07:39:21 integration-slave1410 puppet-agent[16320]: (/Stage[main]/Mediawiki::Packages/Package[php-mail-mime]/ensure) ensure changed 'purged' to 'present'
Apr  1 07:39:23 integration-slave1410 puppet-agent[16320]: (/Stage[main]/Mediawiki::Packages/Package[libtiff-tools]/ensure) ensure changed 'purged' to 'present'
hashar subscribed.Apr 1 2015, 12:03 PM

I suspect an Apache configuration file that include some keywords from one of the module. Potentially Apache would be reloaded before the module actually got installed which would cause the configuration is broken error.

A way to verify the apache config is to run apachectl configtest which should spurts the appropriate error.

Later in the puppet run there is:

(/Stage[main]/Apache::Mod::Rewrite/Apache::Mod_conf[rewrite]/Exec[ensure_present_mod_rewrite]/returns) executed successfully
(/Stage[main]/Apache/Service[apache2]) Triggered 'refresh' from 22 events

So maybe that is just mod_rewrite being missed on the first refresh.

Krinkle renamed this task from Puppet (silently) fails to setup apache on some integration-slave14xx instances to Puppet (silently) fails to setup apache on new trusty instances.Apr 2 2015, 10:04 PM
Krinkle added a parent task: T94916: Re-create ci slaves (April 2015).
Krinkle moved this task from Untriaged to Backlog on the Continuous-Integration-Infrastructure board.Apr 7 2015, 2:30 PM
Krinkle updated the task description. (Show Details)Apr 23 2015, 3:28 AM
hashar closed this task as Resolved.Oct 21 2015, 7:10 PM
hashar claimed this task.

I haven't hit that issue when building a new Trusty slave.

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits