If I try to purge apache2 on toolsbeta-exec-202 (apt-get purge apache2-bin), apt-get instead installs (!) php5-cgi and uninstalls apache2, apache2-bin and libapache2-mod-php5. So I assume one of:

[tim@passepartout ~/src/operations/puppet]$ git grep php modules/toollabs/manifests/exec_environ.pp 
modules/toollabs/manifests/exec_environ.pp:        'php5-cli',
modules/toollabs/manifests/exec_environ.pp:        'php5-curl',
modules/toollabs/manifests/exec_environ.pp:        'php5-gd',
modules/toollabs/manifests/exec_environ.pp:        'php5-imagick',                # T71078.
modules/toollabs/manifests/exec_environ.pp:        'php5-intl',                   # T57652
modules/toollabs/manifests/exec_environ.pp:        'php5-mcrypt',
modules/toollabs/manifests/exec_environ.pp:        'php5-mysql',
modules/toollabs/manifests/exec_environ.pp:        'php5-pgsql',                  # For access to OSM db
modules/toollabs/manifests/exec_environ.pp:        'php5-redis',
modules/toollabs/manifests/exec_environ.pp:        'php5-sqlite',
modules/toollabs/manifests/exec_environ.pp:        'php5-xdebug',                 # T72313
modules/toollabs/manifests/exec_environ.pp:        'php5-xsl',
modules/toollabs/manifests/exec_environ.pp:        'phpunit',
[tim@passepartout ~/src/operations/puppet]$

depends on a PHP environment package, and if you don't specify one, Puppet/apt-get installs by default libapache2-mod-php5 instead of php5-cgi.

So we could:

1. Add service { 'apache2': ensure => stopped, … } to exec_environ.pp.
2. Add package { 'php5-cgi': ensure => installed, before => Package['php5-cli', 'curl', …], } to exec_environ.pp.

I lean slightly towards 2. because it feels "lighter" to me than 1.

(If someone could spell out more clearly which package/requirement causes this and how to find that out, much appreciated.)

valhallasw@tools-exec-1201:~$ apt-cache rdepends --installed apache2 --recurse

would give all local packages that depend on apache2 (recursively):

apache2
Reverse Depends:
    apache2-mpm-itk
    apache2-mpm-event
    apache2-mpm-prefork

apache2-mpm-itk
Reverse Depends:
  libapache2-mod-php5filter

apache2-mpm-event
Reverse Depends:

apache2-mpm-prefork
Reverse Depends:
 |libapache2-mod-php5filter

apache2-mpm-worker
Reverse Depends:

libapache2-mod-php5filter
Reverse Depends:

(with a whole lot of duplicates, for some reason).

libapache2-mod-php5filter provides phpapi-20090626, which is what all the php- packages depend on.

As far as the solution is concerned: another option is

3. add package { 'apache2': ensure => absent }

but on a whole I lean towards (1) because the issue is not so much that apache2 is installed, it's that it's running. And even that is not really a huge issue, it's just a bit weird.

This problem seems to be gone on the new Debian Stretch grid nodes, but there is a similar issue with lighttpd running where it is installed.

Change 481344 had a related patch set uploaded (by BryanDavis; owner: Bryan Davis):
[operations/puppet@production] toolforge: disable lighttpd service on webgrid nodes

https://gerrit.wikimedia.org/r/481344

Change 481344 merged by Bstorm:
[operations/puppet@production] toolforge: disable lighttpd service on webgrid nodes

https://gerrit.wikimedia.org/r/481344