jforrester@deployment-parsoid11:~$ sudo puppet agent --test --noop Info: Using configured environment 'production' Info: Retrieving pluginfacts Info: Retrieving plugin Info: Loading facts Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Evaluation Error: Error while evaluating a Resource Statement, Conftool::Scripts::Safe_service_restart[php7.2-fpm]: has no parameter named 'lvs_services' expects a value for parameter 'services' (file: /etc/puppet/modules/profile/manifests/mediawiki/php/restarts.pp, line: 38) on node deployment-parsoid11.deployment-prep.eqiad.wmflabs Warning: Not using cache on failed catalog Error: Could not retrieve catalog; skipping run
|operations/puppet||production||+1 -1||Fix incorrect name of safe_service_restart parameter|
cherry-picked the above, your instance now has its puppet catalog being successfully compiled. I've left a copy of the output of the first two runs at /root/first-successful-puppet-run-20200307 and /root/second-successful-puppet-run-20200307 in case you were interested in what it did.
Remaining error in there is about failing to start php7.2-fpm - it seems to be trying to allocate 4GB of RAM, in an instance that has that much total. Sounds like we need to tweak profile::mediawiki::apc_shm_size - or replace it with a bigger instance.
Let's just set it to 128M first and see what happens. It's the default in a bunch of places, f.e.
hieradata/regex.yaml: profile::mediawiki::apc_shm_size: 128M
I'd already set it in horizon this morning to 1G; bumped down to 128M now: https://gerrit.wikimedia.org/r/plugins/gitiles/cloud/instance-puppet/+/c253dc43b7b99edcf855f10bdfb875a1b228997e%5E%21/