taavi@deployment-mediawiki-07:~$ sudo run-puppet-agent Info: Using configured environment 'production' Info: Retrieving pluginfacts Info: Retrieving plugin Info: Retrieving locales Info: Loading facts Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Evaluation Error: Error while evaluating a Resource Statement, Evaluation Error: Error while evaluating a Function Call, htpasswd(): salt must be 8 characters (file: /etc/puppet/modules/profile/manifests/mediawiki/php/monitoring.pp, line: 36, column: 24) on node deployment-mediawiki-07.deployment-prep.eqiad1.wikimedia.cloud Warning: Not using cache on failed catalog Error: Could not retrieve catalog; skipping run
Description
Description
Event Timeline
Comment Actions
This started happening after https://gerrit.wikimedia.org/r/c/operations/puppet/+/666787 was merged.
Comment Actions
The affected hosts are
- deployment-parsoid11.deployment-prep.eqiad.wmflabs
- deployment-jobrunner03.deployment-prep.eqiad.wmflabs
- deployment-mediawiki-09.deployment-prep.eqiad.wmflabs
- deployment-mediawiki-07.deployment-prep.eqiad.wmflabs
@Legoktm Do you have any advice on the best way to fix this?
Comment Actions
Yeah, we just need to truncate or pad the profile::mediawiki::php::monitoring::salt hiera so its 8 characters. I'll do that. This has been silently broken since whenever I guess, just now Puppet will complain about it :/
Comment Actions
After trying to figure out where it was set...eventually found it in Horizon: https://gerrit.wikimedia.org/r/plugins/gitiles/cloud/instance-puppet/+/e908a3584339ebed4fdc5e6abdacf775a8bb7212%5E%21/#F0
Ran puppet on all the failing hosts and it passed.