Page MenuHomePhabricator

ruby2.5 2.5.5-3+deb10u5 breaks Puppet
Closed, ResolvedPublic

Description

Seen with today's Cloud VPS unattended-upgrades round. ruby2.5 2.5.5-3+deb10u5 fails with the following error:

taavi@toolsbeta-test-k8s-haproxy-3:~$ sudo run-puppet-agent
Info: Using configured environment 'production'
Info: Retrieving pluginfacts
Error: /File[/var/lib/puppet/facts.d]: Failed to generate additional resources using 'eval_generate': Failed to open TCP connection to :8140 (Connection refused - connect(2) for "" port 8140)
Error: /File[/var/lib/puppet/facts.d]: Could not evaluate: Could not retrieve file metadata for puppet:///pluginfacts: Failed to open TCP connection to :8140 (Connection refused - connect(2) for "" port 8140)
Info: Retrieving plugin
Error: /File[/var/lib/puppet/lib]: Failed to generate additional resources using 'eval_generate': Failed to open TCP connection to :8140 (Connection refused - connect(2) for "" port 8140)
Error: /File[/var/lib/puppet/lib]: Could not evaluate: Could not retrieve file metadata for puppet:///plugins: Failed to open TCP connection to :8140 (Connection refused - connect(2) for "" port 8140)
Info: Retrieving locales
Error: /File[/var/lib/puppet/locales]: Failed to generate additional resources using 'eval_generate': Failed to open TCP connection to :8140 (Connection refused - connect(2) for "" port 8140)
Error: /File[/var/lib/puppet/locales]: Could not evaluate: Could not retrieve file metadata for puppet:///locales: Failed to open TCP connection to :8140 (Connection refused - connect(2) for "" port 8140)
Info: Loading facts
Info: Caching catalog for toolsbeta-test-k8s-haproxy-3.toolsbeta.eqiad1.wikimedia.cloud
Info: Applying configuration version '(55e7ea9d8b) root - prometheus::blackbox: detect ip_families automatically'
Error: /Stage[main]/Puppet::Agent/File[/etc/puppetlabs/facter/facter.conf]: Could not evaluate: Could not retrieve file metadata for puppet:///modules/puppet/facter.conf: Failed to open TCP connection to :8140 (Connection refused - connect(2) for "" port 8140)
Error: /Stage[main]/Prometheus::Node_puppet_agent/File[/usr/local/bin/prometheus-puppet-agent-stats]: Could not evaluate: Could not retrieve file metadata for puppet:///modules/prometheus/usr/local/bin/prometheus-puppet-agent-stats.py: Failed to open TCP connection to :8140 (Connection refused - connect(2) for "" port 8140)
Error: /Stage[main]/Profile::Puppet::Agent/File[/usr/local/share/bash/puppet-common.sh]: Could not evaluate: Could not retrieve file metadata for puppet:///modules/profile/puppet/bin/puppet-common.sh: Failed to open TCP connection to :8140 (Connection refused - connect(2) for "" port 8140)
Error: /Stage[main]/Profile::Puppet::Agent/File[/usr/local/bin/puppet-enabled]: Could not evaluate: Could not retrieve file metadata for puppet:///modules/profile/puppet/bin/puppet-enabled: Failed to open TCP connection to :8140 (Connection refused - connect(2) for "" port 8140)
Error: /Stage[main]/Profile::Puppet::Agent/File[/usr/local/sbin/disable-puppet]: Could not evaluate: Could not retrieve file metadata for puppet:///modules/profile/puppet/bin/disable-puppet: Failed to open TCP connection to :8140 (Connection refused - connect(2) for "" port 8140)
Error: /Stage[main]/Profile::Puppet::Agent/File[/usr/local/sbin/enable-puppet]: Could not evaluate: Could not retrieve file metadata for puppet:///modules/profile/puppet/bin/enable-puppet: Failed to open TCP connection to :8140 (Connection refused - connect(2) for "" port 8140)
Error: /Stage[main]/Profile::Puppet::Agent/File[/usr/local/sbin/run-puppet-agent]: Could not evaluate: Could not retrieve file metadata for puppet:///modules/profile/puppet/bin/run-puppet-agent: Failed to open TCP connection to :8140 (Connection refused - connect(2) for "" port 8140)
Error: /Stage[main]/Profile::Puppet::Agent/File[/usr/local/sbin/run-no-puppet]: Could not evaluate: Could not retrieve file metadata for puppet:///modules/profile/puppet/bin/run-no-puppet: Failed to open TCP connection to :8140 (Connection refused - connect(2) for "" port 8140)
Error: /Stage[main]/Sslcert/File[/usr/local/sbin/x509-bundle]: Could not evaluate: Could not retrieve file metadata for puppet:///modules/sslcert/x509-bundle.py: Failed to open TCP connection to :8140 (Connection refused - connect(2) for "" port 8140)
Error: /Stage[main]/Apt/File[/usr/local/sbin/dist-upgrade]: Could not evaluate: Could not retrieve file metadata for puppet:///modules/apt/dist-upgrade.sh: Failed to open TCP connection to :8140 (Connection refused - connect(2) for "" port 8140)
Error: /Stage[main]/Resolvconf/File[/sbin/resolvconf]: Could not evaluate: Could not retrieve file metadata for puppet:///modules/resolvconf/resolvconf.dummy: Failed to open TCP connection to :8140 (Connection refused - connect(2) for "" port 8140)
Error: /Stage[main]/Resolvconf/File[/etc/dhcp/dhclient-enter-hooks.d/nodnsupdate]: Could not evaluate: Could not retrieve file metadata for puppet:///modules/resolvconf/nodnsupdate: Failed to open TCP connection to :8140 (Connection refused - connect(2) for "" port 8140)
Error: /Stage[main]/Profile::Environment/File[/etc/profile.d/field.sh]: Could not evaluate: Could not retrieve file metadata for puppet:///modules/base/environment/field.sh: Failed to open TCP connection to :8140 (Connection refused - connect(2) for "" port 8140)
Error: /Stage[main]/Profile::Environment/File[/etc/profile.d/systemd-environment.sh]: Could not evaluate: Could not retrieve file metadata for puppet:///modules/profile/environment/systemd-environment.sh: Failed to open TCP connection to :8140 (Connection refused - connect(2) for "" port 8140)
Error: /Stage[main]/Profile::Environment/File[/usr/local/bin/gen_fingerprints]: Could not evaluate: Could not retrieve file metadata for puppet:///modules/base/environment/gen_fingerprints: Failed to open TCP connection to :8140 (Connection refused - connect(2) for "" port 8140)
Error: /Stage[main]/Profile::Ssh::Server/File[/usr/sbin/ssh-key-ldap-lookup]: Could not evaluate: Could not retrieve file metadata for puppet:///modules/profile/ssh/server/ssh-key-ldap-lookup.py: Failed to open TCP connection to :8140 (Connection refused - connect(2) for "" port 8140)
Error: /Stage[main]/Debdeploy::Client/File[/usr/local/bin/apt-upgrade-activity]: Could not evaluate: Could not retrieve file metadata for puppet:///modules/base/apt-upgrade-activity.py: Failed to open TCP connection to :8140 (Connection refused - connect(2) for "" port 8140)
Error: /Stage[main]/Profile::Auto_restarts/File[/usr/local/sbin/wmf-auto-restart]: Could not evaluate: Could not retrieve file metadata for puppet:///modules/base/wmf-auto-restart.py: Failed to open TCP connection to :8140 (Connection refused - connect(2) for "" port 8140)
Error: /Stage[main]/Prometheus::Node_debian_version/File[/usr/local/bin/prometheus-debian-version]: Could not evaluate: Could not retrieve file metadata for puppet:///modules/prometheus/usr/local/bin/prometheus-debian-version.sh: Failed to open TCP connection to :8140 (Connection refused - connect(2) for "" port 8140)
Error: /Stage[main]/Profile::Base::Labs/File[/etc/default/acct]: Could not evaluate: Could not retrieve file metadata for puppet:///modules/base/labs-acct.default: Failed to open TCP connection to :8140 (Connection refused - connect(2) for "" port 8140)
Error: /Stage[main]/Profile::Base::Labs/File[/etc/default/nfs-common]: Could not evaluate: Could not retrieve file metadata for puppet:///modules/base/labs/nfs-common.default: Failed to open TCP connection to :8140 (Connection refused - connect(2) for "" port 8140)
Error: /Stage[main]/Profile::Base::Labs/File[/usr/local/sbin/notify_maintainers.py]: Could not evaluate: Could not retrieve file metadata for puppet:///modules/base/labs/notify_maintainers.py: Failed to open TCP connection to :8140 (Connection refused - connect(2) for "" port 8140)
Notice: /Stage[main]/Profile::Base::Labs/File[/usr/local/sbin/puppet_alert.py]: Dependency File[/usr/local/sbin/notify_maintainers.py] has failures: true
Warning: /Stage[main]/Profile::Base::Labs/File[/usr/local/sbin/puppet_alert.py]: Skipping because of failed dependencies

Downgrading libruby2.5 and ruby2.5 to 2.5.5-3+deb10u4 makes the Puppet run work again.

Event Timeline

Change 927971 had a related patch set uploaded (by David Caro; author: David Caro):

[operations/puppet@production] wmcs.instance: pin ruby2.5

https://gerrit.wikimedia.org/r/927971

Mentioned in SAL (#wikimedia-operations) [2023-06-07T08:22:27Z] <moritzm> uploaded ruby 2.5.5-3+deb10u5+wmf1 to apt.wikimedia.org, unbreaking Puppet runs after latest Ruby update for Buster T338294

Before I file another task, is this the cause of all the Puppet failure emails I received an hour ago?

Before I file another task, is this the cause of all the Puppet failure emails I received an hour ago?

Yes. Sorry about those!

Change 927971 abandoned by David Caro:

[operations/puppet@production] wmcs.instance: pin ruby2.5

Reason:

Not needed, a new package with the fix was released (thanks @mmuhlenhoff!)

https://gerrit.wikimedia.org/r/927971

The fix is out already, if you are still seeing issues you can try updating and upgrading ruby2.5 and libruby2.5 to it's latest version:

sudo apt update
sudo apt install --upgrade ruby2.5 libruby2.5

The new package version is 2.5.5-3+deb10u5+wmf1.

taavi claimed this task.

Mentioned in SAL (#wikimedia-operations) [2023-06-15T14:00:31Z] <moritzm> remove ruby2.5 2.5.5-3+deb10u5+wmf1 (superseded by corrected Debian build 2.5.5-3+deb10u6 T338294