Page MenuHomePhabricator

Puppet failing on wikistats.analytics.eqiad.wmflabs: /usr/local/sbin/x509-bundle error
Closed, ResolvedPublic

Description

Notice: /Stage[main]/Profile::Tlsproxy::Service/Tlsproxy::Localssl[yarn.wikimedia.org]/Sslcert::Certificate[yarn.wikimedia.org]/Sslcert::Chainedcert[yarn.wikimedia.org]/Exec[x509-bundle yarn.wikimedia.org-chain]/returns:     certpath.pop(0)
Notice: /Stage[main]/Profile::Tlsproxy::Service/Tlsproxy::Localssl[yarn.wikimedia.org]/Sslcert::Certificate[yarn.wikimedia.org]/Sslcert::Chainedcert[yarn.wikimedia.org]/Exec[x509-bundle yarn.wikimedia.org-chain]/returns: IndexError: pop from empty list
Error: '/usr/local/sbin/x509-bundle --skip-root --skip-first -c /etc/ssl/localcerts/yarn.wikimedia.org.crt -o /etc/ssl/localcerts/yarn.wikimedia.org.chain.crt' returned 1 instead of one of [0]
Error: /Stage[main]/Profile::Tlsproxy::Service/Tlsproxy::Localssl[yarn.wikimedia.org]/Sslcert::Certificate[yarn.wikimedia.org]/Sslcert::Chainedcert[yarn.wikimedia.org]/Exec[x509-bundle yarn.wikimedia.org-chain]/returns: change from 'notrun' to ['0'] failed: '/usr/local/sbin/x509-bundle --skip-root --skip-first -c /etc/ssl/localcerts/yarn.wikimedia.org.crt -o /etc/ssl/localcerts/yarn.wikimedia.org.chain.crt' returned 1 instead of one of [0]
Notice: /Stage[main]/Profile::Tlsproxy::Service/Tlsproxy::Localssl[yarn.wikimedia.org]/Sslcert::Certificate[yarn.wikimedia.org]/Sslcert::Chainedcert[yarn.wikimedia.org]/File[/etc/ssl/localcerts/yarn.wikimedia.org.chain.crt]: Dependency Exec[x509-bundle yarn.wikimedia.org-chain] has failures: true

This smells like some hiera that needs to be tuned for the Cloud VPS environment or possibly something that needs to be refactored in the applied roles: https://openstack-browser.toolforge.org/server/wikistats.analytics.eqiad.wmflabs

Event Timeline

bd808 created this task.Jun 15 2020, 5:17 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 15 2020, 5:17 PM
fdans assigned this task to elukey.Jun 18 2020, 3:55 PM
fdans added a subscriber: fdans.

For this site, the puppet configuration needs to skip TLS deployment.

fdans claimed this task.Jun 18 2020, 3:56 PM
fdans added a project: Analytics-Kanban.
fdans added a subscriber: elukey.
bd808 added a comment.Jun 26 2020, 9:15 PM

Still busted:

root@wikistats:~# puppet agent -tv
Info: Using configured environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Retrieving locales
Info: Loading facts
Info: Caching catalog for wikistats.analytics.eqiad.wmflabs
Info: Applying configuration version '(c7ec95403c) Brooke Storm - Revert "unattendedupgrades: allow configurable kernel cleanup"'
Notice: The LDAP client stack for this host is: classic/sudoldap
Notice: /Stage[main]/Profile::Ldap::Client::Labs/Notify[LDAP client stack]/message: defined 'message' as 'The LDAP client stack for this host is: classic/sudoldap'
Error: Could not set home on user[stats]: Execution of '/usr/sbin/usermod -d /var/lib/stats stats' returned 6: usermod: user 'stats' does not exist in /etc/passwd
Error: /Stage[main]/Statistics::User/User[stats]/home: change from '/home/stats' to '/var/lib/stats' failed: Could not set home on user[stats]: Execution of '/usr/sbin/usermod -d /var/lib/stats stats' returned 6: usermod: user 'stats' does not exist in /etc/passwd
Notice: /Stage[main]/Statistics::User/File[/var/lib/stats/.git-credentials]: Dependency User[stats] has failures: true
Warning: /Stage[main]/Statistics::User/File[/var/lib/stats/.git-credentials]: Skipping because of failed dependencies
Notice: /Stage[main]/Httpd/Httpd::Mod_conf[cgi]/Exec[ensure_present_mod_cgi]/returns: executed successfully
Info: /Stage[main]/Httpd/Httpd::Mod_conf[cgi]/Exec[ensure_present_mod_cgi]: Scheduling refresh of Service[apache2]
Warning: /Stage[main]/Statistics::User/Git::Userconfig[stats]/File[/var/lib/stats/.gitconfig]: Skipping because of failed dependencies
Warning: /Stage[main]/Statistics::Published/File[/srv/published-rsynced]: Skipping because of failed dependencies
Warning: /Stage[main]/Statistics::Published/Cron[hardsync-published]: Skipping because of failed dependencies
Warning: /Stage[main]/Statistics::Published/Rsync::Server::Module[published-destination]/File[/etc/rsync.d/frag-published-destination]: Skipping because of failed dependencies
Warning: /Stage[main]/Rsync::Server/Exec[compile fragments]: Skipping because of failed dependencies
Warning: /Stage[main]/Rsync::Server/Service[rsync]: Skipping because of failed dependencies
Warning: /Stage[main]/Statistics::Published/Rsync::Server::Module[published-destination]/Ferm::Service[rsyncd_access_published-destination]/File[/etc/ferm/conf.d/10_rsyncd_access_published-destination]: Skipping because of failed dependencies
Notice: /Stage[main]/Httpd/Service[apache2]: Triggered 'refresh' from 1 event
Warning: /Stage[main]/Ferm/Service[ferm]: Skipping because of failed dependencies
Notice: /Stage[main]/Profile::Tlsproxy::Service/Tlsproxy::Localssl[yarn.wikimedia.org]/Sslcert::Certificate[yarn.wikimedia.org]/Sslcert::Chainedcert[yarn.wikimedia.org]/Exec[x509-bundle yarn.wikimedia.org-chain]/returns: Traceback (most recent call last):
Notice: /Stage[main]/Profile::Tlsproxy::Service/Tlsproxy::Localssl[yarn.wikimedia.org]/Sslcert::Certificate[yarn.wikimedia.org]/Sslcert::Chainedcert[yarn.wikimedia.org]/Exec[x509-bundle yarn.wikimedia.org-chain]/returns:   File "/usr/local/sbin/x509-bundle", line 140, in <module>
Notice: /Stage[main]/Profile::Tlsproxy::Service/Tlsproxy::Localssl[yarn.wikimedia.org]/Sslcert::Certificate[yarn.wikimedia.org]/Sslcert::Chainedcert[yarn.wikimedia.org]/Exec[x509-bundle yarn.wikimedia.org-chain]/returns:     main()
Notice: /Stage[main]/Profile::Tlsproxy::Service/Tlsproxy::Localssl[yarn.wikimedia.org]/Sslcert::Certificate[yarn.wikimedia.org]/Sslcert::Chainedcert[yarn.wikimedia.org]/Exec[x509-bundle yarn.wikimedia.org-chain]/returns:   File "/usr/local/sbin/x509-bundle", line 119, in main
Notice: /Stage[main]/Profile::Tlsproxy::Service/Tlsproxy::Localssl[yarn.wikimedia.org]/Sslcert::Certificate[yarn.wikimedia.org]/Sslcert::Chainedcert[yarn.wikimedia.org]/Exec[x509-bundle yarn.wikimedia.org-chain]/returns:     certpath.pop(0)
Notice: /Stage[main]/Profile::Tlsproxy::Service/Tlsproxy::Localssl[yarn.wikimedia.org]/Sslcert::Certificate[yarn.wikimedia.org]/Sslcert::Chainedcert[yarn.wikimedia.org]/Exec[x509-bundle yarn.wikimedia.org-chain]/returns: IndexError: pop from empty list
Error: '/usr/local/sbin/x509-bundle --skip-root --skip-first -c /etc/ssl/localcerts/yarn.wikimedia.org.crt -o /etc/ssl/localcerts/yarn.wikimedia.org.chain.crt' returned 1 instead of one of [0]
Error: /Stage[main]/Profile::Tlsproxy::Service/Tlsproxy::Localssl[yarn.wikimedia.org]/Sslcert::Certificate[yarn.wikimedia.org]/Sslcert::Chainedcert[yarn.wikimedia.org]/Exec[x509-bundle yarn.wikimedia.org-chain]/returns: change from 'notrun' to ['0'] failed: '/usr/local/sbin/x509-bundle --skip-root --skip-first -c /etc/ssl/localcerts/yarn.wikimedia.org.crt -o /etc/ssl/localcerts/yarn.wikimedia.org.chain.crt' returned 1 instead of one of [0]
Notice: /Stage[main]/Profile::Tlsproxy::Service/Tlsproxy::Localssl[yarn.wikimedia.org]/Sslcert::Certificate[yarn.wikimedia.org]/Sslcert::Chainedcert[yarn.wikimedia.org]/File[/etc/ssl/localcerts/yarn.wikimedia.org.chain.crt]: Dependency Exec[x509-bundle yarn.wikimedia.org-chain] has failures: true
Warning: /Stage[main]/Profile::Tlsproxy::Service/Tlsproxy::Localssl[yarn.wikimedia.org]/Sslcert::Certificate[yarn.wikimedia.org]/Sslcert::Chainedcert[yarn.wikimedia.org]/File[/etc/ssl/localcerts/yarn.wikimedia.org.chain.crt]: Skipping because of failed dependencies
Warning: /Stage[main]/Nginx/Service[nginx]: Skipping because of failed dependencies
Info: Stage[main]: Unscheduling all events on Stage[main]
Notice: Applied catalog in 13.47 seconds
Ottomata closed this task as Resolved.Aug 17 2020, 3:42 PM
Ottomata added a subscriber: Ottomata.

We realized today that we don't need this instance at all. Deleted it.