Page MenuHomePhabricator

Puppet broken on deployment-netbox, looks like it thinks its a prod box
Closed, ResolvedPublic

Description

krenair@deployment-netbox:~$ sudo -i puppet agent -tv
Warning: Setting configtimeout is deprecated. 
   (at /usr/lib/ruby/vendor_ruby/puppet/settings.rb:1146:in `issue_deprecation_warning')
Info: Using configured environment 'future'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Info: Caching catalog for deployment-netbox.deployment-prep.eqiad.wmflabs
Notice: /Stage[main]/Base::Environment/Tidy[/var/tmp/core]: Tidying 0 files
Info: Applying configuration version '1515118751'
Notice: /Stage[main]/Profile::Netbox/Postgresql::User[replication@netmon2001]/Exec[create_user-replication@netmon2001]/returns: could not change directory to "/root": Permission denied
Notice: /Stage[main]/Profile::Netbox/Postgresql::User[replication@netmon2001]/Exec[create_user-replication@netmon2001]/returns: createuser: could not connect to database postgres: could not connect to server: No such file or directory
Notice: /Stage[main]/Profile::Netbox/Postgresql::User[replication@netmon2001]/Exec[create_user-replication@netmon2001]/returns: 	Is the server running locally and accepting
Notice: /Stage[main]/Profile::Netbox/Postgresql::User[replication@netmon2001]/Exec[create_user-replication@netmon2001]/returns: 	connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"?
Error: /usr/bin/createuser --no-superuser --no-createdb --no-createrole replication returned 1 instead of one of [0]
Error: /Stage[main]/Profile::Netbox/Postgresql::User[replication@netmon2001]/Exec[create_user-replication@netmon2001]/returns: change from notrun to 0 failed: /usr/bin/createuser --no-superuser --no-createdb --no-createrole replication returned 1 instead of one of [0]
Notice: /Stage[main]/Profile::Netbox/Postgresql::User[netbox@netmon2001]/Exec[create_user-netbox@netmon2001]/returns: could not change directory to "/root": Permission denied
Notice: /Stage[main]/Profile::Netbox/Postgresql::User[netbox@netmon2001]/Exec[create_user-netbox@netmon2001]/returns: createuser: could not connect to database postgres: could not connect to server: No such file or directory
Notice: /Stage[main]/Profile::Netbox/Postgresql::User[netbox@netmon2001]/Exec[create_user-netbox@netmon2001]/returns: 	Is the server running locally and accepting
Notice: /Stage[main]/Profile::Netbox/Postgresql::User[netbox@netmon2001]/Exec[create_user-netbox@netmon2001]/returns: 	connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"?
Error: /usr/bin/createuser --no-superuser --no-createdb --no-createrole netbox returned 1 instead of one of [0]
Error: /Stage[main]/Profile::Netbox/Postgresql::User[netbox@netmon2001]/Exec[create_user-netbox@netmon2001]/returns: change from notrun to 0 failed: /usr/bin/createuser --no-superuser --no-createdb --no-createrole netbox returned 1 instead of one of [0]
Notice: /Stage[main]/Profile::Netbox/Postgresql::User[netbox@localhost]/Exec[create_user-netbox@localhost]/returns: could not change directory to "/root": Permission denied
Notice: /Stage[main]/Profile::Netbox/Postgresql::User[netbox@localhost]/Exec[create_user-netbox@localhost]/returns: createuser: could not connect to database postgres: could not connect to server: No such file or directory
Notice: /Stage[main]/Profile::Netbox/Postgresql::User[netbox@localhost]/Exec[create_user-netbox@localhost]/returns: 	Is the server running locally and accepting
Notice: /Stage[main]/Profile::Netbox/Postgresql::User[netbox@localhost]/Exec[create_user-netbox@localhost]/returns: 	connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"?
Error: /usr/bin/createuser --no-superuser --no-createdb --no-createrole netbox returned 1 instead of one of [0]
Error: /Stage[main]/Profile::Netbox/Postgresql::User[netbox@localhost]/Exec[create_user-netbox@localhost]/returns: change from notrun to 0 failed: /usr/bin/createuser --no-superuser --no-createdb --no-createrole netbox returned 1 instead of one of [0]
Notice: /Stage[main]/Profile::Netbox/Postgresql::User[prometheus@localhost]/Exec[create_user-prometheus@localhost]/returns: could not change directory to "/root": Permission denied
Notice: /Stage[main]/Profile::Netbox/Postgresql::User[prometheus@localhost]/Exec[create_user-prometheus@localhost]/returns: createuser: could not connect to database postgres: could not connect to server: No such file or directory
Notice: /Stage[main]/Profile::Netbox/Postgresql::User[prometheus@localhost]/Exec[create_user-prometheus@localhost]/returns: 	Is the server running locally and accepting
Notice: /Stage[main]/Profile::Netbox/Postgresql::User[prometheus@localhost]/Exec[create_user-prometheus@localhost]/returns: 	connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"?
Error: /usr/bin/createuser --no-superuser --no-createdb --no-createrole prometheus returned 1 instead of one of [0]
Error: /Stage[main]/Profile::Netbox/Postgresql::User[prometheus@localhost]/Exec[create_user-prometheus@localhost]/returns: change from notrun to 0 failed: /usr/bin/createuser --no-superuser --no-createdb --no-createrole prometheus returned 1 instead of one of [0]
Notice: /Stage[main]/Profile::Netbox/Postgresql::Db[netbox]/Exec[create_postgres_db_netbox]/returns: could not change directory to "/root": Permission denied
Notice: /Stage[main]/Profile::Netbox/Postgresql::Db[netbox]/Exec[create_postgres_db_netbox]/returns: createdb: could not connect to database template1: could not connect to server: No such file or directory
Notice: /Stage[main]/Profile::Netbox/Postgresql::Db[netbox]/Exec[create_postgres_db_netbox]/returns: 	Is the server running locally and accepting
Notice: /Stage[main]/Profile::Netbox/Postgresql::Db[netbox]/Exec[create_postgres_db_netbox]/returns: 	connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"?
Error: /usr/bin/createdb --owner='netbox' 'netbox' returned 1 instead of one of [0]
Error: /Stage[main]/Profile::Netbox/Postgresql::Db[netbox]/Exec[create_postgres_db_netbox]/returns: change from notrun to 0 failed: /usr/bin/createdb --owner='netbox' 'netbox' returned 1 instead of one of [0]
Notice: /Stage[main]/Profile::Netbox/Letsencrypt::Cert::Integrated[netbox]/Exec[acme-setup-acme-netbox]/returns: Getting ACME cert /etc/acme/cert/netbox.crt
Notice: /Stage[main]/Profile::Netbox/Letsencrypt::Cert::Integrated[netbox]/Exec[acme-setup-acme-netbox]/returns: Traceback (most recent call last):
Notice: /Stage[main]/Profile::Netbox/Letsencrypt::Cert::Integrated[netbox]/Exec[acme-setup-acme-netbox]/returns:   File "/usr/local/sbin/acme-setup", line 509, in <module>
Notice: /Stage[main]/Profile::Netbox/Letsencrypt::Cert::Integrated[netbox]/Exec[acme-setup-acme-netbox]/returns:     main()
Notice: /Stage[main]/Profile::Netbox/Letsencrypt::Cert::Integrated[netbox]/Exec[acme-setup-acme-netbox]/returns:   File "/usr/local/sbin/acme-setup", line 505, in main
Notice: /Stage[main]/Profile::Netbox/Letsencrypt::Cert::Integrated[netbox]/Exec[acme-setup-acme-netbox]/returns:     key_uid, key_gid)
Notice: /Stage[main]/Profile::Netbox/Letsencrypt::Cert::Integrated[netbox]/Exec[acme-setup-acme-netbox]/returns:   File "/usr/local/sbin/acme-setup", line 478, in acme_setup
Notice: /Stage[main]/Profile::Netbox/Letsencrypt::Cert::Integrated[netbox]/Exec[acme-setup-acme-netbox]/returns:     exp_rand, chal_dir, acme_user, svc, force_crt)
Notice: /Stage[main]/Profile::Netbox/Letsencrypt::Cert::Integrated[netbox]/Exec[acme-setup-acme-netbox]/returns:   File "/usr/local/sbin/acme-setup", line 402, in ensure_crt_acme
Notice: /Stage[main]/Profile::Netbox/Letsencrypt::Cert::Integrated[netbox]/Exec[acme-setup-acme-netbox]/returns:     ensure_real_fs(tls_crt, 0o644, 0, 0, False, cert_create, cert_force)
Notice: /Stage[main]/Profile::Netbox/Letsencrypt::Cert::Integrated[netbox]/Exec[acme-setup-acme-netbox]/returns:   File "/usr/local/sbin/acme-setup", line 205, in ensure_real_fs
Notice: /Stage[main]/Profile::Netbox/Letsencrypt::Cert::Integrated[netbox]/Exec[acme-setup-acme-netbox]/returns:     creator()
Notice: /Stage[main]/Profile::Netbox/Letsencrypt::Cert::Integrated[netbox]/Exec[acme-setup-acme-netbox]/returns:   File "/usr/local/sbin/acme-setup", line 394, in cert_create
Notice: /Stage[main]/Profile::Netbox/Letsencrypt::Cert::Integrated[netbox]/Exec[acme-setup-acme-netbox]/returns:     acme_challenge(id, cert_dir, acct_key, csr, chal_dir, acme_user)
Notice: /Stage[main]/Profile::Netbox/Letsencrypt::Cert::Integrated[netbox]/Exec[acme-setup-acme-netbox]/returns:   File "/usr/local/sbin/acme-setup", line 369, in acme_challenge
Notice: /Stage[main]/Profile::Netbox/Letsencrypt::Cert::Integrated[netbox]/Exec[acme-setup-acme-netbox]/returns:     % (" ".join(args), p.returncode, p_err))
Notice: /Stage[main]/Profile::Netbox/Letsencrypt::Cert::Integrated[netbox]/Exec[acme-setup-acme-netbox]/returns: Exception: Command >>/usr/local/sbin/acme_tiny.py --account-key /etc/acme/acct/acct.key --csr /etc/acme/csr/netbox.pem --acme-dir /var/acme/challenge<< failed, exit code 1, stderr:
Notice: /Stage[main]/Profile::Netbox/Letsencrypt::Cert::Integrated[netbox]/Exec[acme-setup-acme-netbox]/returns: Parsing account key...
Notice: /Stage[main]/Profile::Netbox/Letsencrypt::Cert::Integrated[netbox]/Exec[acme-setup-acme-netbox]/returns: Parsing CSR...
Notice: /Stage[main]/Profile::Netbox/Letsencrypt::Cert::Integrated[netbox]/Exec[acme-setup-acme-netbox]/returns: Registering account...
Notice: /Stage[main]/Profile::Netbox/Letsencrypt::Cert::Integrated[netbox]/Exec[acme-setup-acme-netbox]/returns: Already registered!
Notice: /Stage[main]/Profile::Netbox/Letsencrypt::Cert::Integrated[netbox]/Exec[acme-setup-acme-netbox]/returns: Verifying netbox.wikimedia.org...
Notice: /Stage[main]/Profile::Netbox/Letsencrypt::Cert::Integrated[netbox]/Exec[acme-setup-acme-netbox]/returns: Traceback (most recent call last):
Notice: /Stage[main]/Profile::Netbox/Letsencrypt::Cert::Integrated[netbox]/Exec[acme-setup-acme-netbox]/returns:   File "/usr/local/sbin/acme_tiny.py", line 234, in <module>
Notice: /Stage[main]/Profile::Netbox/Letsencrypt::Cert::Integrated[netbox]/Exec[acme-setup-acme-netbox]/returns:     main(sys.argv[1:])
Notice: /Stage[main]/Profile::Netbox/Letsencrypt::Cert::Integrated[netbox]/Exec[acme-setup-acme-netbox]/returns:   File "/usr/local/sbin/acme_tiny.py", line 230, in main
Notice: /Stage[main]/Profile::Netbox/Letsencrypt::Cert::Integrated[netbox]/Exec[acme-setup-acme-netbox]/returns:     signed_crt = get_crt(args.account_key, args.csr, args.acme_dir, log=LOGGER, CA=args.ca)
Notice: /Stage[main]/Profile::Netbox/Letsencrypt::Cert::Integrated[netbox]/Exec[acme-setup-acme-netbox]/returns:   File "/usr/local/sbin/acme_tiny.py", line 159, in get_crt
Notice: /Stage[main]/Profile::Netbox/Letsencrypt::Cert::Integrated[netbox]/Exec[acme-setup-acme-netbox]/returns:     wellknown_path, wellknown_url))
Notice: /Stage[main]/Profile::Netbox/Letsencrypt::Cert::Integrated[netbox]/Exec[acme-setup-acme-netbox]/returns: ValueError: Wrote file to /var/acme/challenge/cgjbFzMJhHeLkGrzb-ifXPT9ZZzgU1jIY4XyoQAkMeo, but couldn't download http://netbox.wikimedia.org/.well-known/acme-challenge/cgjbFzMJhHeLkGrzb-ifXPT9ZZzgU1jIY4XyoQAkMeo
Error: /usr/local/sbin/acme-setup -i netbox -s netbox.wikimedia.org --key-user root --key-group root -m acme -w apache2 returned 1 instead of one of [0]
Error: /Stage[main]/Profile::Netbox/Letsencrypt::Cert::Integrated[netbox]/Exec[acme-setup-acme-netbox]/returns: change from notrun to 0 failed: /usr/local/sbin/acme-setup -i netbox -s netbox.wikimedia.org --key-user root --key-group root -m acme -w apache2 returned 1 instead of one of [0]
Notice: Applied catalog in 9.79 seconds

trying to connect to netmon2001 possibly? and trying to get an LE cert for netbox.wikimedia.org?
also all those permissions errors relating to the user running puppet, what is that about?
Looks related to T170144

Event Timeline

Krenair triaged this task as Medium priority.Jan 5 2018, 2:21 AM
Krenair created this task.

Seems like deployment-netbox fails to setup the LetsEncrypt certificate because it is coded to use the production URL (netbox.wikimedia.org). @ayounsi if you don't need deployment-netbox, maybe consider dropping it entirely? Else I guess some hiera/puppet work is needed so one can change the netbox domain.

greg added a subscriber: greg.

(Adding SRE since this box is in support of their evaluation of netbox)

Indeed, the instance is not needed anymore. I shut it down and will delete it in a few days.