Page MenuHomePhabricator

Puppet fails on tools-elastic-01, tools-elastic-02 and tools-elastic-03: "Class[Nginx] is already declared"
Closed, ResolvedPublic

Description

tools-elastic-01: Info: Retrieving pluginfacts
tools-elastic-01: Info: Retrieving plugin
tools-elastic-01: Info: Loading facts
tools-elastic-01: Error: Could not retrieve catalog from remote server: Error 400 on SERVER: Duplicate declaration: Class[Nginx] is already declared in file /etc/puppet/modules/elasticsearch/manifests/https.pp:25; cannot redeclare at /etc/puppet/modules/role/manifests/toollabs/elasticsearch.pp:11 on node tools-elastic-01.tools.eqiad.wmflabs
tools-elastic-01: Warning: Not using cache on failed catalog
tools-elastic-01: Error: Could not retrieve catalog; skipping run
pdsh@passepartout: tools-elastic-01: ssh exited with exit code 1
tools-elastic-02: Info: Retrieving pluginfacts
tools-elastic-02: Info: Retrieving plugin
tools-elastic-02: Info: Loading facts
tools-elastic-02: Error: Could not retrieve catalog from remote server: Error 400 on SERVER: Duplicate declaration: Class[Nginx] is already declared in file /etc/puppet/modules/elasticsearch/manifests/https.pp:25; cannot redeclare at /etc/puppet/modules/role/manifests/toollabs/elasticsearch.pp:11 on node tools-elastic-02.tools.eqiad.wmflabs
tools-elastic-02: Warning: Not using cache on failed catalog
tools-elastic-02: Error: Could not retrieve catalog; skipping run
pdsh@passepartout: tools-elastic-02: ssh exited with exit code 1
tools-elastic-03: Info: Retrieving pluginfacts
tools-elastic-03: Info: Retrieving plugin
tools-elastic-03: Info: Loading facts
tools-elastic-03: Error: Could not retrieve catalog from remote server: Error 400 on SERVER: Duplicate declaration: Class[Nginx] is already declared in file /etc/puppet/modules/elasticsearch/manifests/https.pp:25; cannot redeclare at /etc/puppet/modules/role/manifests/toollabs/elasticsearch.pp:11 on node tools-elastic-03.tools.eqiad.wmflabs
tools-elastic-03: Warning: Not using cache on failed catalog
tools-elastic-03: Error: Could not retrieve catalog; skipping run
pdsh@passepartout: tools-elastic-03: ssh exited with exit code 1

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald Transcript

Change 281464 had a related patch set uploaded (by Rush):
toollabs elastic don't use nginx light

https://gerrit.wikimedia.org/r/281464

Change 281464 merged by Rush:
toollabs elastic don't use nginx light

https://gerrit.wikimedia.org/r/281464

The error now moved to:

[tim@passepartout ~]$ echo tools-elastic-0{1..3}.tools.eqiad.wmflabs | pdsh -f 1 -w - sudo puppet agent -t                                                              tools-elastic-01: Info: Retrieving pluginfacts
tools-elastic-01: Info: Retrieving plugin
tools-elastic-01: Info: Loading facts
tools-elastic-01: Info: Caching catalog for tools-elastic-01.tools.eqiad.wmflabs
tools-elastic-01: Info: Applying configuration version '1459793448'
tools-elastic-01: Error: /Stage[main]/Elasticsearch::Https/Base::Expose_puppet_certs[/etc/nginx]/File[/etc/nginx/ssl/cert.pem]: Could not evaluate: Could not retrieve information from environment production source(s) file:/var/lib/puppet/server/ssl/certs/tools-elastic-01.tools.eqiad.wmflabs.pem
tools-elastic-01: Error: Could not set 'present' on ensure: No such file or directory @ dir_s_rmdir - /etc/nginx/sites-available/elasticsearch20160404-7927-h2l1eg.lock at 42:/etc/puppet/modules/nginx/manifests/site.pp
tools-elastic-01: Notice: /Stage[main]/Role::Toollabs::Elasticsearch/Nginx::Site[elasticsearch]/File[/etc/nginx/sites-enabled/elasticsearch]: Dependency File[/etc/nginx/sites-available/elasticsearch] has failures: true
tools-elastic-01: Error: Could not set 'present' on ensure: No such file or directory @ dir_s_rmdir - /etc/nginx/sites-available/elasticsearch20160404-7927-h2l1eg.lock at 42:/etc/puppet/modules/nginx/manifests/site.pp
tools-elastic-01: Wrapped exception:
tools-elastic-01: No such file or directory @ dir_s_rmdir - /etc/nginx/sites-available/elasticsearch20160404-7927-h2l1eg.lock
tools-elastic-01: Error: /Stage[main]/Role::Toollabs::Elasticsearch/Nginx::Site[elasticsearch]/File[/etc/nginx/sites-available/elasticsearch]/ensure: change from absent to present failed: Could not set 'present' on ensure: No such file or directory @ dir_s_rmdir - /etc/nginx/sites-available/elasticsearch20160404-7927-h2l1eg.lock at 42:/etc/puppet/modules/nginx/manifests/site.pp
tools-elastic-01: Warning: /Stage[main]/Role::Toollabs::Elasticsearch/Nginx::Site[elasticsearch]/File[/etc/nginx/sites-enabled/elasticsearch]: Skipping because of failed dependencies
tools-elastic-01: Error: /Stage[main]/Elasticsearch::Https/Base::Expose_puppet_certs[/etc/nginx]/File[/etc/nginx/ssl/server.key]: Could not evaluate: Could not retrieve information from environment production source(s) file:/var/lib/puppet/server/ssl/private_keys/tools-elastic-01.tools.eqiad.wmflabs.pem
tools-elastic-01: Notice: /Stage[main]/Nginx/Service[nginx]: Dependency File[/etc/nginx/sites-available/elasticsearch] has failures: true
tools-elastic-01: Notice: /Stage[main]/Role::Toollabs::Elasticsearch/File[/etc/nginx/elasticsearch.htpasswd]: Dependency File[/etc/nginx/sites-available/elasticsearch] has failures: true
tools-elastic-01: Warning: /Stage[main]/Nginx/Service[nginx]: Skipping because of failed dependencies
tools-elastic-01: Warning: /Stage[main]/Role::Toollabs::Elasticsearch/File[/etc/nginx/elasticsearch.htpasswd]: Skipping because of failed dependencies
tools-elastic-01: Notice: Finished catalog run in 23.00 seconds
pdsh@passepartout: tools-elastic-01: ssh exited with exit code 4
tools-elastic-02: Info: Retrieving pluginfacts
tools-elastic-02: Info: Retrieving plugin
tools-elastic-02: Info: Loading facts
tools-elastic-02: Info: Caching catalog for tools-elastic-02.tools.eqiad.wmflabs
tools-elastic-02: Info: Applying configuration version '1459793448'
tools-elastic-02: Error: /Stage[main]/Elasticsearch::Https/Base::Expose_puppet_certs[/etc/nginx]/File[/etc/nginx/ssl/cert.pem]: Could not evaluate: Could not retrieve information from environment production source(s) file:/var/lib/puppet/server/ssl/certs/tools-elastic-02.tools.eqiad.wmflabs.pem
tools-elastic-02: Error: Could not set 'present' on ensure: No such file or directory @ dir_s_rmdir - /etc/nginx/sites-available/elasticsearch20160404-4257-1o91yff.lock at 42:/etc/puppet/modules/nginx/manifests/site.pp
tools-elastic-02: Error: Could not set 'present' on ensure: No such file or directory @ dir_s_rmdir - /etc/nginx/sites-available/elasticsearch20160404-4257-1o91yff.lock at 42:/etc/puppet/modules/nginx/manifests/site.pp
tools-elastic-02: Wrapped exception:
tools-elastic-02: No such file or directory @ dir_s_rmdir - /etc/nginx/sites-available/elasticsearch20160404-4257-1o91yff.lock
tools-elastic-02: Notice: /Stage[main]/Role::Toollabs::Elasticsearch/Nginx::Site[elasticsearch]/File[/etc/nginx/sites-enabled/elasticsearch]: Dependency File[/etc/nginx/sites-available/elasticsearch] has failures: true
tools-elastic-02: Error: /Stage[main]/Role::Toollabs::Elasticsearch/Nginx::Site[elasticsearch]/File[/etc/nginx/sites-available/elasticsearch]/ensure: change from absent to present failed: Could not set 'present' on ensure: No such file or directory @ dir_s_rmdir - /etc/nginx/sites-available/elasticsearch20160404-4257-1o91yff.lock at 42:/etc/puppet/modules/nginx/manifests/site.pp
tools-elastic-02: Warning: /Stage[main]/Role::Toollabs::Elasticsearch/Nginx::Site[elasticsearch]/File[/etc/nginx/sites-enabled/elasticsearch]: Skipping because of failed dependencies
tools-elastic-02: Error: /Stage[main]/Elasticsearch::Https/Base::Expose_puppet_certs[/etc/nginx]/File[/etc/nginx/ssl/server.key]: Could not evaluate: Could not retrieve information from environment production source(s) file:/var/lib/puppet/server/ssl/private_keys/tools-elastic-02.tools.eqiad.wmflabs.pem
tools-elastic-02: Notice: /Stage[main]/Nginx/Service[nginx]: Dependency File[/etc/nginx/sites-available/elasticsearch] has failures: true
tools-elastic-02: Notice: /Stage[main]/Role::Toollabs::Elasticsearch/File[/etc/nginx/elasticsearch.htpasswd]: Dependency File[/etc/nginx/sites-available/elasticsearch] has failures: true
tools-elastic-02: Warning: /Stage[main]/Nginx/Service[nginx]: Skipping because of failed dependencies
tools-elastic-02: Warning: /Stage[main]/Role::Toollabs::Elasticsearch/File[/etc/nginx/elasticsearch.htpasswd]: Skipping because of failed dependencies
tools-elastic-02: Notice: Finished catalog run in 19.38 seconds
pdsh@passepartout: tools-elastic-02: ssh exited with exit code 4
tools-elastic-03: Info: Retrieving pluginfacts
tools-elastic-03: Info: Retrieving plugin
tools-elastic-03: Info: Loading facts
tools-elastic-03: Info: Caching catalog for tools-elastic-03.tools.eqiad.wmflabs
tools-elastic-03: Info: Applying configuration version '1459793448'
tools-elastic-03: Error: /Stage[main]/Elasticsearch::Https/Base::Expose_puppet_certs[/etc/nginx]/File[/etc/nginx/ssl/cert.pem]: Could not evaluate: Could not retrieve information from environment production source(s) file:/var/lib/puppet/server/ssl/certs/tools-elastic-03.tools.eqiad.wmflabs.pem
tools-elastic-03: Error: Could not set 'present' on ensure: No such file or directory @ dir_s_rmdir - /etc/nginx/sites-available/elasticsearch20160404-7069-y73dck.lock at 42:/etc/puppet/modules/nginx/manifests/site.pp
tools-elastic-03: Error: Could not set 'present' on ensure: No such file or directory @ dir_s_rmdir - /etc/nginx/sites-available/elasticsearch20160404-7069-y73dck.lock at 42:/etc/puppet/modules/nginx/manifests/site.pp
tools-elastic-03: Wrapped exception:
tools-elastic-03: No such file or directory @ dir_s_rmdir - /etc/nginx/sites-available/elasticsearch20160404-7069-y73dck.lock
tools-elastic-03: Notice: /Stage[main]/Role::Toollabs::Elasticsearch/Nginx::Site[elasticsearch]/File[/etc/nginx/sites-enabled/elasticsearch]: Dependency File[/etc/nginx/sites-available/elasticsearch] has failures: true
tools-elastic-03: Error: /Stage[main]/Role::Toollabs::Elasticsearch/Nginx::Site[elasticsearch]/File[/etc/nginx/sites-available/elasticsearch]/ensure: change from absent to present failed: Could not set 'present' on ensure: No such file or directory @ dir_s_rmdir - /etc/nginx/sites-available/elasticsearch20160404-7069-y73dck.lock at 42:/etc/puppet/modules/nginx/manifests/site.pp
tools-elastic-03: Warning: /Stage[main]/Role::Toollabs::Elasticsearch/Nginx::Site[elasticsearch]/File[/etc/nginx/sites-enabled/elasticsearch]: Skipping because of failed dependencies
tools-elastic-03: Error: /Stage[main]/Elasticsearch::Https/Base::Expose_puppet_certs[/etc/nginx]/File[/etc/nginx/ssl/server.key]: Could not evaluate: Could not retrieve information from environment production source(s) file:/var/lib/puppet/server/ssl/private_keys/tools-elastic-03.tools.eqiad.wmflabs.pem
tools-elastic-03: Notice: /Stage[main]/Nginx/Service[nginx]: Dependency File[/etc/nginx/sites-available/elasticsearch] has failures: true
tools-elastic-03: Notice: /Stage[main]/Role::Toollabs::Elasticsearch/File[/etc/nginx/elasticsearch.htpasswd]: Dependency File[/etc/nginx/sites-available/elasticsearch] has failures: true
tools-elastic-03: Warning: /Stage[main]/Nginx/Service[nginx]: Skipping because of failed dependencies
tools-elastic-03: Warning: /Stage[main]/Role::Toollabs::Elasticsearch/File[/etc/nginx/elasticsearch.htpasswd]: Skipping because of failed dependencies
tools-elastic-03: Notice: Finished catalog run in 16.12 seconds
pdsh@passepartout: tools-elastic-03: ssh exited with exit code 4
[tim@passepartout ~]$

yep thanks, merged during a meeting then lunch but I'm coming back around to it

Change https://gerrit.wikimedia.org/r/#/c/281824/ has been merged, which should correct this issue. Let me know if it is not the case. I'll cleanup the local patch referenced on T131906 and close this other task if all looks good.