Maniphest T319681

Puppet failure on deploy-1004.devtools.eqiad1.wikimedia.cloud
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	dancy
	Oct 6 2022, 3:24 PM

Description

Puppet is failing on deploy-1004.devtools.eqiad1.wikimedia.cloud:

ERR: Could not retrieve catalog from remote server: Error 500 on SERVER:
Server Error:
Evaluation Error: Error while evaluating a Resource Statement,
Evaluation Error: Error while evaluating a Resource Statement,
Duplicate declaration:
   File[/usr/bin/scap] is already declared at (file: /etc/puppet/modules/scap/manifests/init.pp, line: 45);
   cannot redeclare (file: /etc/puppet/modules/scap/manifests/master.pp, line: 69)
   (file: /etc/puppet/modules/scap/manifests/master.pp, line: 69, column: 5)
   on node deploy-1004.devtools.eqiad1.wikimedia.cloud

Backtracking of the issue:

operations/puppet/modules/scap/manifests/init.pp defines class scap which will define the /usr/bin/scap file resource if $is_master is false.
operations/puppet/modules/scap/manifests/master.pp defines class scap::master which unconditionally defines the /usr/bin/scap file resource.
operations/puppet/modules/profile/manifests/mediawiki/scap_client.pp defines class profile::mediawiki::scap_client which has parameter $is_master=lookup('profile::mediawiki::scap_client::is_master').
the instance deploy-1004.devtools.eqiad1.wikimedia.cloud has role role::deployment_server.

Grepping through operations/puppet for profile::mediawiki::scap_client::is_master:

./modules/profile/manifests/mediawiki/scap_client.pp:8:    Boolean      $is_master    = lookup('profile::mediawiki::scap_client::is_master')
./hieradata/cloud/eqiad1/deployment-prep/hosts/deployment-deploy03.yaml:1:profile::mediawiki::scap_client::is_master: true
./hieradata/cloud.yaml:335:profile::mediawiki::scap_client::is_master: false
./hieradata/role/common/deployment_server/kubernetes.yaml:128:profile::mediawiki::scap_client::is_master: true
./hieradata/common/profile/mediawiki/scap_client.yaml:1:profile::mediawiki::scap_client::is_master: false

When it comes down to it, profile::mediawiki::scap_client::is_master should be true for any node that uses the role::deployment_server class.

Is there a way to change $is_master=lookup('profile::mediawiki::scap_client::is_master') into something that finds out if class role::deployment_server has been used?

Related Objects

Mentioned Here: T120165: Implement role based hiera lookups for labs

Event Timeline

dancy created this task.Oct 6 2022, 3:24 PM

Restricted Application added a project: Infrastructure-Foundations. · View Herald TranscriptOct 6 2022, 3:24 PM

Restricted Application added a subscriber: Aklapper. · View Herald Transcript

hashar subscribed.Oct 6 2022, 3:26 PM

dancy updated the task description. (Show Details)Oct 6 2022, 3:28 PM

dancy updated the task description. (Show Details)Oct 7 2022, 12:44 AM

hashar updated the task description. (Show Details)Oct 7 2022, 7:49 AM

Mentioned in SAL (#wikimedia-releng) [2022-10-18T19:13:53Z] <hashar> devtools: unbreak puppet on deploy-1004.devtools.eqiad1.wikimedia.cloud by applying profile::mediawiki::scap_client::is_master: true # T319681

https://gerrit.wikimedia.org/r/c/operations/puppet/+/844515 added profile::mediawiki::scap_client::is_master: true to hieradata/cloud/eqiad1/devtools/common.yaml

On the Puppet master it resolves to true:

$ sudo puppet lookup --compile --explain --node deploy-1004.devtools.eqiad1.wikimedia.cloud profile::mediawiki::scap_client::is_master
Searching for "profile::mediawiki::scap_client::is_master"
  Global Data Provider (hiera configuration version 5)
    Using configuration "/etc/puppet/hiera.yaml"
    Hierarchy entry "Http Yaml"
      URI "http://puppet-enc.cloudinfra.wmcloud.org:8100/v1/devtools/node/deploy-1004.devtools.eqiad1.wikimedia.cloud"
        Original uri: "http://puppet-enc.cloudinfra.wmcloud.org:8100/v1/%{::wmcs_project}/node/%{facts.fqdn}"
        Found key: "profile::mediawiki::scap_client::is_master" value: true

I have then removed the entry from the instance in Horizon ( https://horizon.wikimedia.org/project/instances/78f652ff-ae4b-4b0d-9c6c-1097efd258d2/ ).

$ sudo puppet lookup --compile --explain --node deploy-1004.devtools.eqiad1.wikimedia.cloud profile::mediawiki::scap_client::is_master
11, column: 33)
Searching for "profile::mediawiki::scap_client::is_master"
  Global Data Provider (hiera configuration version 5)
    Using configuration "/etc/puppet/hiera.yaml"
    Hierarchy entry "Http Yaml"
      URI "http://puppet-enc.cloudinfra.wmcloud.org:8100/v1/devtools/node/deploy-1004.devtools.eqiad1.wikimedia.cloud"
        Original uri: "http://puppet-enc.cloudinfra.wmcloud.org:8100/v1/%{::wmcs_project}/node/%{facts.fqdn}"
        No such key: "profile::mediawiki::scap_client::is_master"
    Hierarchy entry "cloud hierarchy"
      Path "/etc/puppet/hieradata/cloud/eqiad1/devtools/hosts/deploy-1004.yaml"
        Original path: "cloud/%{::wmcs_deployment}/%{::wmcs_project}/hosts/%{::hostname}.yaml"
        Path not found
      Path "/etc/puppet/hieradata/cloud/eqiad1/devtools/common.yaml"
        Original path: "cloud/%{::wmcs_deployment}/%{::wmcs_project}/common.yaml"
        Found key: "profile::mediawiki::scap_client::is_master" value: true

It has found it from /etc/puppet/hieradata/cloud/eqiad1/devtools/common.yaml which is managed by Puppet.

I then ran Puppet on deploy-1004 and it passes.

Puppet failure on deploy-1004.devtools.eqiad1.wikimedia.cloudClosed, ResolvedPublicActions

Description

Related Objects

Event Timeline

Puppet failure on deploy-1004.devtools.eqiad1.wikimedia.cloud
Closed, ResolvedPublic
Actions