Improve sre.hosts.decommission (additionally find host yaml files)
Open, MediumPublic
Actions

Assigned To

None

Authored By

	elukey
	Jul 7 2020, 10:19 AM

Description

Today while running the decom cookbook for archiva1001 I had this interesting output:

Looking for matches in puppetmaster1001.eqiad.wmnet:/var/lib/git/operations/puppet
hieradata/common/lvs/interfaces.yaml:      'lvs1013': 'enp4s0f1:208.80.154.167'
hieradata/common/lvs/interfaces.yaml:      'lvs1014': 'enp4s0f0:208.80.154.168'
hieradata/common/lvs/interfaces.yaml:      'lvs1015': 'enp4s0f1:208.80.154.169'
hieradata/role/eqiad/wmcs/openstack/eqiad1/labweb.yaml:        host: 208.80.154.160
hieradata/role/eqiad/wmcs/openstack/eqiad1/labweb.yaml:        host: 208.80.154.160
modules/role/templates/mariadb/grants/production-m5.sql.erb:GRANT SELECT, INSERT, UPDATE, DELETE ON striker.* TO 'striker'@'208.80.154.160'
modules/role/templates/mariadb/grants/production-m5.sql.erb:GRANT ALL ON striker.* TO 'striker_admin'@'208.80.154.160'
modules/role/templates/mariadb/grants/production-m5.sql.erb:GRANT ALL ON `labswiki`.* TO 'wikiadmin'@'208.80.154.160'
modules/role/templates/mariadb/grants/production-m5.sql.erb:GRANT DELETE, INSERT, SELECT, UPDATE ON `labswiki`.* TO 'wikiuser'@'208.80.154.160'
Looking for matches in puppetmaster1001.eqiad.wmnet:/srv/private
Looking for matches in deploy1001.eqiad.wmnet:/srv/mediawiki-staging
Found match(es) in the Puppet or mediawiki-config repositories (see above), proceed anyway?

That was really confusing at first, but then I realized that archiva1001's IP was 208.80.154.16, and in the cookbook it seems that we do a simple grep $ip that may match multiple things.

The above list also didn't match a hiera host override, archiva1001.yaml. Not the end of the world but it would be great to add a check for it :)

Details

	Subject	Repo	Branch	Lines +/-
	Add pytest and a simple test for decommission	operations/cookbooks	master	+29 -4

Customize query in gerrit

Related Objects

Mentioned In: rCCKBb49308df9687: Add pytest and a simple test for decommission

Event Timeline

elukey created this task.Jul 7 2020, 10:19 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 7 2020, 10:19 AM

Dzahn renamed this task from Improve sre.hosts.decommission to Improve sre.hosts.decommission (additionally find host yaml files).Jul 8 2020, 6:17 PM

Dzahn added a project: SRE-tools.

@elukey indeed that's confusing, we can surely improve it. I think we could first improve the message for what we're searching for, so that it's easier to spot false positives, and then probably match IP and hostname more strictly (as "words") and look also for file names with the hostname in it. Thoughts?

Yes I think that matching the IP and hostname more strictly is a good idea, when I saw the issue I tried to see if it was a quick patch for the cookbook but it didn't seem a 5 minute thing from a quick look. Looking for filenames is also good, we could limit the scope to hiera's host yaml files if it is easier, it will be basically 99.9% of the use cases in my opinion :)

herron triaged this task as Medium priority.Jul 27 2020, 1:41 PM

Change 631448 had a related patch set uploaded (by Alexandros Kosiaris; owner: Alexandros Kosiaris):
[operations/cookbooks@master] Add pytest and a simple test for decommission

https://gerrit.wikimedia.org/r/631448

gerritbot added a project: Patch-For-Review.Oct 5 2020, 12:31 PM

Change 631448 merged by jenkins-bot:
[operations/cookbooks@master] Add pytest and a simple test for decommission

https://gerrit.wikimedia.org/r/631448

Maintenance_bot removed a project: Patch-For-Review.Oct 6 2020, 1:10 PM

Aklapper added a project: Infrastructure-Foundations.Jun 21 2021, 8:59 PM

akosiaris mentioned this in rCCKBb49308df9687: Add pytest and a simple test for decommission.Dec 14 2022, 3:26 PM