Page MenuHomePhabricator

Have puppet-merge on puppetmaster1001 publish the official sha1 after merging
Closed, ResolvedPublic

Description

There are occasional cases where the puppet head on the prod puppetmasters (puppetmaster1001.eqiad.wmnet et al) is not the same sha1 as the head in the operations/puppet repo hosted on Gerrit.

Our VM-hosted puppetmasters just blindly pull down the latest repo from Gerrit. So occasionally we'll get out of sync from prod.

@fgiunchedi (and others) suggest that we have the puppet-merge script on prod publish the latest sha1 to etcd, which can then be made available via URL on noc.wm.o. Then our existing update cron on VM puppetmasters can grab the sha1 from a URL and sync to that.

I don't immediately know how to do the publishing side of this, but I like the idea!

Details

Event Timeline

Andrew created this task.Mar 27 2019, 2:18 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 27 2019, 2:18 PM
Andrew added a subscriber: Krenair.Mar 27 2019, 2:20 PM
Krenair added a comment.EditedApr 12 2019, 11:24 PM

@fgiunchedi noc.wm.o's main function IIRC was the mediawiki config viewer that was obsolete upon the publishing of mediawiki-config.git, I'm not sure it's a good idea to expand it.
One thing that did seem relevant on the noc index page is this:

Conftool configuration
    Config files of Wikimedia's etcd used for pooling services in LVS

It's a link to https://config-master.wikimedia.org/. Maybe we can write to etcd from puppet-merge and have it exposed there? Any pointers on how one might go about doing that?

Instead of extending the existing system of prod puppetmasters SSHing into labs central puppetmasters (to push updates) over to the new central puppetmasters within the labs realm, let's implement this.

Change 504082 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] puppet-merge: write the latest puppet repo sha-1 to config-master

https://gerrit.wikimedia.org/r/504082

Change 504082 merged by Andrew Bogott:
[operations/puppet@production] puppet-merge: write the latest puppet repo sha-1 to config-master

https://gerrit.wikimedia.org/r/504082

andrew@tools-puppetmaster-01:~$ curl https://config-master.wikimedia.org/puppet-sha1.txt
440fab7e0ee0bd63fdd12492bdffeeb46f504ab5

Change 504825 had a related patch set uploaded (by Alex Monk; owner: Alex Monk):
[operations/puppet@production] git-sync-upstream: Rebase on top of prod's copy of the puppet repo

https://gerrit.wikimedia.org/r/504825

Krenair added a comment.EditedApr 18 2019, 12:57 AM

I think for this to make sense we should require labs/private repository to also exist on a trusted instance somewhere and have the trusted sha1 published in this manner too. Doesn't have to be a prod puppetmaster, the labs puppetmaster would be sufficient - as long as it requires someone privileged to log in to a very restricted host, sudo, and do something to provide the commit a second layer of approval after it's merged in gerrit.

I think for this to make sense we should require labs/private repository to also exist on a trusted instance somewhere

Yeah, I think that's right. I sort of want to just do it on the central puppetmaster to avoid duplication of work but that might be unpopular.

Change 504825 merged by Andrew Bogott:
[operations/puppet@production] git-sync-upstream: Rebase on top of prod's copy of the puppet repo

https://gerrit.wikimedia.org/r/504825