Page MenuHomePhabricator

Prevents labs/private to disappear from integration and deployment-prep puppet master
Closed, ResolvedPublic

Description

Some patch in puppet caused labs/private to be set as absent:

https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/602327/3/modules/puppetmaster/manifests/gitclone.pp,unified

Luckily deployment-puppetmaster04.deployment-prep.eqiad.wmflabs and integration-puppetmaster-02.integration.eqiad.wmflabs had merge conflicts preventing them from automagically applying the faulty puppet.git patch

Event Timeline

On both, I have made a copy to /var/lib/git/labs/private-back

hashar triaged this task as Unbreak Now! priority.Jun 4 2020, 1:43 PM

I have:

  • disabled puppet agent on both puppet master
  • rebased /var/lib/git/operations/puppet on both puppet master.

Confirmed the faulty commit got reverted ;)

Enabled puppet again and ran it again.

The one on beta fails because of a missing etcd::autogen_pwd_seed. I have made a dummy
puppet patch in hieradata/cloud/eqiad1/deployment-prep/common.yaml

hashar claimed this task.

Looks good to me now. We have been lucky to have both puppet master blocked on a merge conflict!